Joined: 11/4/2007
Posts: 1772
Location: Australia, Victoria
As some may have noticed, I've been advertising that I've been encoding movies with a specific script that I've claimed is literally as tough as possible while having no quality loss. Put simply, it's The Stig the A Flygons Guide to Encoding script. It's incredibly slow, but as far as we (The several encoders that I've discussed this with) are aware, it is the toughest possible attainable encoding script using directx264.
For those too lazy to click the page, I'll insert the script below.
The first thing that experienced encoders will notice is the very unusual deldup value, quite simply, this will ensure that as many frames are chucked out as possible without getting rid legitimate frames... I'm actually a bit confused as to how it works still, sgrunt and Aktan brought the issue up with me, I tested it, and was amazed at how well it worked. The only downside is that the value will make incredibly low motion scenes mess around with the playback position, this is especially noticeable inside credits sequences (I had VLC jumping between 10 seconds at a time simply because it couldn't tell where the hell it was), however, this shouldn't cause any compatibility and you're usually just focusing on the movie itself anyway.
Closer inspection reveals absurdly high keyint and merange values, the former helping with making VLC spazz out with its jumping around of the playback slider. Basically, the keyint is set as high as I feel safe without sacrificing player compatibility, and is probably the only value in the script that can be used to make the encode smaller, but I really wouldn't recommend adjusting it too much, it's like adjusting the CPU voltage while overclocking, you can do it but it's dangerous. The merange value is really the heart and soul of this script, it probably makes the largest difference compared to standard scripts, while standard scripts usually have a merange equal to 64, this script has a range that is as high as the largest x or y resolution of the video (Going any higher is pointless, the range is measured in pixels) which really chops down the size... but really kicks up the encoding time.
Ah, the encoding time, that is the problem with this script, it's bloody slow, it's like comparing a solar powered car to a V8 engine powered Holden... without sounding like someone out of Top Gear, it's basically mindnumbingly slow on anything that has very complex motion, 3D games and games with heavy parallax scrolling are the main offenders here, the script is best used on encodes for 8-bit consoles (Such as the NES, SMS or Game Boy), and simpler 16-bit games (Such as early Mega Drive and SNES titles). Using it on anything else is basically suicide, it saves space, but it really does get to the point where it's not worth the time spent (For example, Putter Golf encoded at 0.20fps on a Dual Core 1.6ghz machine simply because of the scrolling water in the background... I ended up cancelling the encode it was that bad). This script also isn't recommended for anyone that has a weak machine, it's extremely recommended you have a very high end Dual Core machine or a pretty decent Quad Core machine, now, I can see the Aktan alarms being set off, but I'm sorry, I just cannot see the quality degradation on 2-4 threads (But I wouldn't recommend going above 4 anyway, just in case), and that of course, you can't lose quality on lossless. I'd recommend a crf value between 18 to 20 for more complex games, while simpler ones keeping at 0 simply because it'll be tiny anyway.
Anyway, the point of this topic? I want to get people with powerful machines into using this script for 8-bit consoles and early 16-bit titles, I noticed I was the only one making them and was a bit disappointed, especially considering how simple it is to just use this script.
I hope this is all a joke. --crf 0 is lossless, and bframes should not be used with lossless. Why are you using lossless anyway? You can use --crf 12 with no visible quality loss. --merange 512 is absurd and does not actually improve quality.
Joined: 11/4/2007
Posts: 1772
Location: Australia, Victoria
If this is a joke, strip me of my Publisher status and stab me in the heart. This isn't a joke.
I do ask why b-frames should not be used with lossless video encoding, I've not seen any issues arise from it, and I am not the only encoder to engage in this practice, Aktans encodes have historically (And are partially the inspiration for this script) been losslessly encoded while using b-frames and no one has reported any issues, I would very highly trust Aktans encoding methods... I daresay he is far more skilled then me, he has taught me quite a bit.
As to why we actually encode in lossless? Mainly because the encodes are so small anyway that keeping them lossless is basically worth that small increase in size for small increase in quality... it's basically a 'why the hell not' scenario.
The merange is that large not to improve quality, but to reduce filesize... that's the entire point of this script, reducing the filesize as much as possible, the quality should be roughly the same as other crf encoded scripts (Assuming the value is the same).
I would not have made this post if I didn't have full confidence in the script in question, and several movies have been published with similar scripts in the past (Albeit, some are using slightly older and less optimized versions of the script).
Joined: 10/28/2007
Posts: 1360
Location: The dark horror in the back of your mind
I'll take the opportunity to remind people that there is no such thing as a perfect encoding script; many of the options there just happen to be what we have found work well for reducing file sizes - in particular, most of the options there were chosen with the intent of using --crf 20 (our de facto standard for most encodes).
If you're interested in tighter encoding standards, do try out the above, but I strongly recommend using a substantially lower value of --merange for speed purposes (64 tends to be the upper limit in practice, though you can probably get away with values as low as 16; at some point I will probably investigate the speed-to-size tradeoff there).
EDIT: If you want even faster encoding, use --me umh instead of --me tesa; this will result in a small file size penalty. It's up to you where you want to draw the line; personally I would prefer a slightly larger encode available at publication time than a slightly smaller encode that we need to wait an extra few days for (especially notable for very long encodes).
As to why we actually encode in lossless? Mainly because the encodes are so small anyway that keeping them lossless is basically worth that small increase in size for small increase in quality... it's basically a 'why the hell not' scenario.
It would be interesting to see some concrete statistics about this (completely out of honest curiosity and nothing else). What kind of file size differences are we talking about with lossless vs. lossy-with-almost-no-visible-visual-artifacts encodes? Examples from some actual TASes would be best (it doesn't have to be full TASes, as it would take a lot of time to do this testing, just some segments which demonstrate the effect).
It wouldn't be surprising, however, if there was little difference between lossless and lossy, due to the nature of the source material (ie. pixel graphics). It's the same with PNG vs JPEG: If we are dealing with photographs or drawings with millions of colors (with color shades, gradients, etc), it's next to impossible for PNG to match a JPEG which still shows no significant amount of visual artifacts. However, when we are talking about things like line graphics or pixel graphics (especially with a very small palette), it becomes quite hard for JPEG to match the sizes of PNG without introducing visible artifacts. (In some cases the sizes might be about the same, but then it makes more sense to use PNG because it is, after all, lossless, so there's no advantage in using JPEG).
I would imagine, however, that TASes of more modern consoles, which use millions of colors and may even be 3D, would benefit more from lossy encoding. Sometimes even eg. SNES TASes might do, depending on how many colors the game uses (as well as how much transparency layers...)
Joined: 11/4/2007
Posts: 1772
Location: Australia, Victoria
That's actually a very interesting point and a slight coincidence, I will be making some test footages in the future demonstrating how well various settings works across certain types of footage (Shadow of the Beast being a notable run for these sort of tests, in my mind).
I think what the interesting issue here is... since there is colourspace reduction in our standard encodes, it can be pretty difficult to actually see the difference between lossy and lossless, especially on simpler games. It's all really an opinionated tradeoff, I fully admit.
I'll probably get some testbed AVIs made in the near future for public use, for the sake of convenience so that everyone can have a standardized testbed.
How do you handle threads on multicore systems that go beyond 2 CPUs?
Also, how is it with bitrate vs crf these days? Is using the bitrate still preferred or has crf won the fight? Although I've initially been against using crf as the filesize is practically uncontrollable, I however learned to appreciate it after resigning here for its quality and speed (in comparision to a fullblown 3pass) when size just isn't a matter to worry about. To be honest, having to comply with the (former) 4.0MB/min limit led to at least 1 subpar encode I had to publish, Mischief Makers on N64 being the one I remember most.
Joined: 11/4/2007
Posts: 1772
Location: Australia, Victoria
You're not the only one that has had a fair amount of grief from the old 4MB/min limit, I mainly point out Metal Slug X (Which I shouldn't have even actually published... impatience got me down on this one, and upon checking who was the one that was actually encoding that, I apologize to you, ShinyDoofy) and Crusader of Centy.
People tend to lean towards crf nowdays, but bitrate is still used by some encoders. About the whole threads thing, well, I'd just assume that the encoder would change the amount of threads to go with how many cores and/or CPUs they have, the addition to the script is actually due to my own self-confessed paranoia.
Default: auto (1.5 * logical processors, rounded up)
Enables parallel encoding by using more than 1 thread to increase speed on multi-core systems. The quality loss from multiple threads is mostly negligible unless using very high numbers of threads (say, above 16). The speed gain should be slightly less than linear until you start using more than 1 thread per 40px of vertical video, at which point the gain from additional threads sharply decreases.
x264 currently has an internal limit on the number of threads set at 128, realistically you should never set it this high.
Recommendation: Default
Going by that, limiting threads at 4 or 5, if you don't trust auto, would bring no adverse effects.
Warp wrote:
Edit: I think I understand now: It's my avatar, isn't it? It makes me look angry.
Joined: 10/28/2007
Posts: 1360
Location: The dark horror in the back of your mind
moozooh wrote:
Going by that, limiting threads at 4 or 5, if you don't trust auto, would bring no adverse effects.
The motivation for settings --threads low, recently, has been to allow us to run several encodes simultaneously. If I'm not attempting that, I just leave it alone.
As with most of the other options, it's up to the encoder.
Default: auto (1.5 * logical processors, rounded up)
Btw, one could ask why have more threads than processors? (With the formula above, even with one single-core processor you would get two threads. With a dual-core you would get 3 threads. But why? What's the point?)
The idea is, AFAIK, that since video encoding is quite I/O-heavy, using more threads than processors actually helps speeding up the process. This is because if there were only as many threads as processors, any thread which is currently doing I/O would have its processor sitting idle (because I/O is so enormously slower than the CPU that the latter has basically nothing to do while it waits for the I/O to finish). Thus if you run 2 threads on one single-core processor, there's a good chance that one of the threads will get this free CPU time for its calculations while the other one is waiting for I/O, hence the CPU will get fully used. The same is true for more processors/cores, naturally. (In fact, with more CPUs and hence more threads the efficiency rate may even be better because it lessens the possibility of the single-core case where both threads are waiting for I/O at the same time.)
As some may have noticed, I've been advertising that I've been encoding movies with a specific script that I've claimed is literally as tough as possible while having no quality loss.
good. so maybe you can make a proper encode of my Turok 3 TAS. the currently published movie has sound desyncs and lacks antialiasing and anisotropic filtering.
It's incredibly slow, but as far as we (The several encoders that I've discussed this with) are aware, it is the toughest possible attainable encoding script using directx264.
too bad we can't use another codec, like the msu capture codec, which is specifically designed for old videogamish content. it probably encodes 200 times faster and produces smaller files.
have you tested if there's any difference with using "tesa" and esa? personally, i didn't notice any difference, except sometimes tesa even produces larger files (strange, i know). you also have subme 10 in your script, but i'm unsure if it's actually better than subme 7. i think the me range is too high too. 64 is usually enough, except maybe for really fast motion games.
Flygon wrote:
That's actually a very interesting point and a slight coincidence, I will be making some test footages in the future demonstrating how well various settings works across certain types of footage (Shadow of the Beast being a notable run for these sort of tests, in my mind).
a lossless test movie of heart of the alien (http://tasvideos.org/1305M.html) might be good too. the game has such low textures and low framerate that it will be a lot smaller in lossless. the encode is 12.46mb, i made a lossless with that weird msu codec and it was 7 mb, but x264 will probably create a little larger file.
As some may have noticed, I've been advertising that I've been encoding movies with a specific script that I've claimed is literally as tough as possible while having no quality loss.
good. so maybe you can make a proper encode of my Turok 3 TAS. the currently published movie has sound desyncs and lacks antialiasing and anisotropic filtering.
What do game emulation settings have anything to do with MPEG-4 video encoding settings? It's a completely different issue.
Default: auto (1.5 * logical processors, rounded up)
Btw, one could ask why have more threads than processors? (With the formula above, even with one single-core processor you would get two threads. With a dual-core you would get 3 threads. But why? What's the point?)
The idea is, AFAIK, that since video encoding is quite I/O-heavy, using more threads than processors actually helps speeding up the process. This is because if there were only as many threads as processors, any thread which is currently doing I/O would have its processor sitting idle (because I/O is so enormously slower than the CPU that the latter has basically nothing to do while it waits for the I/O to finish). Thus if you run 2 threads on one single-core processor, there's a good chance that one of the threads will get this free CPU time for its calculations while the other one is waiting for I/O, hence the CPU will get fully used. The same is true for more processors/cores, naturally. (In fact, with more CPUs and hence more threads the efficiency rate may even be better because it lessens the possibility of the single-core case where both threads are waiting for I/O at the same time.)
The reason for more threads than CPU is due to the fact that there are dependency on certain parts to be encoded first before the thread can finish. Due to these dependency, not all threads would use 100% CPU all the time. Also I think it is off. I could have sworn it was rounded down, not up so that a single core only gets 1 thread.
Johannes is right about the whole B frames in lossless mode thing, Flygon. When I loaded the two videos that you mentioned were encoded with the script (Super Mario Land and NES Tetris), in AVIDemux, if you go under Tools, Bitrate Histogram, it'll tell you the # of each of I,P, and B. For Super Mario Land, its 76 I frames and 43296 P frames, with 0 B, and in Tetris, 8 I frames, 3485 P frames, 0 B. Granted, I don't think the graph was perfectly accurate, as I had to rebuild the keyframes to get a more accurate number (possibly due to the lossless compression, which doesn't display right in the player at all), but when I skimmed through the Tetris video, I didn't see any B frames pop up at all. So, for lossless encodes I'm sure you can omit any B frames..it might or might not speed up the encode process.
I'll be experimenting with your script sometime soon...just as soon as I can get my VB AVISynth script hassle free with no memory issues. :/
Edit: I should mention that x264 lossless isn't really lossless, due to the YV12 colorspace. Something else to consider :P
We can't really escape YV12 conversion, can we? Unless we publish lossless videos as they are, that is.
Yeah, which is one reason why it'd be pointless to convert losslessly anyway.
But, I have been browsing around doom9's forums trying to find information on a supposed yv24 standard for h264. I'm hoping I can find an encoder for it..I am a bit frustrated that I'd have to doublescale any videos in order to preserve quality..but at the same time I realize video submissions are limited to original resolution only. >.<
Johannes is right about the whole B frames in lossless mode thing, Flygon. When I loaded the two videos that you mentioned were encoded with the script (Super Mario Land and NES Tetris), in AVIDemux, if you go under Tools, Bitrate Histogram, it'll tell you the # of each of I,P, and B. For Super Mario Land, its 76 I frames and 43296 P frames, with 0 B, and in Tetris, 8 I frames, 3485 P frames, 0 B. Granted, I don't think the graph was perfectly accurate, as I had to rebuild the keyframes to get a more accurate number (possibly due to the lossless compression, which doesn't display right in the player at all), but when I skimmed through the Tetris video, I didn't see any B frames pop up at all. So, for lossless encodes I'm sure you can omit any B frames..it might or might not speed up the encode process.
I'll be experimenting with your script sometime soon...just as soon as I can get my VB AVISynth script hassle free with no memory issues. :/
Edit: I should mention that x264 lossless isn't really lossless, due to the YV12 colorspace. Something else to consider :P
The param there won't hurt it like Johannes found out. x264 just ignores it. Flygon should know very well by now that it isn't really lossless since I've been drilling that into him. XD
Well, it's not that they're limited, it's just the established convention. We often increase the encode's resolution on games that can be rendered at higher resolution with no fidelity loss (most N64 games, for example).
Still, even without doublescaling it's not at all pointless to convert to YV12 lossless as an intermediary step for several reasons:
1) it makes editing and processing easier (no need to pipe the emulator output several times in a row, especially in case with N64 games rendered at high resolution);
2) it allows for higher-quality streaming encodes on YT and other sites.
Warp wrote:
Edit: I think I understand now: It's my avatar, isn't it? It makes me look angry.
Yeah, which is one reason why it'd be pointless to convert losslessly anyway.
But, I have been browsing around doom9's forums trying to find information on a supposed yv24 standard for h264. I'm hoping I can find an encoder for it..I am a bit frustrated that I'd have to doublescale any videos in order to preserve quality..but at the same time I realize video submissions are limited to original resolution only. >.<
Well, it's not that they're limited, it's just the established convention. We often increase the encodes' resolution on games that can be rendered at higher resolution with no fidelity loss (most N64 games, for example).
Still, even without doublescaling it's not at all pointless to convert to YV12 lossless as an intermediary step for several reasons:
1) it makes editing and processing easier (no need to pipe the emulator output several times in a row, especially in case with N64 games rendered at high resolution);
2) it allows for higher-quality streaming encodes on YT and other sites.
Warp wrote:
Edit: I think I understand now: It's my avatar, isn't it? It makes me look angry.
