Post subject: Modern HQ & RGB
Joined: 10/14/2013
Posts: 335
Location: Australia
For publications we currently offer two main types of downloadable encodes: MP4 Compatibility uses YUV color encoding, limited levels and 4:2:0 chroma. MKV Modern HQ uses YUV color encoding, full levels and 4:4:4 chroma. With all the changes made to encoders and players alike over the recent years, Modern HQ just doesn't feel very "modern" any more. x264 has supported RGB for a long time, so my question is: why are we using 4:4:4 as our HQ option when we could be offering RGB? It would further separate the two encode types and we could still keep all the other (non-colorspace) settings identical and so the file size should theoretically be similar. The only difference is we'd be removing a needless RGB>YUV conversion from the process. To be safe, I did a number of tests to see how x264 worked when handling lossless content in RGB. The resulting files (when decoded to RGB24) were identical to the same encodes produced with the Lagarith codec in RGB. Interestingly, I used this run as my test and the resulting video stream (with no audio data) was 30.4mb. Combining that with FLAC audio leads to a final file size of 64.5mb (although this had no logo or subtitles), which gave me a second thought: Lossless RGB. I'd love to see a third (optional, see below) encode option on publications for Lossless RGB. If I were not making encodes myself, it's the one I'd download. Internet connections are getting faster and hard drive capacities larger. Digital retailers are selling music in lossless (and sometimes 24-bit) formats. Everything seems to be evolving in a very quality-driven manner and so I think we should too. Excluding detail changes (such as branch name) or errors, having Lossless RGB should prevent the need for any future encode updates and give people the best downloadable watching experience possible. Regarding it being optional: I'd love to see this introduced as a standard for every publication on the site- but it's a huge task. Not every game is going to be able to get a lossless encode right away (3D games for instance, would be harder), and not everyone is going to be able to upload large files (Metroid was small, but longer encodes and more complex games could easily get large very fast). From an encoding perspective, it's another thing that's going to take time, although the good news is that the first pass (frame decimation) data generated for the Modern HQ can be re-used for the lossless encode to save time. Having said that, it would be nice to see some publications slowly pop up with "Lossless MKV" or some similar term as a third download option. Even if they're few in number to begin with, it would have to start somewhere. I feel like at the very least we should have an RGB encode, whether it's new adjustment to the Modern HQ encode or a new lossless option.
I'm not as active as I once was, but I can be reached here if I should be needed.
Site Admin, Skilled player (1234)
Joined: 4/17/2010
Posts: 11251
Location: RU
Ah, I thought the size would be comparable to 444. Since it's an extra thing that is also optional, I agree that it should be done instead of any ExtraHQ encode for 2D footage.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Joined: 10/14/2013
Posts: 335
Location: Australia
feos wrote:
Ah, I thought the size would be comparable to 444.
I'll do a size test between our current Modern HQ method and the same method with RGB and post the results shortly. Edit, Results: Lagarith source file: 701 MB (735,481,802 bytes) Modern HQ 444 (YUV BT.601): 33.0 MB (34,685,337 bytes) Modern HQ RGB: 53.9 MB (56,564,916 bytes) Lossless RGB: 81.5 MB (85,467,011 bytes) All results were calculated using a dump of this run trimmed to the run's length, with no logo present or subtitles added, and no audio track present (because I wanted to do this quickly). The results were more varied than I had expected and have altered my opinions slightly. The first thing worth noting is that in this case, RGB is around 160% larger. This means my similar size hypothesis was incorrect. The second thing worth noting is the size difference between the Lagarith source dump, the Lossless x264 RGB and our current Modern HQ 444, which helps deal with the assumption that lossless encodes are going to be astronomical. In this test, the result is less than 3x the size of the Modern HQ 444 and roughly 8.6x smaller than the initial lossless dumps. Using these results as an indicator, I'd be more inclined to suggest the adding of Lossless RGB as an optional encode as opposed to replacing Modern HQ with the RGB variant, I don't see a point in bumping up the size by what appears to be a significant amount when a little more could present the ideal scenario.
I'm not as active as I once was, but I can be reached here if I should be needed.
Site Admin, Skilled player (1234)
Joined: 4/17/2010
Posts: 11251
Location: RU
Camstudio Lossless Codec doesn't seem to allow command line usage. But it has a VS solution available, so maybe it could be recompiled into an executional, or there could be a wrapper for it that could translate the calls. But still I don't think it'd be able to overcome the 2GB problems of VideoForWindows... Can you test FFV1 too?
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Joined: 10/14/2013
Posts: 335
Location: Australia
Here's FFV1 (using the settings from over here) and Camstudio using LZO (as I was unable to get GZIP to function properly). Lossless Camstudio LZO: 311 MB (326,661,150 bytes) FFV1: 707 MB (742,030,710 bytes) Compared to x264 they both encode a lot faster. I got over 200fps with FFV1, compared to 8fps with x264 (which was the same rate I got for encoding the Modern HQ, for reference), but it looks like neither comes close to x264 RGB in terms of file size. Compatibility could also be a factor here. If someone can play the Modern HQ, then they could play x264 lossless without need of additional codecs.
I'm not as active as I once was, but I can be reached here if I should be needed.
Site Admin, Skilled player (1234)
Joined: 4/17/2010
Posts: 11251
Location: RU
Ahem! <fsvgm777> Codecs: <fsvgm777> D..... = Decoding supported <fsvgm777> .E.... = Encoding supported <fsvgm777> ..V... = Video codec <fsvgm777> ..A... = Audio codec <fsvgm777> ..S... = Subtitle codec <fsvgm777> ...I.. = Intra frame-only codec <fsvgm777> ....L. = Lossy compression <fsvgm777> .....S = Lossless compression <fsvgm777> D.V..S cscd CamStudio (decoders: camstudio )
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Joined: 10/14/2013
Posts: 335
Location: Australia
Ah, of course! But don't forget about hardware players and non-PC devices too, there's a chance they might not be ffmpeg based and may be limited to specific codecs. CSCD's still producing significantly larger files (although not as drastic as FFV1 or Lagarith) and depending on the footage it can much slower to seek through. This can be fixed by adding keyframes every second or so, but it could bulk up the size considerably. In comparison, x264's main drawback here is speed, but we're already toggling on lots of slow options to shrink the size of our current downloadable encodes. If this is a theoretical new option, it'd suit the site's existing procedures to follow on and provide the best-compressed file possible. The fact all the downloadable encodes would share the same video format would be an extra bonus in some situations, for instance:
  • We would not need any additional software in the package to make the video portion of these encodes.
  • Users who can play our current Modern HQ encodes would likely be able to play our lossless encodes without the need of any additional software, codecs, or hardware upgrades.
  • Troubleshooting for user playback issues would remain somewhat universal.
Edit: x264's encoding framerate isn't always terrible. When test-encoding NES Super Mario Bros. and NES Castlevania at the same time, both encodes averaged around 60-75fps with the same settings as before.
I'm not as active as I once was, but I can be reached here if I should be needed.
Site Admin, Skilled player (1234)
Joined: 4/17/2010
Posts: 11251
Location: RU
Dedupped cscd at level 9 (guess how I got it!) for [3019] SNES Super Mario World "warps" by BrunoVisnadi, Amaraticando in 09:57.10 is 125 MB (132,049,263 bytes). But I only stopped the video at 11:21, so a little bit of overhead is there. Still pretty obviously worse than x264.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Joined: 10/14/2013
Posts: 335
Location: Australia
feos wrote:
Dedupped cscd at level 9 (guess how I got it!) for [3019] SNES Super Mario World "warps" by BrunoVisnadi, Amaraticando in 09:57.10 is 125 MB (132,049,263 bytes).
That's significantly smaller than I expected! Did you make a command line tool? Also for comparison, my tests of the same file ran for 15:07 and were not dedupped.
I'm not as active as I once was, but I can be reached here if I should be needed.
Site Admin, Skilled player (1234)
Joined: 4/17/2010
Posts: 11251
Location: RU
I dumped a JMD. The rest should be easy to figure out.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Joined: 10/14/2013
Posts: 335
Location: Australia
Assuming %FPS% represents the frame rate of the source value multiplied by 10, this is the lossless command I've been using for my tests (isolated from the batch system). I use AVS to load the video because it's the input format we'll likely be dealing with:
".\programs\x264-10_x64" --threads auto --qp 0 --keyint %FPS% --ref 16 --bframes 16 --b-adapt 2 --trellis 2 --direct auto --me tesa --merange 64 --subme 11 --partitions all --no-dct-decimate --no-fast-pskip --output-csp rgb --range pc --input-csp rgb24 --input-range pc --demuxer avs -o ".\temp\video.mkv" ".\source.avs"
Additionally, I'd probably say this should be coupled with FLAC for the resulting file, which can be encoded like this:
".\programs\avs2pipemod" -wav ".\source.avs" | ".\programs\flac_x64" - --best -o ".\temp\audio.flac"
And muxed like this:
".\programs\mkvmerge" -o ".\output\output.mkv" ".\temp\video.mkv" ".\temp\audio.flac"
Edit: Changed "x264_x64" to "x264-10_x64". The build I have handles the 10-bit encoding all in the main executable, but if you're using a version prior to that update you'll need to use the correct executable.
I'm not as active as I once was, but I can be reached here if I should be needed.
Publisher
Joined: 4/23/2009
Posts: 1283
TheCoreyBurton wrote:
Ah, of course! But don't forget about hardware players and non-PC devices too, there's a chance they might not be ffmpeg based and may be limited to specific codecs.
You know, this goes for RGB H.264 too... I don't think RGB in H.264 is actually standard, just the most popular encoder, x264, added a way to do it. I could be wrong though. Even if I am wrong, I believe the profile needed to playback RGB H.264 would be one of the highest and not many players support that.
Joined: 10/14/2013
Posts: 335
Location: Australia
That is a very good point.
I'm not as active as I once was, but I can be reached here if I should be needed.
Site Admin, Skilled player (1234)
Joined: 4/17/2010
Posts: 11251
Location: RU
Adding RGB lossless encode doesn't mean we remove other ones. But another thing that needs to be tested is vp9. And it has a lossless mode as well.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Site Admin, Skilled player (1234)
Joined: 4/17/2010
Posts: 11251
Location: RU
Tested a couple dumping options. Requirements: fast encoding, fast seeking, small filesize. Tested on a toaster, Sonic 3 & Knuckles running for 10k frames without input. Lagarith Command: Video for Windows (2GB split unavoidable, also codec is Windows only) Keyint: 1 (can't be changed) Seeking backwards: instant. Size: 256MB (100%) Speed: 140fps (100%) libx264rgb Command: ffmpeg -c:a pcm_s16le -c:v libx264rgb -qp 0 -preset ultrafast -g 1 -pix_fmt rgb24 -f avi Keyint: 1 Seeking backwards: instant. Size: 911MB (356%) Speed: 130fps (93%) Command: ffmpeg -c:a pcm_s16le -c:v libx264rgb -qp 0 -preset ultrafast -g 30 -pix_fmt rgb24 -f avi Keyint: 30 Seeking backwards: instant. Size: 177MB (69%) Speed: 130fps (93%) Command: ffmpeg -c:a pcm_s16le -c:v libx264rgb -qp 0 -preset ultrafast -g 60 -pix_fmt rgb24 -f avi Keyint: 60 Seeking backwards: fast. Size: 163MB (63%) Speed: 130fps (93%) Command: ffmpeg -c:a pcm_s16le -c:v libx264rgb -qp 0 -preset ultrafast -pix_fmt rgb24 -f avi Keyint: 250 (default) Seeking backwards: slow! Size: 153MB (59%) Speed: 130fps (93%) Command: ffmpeg -c:a pcm_s16le -c:v libx264rgb -qp 0 -preset ultrafast -g 600 -pix_fmt rgb24 -f avi Keyint: 600 Seeking backwards: really slow! Size: 151MB (58%) Speed: 130fps (93%) FFV1 Command (what Dolphin uses): ffmpeg -c:a pcm_s16le -c:v ffv1 -pix_fmt bgr0 -level 1 -g 1 -coder 1 -context 1 -f avi Keyint: 1 Seeking backwards: instant. Size: 331MB (129%) Speed: 80fps (57%) Command: ffmpeg -c:a pcm_s16le -c:v ffv1 -pix_fmt bgr0 -level 1 -g 30 -coder 1 -context 1 -f avi Keyint: 30 Seeking backwards: instant. Size: 308MB (120%) Speed: 80fps (57%)
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Joined: 10/14/2013
Posts: 335
Location: Australia
I was doing some work on a publication yesterday (dumped with FFV1 at 4K) and noticed the seeking was frustratingly slow. Given the resolution, I expected as much but it made me think about this post. I did some overnight tests and noted down a few things: FFV1: Command: ffmpeg -i "framedump0.avi" -c:v ffv1 -pix_fmt bgr0 -level 1 -g 1 -coder 1 -context 1 "output-ffv1.avi" File size: 891 MB (934,546,882 bytes) Approximate encoding time: 10 minutes. Approximate seek time: ~0.5 seconds per frame. Lagarith: Command: None. Opened FFV1 in Virtualdub and saved with Lagarith codec. File size: 1.01 GB (1,088,373,084 bytes) Approximate encoding time: 6 minutes. Approximate seek time: Instant. x264RGB "1": Command: ffmpeg -i "framedump0.avi" -c:v libx264rgb -qp 0 -preset ultrafast -g 1 -pix_fmt rgb24 -context 1 "output-x264rgb-30.avi" File size: 1.68 GB (1,813,903,120 bytes) Approximate encoding time: 1 minute. Random seek time: Instant. x264RGB "30": Command: ffmpeg -i "framedump0.avi" -c:v libx264rgb -qp 0 -preset ultrafast -g 30 -pix_fmt rgb24 -context 1 "output-x264rgb-30.avi" File size: 1.22 GB (1,313,018,268 bytes) Approximate encoding time: 2 minutes. Random seek time: ~1.5 seconds per frame. Before discussing the results, there's a few things worth mentioning: The encoding times come from the file properties and are only accurate to the nearest minute. The seek times are based on an external capture of the PC doing the seeking and could be slightly askew. "-c:a pcm_s16le" isn't needed in these cases as Dolphin dumps video and audio separately. The game runs at 30fps and so every second frame is a duplicate. Currently, I re-encode Dolphin's FFV1 dumps to FFV1 (with the settings above) to make them compatible with AVISource. I'm replacing that portion of my process with the x264RGB "1" command line, because in terms of encoding speed and seek time it's better to deal with. My main observation on the previous conclusion is that with the keyint being that large, the file becomes downright horrendous to work with when seeking is required. FFV1's seek time was already a nuisance and tripling it only makes things worse. For this particular dump, the file is also larger when encoded this way, and so saying Dolphin must switch to this preset might be a bit hasty. I have a lot of HDD space to work with though and so I can see how better compression would benefit other users more than myself. Perhaps the best option here is the ability to choose a custom command line? Combine that with a couple of presets (such as the three tested here, for instance) for users less familiar with ffmpeg and then the situation is user friendly and the possibilities are endless.
I'm not as active as I once was, but I can be reached here if I should be needed.
Site Admin, Skilled player (1234)
Joined: 4/17/2010
Posts: 11251
Location: RU
Good point. You might want to try other keyints, starting with 2 and waiting for seeking to stop being instant. Maybe somewhere at around 10 it becomes a problem? I dunno, a wild guess.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Publisher
Joined: 4/23/2009
Posts: 1283
I should note that H.264 in AVI is usually a hacky workaround for B-Frames (since AVI doesn't support B-Frames). Lossless has no B-Frames so hopefully the problem isn't there but usually it means with the hack that frame 0 may be duplicated a few frames. Aka, maybe do a test to see the frame count in the x264 RGB vs LAGs vs FFV1? Edit: Also do a test to see if a scene change starts at the same frame for all three?
Site Admin, Skilled player (1234)
Joined: 4/17/2010
Posts: 11251
Location: RU
Is it fine to use consecutive footage as is for such a test, or do I have to add cuts to it via avisynth?
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Publisher
Joined: 4/23/2009
Posts: 1283
Hmm good question. It may work fine in sequence but not so fine when out of sequence. I don't know, try both? I think random access is the ultimate test as I usually do see the problem after random access. Aka seek to some random frame, then going back to frame 0 yields a grey screen. I guess this is less of a problem since an encoder always encode in sequence, but on the other hand if you need to do any video editing, like cut out parts here and there, it be not in sequence anymore.
Joined: 10/14/2013
Posts: 335
Location: Australia
I was worried about the "hack" nature of h264 in AVI due to some bad experiences, but surprisingly didn't have any issues at first (I assumed because VFW was not involved). I did two test encodes on my current PC today. In both cases I used AVISource to load the footage, but only used ExactDedup on the second script. The output of both encodes was the same as if I'd used a Lagarith source. I did the same tests on my second PC (which recently got a HDD replaced and therefore has a clean installation of Windows) and got different results. The only things installed on that PC were the OS (Windows 7 x64), AVS+, ffdshow and all the appropriate pre-requisites to get these and the encoding package running (such as MSVC++ 2010). Regarding the results, the non-dedupped encode looked the same, but the dedupped encode started with some garbage frames. The garbage frames are additional frames added at the start of the encode. They don't replace existing frames and therefore the audio sync is noticeably out when they're present: Randomly seeking through the non-dedupped script showed no issues. Seeking to the first non-logo frame of the dedupped script showed the garbage frame at first, but after seeking elsewhere in the clip (and then seeking back), the frame was gone. I tried reopening the script and the problem was still resolved. I rebooted the PC and opened the script again and as expected, the problem was back (to which seeking again solved the issue). The solution seems to have been to uninstall ffdshow (which is out of date anyway) and instead install LAV filters.* I'd need someone with a clean system to try encounter the error and solve it this way to be sure this is the correct solution, though on the test PC the script produces the expected output in any test situation I've tried, including the above situations where the output was previously broken. My main concern here is that the correct decoding of h264 in AVI looks to vary depending on installed codecs or system. Whilst this is true for most formats, there's usually some form of error message or some obvious sign that something's wrong; this isn't the case here. If you were to preview the footage in Virtualdub via AVS, the dump would look fine. If the problem did occur, seeking past it for a few frames and going back would fix it. If there were doubts and the script was reloaded to check, it would be fixed on reload. This could be problematic as it has the potential to go unnoticed in some situations. * I initially installed LAV filters without removing ffdshow (as that is the codec setup on my main PC), but this didn't solve the issue and so the safest bet is to remove ffdshow altogether. Note: There were a number of edits originally appended to the end to this post, which lead to the point being worded in a confusing and unnecessary way. All of the edits all have therefore been re-worded and added in to the original post.
I'm not as active as I once was, but I can be reached here if I should be needed.
Site Admin, Skilled player (1234)
Joined: 4/17/2010
Posts: 11251
Location: RU
BTW, with this installation of lav filters and without ffdshow, regular avisynth doesn't want to load an ffv1 clip for me, even though ffv1 is checked in lav's video config. As for h264, we can ship lav filters with bizhawk prereqs.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Joined: 10/14/2013
Posts: 335
Location: Australia
feos wrote:
with this installation of lav filters and without ffdshow, regular avisynth doesn't want to load an ffv1 clip for me
Does encoding using this command produce a file that loads correctly? It likely won't make a difference if you're using the Bizhawk command but it's probably better to be safe and check everything.
feos wrote:
As for h264, we can ship lav filters with bizhawk prereqs.
So h264 still loads with the removal of ffdshow (and installation of LAV filters)? If so, this is great news. We may also probably want to warn users somehow of the potential problem of having ffdshow installed at the same time. It's interesting as I had the same version of ffdshow and LAV on both this and the other PC. This one decodes fine, but the other one required removal of ffdshow first. It's like ffdshow took decoding priority over LAV on the other PC (and so had to be removed) but LAV took priority here. Maybe there's a way to force priority or decoder if this is true (while still using AVISource).
I'm not as active as I once was, but I can be reached here if I should be needed.
Publisher
Joined: 4/23/2009
Posts: 1283
Couldn't you just uncheck H.264 as decoding option in ffdshow settings?
Joined: 10/14/2013
Posts: 335
Location: Australia
Aktan wrote:
Couldn't you just uncheck H.264 as decoding option in ffdshow settings?
Probably. I didn't think of this at the time, but it would have likely been the easier and better option.
I'm not as active as I once was, but I can be reached here if I should be needed.