Posted: 3/6/2012 9:52 AM
(Edited: 3/27/2012 5:46 AM)
Post subject: 10-bit 4:4:4 publication encode issues
Quote
Lex
Joined: 6/25/2007
Posts: 732
Location: Vancouver, British Columbia, Canada
I tediously went through the entire list of published 10-bit 4:4:4 encodes here ( http://tasvideos.org/Ilari/Encodes10Bit444.html ) and checked for issues by downloading each 10-bit 4:4:4 encode. Here are my findings. These lists are only concerned with the 10-bit 4:4:4 encodes on the linked pages. I didn't check the 8-bit encodes for these publications because I don't care enough about them and they probably have their own sets of issues. --- This first list denotes 10-bit 4:4:4 encodes which use limited-range YUV. All of these are also flagged as full-range, causing a very visible range problem in correct video playback systems. I checked by using LAV Video Decoder 0.47 with output range set to "untouched" and all output colorspaces disabled except RGB24 and RGB32 in MPC-HC with madVR. If the video showed limited range, I then switched output range in LAV Video Decoder to "PC" to check if the encode belongs in the next list rather than this one. The name in brackets is the encoder who made the video. I don't mean any offense by including this data. I'm just trying to help tasvideos produce quality publications. http://tasvideos.org/10M.html (ledauphinbenoit) http://tasvideos.org/24M.html (ledauphinbenoit) http://tasvideos.org/46M.html (ledauphinbenoit) http://tasvideos.org/88M.html (ledauphinbenoit) http://tasvideos.org/252M.html (ledauphinbenoit) http://tasvideos.org/637M.html (ledauphinbenoit) http://tasvideos.org/651M.html (ledauphinbenoit) http://tasvideos.org/1849M.html (Lex; won't re-dump since this run was hell to dump, but if the full-range flag can be toggled, that could work) http://tasvideos.org/1918M.html (omnipotententity) http://tasvideos.org/1921M.html (ledauphinbenoit) http://tasvideos.org/1924M.html (Dacicus) http://tasvideos.org/1927M.html (grunt) http://tasvideos.org/1928M.html (ledauphinbenoit) http://tasvideos.org/1930M.html (ledauphinbenoit) http://tasvideos.org/1934M.html (ledauphinbenoit) http://tasvideos.org/1935M.html (Dacicus) http://tasvideos.org/1936M.html (ledauphinbenoit) http://tasvideos.org/1941M.html (ledauphinbenoit) http://tasvideos.org/1943M.html (ledauphinbenoit) http://tasvideos.org/1944M.html (ledauphinbenoit) http://tasvideos.org/1945M.html (Dacicus) http://tasvideos.org/1946M.html (Dacicus) http://tasvideos.org/1950M.html (ledauphinbenoit) http://tasvideos.org/1959M.html (omnipotententity) http://tasvideos.org/1967M.html (Dacicus) http://tasvideos.org/1976M.html (ledauphinbenoit) http://tasvideos.org/1978M.html (omnipotententity) http://tasvideos.org/1981M.html (ledauphinbenoit) <-- This was the last encode on the list at the time I made this post. http://tasvideos.org/1997M.html (omnipotententity; also, slightly late audio with some splitters and soft subtitle issues; recommend using mkv with .ass subtitles) --- This second list denotes 10-bit 4:4:4 encodes which use limited-range YUV and are also flagged that way. This is not ideal, but at least they look good enough when played with range upscaling, which correct video playback systems will do. These, I would say, are low priority. http://tasvideos.org/1882M.html (grunt) http://tasvideos.org/1917M.html (turska) http://tasvideos.org/1937M.html (omnipotententity) http://tasvideos.org/1940M.html (omnipotententity) --- other oddities: http://tasvideos.org/756M.html (8-bit encode (by adelikat) uses limited range with limited range flag set in stream; is avi. also, archive.org links are swapped, causing me to have downloaded the 8-bit 4:2:0 encode instead of the 10-bit 4:4:4 encode at first, hence the analysis of the 8-bit encode) http://tasvideos.org/1269M.html (BrandonE; a/v desync; very late audio) http://tasvideos.org/1588M.html (BrandonE; a/v desync; very late audio) --- I will be updating this post as encodes get fixed. If you fix something, please post and I'll update.
Posted: 3/6/2012 8:56 PM
Quote
Dacicus
Editor, Player (69)
Joined: 6/22/2005
Posts: 1050
Thanks for the heads up. Those were my first 10-bit 4:4:4 encodes, and I've updated x264 and my batch file since then. I'll re-encode them.
Current Projects: TAS: Wizards & Warriors III.
Posted: 3/6/2012 8:59 PM
Post subject: Re: 10-bit 4:4:4 publication encode issues
Quote
natt
Editor, Emulator Coder, Site Developer
Joined: 5/11/2011
Posts: 1108
Location: Murka
Very useful work. Re 756M: So the problem is that the two encodes are switched in expected order on the movie page? (What determines that order anyway?) Re list2: I don't really see this as any problem at all. I know color theory is very complex and I'm glossing over quite a few things, but consider the following approximate quantities: 2k: number of colors genesis can represent 33k: number of colors GBA can represent 260k: number of colors NDS can represent 2.6M: number of colors 8 bit limited range YUV can represent 4.0M: number of colors 8 bit full range YUV can represent 17M: max number of colors any of our current emulators could theoretically represent (8 bit RGB) 170M: max number of colors 10 bit limited range YUV can represent 260M: max number of colors 10 bit full range YUV can represent There should be minimal error using 10 bit limited range to represent anything that was originally RGB24. (nb: for YUV spaces, i'm restricting to colors that are inside the RGB space)
Posted: 3/7/2012 3:41 AM
Quote
feos
Site Admin, Skilled player (1255)
Joined: 4/17/2010
Posts: 11492
Location: Lake Char­gogg­a­gogg­man­chaugg­a­gogg­chau­bun­a­gung­a­maugg
NES TAS of 2011
It would be nice if anyone fixes [1920] NES Battletoads "warps, 2 players" by feos & MESHUGGAH in 11:04.72. I have configured my MPC to play 444 encodes correctly, but when I try to watch THIS encode, it shuts down and can't play even audio. Using LAV video output (other 444 encodes play fine). It was said that the problem was in MP4Box.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Posted: 3/7/2012 6:54 AM
Quote
creaothceann
He/Him
Editor
Joined: 4/7/2005
Posts: 1874
Location: Germany
feos wrote:
It would be nice if anyone fixes [1920] NES Battletoads "warps, 2 players" by feos & MESHUGGAH in 11:04.72. I have configured my MPC to play 444 encodes correctly, but when I try to watch THIS encode, it shuts down and can't play even audio. Using LAV video output (other 444 encodes play fine).
It plays fine in my MPC-HC. - k-lite Mega 8.4.4 + update 8.4.9 - Haali splitter - LAV 0.46 video decoder - LAV 0.46 audio decoder - MPC-HC 1.6.1.4074 - madVR
Posted: 3/7/2012 7:04 AM
Quote
RachelB
She/Her
Player (130)
Joined: 12/3/2011
Posts: 1579
Doesn't play for me in mpc-hc, using lav, and madvr. It doesn't display anything, and it just jumps to the end of the video. Interestingly, it stays open, even though i have it configured to close when finished playing. No errors or anything, either. Plays fine in vlc though.
Posted: 3/7/2012 2:02 PM
Quote
Lex
Joined: 6/25/2007
Posts: 732
Location: Vancouver, British Columbia, Canada
It plays fine for me too. It's probably just whatever splitter you're using being unable to handle the silly characters that old version of mp4box wrote as the stream titles. Try configuring MPC-HC to use LAV Splitter Source and configuring LAV Splitter Source to handle mp4 files (probably not necessary; should be enabled by default).
Posted: 3/7/2012 4:01 PM
Quote
feos
Site Admin, Skilled player (1255)
Joined: 4/17/2010
Posts: 11492
Location: Lake Char­gogg­a­gogg­man­chaugg­a­gogg­chau­bun­a­gung­a­maugg
NES TAS of 2011
I could only find where to set external filters: LAV Video Decoder and Source Splitter, and make them preferred, but don't see how to configure the latter for mp4. Doesn't work in total.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Posted: 3/7/2012 4:11 PM
Quote
Lex
Joined: 6/25/2007
Posts: 732
Location: Vancouver, British Columbia, Canada
You have to disable whatever other mp4 splitter that's being used. If it's the internal MPC-HC mp4 splitter, it would be this one. You can check what filters are currently in use like this (loaded video in screenshot is irrelevant): Here, you can see I have Haali splitter ("Haali Media Source") in use for this particular file, which happens to be an mkv. If the loaded file was an mp4 file (such as the file we're discussing), you would see "LAV Splitter Source" there instead in my setup. With MPC-HC's internal mp4 splitter, you would see "MPC MP4/MOV Source". I can reproduce your issue, and I'm pretty sure that's it.
Posted: 3/7/2012 4:19 PM
Quote
feos
Site Admin, Skilled player (1255)
Joined: 4/17/2010
Posts: 11492
Location: Lake Char­gogg­a­gogg­man­chaugg­a­gogg­chau­bun­a­gung­a­maugg
NES TAS of 2011
It works after I disabled internal mp4 filter, even with only LAV Video Decoder preferred. Thanks.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Posted: 3/7/2012 4:23 PM
Quote
natt
Editor, Emulator Coder, Site Developer
Joined: 5/11/2011
Posts: 1108
Location: Murka
feos wrote:
It would be nice if anyone fixes [1920] NES Battletoads "warps, 2 players" by feos & MESHUGGAH in 11:04.72. I have configured my MPC to play 444 encodes correctly, but when I try to watch THIS encode, it shuts down and can't play even audio. Using LAV video output (other 444 encodes play fine). It was said that the problem was in MP4Box.
Corrected
Posted: 3/7/2012 8:29 PM
Post subject: Re: 10-bit 4:4:4 publication encode issues
Quote
natt
Editor, Emulator Coder, Site Developer
Joined: 5/11/2011
Posts: 1108
Location: Murka
Lex wrote:
http://tasvideos.org/1588M.html (BrandonE; a/v desync; very late audio)
It looks like there was some sort of muxing issue. I get various forms of broken depending on what mp4 splitter I use. I think I've corrected the problem, but because this is unfamiliar territory I'd appreciate someone else checking on their player before I update the central record. 22MiB: http://www.mediafire.com/?kv7itwbk0avrbkw
Posted: 3/8/2012 2:02 AM
Quote
Lex
Joined: 6/25/2007
Posts: 732
Location: Vancouver, British Columbia, Canada
Isn't the obvious solution to just use mkv instead? :P Just because the original is mp4 doesn't mean the fixed one has to be, I think. Anyway, yeah, that seems to have fixed it here. I tested with LAV Splitter and MPC-HC's internal mp4 splitter. Nice job!
Posted: 3/8/2012 2:10 AM
Quote
natt
Editor, Emulator Coder, Site Developer
Joined: 5/11/2011
Posts: 1108
Location: Murka
Lex wrote:
Isn't the obvious solution to just use mkv instead? :P Just because the original is mp4 doesn't mean the fixed one has to be, I think. Anyway, yeah, that seems to have fixed it here. I tested with LAV Splitter and MPC-HC's internal mp4 splitter. Nice job!
How to use mkv though? Timecodes are required for the video track, and I'm sure they could be made from an NHML file, but I don't have a tool to do so.
Posted: 3/8/2012 2:13 AM
Quote
Lex
Joined: 6/25/2007
Posts: 732
Location: Vancouver, British Columbia, Canada
I discovered yesterday that Aegisub can save a timecodes file in mkv timecodes v2 format, and tested just now whether it works from mp4, and it does! :) It's very easy to do. Just load the video in Aegisub and click "Video" -> "Save Timecodes File...". Edit: Hmm. I tried to mux an mkv from using that video's timecodes file, video track, and audio track. It didn't work out very well. The video track ran at the expected rate and looked very good, but the audio was really early. Was something crazy done on a container level to align the audio to the video? Edit: I added a container-level audio track delay of 666 ms and now the a/v sync seems to be perfect. The timecodes file tipped me off to a good delay value, as 666 was the first timecode in the file. Here it is in glorious synced mkv format. I tested it using Haali Splitter, LAV Splitter, MPC-HC's internal matroska splitter, and even VLC (2.0). All splitters worked perfectly for this video. :)
Posted: 3/8/2012 5:36 AM
Quote
natt
Editor, Emulator Coder, Site Developer
Joined: 5/11/2011
Posts: 1108
Location: Murka
Fixed: http://tasvideos.org/1269M.html (BrandonE; a/v desync; very late audio) http://tasvideos.org/1588M.html (BrandonE; a/v desync; very late audio)
Posted: 3/8/2012 11:23 PM
Quote
ledauphinbenoit
Joined: 8/29/2005
Posts: 148
Location: Dayton, OH
Is there any way to easily/systematically detect when this issue occurs? Lex stated he used some very specific settings, but I'm not sure if in the end it wasn't just an eyeball test to see if the range was limited. If we can test it, I would suggest we need to add this to the encoding guide wiki pages. I examine the log/output files for each of my encodes and no errors were present (since this was the result of a x264 bug). Not to mention that Grunt and Ilari, as well as others watched the encodes and did not notice the problem. This is likely due to not having our settings in a state where we could see the problems. I'm less concerned about having to do some re-encode some movies than I am about preventing something like this from happening in the future. Was this issue really detectable by anyone except someone looking for it? Also, the encoding guide has no mention of 10bit444 or any kind of standards for it. It has been stated elsewhere that eventually this might become the "standard" encode for each movie. Like Lex, I want to make encodes of the highest quality, should there be more criteria listed somewhere of what is unacceptable?
Posted: 3/9/2012 12:32 AM
Quote
Lex
Joined: 6/25/2007
Posts: 732
Location: Vancouver, British Columbia, Canada
Yes, the issue is obvious in any video playback system which properly handles the range flag. Those other people who watched the encodes either: were using incorrect video playback systems, didn't notice, didn't care enough to want it fixed, or wanted it fixed but didn't know how to go about suggesting (how) to do it. As far as I know, there's no ready-made program which can detect this issue automatically. Since the issue is so obvious in any correct video playback system, an eyeball test with an understanding of color ranges and their surrounding settings should be considered good enough for this particular issue. Considering your recent test video you uploaded to mediafire and linked in #tasvideos on freenode, this issue seems to be fixed for your future encodes. As for past encodes, I'd like to see them fixed, but I can't and won't demand anything, being a lazy guy myself. The main reason I posted this thread was to bring the problem into the open and organize it rather than piping up in IRC every once in a while like I did in the past.
Posted: 3/27/2012 12:18 AM
Quote
ledauphinbenoit
Joined: 8/29/2005
Posts: 148
Location: Dayton, OH
1944M, 1943M, and 1941M have all been fixed. I don't have dumps for any of the earlier encodes which are mine, so I probably won't be doing those anytime soon. There is other stuff I'd like to get to encoding at the moment.
Posted: 3/27/2012 5:46 AM
Quote
Lex
Joined: 6/25/2007
Posts: 732
Location: Vancouver, British Columbia, Canada
Thanks! Those are updated on the list now. It's great to see that so many have been fixed! :D
Posted: 9/13/2012 4:43 PM
Quote
Dacicus
Editor, Player (69)
Joined: 6/22/2005
Posts: 1050
If someone could check this 10-bit encode for 1935M to make sure that it meets site standards and I didn't mess up my scripts since last encoding, I'll replace the bad one and fix the rest of mine: http://www.mediafire.com/?7ialkk1r344qa6q Thanks!
Current Projects: TAS: Wizards & Warriors III.
Posted: 9/14/2012 11:50 AM
Quote
natt
Editor, Emulator Coder, Site Developer
Joined: 5/11/2011
Posts: 1108
Location: Murka
Dacicus wrote:
If someone could check this 10-bit encode for 1935M to make sure that it meets site standards and I didn't mess up my scripts since last encoding, I'll replace the bad one and fix the rest of mine: http://www.mediafire.com/?7ialkk1r344qa6q Thanks!
looks good to me
Posted: 9/23/2012 4:41 AM
Quote
Dacicus
Editor, Player (69)
Joined: 6/22/2005
Posts: 1050
All of my 10-bit encodes from the list have been fixed now.
Current Projects: TAS: Wizards & Warriors III.
Posted: 9/23/2012 4:50 AM
Quote
Aktan
Publisher
Joined: 4/23/2009
Posts: 1283
natt wrote:
How to use mkv though? Timecodes are required for the video track, and I'm sure they could be made from an NHML file, but I don't have a tool to do so.
Old and very late, but MKVMerge reads VFR MP4s fine, you just add the file and mux. The resulting MKV is correct.