Now that madvr actually supports YCgCo, I decided to give it a bit of a test. The run is 1247M. If you'd like the uncompressed dump I worked from, just give me a holler.
I'm comparing a 10 bit 4:4:4: YCbCr (PC.601) encode to a 10 bit 4:4:4 YCgCo (fullrange) encode. The YCbCr encode was done at --crf 18 using my normal quality assessments for publication. The YCgCo encode was two-passed to match the YCbCr on size, and is within 0.2% of that target size.
Command lines and x264 outputs:
avs2pipemod\avs2pipemod.exe -rawvideo=vflip 1247M.avs | dedupc\dedupc_color.exe NUL 60 1 240 160 yuv444p10le | x264\x264_64_10.exe --crf 18.000000 --preset placebo --rc-lookahead 120 --keyint 400 --me umh --subme 10 --merange 32 --input-fmt raw --input-csp i444 --input-res 240x160 --input-depth 10 --output-csp i444 --colorprim bt709 --transfer bt470m --colormatrix bt470bg --input-range pc --sar 1:1 --tcfile-in temp\times.txt -o temp\hi10.h264 -
avs2pipemod\avs2pipemod.exe -rawvideo=vflip 1247M.avs | dedupc\dedupc_color.exe NUL 60 1 240 160 ycgco10 | x264\x264_64_10.exe --bitrate 337 --preset placebo --rc-lookahead 120 --keyint 400 --me umh --subme 10 --merange 32 --input-fmt raw --input-csp i444 --input-res 240x160 --input-depth 10 --output-csp i444 --colorprim bt709 --transfer bt470m --colormatrix YCgCo --input-range pc --sar 1:1 --slow-firstpass --pass 1 --stats temp\ycgco.stats --tcfile-in temp\times.txt -o temp\ycgco1.h264 -
avs2pipemod\avs2pipemod.exe -rawvideo=vflip 1247M.avs | dedupc\dedupc_color.exe NUL 60 1 240 160 ycgco10 | x264\x264_64_10.exe --bitrate 337 --preset placebo --rc-lookahead 120 --keyint 400 --me umh --subme 10 --merange 32 --input-fmt raw --input-csp i444 --input-res 240x160 --input-depth 10 --output-csp i444 --colorprim bt709 --transfer bt470m --colormatrix YCgCo --input-range pc --sar 1:1 --slow-firstpass --pass 3 --stats temp\ycgco.stats --tcfile-in temp\times.txt -o temp\ycgco3.h264 -
x264 output (YCbCr)
raw [info]: 240x160p 1:1 @ 25/1 fps (cfr)
timecode [info]: automatic timebase generation 1/60
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.2 AVX
x264 [info]: profile High 4:4:4 Predictive, level 2.1, 4:4:4 10-bit
avs2pipemod[info]: invoking FlipVertical() ...
avs2pipemod[info]: writing 24551 frames of 240x160 BGR rawvideo.
avs2pipemod[info]: finished, wrote 24551 frames [100%].
avs2pipemod[info]: total elapsed time is 166.957 sec [147.050fps].
dedupc[info]: wrote 20309 out of 24551 frames
x264 [info]: frame I:67 Avg QP:32.76 size: 18998
x264 [info]: frame P:4730 Avg QP:38.65 size: 1533
x264 [info]: frame B:15512 Avg QP:41.77 size: 563
x264 [info]: consecutive B-frames: 4.5% 4.7% 9.2% 23.1% 11.7% 10.5% 6.3% 5.
5% 3.0% 5.0% 5.2% 4.8% 2.4% 2.3% 0.6% 0.4% 0.6%
x264 [info]: mb I I16..4: 3.1% 11.7% 85.1%
x264 [info]: mb P I16..4: 1.1% 1.3% 1.0% P16..4: 28.8% 5.0% 7.1% 0.1% 0
.4% skip:55.1%
x264 [info]: mb B I16..4: 0.1% 0.1% 0.1% B16..8: 16.8% 3.2% 2.9% direct:
1.6% skip:75.3% L0:48.7% L1:44.8% BI: 6.5%
x264 [info]: 8x8 transform intra:32.4% inter:11.0%
x264 [info]: direct mvs spatial:97.5% temporal:2.5%
x264 [info]: coded y,u,v intra: 43.8% 30.4% 28.1% inter: 6.6% 2.8% 3.1%
x264 [info]: i16 v,h,dc,p: 64% 32% 4% 0%
x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 6% 5% 84% 1% 1% 1% 1% 1% 1%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 21% 18% 16% 6% 7% 7% 7% 8% 10%
x264 [info]: Weighted P-Frames: Y:4.2% UV:3.0%
x264 [info]: ref P L0: 36.2% 10.4% 12.7% 3.7% 4.9% 4.0% 5.5% 2.0% 2.6% 2.
4% 2.8% 2.6% 2.9% 2.7% 4.0% 0.7%
x264 [info]: ref B L0: 65.3% 10.4% 6.6% 3.2% 2.6% 2.2% 2.1% 1.1% 1.1% 1.
1% 1.0% 1.1% 1.1% 0.8% 0.4%
x264 [info]: ref B L1: 89.1% 10.9%
x264 [info]: kb/s:337.09
x264 output (YCgCo second pass)
raw [info]: 240x160p 1:1 @ 25/1 fps (cfr)
timecode [info]: automatic timebase generation 1/60
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.2 AVX
avs2pipemod[info]: invoking FlipVertical() ...
avs2pipemod[info]: writing 24551 frames of 240x160 BGR rawvideo.
x264 [info]: profile High 4:4:4 Predictive, level 2.1, 4:4:4 10-bit
avs2pipemod[info]: finished, wrote 24551 frames [100%].
avs2pipemod[info]: total elapsed time is 152.812 sec [160.661fps].
dedupc[info]: wrote 20309 out of 24551 frames
x264 [info]: frame I:69 Avg QP:33.44 size: 18409
x264 [info]: frame P:4771 Avg QP:38.90 size: 1536
x264 [info]: frame B:15469 Avg QP:42.06 size: 558
x264 [info]: consecutive B-frames: 4.7% 4.9% 9.3% 23.0% 11.2% 11.0% 6.2% 5.
8% 3.2% 5.1% 5.4% 4.5% 2.3% 2.1% 0.4% 0.4% 0.5%
x264 [info]: mb I I16..4: 2.9% 11.8% 85.2%
x264 [info]: mb P I16..4: 1.2% 1.3% 0.9% P16..4: 28.9% 5.1% 7.1% 0.1% 0
.4% skip:54.9%
x264 [info]: mb B I16..4: 0.1% 0.1% 0.1% B16..8: 16.6% 3.2% 2.8% direct:
1.6% skip:75.5% L0:49.4% L1:44.1% BI: 6.5%
x264 [info]: 8x8 transform intra:31.1% inter:11.2%
x264 [info]: direct mvs spatial:90.3% temporal:9.7%
x264 [info]: coded y,u,v intra: 45.5% 26.5% 33.1% inter: 6.5% 2.1% 3.7%
x264 [info]: i16 v,h,dc,p: 64% 31% 4% 0%
x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 6% 5% 83% 1% 1% 1% 1% 1% 2%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 21% 18% 16% 6% 7% 7% 7% 8% 10%
x264 [info]: Weighted P-Frames: Y:4.5% UV:3.1%
x264 [info]: ref P L0: 36.2% 20.0% 15.7% 6.0% 5.1% 4.3% 3.8% 1.7% 1.4% 1.
2% 1.1% 1.0% 0.8% 0.8% 0.7% 0.1%
x264 [info]: ref B L0: 65.0% 16.1% 7.9% 3.1% 2.2% 1.6% 1.3% 0.6% 0.5% 0.
4% 0.3% 0.3% 0.3% 0.2% 0.1%
x264 [info]: ref B L1: 89.2% 10.8%
x264 [info]: kb/s:336.69
The encodes are on archive.org:
Collection
http://archive.org/details/GbaCastlevaniaAriaOfSorrowusajuliusVersionIn0438.9ByCpadolf
YCbCr
http://archive.org/download/GbaCastlevaniaAriaOfSorrowusajuliusVersionIn0438.9ByCpadolf/castlevaniaaos-tas-cpadolf_10bit444.mkv
YCgCo
http://archive.org/download/GbaCastlevaniaAriaOfSorrowusajuliusVersionIn0438.9ByCpadolf/castlevaniaaos-tas-cpadolf_ycgco.mkv
The YCbCr encode looks slightly more reddish to me when I A/B view them (I guess we'll never get these things perfect...) Other than that, I wouldn't say either one is terribly better than the other. My conclusion is, don't bother with YCgCo for final encodes.