Posted: 7/29/2011 7:33 PM
Quote
creaothceann
He/Him
Editor
Joined: 4/7/2005
Posts: 1874
Location: Germany
Why is dedup needed btw? Identical frames would be caught by the codec.
Posted: 7/29/2011 8:28 PM
Quote
Ilari
Emulator Coder, Skilled player (1114)
Joined: 5/1/2010
Posts: 1217
Rookie of 2010 New systems TASer of 2010 New systems TAS of 2010
creaothceann wrote:
Why is dedup needed btw? Identical frames would be caught by the codec.
Depends on the codec. x264 won't catch them well enough. There is sizable difference between dedupped and non-dedupped sizes if run has lots of duplicate frames.
Posted: 7/29/2011 8:32 PM
Quote
Aktan
Publisher
Joined: 4/23/2009
Posts: 1283
What do you mean by caught? If you mean they detect a duplicate frame, but encode it anyway with some small amount of data, I guess you are correct. But if you meant the duplicate frame is dropped, then you are incorrect, since it is not. There are other benefits of removing duplicate frames. For example, H.264 has a maximum reference size of 16 frames ahead. If the duplicate frames were left there, then the amount of reference frames is actually smaller. Also while the encoded duplicate frames are quite small, they still do take space. Basically, in general, the dedup encode is smaller because we remove duplicates.
Posted: 7/29/2011 8:39 PM
Quote
sgrunt
He/Him
Emulator Coder, Former player
Joined: 10/28/2007
Posts: 1360
Location: The dark horror in the back of your mind
creaothceann wrote:
Identical frames would be caught by the codec.
Not true, in x264's case. Allow me to illustrate. (Most of the files I refer to in the following are available here, for the curious.) I made this image while [thread 11511]testing media player colour spaces[/thread], and converted it into a ten second, 60fps test clip in FFV1/BGR32 (cc10.avi). From here, I applied two workflows.
  • First, I used ffmpeg to directly convert the AVI into a raw YV12 input to be used with x264: 
    ffmpeg -i cc10.avi -pix_fmt yuvj420p -vcodec raw -f rawvideo cc10raw.yv12
    I then encoded this losslessly with x264: 
    x264 --fps 60 --input-res 256x256 --demuxer raw cc10raw.yv12 --crf 0 --fullrange on  -o cc10nodedup.mp4
  • For the second, I used my [wiki DedupC]dedup filter[/wiki] in conjunction with ffmpeg to generate a dedupped raw YV12 for input to x264: 
    ffmpeg -i cc10.avi -pix_fmt rgb24 -vcodec rawvideo -f rawvideo - | ~/dedup times.txt 60 1 256 256 20 | ffmpeg -r 60 -s 256x256 -pix_fmt rgb24 -f rawvideo -i - -pix_fmt yuvj420p -f rawvideo -vcodec rawvideo cc10dedup.yv12
    I then encoded this losslessly with x264: 
    x264 --input-res 256x256 --demuxer raw cc10dedup.yv12 --crf 0 --fullrange on --tcfile-in times.txt -o cc10dedup.mp4
The resulting cc10dedup is 26047 bytes, and the resulting cc10nodedup is 87956 bytes. There's a clear benefit. EDIT: In the interest of full disclosure, 
$ x264 --version
x264 0.116.2037 f8ebd4a
(libswscale 2.0.0)
(ffmpegsource 2.15.4.1)
built on Jul 29 2011, gcc: 4.6.1
configuration: --bit-depth=8
x264 license: GPL version 2 or later
libswscale/ffmpegsource license: GPL version 3 or later
$ ffmpeg -version
ffmpeg version 0.8, Copyright (c) 2000-2011 the FFmpeg developers
  built on Jul 29 2011 14:23:50 with gcc 4.6.1
  configuration: --prefix=/usr --libdir=/usr/lib64 --shlibdir=/usr/lib64 --mandir=/usr/share/man --enable-shared --cc=x86_64-pc-linux-gnu-gcc --disable-static --enable-gpl --enable-version3 --enable-postproc --enable-avfilter --disable-stripping --disable-debug --disable-doc --enable-libmp3lame --enable-libvo-aacenc --enable-libtheora --enable-libvorbis --enable-libx264 --enable-libdc1394 --disable-indev=oss --enable-x11grab --disable-outdev=oss --enable-libfreetype --enable-pthreads --enable-libgsm --enable-libdirac --enable-libschroedinger --enable-libopenjpeg --disable-altivec --disable-avx --cpu=core2 --enable-hardcoded-tables
  libavutil    51.  9. 1 / 51.  9. 1
  libavcodec   53.  7. 0 / 53.  7. 0
  libavformat  53.  4. 0 / 53.  4. 0
  libavdevice  53.  1. 1 / 53.  1. 1
  libavfilter   2. 23. 0 /  2. 23. 0
  libswscale    2.  0. 0 /  2.  0. 0
  libpostproc  51.  2. 0 / 51.  2. 0
ffmpeg 0.8
libavutil    51.  9. 1 / 51.  9. 1
libavcodec   53.  7. 0 / 53.  7. 0
libavformat  53.  4. 0 / 53.  4. 0
libavdevice  53.  1. 1 / 53.  1. 1
libavfilter   2. 23. 0 /  2. 23. 0
libswscale    2.  0. 0 /  2.  0. 0
libpostproc  51.  2. 0 / 51.  2. 0
Posted: 7/30/2011 5:46 PM
Quote
creaothceann
He/Him
Editor
Joined: 4/7/2005
Posts: 1874
Location: Germany
I see. Thank you for the answers. :)