Post by Warp

Posted: 5/20/2010 6:04 PM

Quote

SatoshiLyish

Joined: 3/14/2005
Posts: 43

Aktan wrote:

The param there won't hurt it like Johannes found out. x264 just ignores it. Flygon should know very well by now that it isn't really lossless since I've been drilling that into him. XD

Well, I wasn't sure whether it ignored the command or not. You can never be too sure with programmer's implementation..But in terms of script efficiency, B frames shouldn't even be mentioned if its a lossless encode.

moozooh wrote:

Well, it's not that they're limited, it's just the established convention. We often increase the encode's resolution on games that can be rendered at higher resolution with no fidelity loss (most N64 games, for example). Still, even without doublescaling it's not at all pointless to convert to YV12 lossless as an intermediary step for several reasons: 1) it makes editing and processing easier (no need to pipe the emulator output several times in a row, especially in case with N64 games rendered at high resolution); 2) it allows for higher-quality streaming encodes on YT and other sites.

Oh, well if published runs have doubled the resolution on some occasion, I guess I shouldn't be too concerned about that guideline. As for master encoding before the final process, I now use FFV1 (Final Fantasy VI anyone? ^^) in RGB32 mode, thanks to the suggestion from sgrunt. That way I know everything is bit perfect before final encode. (although I can't imagine uploading one of those to Youtube, unless it was a 1 minute clip or something ^^;)

Posted: 5/20/2010 7:29 PM

Quote

Aktan

Publisher

Joined: 4/23/2009
Posts: 1283

SatoshiLyish wrote:

Well, I wasn't sure whether it ignored the command or not. You can never be too sure with programmer's implementation..But in terms of script efficiency, B frames shouldn't even be mentioned if its a lossless encode.

I was the one who found the bug that lossless mode still had b-frames back in 2008, so I know the fix was to just ignore it. Plus the encoding generally is faster doing lossless than not (because lack of b-frames) which is why Flygon would even do these insane settings. These settings with b-frames would be a lot slower.

SatoshiLyish wrote:

As for master encoding before the final process, I now use FFV1 (Final Fantasy VI anyone? ^^) in RGB32 mode, thanks to the suggestion from sgrunt. That way I know everything is bit perfect before final encode. (although I can't imagine uploading one of those to Youtube, unless it was a 1 minute clip or something ^^;)

Usually I do upload near lossless to YT (YT doesn't support lossless H.264), so what moozooh said is correct. Lossless does have the advantage of having no lossy artifacts also, dispite the color loss =p.

Posted: 5/20/2010 8:24 PM

Quote

SatoshiLyish

Joined: 3/14/2005
Posts: 43

Aktan wrote:

I was the one who found the bug that lossless mode still had b-frames back in 2008, so I know the fix was to just ignore it. Plus the encoding generally is faster doing lossless than not (because lack of b-frames) which is why Flygon would even do these insane settings. These settings with b-frames would be a lot slower.

Very interesting. Also, I think I'll have to take back what I said about lossless being pointless since it wasn't true lossless. I've been reading up on the "Encode to Lossless x264" thread, and I didn't realize that it was possible to get lower file sizes than some q>0 encodes (my experience with all other encodes and codecs proved to be otherwise). I guess anything that increases the quality:size ratio is worth it.

Aktan wrote:

Usually I do upload near lossless to YT (YT doesn't support lossless H.264), so what moozooh said is correct. Lossless does have the advantage of having no lossy artifacts also, dispite the color loss =p.

With Youtube I try to approach the max upload size per clip, but I won't go out of my way wasting space just to do so. ^^; Anyway, I think the best way to preserve your settings is to upload an .flv instead. (currently experimenting with that..I want to somehow get 60fps video on there..which is strange because I remember it used to work..probably when HD came out that changed things) Also, I HAVE figured out a way to encode 'true' lossless to x264, and all that's required is downloaded the latest AVISynth 2.6.0 beta (or alpha?) and using the following script: source=ConvertToYV24(ffdshow_source()) return (source) However, there's currently no decoder that will process the video correctly. What you get is the full luma displaying correctly, with the top left quarter of the chroma displayed on top of it. Strange looking to say the least. The only thing I could see this site using it for is for "future archiving" when yv24 decoding becomes more mainstream/supported. File sizes are a lot bigger. 4:2:0 q0 is 1.9MB, q1 1.6MB; 4:4:4 q0 6.71MB, q1 8.28MB. LOL, guess that just proved lossless encoding efficiency right there.

Posted: 5/21/2010 12:06 AM

Quote

Aktan

Publisher

Joined: 4/23/2009
Posts: 1283

SatoshiLyish wrote:

Very interesting. Also, I think I'll have to take back what I said about lossless being pointless since it wasn't true lossless. I've been reading up on the "Encode to Lossless x264" thread, and I didn't realize that it was possible to get lower file sizes than some q>0 encodes (my experience with all other encodes and codecs proved to be otherwise). I guess anything that increases the quality:size ratio is worth it.

Unfortunately that thread, (which was started by me, right?) is a bit outdated now. After finding out out the different ways I could convert to YV12 from RGB32, in which some retains more or less color, I don't think encoding to lossless can be as small as it used to be. Apparently the way I converted to YV12 before was a simple method making it easier to encode at the cost of the colors are not as accurate. The method I now use to convert to YV12 now retains (all subjective of course) more color, but making the file size larger in which I've noticed most of the time it be smaller to just go lossy.

SatoshiLyish wrote:

Also, I HAVE figured out a way to encode 'true' lossless to x264, and all that's required is downloaded the latest AVISynth 2.6.0 beta (or alpha?) and using the following script: source=ConvertToYV24(ffdshow_source()) return (source) However, there's currently no decoder that will process the video correctly. What you get is the full luma displaying correctly, with the top left quarter of the chroma displayed on top of it. Strange looking to say the least. The only thing I could see this site using it for is for "future archiving" when yv24 decoding becomes more mainstream/supported. File sizes are a lot bigger. 4:2:0 q0 is 1.9MB, q1 1.6MB; 4:4:4 q0 6.71MB, q1 8.28MB. LOL, guess that just proved lossless encoding efficiency right there.

I don't think this works. Even if you convert to YV24 in avisynth, x264 auto converts it to YV12 (at least in the newer versions). I think you are getting the errors you see because x264 assumes it's YV12 and encodes as such. I'm not sure though. I'll do some of my own testing. It is common to see q1 is better than q0 actually. I'm not so surprised by that really since some of the info might be just pure motion in q0, thus saving space.

Posted: 5/21/2010 12:37 AM

Quote

SatoshiLyish

Joined: 3/14/2005
Posts: 43

Aktan wrote:

Unfortunately that thread, (which was started by me, right?) is a bit outdated now. After finding out out the different ways I could convert to YV12 from RGB32, in which some retains more or less color, I don't think encoding to lossless can be as small as it used to be. Apparently the way I converted to YV12 before was a simple method making it easier to encode at the cost of the colors are not as accurate. The method I now use to convert to YV12 now retains (all subjective of course) more color, but making the file size larger in which I've noticed most of the time it be smaller to just go lossy.

I would be very interested in your conversion method. Even if there were YV24 decoders I would have to go all the way down to q40-45 in order to get the same file size as the 4:2:0 material at q20..and in general it ended up looking worse (compression artifacts). Maybe 2x Resolution is the only real way to go about it..and if I have to use 4:2:0 I'd rather get the colors accurate due to a better dithering method.

Aktan wrote:

I don't think this works. Even if you convert to YV24 in avisynth, x264 auto converts it to YV12 (at least in the newer versions). I think you are getting the errors you see because x264 assumes it's YV12 and encodes as such. I'm not sure though. I'll do some of my own testing.

Well, when I started encoding I got a "4:4:4 Predictive" popup, so I'm assuming it was reading and encoding as YV24 material. Also, where is all the extra space coming from in the file sizes? I somehow doubt its all junk data in there.

Posted: 5/21/2010 1:42 PM

Quote

Aktan

Publisher

Joined: 4/23/2009
Posts: 1283

SatoshiLyish wrote:

I would be very interested in your conversion method. Even if there were YV24 decoders I would have to go all the way down to q40-45 in order to get the same file size as the 4:2:0 material at q20..and in general it ended up looking worse (compression artifacts). Maybe 2x Resolution is the only real way to go about it..and if I have to use 4:2:0 I'd rather get the colors accurate due to a better dithering method.

Using AviSynth 2.60 Alpha I do the following lines: ConvertToYV24(chromaresample="point") ConvertToYV12(chromaresample="lanczos4")

SatoshiLyish wrote:

Well, when I started encoding I got a "4:4:4 Predictive" popup, so I'm assuming it was reading and encoding as YV24 material. Also, where is all the extra space coming from in the file sizes? I somehow doubt its all junk data in there.

Ah, but it always says that even when encoding with YV12 source. I really do think the increase in file size is due to garbage data. Garbage data can be hard to encode =p.

Posted: 5/21/2010 3:09 PM

Quote

SatoshiLyish

Joined: 3/14/2005
Posts: 43

Aktan wrote:

Using AviSynth 2.60 Alpha I do the following lines: ConvertToYV24(chromaresample="point") ConvertToYV12(chromaresample="lanczos4")

Thanks, I'll check it out after work today.

Aktan wrote:

Ah, but it always says that even when encoding with YV12 source. I really do think the increase in file size is due to garbage data. Garbage data can be hard to encode =p.

Ugh, I must have selective vision or whatever..I see it popping up with the normal material now. So much for that :/ ...if only there was a proper decoder so I could verify for sure.. *just realizes FFVI had 444P output*..*does a quick encode to rule out ffdshow's faulty decoding*..bah, x264 downsamples to yv12. Least I know for sure ffdshow was reconverting back to YV12 After all that mess. -_-;

Posted: 5/21/2010 7:55 PM

Quote

Aktan

Publisher

Joined: 4/23/2009
Posts: 1283

SatoshiLyish wrote:

Ugh, I must have selective vision or whatever..I see it popping up with the normal material now. So much for that :/ ...if only there was a proper decoder so I could verify for sure.. *just realizes FFVI had 444P output*..*does a quick encode to rule out ffdshow's faulty decoding*..bah, x264 downsamples to yv12. Least I know for sure ffdshow was reconverting back to YV12 After all that mess. -_-;

Another proof that x264 doesn't have 4:4:4 yet is this: http://x264dev.multimedia.cx/?p=332 One of the projects is to add that support!

Posted: 5/23/2010 10:36 PM

Quote

SatoshiLyish

Joined: 3/14/2005
Posts: 43

quote="Aktan"]Another proof that x264 doesn't have 4:4:4 yet is this: http://x264dev.multimedia.cx/?p=332 One of the projects is to add that support![/quote] Sounds good to me! But for now, I'll just work with what I have until it becomes supported. Okay, to get back on track with the thread topic, I've been studying the options for the deldup function. From what I can understand, Flygon's implementation would be minfps=0.1:lthresh=0.1:mbthresh=100:mbmax=1:cthresh=0.1. minfps is simple enough. 0.1fps would be one frame per 10 seconds. This is most likely the culprit of the strange tracking Flygon experienced in VLC, but I don't think that that in itself should be that big of an issue..if it is 1 should work perfectly fine. Lthresh..would be the threshold of the luma (Y) channel. If I'm understanding correctly, the addition of every pixel's SAD will be up to this value. Its defined by X*Y*Lthresh, so, if the value is 0.1 that would mean its 10% of the total amount of pixels. I ran some tests on this, from 100 jumping all the way down to 0.00001, and to be honest I don't think this value means a whole lot. MBthresh has a greater factor in terms of what frames gets dropped. MBthresh and MBmax..if the number of 8x8 blocks in the image with SAD larger than MBthresh (100) exceeds MBmax (1), then the next frame is a unique image and is kept. In the documentation I'd like to point out that it specifically says it has to "exceed" MBmax (technically says MBThresh, but I'm sure that's a typo), so I think an MBmax value of 0 would be better, since the current values would dictate that you'd need two blocks to be different. Here I'll say that only when I set LThresh to 0.00001, with MBThresh at 100, that the encoder kept an excessive amount of frames. (0.00002 still seemed to still work properly) I think it had to do with SAD being under 1 total as to why it wasn't working. CThresh is the threshold of the chroma (U & V) channel. Works mostly the same way as LThresh, only when I tried disabling Lthresh to test Cthresh only, it wouldn't encode. I was considering maybe you wouldn't really need this value and you can assign it a negative in order to disable it and increase encoding speeds, but since SAD calculation appears to be very fast with vast implementation, the benefit would be very minimal. There's also the chance of two colors having the same Luma value, so it wouldn't be the wisest thing. Anyway, I think adequate values for C & LThresh would be 1, since it'd equal the amount of available pixels..anyway because of MBThresh being so low the number doesn't have to be excessively large to begin with...assuming I understand how SAD works (is it just, for CThresh as example, one pixel's chroma minus other pixel's chroma value and stored as an absolute?). MBThresh..it all depends on how sensitive you want the filter to be in keeping non-duplicate frames. Since I work with FFV1 (lossless RGB32 compression), assigning it to 1 would make it the most sensitive to any changes in the video, but I could very well see it keeping all the frames in an already compressed video. More tests would need to be run for this, but I think the current value of 100 works fine.

Posted: 5/23/2010 10:38 PM

Quote

sgrunt

He/Him

Emulator Coder, Former player

Joined: 10/28/2007
Posts: 1360
Location: The dark horror in the back of your mind

To note: I originally came up with those settings for deldup; it was the first set I came up with that dealt reasonably well with the two videos for which problems had previously been observed (Bisqwit's no-friction Lunar Ball run and my CK5 run). They could possibly be fine tuned further.

Posted: 5/23/2010 11:10 PM

Quote

SatoshiLyish

Joined: 3/14/2005
Posts: 43

sgrunt wrote:

To note: I originally came up with those settings for deldup; it was the first set I came up with that dealt reasonably well with the two videos for which problems had previously been observed (Bisqwit's no-friction Lunar Ball run and my CK5 run). They could possibly be fine tuned further.

hmm..granted I've only tested with low motion/similar colored movement in NES Zelda, so I'll look into those runs for further testing. I started looking into other values, like subme and merange. I do feel 512 for merange is excessive, unless I know the character will wrap around to the other side of the screen (LoZ 2nd Quest Glitched as an example) . That also depends on if the encoder is scanning the range of a pixel on the edge of the screen, if it compensates the range of pixels off the edge by wrapping around to the other side. Subme..sounds a lot like the pel option in AviSynth for MVAnalyse. Is it really necessary to scan subpixels for games with bitmap material? I could see it being useful for polygonal games, but not for games on older systems like SNES or Gameboy.

Posted: 5/24/2010 2:31 AM

Quote

Flygon

Joined: 11/4/2007
Posts: 1772
Location: Australia, Victoria

About the merange, as far as I am aware, it actually caps off at 256 for NES games... I just put it at 512 because I like powers of two and that it works well for extremely basic Mega Drive games (Where it caps off at 320). Basically, from what I recall, it's pixel based. I felt I should clear this up, just in case.

Posted: 5/24/2010 5:32 AM

Quote

SatoshiLyish

Joined: 3/14/2005
Posts: 43

Flygon wrote:

About the merange, as far as I am aware, it actually caps off at 256 for NES games... I just put it at 512 because I like powers of two and that it works well for extremely basic Mega Drive games (Where it caps off at 320). Basically, from what I recall, it's pixel based. I felt I should clear this up, just in case.

What kind of an improvement is it? Is it lower file size? Better visuals? I can't imagine it being very efficient, nor very beneficial..its akin to losing your keys in a parking lot. You can scour every inch of the lot from left to right, until you find them, or you can search the areas you remember being last, and most likely find them that way in a lot less time. Both results in you just merely "finding your keys". Most game animations wouldn't even move that fast within one frame, unless it teleported or scrolled off one edge of the screen to the other. Its the only instances I can think of for needing (and that's stretching it) to use such a high value. Anyway, I did more testing with subme..the higher values significantly chops down the file size. subme 5 starts to get pretty close to 10, but I haven't tested fast motion video yet, so I'm sure there's a pretty good reason for implementing it. I'm curious about the quality with high b-frame rates, so I'll start messing with that.

Posted: 5/24/2010 7:31 AM

Quote

Flygon

Joined: 11/4/2007
Posts: 1772
Location: Australia, Victoria

I will admit that the value, for a normal movie, is very excessive, but the point really is just to chop down the filesize as much as possible. I just use it because I'm patient with how long it takes to encode movies (I am happy to wait at 1.5fps for a NES game to encode), but I will admit that a merange of 64 does make more sense for more advanced consoles. I do know that not many game have enough motion to really require such a high value... but, well, it does cut down some size either way. I should probably make a proof of example encode with the Chessmasters game that's currently on the workbench, for comparing the various meranges on an incredibly basic game (However much so basic is an overstatement). That, and such tests are extremely dial-up friendly, heh.

Posted: 5/24/2010 11:08 AM

Quote

moozooh

Senior Moderator

Joined: 8/4/2005
Posts: 5770
Location: Away

ME range isn't about the amount of motion, it's about pixels traveling a maximum distance to be still recognized as vectors. For instance, if at least something in your game is scrolling at 32 pixels/frame, then merange=32 is warranted to pick it up. If nothing does, there will be no benefit whatsoever. But I'm fairly sure that the motion search value is essentially multiplied by the amount of reference frames; i. e., with merange=32 and ref=16 it will search not only the 32 pixels around the given pixel for one with the same attributes compared to the previous frame, but comparing to 16 previous frames. This needs confirmation, though.

Warp wrote:

Edit: I think I understand now: It's my avatar, isn't it? It makes me look angry.

Posted: 6/30/2010 3:08 AM

Quote

C0DE_RED

Banned User, Former player

Joined: 5/22/2010
Posts: 80
Location: WAR TOWN, Georgia (USA)

Flygon: That command prompt you posted works best on Linux, right?! If so, what's the best Linux OS to use for encoding? I thought Mandriva and Ubuntu isn't so bad.

Do I look like someone who really cares what God thinks?! "GI Joe: The Atlantis Factor" - May be improved in the future.

Posted: 6/30/2010 3:22 AM

Quote

sgrunt

He/Him

Emulator Coder, Former player

Joined: 10/28/2007
Posts: 1360
Location: The dark horror in the back of your mind

There's more to encoding than just running that script, though most of the tools for currently-employed encoding methods do indeed originate from Linux. Which distribution it is you're using probably doesn't matter significantly as long as you're able to install (at least) x264, mencoder, and an audio encoder of your choice (for which packages should be available). Most of the command lines in the Windows guides would be applicable with those packages installed. (The Linux guides there are currently rather obsolete, so don't bother with them.)

Posted: 6/30/2010 3:25 AM
(Edited: 6/30/2010 6:05 AM)

Quote

Ilari

Emulator Coder, Skilled player (1141)

Joined: 5/1/2010
Posts: 1217

C0DE RED wrote:

Flygon: That command prompt you posted works best on Linux, right?! If so, what's the best distro to use for encoding? I thought Mandriva and Ubuntu isn't so bad.

If you refer to script in first post, then its definitely for Windows. --deldup and --versioninfo are Direct264-only options (and Direct264 only is for Windows, but it seems to run fine in Wine, even on x64). Also the 'start /wait' part is dead giveaway. The linux version would be 'wine /path/to/x264.exe <rest of arguments here>'. Or if you don't need deldup (bit larger encodes, but you can use "native" x264), 'x264 <rest of arguments removing deldup and versioninfo>'. Also note that x264 may or may not support input from AVI and other containers (direct264 does) depending on what external libraries it is configured to use. If it doesn't support container input, you have to feed the data as raw YUV420p (ffmpeg or mplayer can generate it, I use mplayer for that becuase piping the data directly to x264 works, whereas I couldn't get that working with ffmpeg). On Linux, mplayer can be used for encoding too (don't ask me how, I have never gotten result that even passes basic sanity checks, such as A/V sync; others that use mplayer for encoding should know). Linux distro doesn't matter much apart from wheither there's prebuilt x264 for it (if not, you need to download direct264 and install wine or build x264 from source). But yeah, as said, there's much more to encoding than just the video compression, like: * Dumping the video (some emulators are easy in this regard, others are not-so-easy) * Fixing A/V desyncs in source material (if any, but dumping methods that produce severe desyncing should not be used) * Inserting subtitles * Inserting logo * encoding audio * final muxing If you want to try encoding stuff, I think it is good idea to first develop methods so that you can get encodes that have logo, good video (at least no blatant errors in colors or picture jumping all over the place), good subtitles, good audio (no staticy mess) and A/V sync. Use very fast encoding settings on this stage so you can test faster... Then work on encode sizes. Also As said, Windows encoding guides are mostly better than for Linux. Those guides usually can be fairly easily be (mostly) adopted to work on Linux (e.g. mencoder is available for both).

Posted: 6/30/2010 4:08 AM

Quote

BadPotato

Joined: 1/26/2009
Posts: 558
Location: Canada - Québec

Ilari wrote:

The linux version would be 'wine /path/to/x264.exe <rest>'. Or if you don't need deldup (bit larger encodes, but you can use "native" x264), 'x264 <rest>'.

I'm very disapointed that this deldup story is available for windows/wine only(similar feeling with the avisynth script, actually)... are you sure there aren't an equivalent for linux or with mencoder? edith: I'm not an expert either, but that "bit of difference" was quite noticeable in my previous experience.