Post by eternaljwh

Posted: 1/8/2009 3:01 PM
(Edited: 1/28/2009 5:13 PM)

Post subject: File size to length ratio on encodes.

Quote

Johannes

Former player

Joined: 12/1/2007
Posts: 425

I think the file size to length ratio limit serves no purpose other than making the encodes of the games that are complex to encode look bad.

SM64.org

Posted: 1/8/2009 4:17 PM

Quote

I suspect this is one place where, again, we need people to make judgment calls. Experienced encoders should be able to say "Look, it looks like crap if I'm at this encode level, but if I bump it up a bit, it looks much better, at the cost of a 40% increased file size" or whatever, regardless of the console in question. On the flipside, I'm sure there are N64 games that wouldn't benefit significantly from larger filesizes, and there's no need to force our dialup users to suffer exorbitantly long download waits. Remember how the original Gunstar Heroes run wasn't encoded because we couldn't at the time achieve an acceptable file size to quality ratio? That's why I'm saying it should just be a judgment call for the game in question.

Pyrel - an open-source rewrite of the Angband roguelike game in Python.

Posted: 1/8/2009 4:22 PM

Quote

Johannes

Former player

Joined: 12/1/2007
Posts: 425

I'm not saying all games should have higher file sizes, only the ones that actually need it.

SM64.org

Posted: 1/8/2009 4:23 PM

Quote

Xkeeper

Banned User, Former player

Joined: 12/23/2004
Posts: 1850

Johannes wrote:

I'm not saying all games should have higher file sizes, only the ones that actually need it.

I think you entirely missed his point.

Perma-banned

Posted: 1/8/2009 4:29 PM

Quote

Johannes

Former player

Joined: 12/1/2007
Posts: 425

Xkeeper wrote:

Johannes wrote:
I'm not saying all games should have higher file sizes, only the ones that actually need it.
I think you entirely missed his point.

On the flipside, I'm sure there are N64 games that wouldn't benefit significantly from larger filesizes, and there's no need to force our dialup users to suffer exorbitantly long download waits. I interpreted this as that he thought I meant all games need higher file sizes. My point was that higher file sizes should be allowed when there's need for it, while games that don't need it stay the same. Did I get it wrong?

SM64.org

Posted: 1/8/2009 5:04 PM

Quote

nineko

Banned User

Joined: 8/2/2008
Posts: 420
Location: italy

Sometimes I feel a higher bitrate would be welcome, yes. File size shouldn't really be a limit nowadays, since connections are faster and faster, and hard drives are bigger and bigger. Quality matters, in my opinion, and now that there are archive.org mirrors for most of the movie files, getting them shouldn't be a problem too since you're no longer forced to use the starving torrents.

Gone.

Posted: 1/8/2009 7:22 PM

Quote

Derakon

Joined: 7/2/2007
Posts: 3960

Johannes: it sounded to me like you were saying "All N64 games should be allowed higher filesizes", while I was saying "All games that would look crappy at low filesizes should be allowed higher filesizes". That's the only significant distinction between what I said and what you said. I don't think I've seen an 8-bit or 16-bit TAS yet that would significantly benefit from a higher filesize, but that doesn't mean there won't be one in the future (and I pulled out Gunstar Heroes as an example of a time when we could've used this rule in the past). I do think it quite plausible that there are games out there that don't benefit significantly from a higher filesize, though, and I don't want to see our dialup users get punished just because the encoders weren't as efficient as they could be.

Pyrel - an open-source rewrite of the Angband roguelike game in Python.

Posted: 1/8/2009 8:12 PM

Quote

Johannes

Former player

Joined: 12/1/2007
Posts: 425

Derakon wrote:

Johannes: it sounded to me like you were saying "All N64 games should be allowed higher filesizes", while I was saying "All games that would look crappy at low filesizes should be allowed higher filesizes". That's the only significant distinction between what I said and what you said. I don't think I've seen an 8-bit or 16-bit TAS yet that would significantly benefit from a higher filesize, but that doesn't mean there won't be one in the future (and I pulled out Gunstar Heroes as an example of a time when we could've used this rule in the past). I do think it quite plausible that there are games out there that don't benefit significantly from a higher filesize, though, and I don't want to see our dialup users get punished just because the encoders weren't as efficient as they could be.

Only the complex 3D games need higher file sizes. I never said any 8- or 16-bit game did. Only the complex 3D games do. Simple games don't need to have unnecessarily high file sizes just because complex 3D games should be allowed to have higher file sizes when needed. NES games typically look good at 1-2 MB/min. N64 games need much larger file sizes to look good. I think it's about time we stop caring so much about dialup users. There aren't many of them left, and soon there will be none. Either way, only the few games that will truly benefit from larger file sizes, and therefore should be allowed to have larger file sizes, will actually have larger file sizes, and the quality difference is worth the slower downloads. It's starting to seem you're arguing just for the sake of arguing.

SM64.org

Posted: 1/8/2009 8:17 PM

Quote

Derakon

Joined: 7/2/2007
Posts: 3960

Well that last line came out of nowhere, and I don't think it was warranted. But you're entitled to your opinion. I'm simply trying to make a distinction that you aren't, as evidenced by your line "N64 games need much larger file sizes to look good." This is not universally true. While it is true in general, I wouldn't be surprised if, say, Paper Mario would look fine with our current "standard" bitrate. My main goal here is to make certain that we don't end up just inflating file sizes unnecessarily.

Pyrel - an open-source rewrite of the Angband roguelike game in Python.

Posted: 1/8/2009 8:28 PM

Quote

Johannes

Former player

Joined: 12/1/2007
Posts: 425

Derakon wrote:

Well that last line came out of nowhere, and I don't think it was warranted. But you're entitled to your opinion. I'm simply trying to make a distinction that you aren't, as evidenced by your line "N64 games need much larger file sizes to look good." This is not universally true. While it is true in general, I wouldn't be surprised if, say, Paper Mario would look fine with our current "standard" bitrate. My main goal here is to make certain that we don't end up just inflating file sizes unnecessarily.

Sorry if I'm being rude. I was never encouraging inflating file sizes unnecessarily though.

SM64.org

Posted: 1/8/2009 8:34 PM

Quote

ShinyDoofy

Former player

Joined: 12/5/2007
Posts: 716

While in general I can perfectly understand Johannes' idea, that's what exceptions were made for. 4MB/minute might not always be a good limit, but certainly one that most game look fine with. If a game such as F-Zero X really looks better with a higher bitrate (just ran a test encode and found 470kbps of video for ingame scenes only to be damn near publishable quality at 320x240), a publisher can always ask an admin for an exception. Generally raising this limit would potentially result in a lazy encoder just going with a higher bitrate instead of trying harder.

Posted: 1/9/2009 7:52 PM

Quote

bkDJ

Joined: 7/26/2006
Posts: 1215

1st pass should be done with a crf of the encoder's choosing (picking 22 to 26 seems reasonable) and pass2 should just use that bitrate. That way you are encoding to a certain quality and don't have to guess a good bitrate. Oftentimes, it will be less than the 414 kbps limit (assuming 128kbps sound) and mroe efficient than a guess. Also since pass2 is like 20% better than pass1, you can also do like pass1 at crf19-23 and make pass2 20% lower bitrate and be near exactly the same quality. All this gives every game an equal chance to look good at a reasonable filesize. Naturally the more demanding 3D games' encodes will be bigger this way but not "bloated." My 2 cents.

Posted: 1/10/2009 12:06 PM

Quote

Johannes

Former player

Joined: 12/1/2007
Posts: 425

eternaljwh wrote:

Encoded TASes take up roughly 40G of my disk, about half that drive.

That's your problem. We shouldn't have to make bad looking video just so you can fit more on your HDD.

ShinyDoofy wrote:

(just ran a test encode and found 470kbps of video for ingame scenes only to be damn near publishable quality at 320x240)

The problem is that what's currently considered "publishable" quality for complex 3D games is not good enough. The complex 3D games generally use too low bitrates. NES game encodes nearly always look transparent, while encodes of complex 3D games look very lossy. Look at the Mischief Makers encode, for example - the intro starts out extremely blurry and suddenly sharpens. This is a clear sign of a too low bitrate.

ShinyDoofy wrote:

Generally raising this limit would potentially result in a lazy encoder just going with a higher bitrate instead of trying harder.

Anything above 2 MB/min is overkill for NES games, but the limit is 4 MB. Has this resulted in people using unecessarily high bitrates for NES? No. In my opinion, we should encourage higher bitrates for the complex 3D games, so they don't look very lossy while the simpler games look perfect. There's no reason the video quality of encodes should depend on the game's complexity. Encoders shouldn't have to ask for special permission to use a bitrate as high as needed to make the encode look decent. This does not mean the encoders should be lazy and go with unecessarily high bitrates.

bkDJ wrote:

1st pass should be done with a crf of the encoder's choosing (picking 22 to 26 seems reasonable) and pass2 should just use that bitrate. That way you are encoding to a certain quality and don't have to guess a good bitrate. Oftentimes, it will be less than the 414 kbps limit (assuming 128kbps sound) and mroe efficient than a guess. Also since pass2 is like 20% better than pass1, you can also do like pass1 at crf19-23 and make pass2 20% lower bitrate and be near exactly the same quality. All this gives every game an equal chance to look good at a reasonable filesize. Naturally the more demanding 3D games' encodes will be bigger this way but not "bloated." My 2 cents.

Not a bad idea. However, with that method, bitrate allocation won't be as accurate as it would be with the same bitrate for both passes, but that can be solved by doing a third pass.

SM64.org

Posted: 1/10/2009 12:43 PM

Quote

moozooh

Senior Moderator

Joined: 8/4/2005
Posts: 5770
Location: Away

I'm afraid you're overlooking a lot. First of all, about 90% of the encodes for any platform don't even come close to the limit you're talking about. Next, "complex 3D games" and "simple 2D games" doesn't tell anything about encoding complexity. Not a single thing. I don't know your experience in video encoding, but what really kills codecs are things like scaling, fast randomly moving objects, multi-layered backgrounds, particle bursts or small areas of a sprite shifting against each other, gradual fade-ins/fadeouts and various color oscillation. Generally, 3D games presented on the consoles we have an ability to TAS on currently only exhibit the first and the last item of that list. On the contrary, gradients and general blur seen as a result of texture filtration actually save bits because blurred image is easier to compress using MPEG family codecs. And, the most important one yet, the limit can be raised if it's proven to be impossible to create a decently-looking encode within the 4MB/min limit. What I do agree, however, is that in certain cases, a small bitrate increase (especially for audio) would be welcome. And while I'm at it, I'd also like to point out that "simple" platforms' sound does never mean it's simple for a codec to handle. Crispness, sharp attacks, noises and such are much harder to compress, and thus require a somewhat higher bitrate.

Warp wrote:

Edit: I think I understand now: It's my avatar, isn't it? It makes me look angry.

Posted: 1/10/2009 1:13 PM

Quote

Johannes

Former player

Joined: 12/1/2007
Posts: 425

What exactly am I overlooking? First of all, about 90% of the encodes for any platform don't even come close to the limit you're talking about. I never said most games need to exceed the limit. The point from the beginning was that the ones that do need to exceed the limit should not look bad. Many N64 encodes look bad. The point was never what was considered complex to encode, it was that games that need it should be given higher bitrate. In my opinion, we should forget about the 4 MB/min limit, and use the lowest bitrate that will actually look good, instead of striving to barely fit the 4 MB/min limit, and possibly sacrificing things like seekability and compability in the process, just to get a somewhat decent looking (and sounding) video file.

SM64.org

Posted: 1/10/2009 3:32 PM

Quote

moozooh

Senior Moderator

Joined: 8/4/2005
Posts: 5770
Location: Away

Johannes wrote:

I never said most games need to exceed the limit. The point from the beginning was that the ones that do need to exceed the limit should not look bad. Many N64 encodes look bad.

How many of them have 400+ kbps for video? Please do tell.

Johannes wrote:

The point was never what was considered complex to encode, it was that games that need it should be given higher bitrate.

My point is, a "complex 3D game" (found 7 instances of this phrase in your previous posts, btw) isn't necessarily complex to encode, and thus doesn't need high bitrate to look good, which you seem to claim so rigorously.

Johannes wrote:

In my opinion, we should forget about the 4 MB/min limit, and use the lowest bitrate that will actually look good, instead of striving to barely fit the 4 MB/min limit

Once again: the limit is there to prevent accidental publishing of bloated files. The limit is not absolute, and it has been raised when needed, which is exactly what you're asking about. This is what you're overlooking! Also, I'd still like you to list the bitrates of N64 encodes you consider bad.

Johannes wrote:

and possibly sacrificing things like seekability and compability in the process, just to get a somewhat decent looking (and sounding) video file

How compatible is "compatible"? How good is "good"? There is no universal solution.

Warp wrote:

Edit: I think I understand now: It's my avatar, isn't it? It makes me look angry.

Posted: 1/10/2009 10:28 PM

Quote

ZeXr0

Joined: 8/27/2006
Posts: 883

You are missing the point. The point is, the bitrate should be ajusted so that the game look good, even if that means having a bitrate that takes 6mb/min. That's pretty much it. Nothing more, nothing less.

Posted: 1/11/2009 12:27 AM

Quote

moozooh

Senior Moderator

Joined: 8/4/2005
Posts: 5770
Location: Away

Wow, guys. :\ The FAQ articles all insist the visual quality must be good. And it is usually good even within the 4 MB/s limit. If it's not good, the bitrate is increased further until it's good, and Bisqwit approves. If it doesn't look good it means the bitrate is low, but that's not the 4 MB/s limit's fault, it's the encoder's fault. The limit is not absolute, it's only there to ensure no bloated files are published when the site administrators' can't supervise the process. It's you who are missing the point.

Warp wrote:

Edit: I think I understand now: It's my avatar, isn't it? It makes me look angry.

Posted: 1/11/2009 12:45 AM

Quote

ZeXr0

Joined: 8/27/2006
Posts: 883

That's pretty much what we are saying. So maybe there should be someone that review the visual quality of the file to know if it should be reeconded because the quality is too low ?

Posted: 1/11/2009 12:54 AM

Quote

moozooh

Senior Moderator

Joined: 8/4/2005
Posts: 5770
Location: Away

Yes, these people are called publishers. Badger them if videos look bad. (Beware, though: harsh replies from adelikat's side are not to be unexpected.) Also, if you read Johannes's first two posts, he was on about a different thing; namely, increasing bitrate limit for N64 or abolishing it altogether.

Warp wrote:

Edit: I think I understand now: It's my avatar, isn't it? It makes me look angry.

Posted: 1/11/2009 12:29 PM

Quote

Johannes

Former player

Joined: 12/1/2007
Posts: 425

moozooh wrote:

Yes, these people are called publishers. Badger them if videos look bad. (Beware, though: harsh replies from adelikat's side are not to be unexpected.) Also, if you read Johannes's first two posts, he was on about a different thing; namely, increasing bitrate limit for N64 or abolishing it altogether.

Maybe I'm just bad at explaining in English..

ZeXr0 wrote:

The point is, the bitrate should be ajusted so that the game look good, even if that means having a bitrate that takes 6mb/min. That's pretty much it. Nothing more, nothing less.

That was my one and only point from the beginning.

SM64.org

Posted: 1/12/2009 5:54 PM

Quote

Raiscan

Joined: 11/11/2006
Posts: 1235
Location: United Kingdom

ZeXr0 wrote:

The point is, the bitrate should be ajusted so that the game look good, even if that means having a bitrate that takes 6mb/min. That's pretty much it. Nothing more, nothing less.

It has always been a case by case basis. The limit is there, but it can be worked around if there is a valid reason for high bitrate. Personally I feel if you need to use more than 6MB/min, you need more aggressive settings.

<adelikat> I am annoyed at my irc statements ending up in forums & sigs

Posted: 1/12/2009 9:39 PM
(Edited: 1/12/2009 10:22 PM)

Quote

Bisqwit

Editor, Active player (296)

Joined: 3/8/2004
Posts: 7469
Location: Arzareth

Everything I should say has probably already been said, but here goes anyway. The 4 MB/minute limit was invented with a modified Harrison&Stetson method, i.e. an arbitrary value was chosen with intuition and verified against a few cases, to determine whether it is a good limit. For different consoles, the average varies. Actually, why not do the math here.

mysql> select s.abbr, min((filesize/1e6)/(length/60))mi, avg((filesize/1e6)/(length/60))av, max((filesize/1e6)/(length/60))ma from movie_file mf,movie m,filetype t,system s where mf.movieid=m.id and m.systemid=s.id and t.filetype=mf.typech and filetypeclass='T' group by s.abbr;
+---------+-------------------+------------------+------------------+
| abbr    | mi                | av               | ma               |
+---------+-------------------+------------------+------------------+
| GB      | 0.572175849272519 |  1.4087359767177 | 2.33612181818182 | 
| GBA     |  1.34804613861386 |  2.8145274068148 | 6.20754240674907 | 
| GBC     | 0.767979931209014 |  1.2569075060325 |  1.8114107364037 | 
| Genesis | 0.975704864864865 |  2.9559998754507 | 6.30503649295637 | 
| N64     |  2.35566726298707 |  2.9585978329765 | 3.87445178347063 | 
| NES     | 0.646402777777778 |  2.1354393788815 |  9.8144029716529 | 
| SGB     |  0.60576729658376 | 0.90576560167766 | 1.43155425373134 | 
| SMS     | 0.870126857142857 |  1.6502879425395 |  2.4848148482854 | 
| SNES    |   1.1508051414501 |  2.8344977178136 | 5.48289923076923 | 
+---------+-------------------+------------------+------------------+

The above is for all publications, including obsoleted AVIs. Here's the same table for currently published movies.

+---------+-------------------+-----------------+------------------+
| abbr    | mi                | av              | ma               |
+---------+-------------------+-----------------+------------------+
| GB      | 0.572175849272519 | 1.3884989196617 | 2.33612181818182 | 
| GBA     |  1.38530359840954 | 2.9467362692611 | 6.20754240674907 | 
| GBC     | 0.767979931209014 | 1.2139071291976 |  1.8114107364037 | 
| Genesis | 0.975704864864865 | 2.8845012522829 | 6.30503649295637 | 
| N64     |  2.57872355200365 | 3.0857642439087 | 3.87445178347063 | 
| NES     | 0.661616585365854 |  1.837647967182 |  9.8144029716529 | 
| SGB     |           0.84024 | 1.0699646815687 | 1.43155425373134 | 
| SMS     | 0.870126857142857 |  1.714521622153 |  2.4848148482854 | 
| SNES    |   1.1508051414501 |  2.724627874046 | 5.48289923076923 | 
+---------+-------------------+-----------------+------------------+

This graph tells us, that for every console, the average encoded AVI/MKV/whatever size has been about 2 MB/minute, give or take some 1.1 MB/minute. For SNES, GBA, Genesis, and surprisingly, NES, exceptions have been made sometimes over the 4 MB/minute limit, with the highest bitrate for SNES having been 5.5 MB per minute. These exceptions usually need to be made for movies with lots of full-screen movement that is not compensated by either motion estimation or frame history referrals, because the screen contents change entirely, and for movies that contain colour schemes that are particularly difficult because of chroma supersampling (for example, black objects on a blue or red background). In general, the visual and aural quality of our AVIs has been very good. ShinyDoofy's most recent encode of Shinryuu's Mega Man 6 movie is an outstanding proof of this claim. Even with a trained eye, I cannot find encoding artifacts in his video with casual inspection. His movie uses 2.2 MB per minute. There have indeed been some movies with not so good quality. In those cases the blame lies either on the too modest bitrate that the encoder used, or on too lazy settings the encoder used for the fear of consuming too much CPU time in the encoding. We encourage our encoders to use the strongest possible encoding settings of the codec in order to produce the best quality per bit value. The MEncoder mantra that I use ifor my encodings with x264, is this: partitions=all:me=umh:frameref=15:me_range=64:subme=7:mixed_refs:trellis=2:ratetol=60:subq=7:direct_pred=auto, together with multi-pass encoding to produce the most optimal distribution of bits within the file. This does not even include B-frames. I don't think B-frames are particularly useful with NES, but they may be useful with 3D content on N64 and PSX (and vector/scaledbitmap content on SNES and GBA). We also encourage our encoders to visually inspect their result and to see if there are encoding artifacts that should be addressed (by changing the codec settings and re-encoding) before the AVI is published. In the rare cases that a larger bitrate is indeed needed -- this happens usually for racing games and for some 3D games -- a developer of the TASVideos site needs to be contacted by the publisher to manually disable the limit temporarily for the duration of filling the publication form. For now, the only person that can do that is me, but I'm hoping to change that fact soon. Regarding Johannes's claim that the ratio is too low for N64...

+---------+-----------------------------------+------------------+
| movieid | gamename                          | mbpermin         |
+---------+-----------------------------------+------------------+
|    1029 | Tony Hawk's Pro Skater 3          | 3.87445178347063 | 
|    1190 | Super Smash Bros.                 | 3.82257774859752 | 
|    1211 | Mischief Makers                   | 3.62135403109398 | 
|     975 | 007 - The World is Not Enough     | 3.49640452173913 | 
|     886 | Wetrix 64                         | 3.43624058924595 | 
|    1229 | Legend of Zelda - Ocarina of Time | 3.35597524190427 | 
|     530 | Tetrisphere                       | 3.30907050706493 | 
|     957 | Duke Nukem 64                     |  3.2286800456013 | 
|    1197 | Super Mario 64                    | 3.14568689450763 | 
|    1238 | Chameleon Twist                   | 3.10092146788991 | 
|    1007 | Goldeneye 007                     | 3.02810797342193 | 
|     962 | Glover                            |       2.96598544 | 
|     963 | Legend of Zelda - Majora's Mask   | 2.81573428140494 | 
|     808 | Gex 64 - Enter the Gecko          | 2.79031657176988 | 
|     946 | Diddy Kong Racing                 | 2.77021529065256 | 
|     850 | Super Mario 64                    | 2.72489028646749 | 
|     759 | Mortal Kombat 4                   | 2.70063864713825 | 
|     576 | Legend of Zelda - Ocarina of Time | 2.69789365087056 | 
|    1095 | Turok - Dinosaur Hunter           | 2.68556397802107 | 
|    1237 | Gex 3: Deep Cover Gecko           | 2.65161661921689 | 
|    1229 | Legend of Zelda - Ocarina of Time | 2.57872355200365 | 
+---------+-----------------------------------+------------------+

Is it indeed the limit that has been too low, or has it been the encoders who have been too modest? I guess Tony Hawk's World Is Gray could have used an exception. Super Smash Bros is a suspect. Zeldas and Marios look quite fine, though if they needed to be encoded at a higher rate, it could be done without offending the threshold. My verdict is: Thus is good.

Posted: 1/12/2009 9:45 PM

Quote

nineko

Banned User

Joined: 8/2/2008
Posts: 420
Location: italy

Just out of curiosity, what is the NES game with a bitrate of 9.8144029716529?

Gone.