Topic by Bisqwit

Posted: 10/28/2005 7:48 AM
(Edited: 1/23/2022 8:43 PM)

Quote

Bisqwit

Editor, Active player (296)

Joined: 3/8/2004
Posts: 7469
Location: Arzareth

[EDIT: Old posts moved from various threads. See the August 14th 2006 post for more recent posts.] My contribution to Mario64 AVI encodings! https://files.tasvideos.org/bisqwit/mupen64-mario64-three-tries-avi.zip This ZIP (~35 MB) contains three AVI files. Do not use download accelerators. Each AVI is 98.4 seconds long and contains the same video, encoded from different sources using the same codec settings (qp_constant=20). Scaling was done inside the emulator; mencoder's "-vf scale" wasn't used. test0-noscaling.avi: 320x240, emulated in 320x240, no scaling, ~36 fps test0-triangle.avi: 320x240, emulated in 640x480, triangle scaler, ~8 fps test0-lanczos.avi: 320x240, emulated in 640x480, lanczos scaler, ~3 fps Please analyze the video quality of these files. I'm mostly interested to know whether the significantly slower recording speed is worth it. Concentrate at places of moire patterns and aliasing artifacts, and also at the general sharpness/blurriness of the image. A hardware antialias setting was also used, I think. The display plugin was glN64. Relevant source code: http://bisqwit.iki.fi/src/mupen64-somefiles.zip Edit: For reference, unscaled 640x480 recording goes at ~9 fps, and results in 2.3 times larger AVI file (when using the same qp_constant).

Posted: 10/28/2005 8:34 AM

Quote

Phil

Active player (410)

Joined: 3/16/2004
Posts: 2623
Location: America, Québec

lanczos seems the best. The triangle version is blurry but maybe if you can make all video the same size, it would be easier to compare.

Posted: 10/28/2005 8:36 AM

Quote

Bisqwit

Editor, Active player (296)

Joined: 3/8/2004
Posts: 7469
Location: Arzareth

Phil wrote:

The triangle version is blurry but maybe if you can make all video the same size, it would be easier to compare.

What do you mean, all video the same size? I encoded them all at the same _codec quality setting_. There should be next to none artifacts introduced by H.264 encoding.

Posted: 10/28/2005 8:38 AM

Quote

NrgSpoon

Joined: 6/14/2004
Posts: 646

I don't see a significant enough difference between the 3 to bother with the higher quality scaling. If the saved video was actually going to be larger than 320x240 then i might see a reason, but if you're going to leave it as it is, you might as well stick with no scaling.

I like my "thank you"s in monetary form.

Posted: 10/28/2005 8:47 AM
(Edited: 1/23/2022 8:43 PM)

Post subject: Graphics comparisons (related to previously posted AVIs)

Quote

Bisqwit

Editor, Active player (296)

Joined: 3/8/2004
Posts: 7469
Location: Arzareth

Graphics comparisons. Left to right: no scaling, triangle, lanczos, cubic. EDIT: I added cubic, catrom and blackman scalings (in that order).

The point of this first image serie is best demonstrated in animation. In the unscaled version, the fence is full of moire artifacts (unwanted animation), whereas the lanczos version is sharp but moireless.

In the second serie, you can see that the triangle scaling is blurrier than the others. (Look at the brick textures and Mario's details.)

In the third serie, you can see that the lanczos version avoids the aliasing artifacts of the unscaled version, while also avoiding the blurriness of the triangle-scaled version. (Look at the grating, and at the HUD.) Triangle scaling is basically the same thing as linear interpolation.

Posted: 10/28/2005 9:42 AM

Quote

Dada

Joined: 11/22/2004
Posts: 1468
Location: Rotterdam, The Netherlands

I think it's also important to note that triangle-scaling blurs out the 2D elements on the screen as well. That's definitely unwanted.

Posted: 10/28/2005 9:48 AM

Post subject: Re: My contribution

Quote

nitsuja

Emulator Coder, Skilled player (1300)

Joined: 12/21/2004
Posts: 2687

Have you tried turning off the hardware antialiasing when doing this? Because you are basically doing software antialiasing on top of the hardware antialiasing, so maybe better quality and/or speed would result from turning it off, or turning it off and reducing from 1280x960. Besides that... Triangle filter is obviously not the correct choice, too blurry, although it does make for the smallest filesize. Lanczos results in noticeably better looking and a smaller AVI than unscaled, so I think of your 3 test AVIs lanczos definitely wins even though it took considerably longer to encode. (Are you certain that this implementation of lanczos scaling cannot be further optimized?) (edit: BTW, I'm surprised that the resulting lanczos-scaled image did not show the extra pixellation and/or blurriness in the HUD elements that normally shows up when rendering at 640x480. But most people are probably going to watch these AVI's at double the resolution, at least, which makes me wonder if it wouldn't be better to just render at 640x480 (with antialiasing, if it helps) and encode to 640x480 at a somewhat lower codec quality setting. Also, is it just me, or is the shading on Mario way too dark in all of these?)

Posted: 10/28/2005 10:20 AM

Post subject: Re: Scaling algorithms

Quote

Bisqwit

Editor, Active player (296)

Joined: 3/8/2004
Posts: 7469
Location: Arzareth

nitsuja wrote:

Have you tried turning off the hardware antialiasing when doing this? Because you are basically doing software antialiasing on top of the hardware antialiasing, so maybe better quality and/or speed would result from turning it off, or turning it off and reducing from 1280x960.

I didn't try turning it off. It's done at no cost and I don't believe it can make the quality _worse_, so I enabled it. > I think of your 3 test AVIs lanczos definitely wins I added also cubic scaling for reference, but it seems to be the blurriest of all. Still untested are hermite, hamming, hanning, blackman, gaussian, quadratic, catrom, mitchell, bessel and sinc scalings. (From Imagemagick.) There seems to be some analysis of these algorithms at: - http://efflare.com/docs/CFX_ImageCR3/reference/advanced/resamplingalgorithms.html - http://www.dylanbeattie.net/magick/filters/result.html I'm using blur setting equivalent to 1.0. > Are you certain that this implementation of lanczos scaling cannot be further optimized? Most certainly it could. I just don't have much expertise on optimizing graphics algorithms. > wouldn't [it] be better to just render at 640x480 (with antialiasing, if it helps) > and encode to 640x480 at a somewhat lower codec quality setting. I'm not sure. > Also, is it just me, or is the shading on Mario way too dark in all of these?) I don't think I can affect that.

Posted: 10/28/2005 10:32 AM
(Edited: 10/28/2005 10:35 AM)

Quote

nitsuja

Emulator Coder, Skilled player (1300)

Joined: 12/21/2004
Posts: 2687

Keep in mind also that 320x240 is not the maximum resolution of the N64. Some games (Turok) go as high as 480x360, I think there might be a few 640x240 or 320x480, and some other games (GoldenEye) change the resolution during the game. 640x480 is the maximum the N64 could technically support.

Posted: 10/28/2005 10:35 AM
(Edited: 10/28/2005 10:44 AM)

Quote

Dada

Joined: 11/22/2004
Posts: 1468
Location: Rotterdam, The Netherlands

nitsuja wrote:

But most people are probably going to watch these AVI's at double the resolution, at least, which makes me wonder if it wouldn't be better to just render at 640x480 (with antialiasing, if it helps) and encode to 640x480 at a somewhat lower codec quality setting.

I think that's stepping away from the original image too much. I'm not sure how you feel about this, but I think that the best thing to do is to get the most authentic emulation of the original image quality. Doubling the model rendering accuracy contradicts this.

Bisqwit wrote:

I didn't try turning it off. It's done at no cost and I don't believe it can make the quality _worse_, so I enabled it.

Isn't it possible that doubly anti-aliasing the image results in a more blurry image?

Posted: 10/28/2005 10:43 AM

Quote

Bisqwit

Editor, Active player (296)

Joined: 3/8/2004
Posts: 7469
Location: Arzareth

Isn't it possible that doubly anti-aliasing the image results in a more blurry image?

Only if the processing is done twice for the same material. I believe the antialiasing enabled nvidia-settings does antialiasing by the primitive level (that is, antialiasing polygons etc), which should only improve the quality of the image, not be a filter for it. Though I don't really know. Edit: Aww, this thread has became hopelessly populated with AVI and mupen64 talk. I wish it stayed in Mario 64. But I can't move the AVI talk to a separate thread, because it's too entangled with the talk about Mario64... (-.-;

Posted: 8/10/2006 7:08 AM

Quote

Bisqwit

Editor, Active player (296)

Joined: 3/8/2004
Posts: 7469
Location: Arzareth

This is offtopic, but re: The A/V sync problems, I will try to release an updated Windows version next weekend, that should have perfect A/V sync with nondistorted audio. (Or, if someone else wants to try it, take this patch <http://bisqwit.iki.fi/src/mupen64-windows-patch-bisqwit.txt> and find the relevant parts. Core ideas: Audio is _not_ modified in the VCR code; video frames are duplicated or skipped as per need; an "audiolead" variable controls the floating point rate control variable "videorate", which is the factor by which to speed/slow down the video and that defaults to 1.0. Video can be scaled to an arbitrary size with a lanczos scaler. The audio is resampled to target rate with a lanczos scaler _after_ the sync is done, but this part is not included in the patch, because it happened on my server remotely.)

Posted: 8/10/2006 8:09 AM

Quote

AzHP

Joined: 6/20/2006
Posts: 142

Hmm...wouldn't that cause skips in the video instead? Up to 5 seconds of skipping according to the patch?

Posted: 8/10/2006 8:17 AM

Quote

Bisqwit

Editor, Active player (296)

Joined: 3/8/2004
Posts: 7469
Location: Arzareth

AzHP wrote:

Hmm...wouldn't that cause skips in the video instead? Up to 5 seconds of skipping according to the patch?

It may cause skipping in the video. However, most N64 games naturally have a low FPS (OoT has 20 fps in 3D scenes, 30 fps in status screens), and we make the avi at 60 fps -- and the skip/duplicate ratios are very subtle -- chances to that two frames in a row skipped are very low. Skipping 1 frame (or usually even 2) will not result in loss of visual information, unless the game actually works at 60 fps speed at that moment. In OoT, I don't think frames were ever skipped; they were sometimes duplicated, instead.

Posted: 8/13/2006 11:14 PM

Post subject: Lanczos scaler and related stuff

Quote

Bisqwit

Editor, Active player (296)

Joined: 3/8/2004
Posts: 7469
Location: Arzareth

[EDIT] I moved posts from a couple of existing threads into this thread. This weekend, I worked a little on Mupen64 AVI recording code, and as a result, portions of it were rewritten -- affecting even the vcr_compress API. http://bisqwit.iki.fi/src/mupen64-vcr-patch2-bisqwit.txt = source code patch, applies to code extracted from mupen64_src-rerecording-v8.7z (after converting CRLFs into LFs) The change introduces my custom perfect noiseless A/V sync into the AVI recording mechanism by duplicating/dropping video frames as explained earlier. It also incorporates a lanczos audio and video scaler. However, it is incomplete! 1. I have not tried to compile it yet, because I have no access to a Windows computer here. 2. The Windows AVI recording dialog needs to be changed a bit. I cannot do this because I lack Windows programming experience. - The AVI recording dialog should ask two extra questions: desired video resolution, and desired audio sampling rate. If the desired video resolution differs from the size the game is rendered at, the video will be scaled using a lanczos scaler before saving into the AVI. Examples speaking in favor of this option have been posted earlier in this thread. And it should ask the desired audio sampling rate. The emulator will use a lanczos scaler to resample the game's outputted audio into the target rate, and will do that reliably even if the rate changes in the middle of game. The lanczos scaler has better quality than the linear sampler that there was before. The changed API also supports implementing encoders into formats that allow changing the video resolution during recording, and formats that allow changing the audio rate during recording. I repeat, this is unfinished code and help is needed to finish it. A breakdown of the changes in this patch: * Makefile: Added a rule for vcr.o, removed vcr_resample.o from the linkage list * config.h: Added VCR_SUPPORT as default, because the code doesn't even compile without it * main_gtk.c: Fixed an error * nesvideos_piece.*: Removed obsolete files * vcr.c: Renamed into vcr.cpp * vcr.cpp: Added a lanczos rescaler, a perfect A/V sync and decreased the number of global variables * vcr.h: Removed non-op function "invalidatedCaptureFrame" * vcr_compress.c*: The semantics of VCRComp_startFile() have changed, updated the files to match it; added also stubs for two notify functions. * vcr_compress.h: Added new functions, changed some, added documentation * vcr_resample.*: Removed obsolete files * main_win.c: Removed mention of non-op "invalidatedCaptureFrame" * Makefile.win: Removed linkage of vcr_recompress.*, updated rules for vcr.c* * mupen64.dev: Removed mention of nesvideos-piece.h

Posted: 8/14/2006 1:02 AM

Quote

L4yer

Joined: 6/12/2006
Posts: 368

Does this finally mean not having to record pcm audio? Can't seem to change it now :(

Posted: 8/14/2006 1:05 AM

Quote

Bisqwit

Editor, Active player (296)

Joined: 3/8/2004
Posts: 7469
Location: Arzareth

L4yer wrote:

Does this finally mean not having to record pcm audio? Can't seem to change it now :(

Do you mean, did I do audio codec selection related changes? No, I didn't.

Posted: 8/14/2006 4:55 AM

Quote

nitsuja

Emulator Coder, Skilled player (1300)

Joined: 12/21/2004
Posts: 2687

It compiled with some minor changes, but nearly everything in vcr.o fails to link:

  [Linker error] undefined reference to `VCR_aiLenChanged' 
  [Linker error] undefined reference to `VCR_aiDacrateChanged' 
  [Linker error] undefined reference to `VCR_coreStopped' 
  [Linker error] undefined reference to `VCR_updateScreen' 
  etc...

It probably has to do with vcr.cpp being the first and only C++ file added to the (windows) project. I've already set IsCPP=1 in mupen64.dev so it's not a problem of linking with gcc instead of g++. Does an "extern "C" {...}" need to be added somewhere?

L4yer wrote:

Does this finally mean not having to record pcm audio? Can't seem to change it now :(

I've never seen any emulator do this. AVISaveOptions is the function that's supposed to bring up the video or audio codec dialog, and it works for video, but for audio any call to AVISaveOptions immediately fails, as far as I can tell.

Posted: 8/14/2006 7:43 AM
(Edited: 8/14/2006 8:28 AM)

Quote

Bisqwit

Editor, Active player (296)

Joined: 3/8/2004
Posts: 7469
Location: Arzareth

nitsuja wrote:

Does an "extern "C" {...}" need to be added somewhere?

Yes, that seems to be the case. I added it now, patch updated. EDIT: Apparently it's more complicated than this. Yes, the problems are resolved by adding extern "C" { ... } at right places, but apparently, the places are scattered around. Many awful hacks. #ifdef __WIN32__ extern ... #endif are found in many functions in that module separately, instead of having a clear API. Each one of those needs the "C" to be added in them.

Posted: 8/14/2006 7:54 AM

Quote

Andy_Olivera

Joined: 3/1/2005
Posts: 46

Is this the same problem we were having with FCEU? Does Mupen run at exactly 60fps? I'm guessing not, so all we need to do is find the precise speed and convert it into a fraction for the AVI and that should solve the problem without skipping/adding frames, right?

Posted: 8/14/2006 7:59 AM

Quote

Bisqwit

Editor, Active player (296)

Joined: 3/8/2004
Posts: 7469
Location: Arzareth

Andy Olivera wrote:

Is this the same problem we were having with FCEU? Does Mupen run at exactly 60fps? I'm guessing not, so all we need to do is find the precise speed and convert it into a fraction for the AVI and that should solve the problem without skipping/adding frames, right?

It is not a matter of FPS. The problem is that audio is only generated at the game's whim. And I have already solved this issue.

Posted: 8/15/2006 12:18 AM

Quote

nitsuja

Emulator Coder, Skilled player (1300)

Joined: 12/21/2004
Posts: 2687

So to set the desired rate and resolution, it's only necessary to change the pointed-to width/height/rate values passed into VCRComp_startFile? Is there anything known that VCRComp_notifyNewRes and VCRComp_notifyNewRate will be useful for, or are they only there in case some future use is found for them?

Posted: 8/15/2006 12:22 AM

Quote

Bisqwit

Editor, Active player (296)

Joined: 3/8/2004
Posts: 7469
Location: Arzareth

nitsuja wrote:

So to set the desired rate and resolution, it's only necessary to change the pointed-to width/height/rate values passed into VCRComp_startFile?

Exactly.

nitsuja wrote:

Is there anything known that VCRComp_notifyNewRes and VCRComp_notifyNewRate will be useful for, or are they only there in case some future use is found for them?

Ogg-vorbis can store audio of arbitrary sampling rate. I don't know if it can cope with changing sampling rates though. But it's there in case one wants to encode a stream where the sampling rate is decided by the game, but that is not known yet by the time VCRComp_startFile() is called. It doesn't work nicely if the game changes the sampling rate later, though... Both mp3 and mpeg4 can technically cope with specifications that change in the middle of the stream, but I don't think many decoders or encoders can handle that.

Posted: 9/21/2006 4:44 AM

Quote

nitsuja

Emulator Coder, Skilled player (1300)

Joined: 12/21/2004
Posts: 2687

There seems to be a problem with the video scaling.... Here is the sort of output I get for 640x480 -> 320x240 (Mario 64 title screen):

The smaller the scaling should be, the more pieces of a smaller segment of the screen are repeated, and (because the height is less) the closer it is to the bottom.

void VCRComp_startFile( const char *filename, long* width, long* height, long* rate, int fps )
{
	// testing (actually a dialog sets these here)
	*width = 320; // 50% scaling
	*height = 240; // 50% scaling
	*rate = 32006; // apparently the default in Mario 64
	
	avi_opened = 1;
	frame = 0;
	infoHeader.biSize = sizeof( BITMAPINFOHEADER );
	infoHeader.biWidth = *width;
	infoHeader.biHeight = *height;
	infoHeader.biPlanes = 1;
	infoHeader.biBitCount = 24;
	infoHeader.biCompression = BI_RGB;
	infoHeader.biSizeImage = *width  *  *height  *  3;
	...

I haven't noticed anything wrong with the A/V sync or sound scaling. (And *rate is not later being set to 48000 each time.)