Posted: 7/24/2013 3:11 AM
Post subject: lsnes (libgambatte core): Frames don't end on VBlank
Quote
MrWint
Skilled player (1040)
Joined: 7/24/2013
Posts: 175
TAS of 2017 Glitchy TAS of 2017 Gameboy TAS of 2017 Gameboy TASer of 2016 Gameboy TAS of 2016 Gameboy TASer of 2014 Lucky TAS of 2014 Gameboy TAS of 2014 Lucky TAS of 2013
This section of the forums is the closest I got finding the right place for this topic. Since lsnes is mainly an SNES emulator and this is supposed to be the section for it, I posted it here and not in the GBx section, even though it's about the libgambatte core. While experimenting with different GBx emulators, I noticed that the frame timing of lsnes is way different from most other emulators, even those who are using the same core (libgambatte). I found the reason in the way lsnes is handling a frame advance: In lsnes, a frame is forced to be exactly 35112 audio samples long, by running gambatte::GB.runFor in a loop until it is reached (see http://repo.or.cz/w/lsnes.git/blob/refs/heads/master:/src/emulation/gambatte/core.cpp) In comparison, the Gambatte (SDL) emulator recognizes the VBlank (indicated by the return value) to determine when a video frame has passed. (see https://github.com/sinamas/gambatte/blob/master/gambatte_sdl/src/gambatte_sdl.cpp) BizHawk is doing something similar to Gambatte, but it ignores the return value under the assumption that a frame can be at most 35112 audio samples long, so a single call of gambatte::GB.runFor ensures to reach the VBlank. (see http://code.google.com/p/bizhawk/source/browse/trunk/BizHawk.Emulation/Consoles/Nintendo/Gameboy/Gambatte.cs) While both BizHawk and lsnes provide consistent frame timing this way (in the sense of being predictable and reproducable), lsnes timing is completely unrelated to VBlanks. It is true that a video frame is produced every 35112 audio samples as described by the Gambatte documentation, but the ability of GB programs to turn off the LCD messes the count up, making the number of audio samples inbetween video frames longer or shorter. I'm by no means an expert in knowing the GB architecture, but that is how it works afaik. This causes lsnes to become out of sync with the actual video frames: An lsnes frame then consists of two halves of video frames, having the VBlank somewhere in the middle of the frame. Since the LCD is turned off especially in the initial loading phase of the GB cart, it is out of sync basically all the time. So what is wrong about being out of sync this way? It largely depends on the method the game uses to read joypad data. If it does it at VBlank (like most games do afaik), the input frames on lsnes can be off. It may even be possible to miss out on input possibilities because of this, when two inputs happen to be in the same lsnes frame, even though they are in different video frames. On the other hand, you might be able to issue inputs that aren't possible on other emulators (they are always possible on console), if it happens the other way round, having two inputs that are in the same video frame separated by the lsnes frames. I suspect it to be unintentional, i.e., a bug. Changing this woud desync all GB movies, though, and it's not really broken as it is now, just... unintuitive. I guess the acutal question I'm trying to ask here is: what to do about it?
Posted: 7/24/2013 5:05 PM
Quote
creaothceann
He/Him
Editor
Joined: 4/7/2005
Posts: 1874
Location: Germany
The most accurate thing to do would be recording the button status whenever the game checks it, imo. It's even possible to convert frame-based movies by playing them back while simultaneously only recording what the game actually reads back. (Converting access-based movies would also be possible, but wouldn't make sense except if lsnes would be discontinued and the existing lsnes movies would have to be converted to BizHawk's format.) The remaining difficulty would be changing the emulators and tools to the slightly different scenario.
Posted: 7/24/2013 5:16 PM
Quote
Ilari
Emulator Coder, Skilled player (1114)
Joined: 5/1/2010
Posts: 1217
Rookie of 2010 New systems TASer of 2010 New systems TAS of 2010
It is done that way to ensure that video output frames happen at consistent intervals (1 video frame every 35112 audio samples) so audio and video will be in sync. Also, if input and video frames are nowhere near aligned and game polls on VBlank, the poll should go in the middle of video frame. Also, even if there are multiple input frames per video frame, those can differ (subframe input).
Posted: 7/25/2013 2:27 AM
Quote
MrWint
Skilled player (1040)
Joined: 7/24/2013
Posts: 175
TAS of 2017 Glitchy TAS of 2017 Gameboy TAS of 2017 Gameboy TASer of 2016 Gameboy TAS of 2016 Gameboy TASer of 2014 Lucky TAS of 2014 Gameboy TAS of 2014 Lucky TAS of 2013
I guess I didn't do a good job in explaining what problem I see with the way lsnes handles input frames, I'll try to give an example: I recently worked on a TAS of Pokemon Blue (see here), and I wanted to create a computer program to search for favourable RNG outcomes. Pokemon Blue reads inputs exactly once every frame during the VBlank interrupt and stores it in memory. Suppose there is a text box and I want to close it on the first frame possible. I run the emulation until I recognize the assembly operations that evaluate the input to determine closing the text box. If the frames are aligned at the VBlanks (like in BizHawk), I know exactly that the frame this operation occurs in is the frame I need to press the button to close it. If the frames are misaligned like in lsnes, that doesn't work: depending on whether the operation is in the first or the second half of the frame (sepated by the VBlank), the relevant input frame can be either that frame or the frame before, which is highly undesirable. To sum up, what happens is that the input of the last frame also influences the next frame, each input is valid for the second half of the frame it is pressed in and the first half of the next frame. creaothceann is right in that it would be most accurate to record buttons not once a frame, but every time it is read by the game. Since this is during VBlank in the games I know (which aren't many, a more elaborate evaluation might be needed to confirm the hypothesis), separating the frames by VBlanks seems to the a more accurate way to handle frame timing. Making every video frame the same length is less important in my opinion, since it isn't the case on the actual console, either.
Posted: 7/25/2013 6:51 AM
Quote
Warepire
He/Him
Editor
Joined: 3/2/2010
Posts: 2178
Location: A little to the left of nowhere (Sweden)
MrWint wrote:
creaothceann is right in that it would be most accurate to record buttons not once a frame, but every time it is read by the game.
Except the game boy offered an alternative to polled input which is interrupt driven input, for a game with interrupt driven input that would simply not work as there will never be a "poll" to catch. Other games poll several times per frame and expect to read the same value more than x number of times during that poll period (it is to defeat false positives of pressed buttons), these games would still work, but I don't think a TASer wants to have to enter the same button combination up to 10 times per frame.
Posted: 7/25/2013 7:04 AM
Quote
Patashu
He/Him
Joined: 10/2/2005
Posts: 4045
Are there any games where being able to do input more than once a frame allows you to do things you could not do before? (Barring the case of subframe reset timing)
My Chiptune music, made in Famitracker: http://soundcloud.com/patashu My twitch. I stream mostly shmups & rhythm games http://twitch.tv/patashu My youtube, again shmups and rhythm games and misc stuff: http://youtube.com/user/patashu
Posted: 7/25/2013 7:09 AM
Quote
Ilari
Emulator Coder, Skilled player (1114)
Joined: 5/1/2010
Posts: 1217
Rookie of 2010 New systems TASer of 2010 New systems TAS of 2010
MrWint wrote:
creaothceann is right in that it would be most accurate to record buttons not once a frame, but every time it is read by the game. Since this is during VBlank in the games I know (which aren't many, a more elaborate evaluation might be needed to confirm the hypothesis), separating the frames by VBlanks seems to the a more accurate way to handle frame timing. Making every video frame the same length is less important in my opinion, since it isn't the case on the actual console, either.
The frames all need to be on average the same length or issues crop up, both for timing of movies and with video dumping. Well, with video dumping, the framerate can technically be changed, but doing such thing leads to problems when trying to encode the thing. Yeah, some sort of constrained VFR mode that kept the clock counter in bounded interval containing the nominal value might work... I don't think GB can generate video frames faster than 35112 cycles per frame (but it can generate those slower).
Posted: 7/25/2013 8:33 AM
Quote
MrWint
Skilled player (1040)
Joined: 7/24/2013
Posts: 175
TAS of 2017 Glitchy TAS of 2017 Gameboy TAS of 2017 Gameboy TASer of 2016 Gameboy TAS of 2016 Gameboy TASer of 2014 Lucky TAS of 2014 Gameboy TAS of 2014 Lucky TAS of 2013
I see that the audio samples are the most accurate way of timing a movie and video frames should be kept at regular intervals, but that does not mean the recorded input sequence needs to have equal-distance steps as well. What lsnes currently does is well-suited for creating accurately timed video files, but (and that's my hypothesis) not as well suited for creating accurate input recordings. You would need to separate video frames from input frames. When creating a video file from an input file, there would be no one-to-one translation of input frames to video frames then. You would basically sample the recording at your desired framerate (based on the number of audio frames that have passed) and use whatever video frame is the most recent for each sampling point in the created video file to make it equidistant even though the input sequence is not. The run-time of a movie is then not only determined by the number of input frames it has, but also by the length of each input frame. BizHawk is indeed not accurate in that regard, as it treats all input frames to have the same length, even though some of them are shorter. On the other hand, the number of "short" frames in a movie is as most as large as the number of times a game messes with the LCD, which is arguably negligible in most instances.
Ilari wrote:
I don't think GB can generate video frames faster than 35112 cycles per frame (but it can generate those slower).
You are probably right about that, the only instance I've seen a video frame being shorter than 35112 audio samples is the very first frame at GB startup, and I'm not too sure about why that is. The reason short input frames occur in e.g. BizHawk is that input frames are capped at 35112 audio samples, and if a longer frame occurs, it is separated into several full frames and one short frame that syncs it up with the VBlanks again.
Warepire wrote:
Except the game boy offered an alternative to polled input which is interrupt driven input, for a game with interrupt driven input that would simply not work as there will never be a "poll" to catch.
You're right, I forgot about that possibility. All of this discussion is only valid for polled input, as interrupt-driven input can't be accurately recorded in any emulator that I'm aware of. It can happen anywhere and would require subframe input capabilites accurate to the CPU cycle.
Posted: 7/26/2013 7:51 AM
Quote
Ilari
Emulator Coder, Skilled player (1114)
Joined: 5/1/2010
Posts: 1217
Rookie of 2010 New systems TASer of 2010 New systems TAS of 2010
MrWint wrote:
All of this discussion is only valid for polled input, as interrupt-driven input can't be accurately recorded in any emulator that I'm aware of. It can happen anywhere and would require subframe input capabilites accurate to the CPU cycle.
Such capabilities would also require movie format that is pretty totally different from most formats. Subframe capabilities accurate to CPU cycle wouldn't be hard to implement, apart from that interface design could turn nasty.