Editor, Experienced Forum User, Published Author, Expert player
(3560)
Joined: 11/30/2014
Posts: 2744
Location: US
Gooftroop wrote:
Hello,
I'm new and having problems recording longer TAS projects, so I have two questions.
Is it normal for Bizhawk / TAStudio to start lagging when recording over 100,000 frames / 5 hours?
And is it possible two TASPROJ. / BK2 from the same game and divided into sections to be put together via the input log?
Sorry for my English - I use a translator.
It's possible BizHawk is running low on memory for longer TASes, you can try turning down the buffer settings in TAStudio under metadata -> savestate history settings.
You can patch together segments by stopping a current movie, and recording a new one using the Record from: 'now' setting in the movie recording dialog. From there you can use TAStudio again and later just copy paste the input logs together.
Editor, Experienced Forum User, Published Author, Expert player
(3560)
Joined: 11/30/2014
Posts: 2744
Location: US
I'm having no success getting any other runs to sync for games I have. Here's a quick summary:
Fire Emblem:desyncs in second stage, doesn't seem fixable fixed in 2.1.1
Fire Emblem Sacred Stones:desyncs about half way through the run in the same fashion as the first game. fixed in 2.1.1
Banjo Kazooie: I can't fix the RNG of the enemies, so damage boosts don't happen.
Rayman Advance: Syncs about half way through with some manual effort, but then breaks. Fixed, new run submitted.
Advance Wars 2: Seems like it might sync in emulator but desyncs in a strange way on console, the game polls input in a weird way.
Mario vs. Donkey Kong: See below
Turok Evolution:bad enemy RNG in first stage. New run submitted, syncs on console.
Mega Man Battle network 2: Bad encounter RNG
So yeah, definitely hitting a road block in terms of resyncs.
I did manage to console verify the Donkey Kong Country 101% run though:
Link to video
It turns out I had a wrong EEPROM setting which was preventing sync at the start. After that I needed to adjust timing of EEPROM writes to be about 10% faster than the GBA Tek baseline to get it to sync.
GBAHawk version 2.1.1 will have a sync setting to account for this. Eventually I'll add a similar setting for Flash, but that is more complicated so just EEPROM for now.
Some time ago RetroEdit sent me a resync of Donkey Kong Country 2, but I don't currently have a cart to test it, seems like it should work though.
Editor, Experienced Forum User, Published Author, Expert player
(3560)
Joined: 11/30/2014
Posts: 2744
Location: US
I haven't had any luck with RNG and resyncing runs, so I'm working on a few more small accuracy issues. One of these is a known error in sprite rendering where too many pixels are rendered. I made a test for this and also a similar thing that happens when hblank interval is free:
Fixing it was pretty simple, but it's possible I'll need to redo sprite rendering in the future if there are other side effects.
The next thing I'll do is fix the weird behavior involving DMA in ROM area.
Editor, Experienced Forum User, Published Author, Expert player
(3560)
Joined: 11/30/2014
Posts: 2744
Location: US
I finished up verifying Metroid Zero Mission runs thanks to RetroEdit's resync script greatly speeding up the process. The new Metroid Fusion Memory Corruption run also worked on console so that was cool. That's all runs of GBA Metroid games verified along with ~50% of associated ROM hack runs, good progress.
Now I will try to fix some runs where RNG doesn't work.
Editor, Experienced Forum User, Published Author, Expert player
(3560)
Joined: 11/30/2014
Posts: 2744
Location: US
GBAHawk version 2.1.0 is released! This version comes with numerous accuracy improvements along with a small but noticeable performance improvement.
With this release I was able to add console verified check marks to Shrek 2 and the baseline run of Metroid Zero Mission. I should be able to do the low % zero mission TAS this week along with the pending 6 scrolls ROM hack.
I tried resyncing Fire Emblem with RetroEdit's script but it desynced at the same point as when I tried manually, so I guess RNG for that game will prevent verification of the current run as is. I also tried Turok Evolution but desynced due to RNG as well. RNG is also preventing me from making progress on Banjo Kazooie.
I'm going to focus on fixing some of those RNG issues as well as making a few test TASes of different games that I haven't previously used for testing.
Still a lot of work and verifications to do, with Mega Man Zero and Battle Network Series, various Castlevanias, Zelda Minish Cap and Mario Super Star Saga. I'm hoping the most tricky and time consuming emulation issues are behind me so I can focus more on the TASes.
Editor, Experienced Forum User, Published Author, Expert player
(3560)
Joined: 11/30/2014
Posts: 2744
Location: US
The quest to find the bug that was keeping Shrek 2 from being console verified is over! Today I found that it was a rather simple issue with timing of the case where a DMA pauses an already running DMA of lower priority. This also fixes the desync with Metroid Zero Mission.
Link to video
With that, I have no TASes known to desync due to emulator accuracy issues. I have to do some work to verify the fix in all edge cases, but it fixes the two desyncs and doesn't break any existing TAS, so it seems solid. At the same time I also redid part of the DMA loop, which gave a ~5% speed up due to skipping a lot of steps when no DMA is running. It's not much but I'll take every fps I can get.
Once I have everything tested and verified good I'll release v2.1.0. Now that Shrek 2 is no longer stuck in the back of my mind I can focus on the long list of relatively less important edge cases like undefined / implementation defined cpu behavior, running DMA in decrement mode in ROM region, running FIFO DMAs very close together, etc.
I also tried to verify Donkey Kong Country but the EEPROM timing was way off, no small or even relatively large change to the baseline timing could possibly fix it. I'm not sure what I'll do about such cases yet. Maybe just use extra blank frames since saving only happens at the very start and very end of the run, but I'd much rather have something cleaner.
Anyway finally time to test some new runs!
Editor, Experienced Forum User, Published Author, Expert player
(3560)
Joined: 11/30/2014
Posts: 2744
Location: US
Testing with Metroid Zero Mission is going well. I already verified a couple of the ROM hack runs and the 100% run. Two of the hacks require a 32MB card which I don't have, so I'll to get one before I can do them. On the baseline run I actually got a desync. Haven't figured out the cause yet, but I can see the desync point pretty clearly and the error would have to be fairly large (~50 cycles) to get a desync in that spot. I would think such a large error at this point can only be caused by something with audio DMA, but nothing obvious sticks out at me so far.
Zero Mission also revealed an emulation error in the ppu blending effect. This was an an easy fix but pretty surprising this case didn't come up before.
Banjo Kazooie continues to reveal bugs as well. It turns out I missed a case of dma pausing another dma resetting access timing to be non-sequential. This happens to occur in Banjo Kazooie and was causing failures in my testing. After fixing that I can finally get past loading and the intro with correct timing. I tested up to about 4 minutes into the run and it seems RNG is still good. So maybe that game is good now, but I can't get correct enemy movement so far trying to manipulate RNG with jumping so not sure yet.
Editor, Experienced Forum User, Published Author, Expert player
(3560)
Joined: 11/30/2014
Posts: 2744
Location: US
We just reached 350 console verified runs! That is out of 3335 currently published runs, so managing to stay above 10%. I think 400 is reachable without too much trouble, mostly just a matter of time. Past that I think resyncability of old runs will start to become a major hurdle on the road to 500.
I also have almost 10% of all GBA runs verified, so that's pretty neat. I'm hopeful to reach 50 total this year, seems doable. The biggest variable here is long it will take me to track down remaining accuracy issues.
So onward to 400.
Editor, Experienced Forum User, Published Author, Expert player
(3560)
Joined: 11/30/2014
Posts: 2744
Location: US
Occasionally I look around for any homebrew or tech demos that I missed testing, and I recently came across this Yoshi's Island tech demo that I hadn't seen before and was broken in GBAHawk:
It turns out I was not doing rotation and scaling correctly in this case, so I went back and redid that code. With a better understanding how it was supposed to work, the rewrite was more accurate and also quite a bit faster since I got rid of the floating point arithmetic. It gives a couple fps boost to games that use mode 7 stuff. Now the demo works correctly:
I'll probably go back and do the same clean up for rotation and scaling of sprites, might get a small performance boost from that too.
Now that I finally have a Metroid Zero Mission cart I will be testing that game and associated ROM hacks.
Editor, Experienced Forum User, Published Author, Expert player
(3560)
Joined: 11/30/2014
Posts: 2744
Location: US
I've made a lot of progress understanding Grunty's Revenge. The game doesn't use halt, and bases RNG off a timer it starts shortly after power on, so everything needs to be perfect to get correct RNG. It's a really good test case.
Debugging this was difficult because I was getting confused by 2 unrelated errors that very coincidentally cancelled each other out. What are the odds! One error was in multiplication timing (6 cycles too long) and the other was a new behaviour in the prefetcher that I just discovered thanks to this game (which ended up resulting in execution being 6 cycles too short.) On ROM boundaries of size 0x20000, the prefetcher completely stops. These boundaries are known to generate forced non-sequential accesses, but it wasn't known that this causes the prefetcher to halt. I guess the prefetcher can only do sequential accesses. There is apparently also an extra cycle penalty when this happens.
After fixing these I was still off a bit, so something else was wrong. It turns out that the prefetcher still runs even when execution is happening from RAM or BIOS. Originally I had this as disabled. This usually doesn't effect anything because execution in these regions is almost always longer than 16 cycles, which is the time it takes the prefetcher to fill its 8 byte buffer and halt. When execution returns to ROM it is either at a different point, or the address before the first one in the prefetcher (since the pipeline is two instructions long) triggering a reset. However Banjo Kazooie jumps to BIOS from ROM and runs only 2 instructions after the end of IRQ code, which is short enough that having the prefetcher still active triggers some bus access delays.
With all of that fixed, now I can get as far as initial loading with exact execution timing. I can do a couple more tests up to first input, but after that it will just be a matter of running the TAS and seeing if it works. Unfortunately I can't get good RNG to resync the current run yet, so I'm a bit stuck. Hopefully I can guess and check my way to success.
All of this also improved sync on Shrek 2, which started desyncing in mid-run around GBAHawk 2.0.1 due to IRQ changes. Console was known to sync further than this so something was up but I didn't know what. With these fixes it now goes back to desyncing at the next to last level, where it originally desynced. Unfortunately these improvements STILL didn't seem to fix it. The search for a solution continues.
Editor, Experienced Forum User, Published Author, Expert player
(3560)
Joined: 11/30/2014
Posts: 2744
Location: US
Cool! How many such soft resets would there be in a complete run? 75 frames is an inexplicably long time, not sure what's causing that, but I guess it's manageable as long as it's consistent.
Editor, Experienced Forum User, Published Author, Expert player
(3560)
Joined: 11/30/2014
Posts: 2744
Location: US
I've been testing Grunty's Revenge, and am getting desyncs that look like they could potentially lead to diagnosing some emulation issues. It appears I am getting bad RNG, and after g0goTBC helpfully told me where to look, I found that the RNG is initially tied to one of the GBA hardware timers for initialization. I confirmed that EEPROM is not an issue, by testing with both an original cart and with a dev cart that doesn't have EEPROM and I got the same results. This is very promising, as it makes this a perfect candidate for some more test ROM injecting. I'll be diving into this over the next couple of weeks, hopefully something interesting shakes out.
Editor, Experienced Forum User, Published Author, Expert player
(3560)
Joined: 11/30/2014
Posts: 2744
Location: US
Link to video
I was able to properly verify Metroid Fusion, so that run is verified, finally completing one of my original goals. I am working on 100% as well.
The resync was done using RetroEdit's resyncing tools, which are incredible and saved me hours of effort. They should greatly speed up future verifications where simple adjustment for lag / loading times are all that is needed. EDIT: 100% done as well, cool!
Editor, Experienced Forum User, Published Author, Expert player
(3560)
Joined: 11/30/2014
Posts: 2744
Location: US
I finished up verifying Wario Land 4. All resyncs were pretty simple. 4/4 runs verified. I'm going to try Metroid Fusion again, then move on to Metroid Zero Mission.
Editor, Experienced Forum User, Published Author, Expert player
(3560)
Joined: 11/30/2014
Posts: 2744
Location: US
I have released version 2.0.3 of GBAHawk. This version adds back in ffmpeg encoding, which was requested by several people. For emulation improvements, I added basic emulation of stop mode, which I suppose isn't very helpful for TAS, but is a usable hardware feature so is good to have implemented. In the process of adding it I realized that stop mode is disabled on GBP, so when I get around to Gamecube detection I will have to account for that. I also fixed the open bus issues and gamepad IRQ issues mentioned in the previous posts, along with some other minor things.
With this, the only known console verification issue is with Shrek 2. I have no leads on it and no way to test anything. I think I will have to simply set it aside for now and test other games. I recently went back to double check Bionicle, and it was indeed just an input timing issue, so nothing fruitful came of it.
There is a new test ROM available for Sprite VRAM access timing that GBAHawk currently doesn't pass. It looks like maybe I just have to cut off VRAM access a bit earlier when HBlank is used for VRAM access, but I'm not yet sure. Aside from that the only known failing test is due to decrementing DMAs in the ROM region, which I might fix for next release but isn't really important.
I will be focusing on console verifications now, starting with Metroid Zero Mission and associated ROM hacks. With little else to go on, I kind of have to hope to see some desyncs on console that look approachable.
Editor, Experienced Forum User, Published Author, Expert player
(3560)
Joined: 11/30/2014
Posts: 2744
Location: US
I finally figured out Incredibles enough to get it to sync. The game initially turns on gamepad IRQs which makes playback difficult, but it turns them off again after the first input. So I found that having an extra start press while the game title is loading turns off the gamepad IRQs before they cause issues with the rest of the run. The code it runs at the IRQ seems like it is not entirely benign though and messes up sync for the rest of the existing movie file from RetroEdit. GBAHawk 2.0.2 had a bug where the interrupt wasn't firing at all, so the desync seems unavoidable. I resynced what I could manually, the video is below. It seems like, barring anything else being wrong, Incredibles should be console verifiable in GBAHawk 2.0.3.
Link to video
I also decided to test Super Monkey Ball Jr as the run is short and the 3D seemed like a good test of timing. The TAS desyncs very badly, but I got a test run of level 1 going and it made it through that. The game uses EEPROM, but doesn't appear to be too sensitive to timing. Comparing the video to emulator, it looks like the EEPROM is going somewhat faster on console, so the game spends a few extra frames on the title screen, but it doesn't effect the rest of the run:
EDIT: video removed, superceded below.
So good progress. I have a couple more things I want to do before releasing 2.0.3, but now that I finally figured out Incredibles I can start making progress again.
Editor, Experienced Forum User, Published Author, Expert player
(3560)
Joined: 11/30/2014
Posts: 2744
Location: US
So I finally managed to track down the source of desyncs on console for Incredibles, and it turns out to be something related to gamepad IRQs. I don't have all the details yet, but what I know so far is that if I modify the ROM to not enable them, the existing test run syncs just fine. Gamepad IRQs are tricky because they are triggered at different times on console compared to emulator, since the input resolution is only 4 scanlines. But it also seems clear that my current implementation is somehow wrong. in the end I may need proper sub-frame input timing so that I can match input to the emulator where it would be console to get sync. I need to do several tests first, at least the problem is known now.
Editor, Experienced Forum User, Published Author, Expert player
(3560)
Joined: 11/30/2014
Posts: 2744
Location: US
Dimon12321 wrote:
Thank you very much for your effort for GBA emulation and verifications! A sceptic question, sorry. Did anyone else verified a movie which has been previously verified by you? I'm just worried that you may be developing GBAHawk with only your improvised verificator and ROM re-writer in mind.
I don't think so, but all my files are available here: https://github.com/alyosha-tas/GBA_replay_files and I use original carts except for homebrew.
So, anyone can check, but that kind of thing is kind of unavoidable with things this niche.
Editor, Experienced Forum User, Published Author, Expert player
(3560)
Joined: 11/30/2014
Posts: 2744
Location: US
It's a new year, and almost 3 years already since I started this thread, wow that time went fast. That time though saw big advances in emulation and console verification on a number of systems. Here's my synopsis of the current state of R&D.
NES: The biggest deficiency left for NES is reset behavior, but few runs use that anyway. Aside from that many / most TASes should be verifiable, even ones that rely on precise power up timing and DMC glitches. Console verification of NES TASes is now pretty standard, and it is far more rare to see one fail than succeed (as long as its being done on NESHawk.) There are some edge cases that are un-emulated in NESHawk, but they are unlikely to ever come up in practice, probably only if some homebrew dev used them for an emulator check. Implementing these would require a major rewrite to NESHawk, and I'm not sure they even effect all hardware revisions. Anyway, NES is in good shape.
SNES: Modern versions of BSNES are now in BizHawk, but even so console verification has limited success. Most desyncs are random, very likely due to the audio chip clock, a well known issue. There are also some deterministic desyncs, though with unknown cause. There have been a few successful verifications along the way, and recently I saw that DwangoAC managed to verify a WIP of SMW 96 exits. Surprisingly, there is even new emulation discoveries still happening, as very recently an un-emulated cpu quirk was found. It's hard to say what is needed to advance verification efforts for SNES, it might even take some kind of hardware mod to tame the SPC clock. Deterministic desyncs make me think that something is not right on the emulation side, although it could still be a bot issue, but where to even look?
GB/C: Nothing much new has come up that I'm aware of for GB/C, its pretty mature.
GBA: Lot's of progress has happened on GBA. Tests from Flevoriux have nailed down most ppu timings and emulation details. Myself and GhostRain0 have found a lot of details related to prefetcher emulation. I've also found a few of other minor emulation details for other system components. All of this effort on the emulation side has led to many new verifications, including of first party games. There is still a ways to go, with some known desyncs and emulation deficiencies. However I'mn hopeful (and pretty confident) that 2024 will see GBA emulation and verificaiton mature to the level of NES and GB/C.
N64: A great amount of emulation effort has taken place for N64 since my original post. The N64 core of the Ares emulator is maturing quickly and is increasingly accurate, with several talented N64 devs contributing. Even cache emulation is now close to cycle accurate, it's an impressive feat. I'm not aware of any actual console testing taking place yet, and perhaps it's still early as there are still many issues, but there are exciting prospects. Maybe I'll do some testing myself later this year.
DS: While it seems like a likely next target for work after GBA and along side N64, the DS is a very complicated system and much work still needs to be done.
Genesis: Bigbass made a new verification pipeline and tools for Genesis verifications and even made the first new verification for Genesis in a long time. This is impressive work and now there is a lot of room for more genesis testing and verification.
I believe that sums up all the major developments. Console verificaiton is overall much more mature than it was 3 years ago, in some cases it is even commonplace. Also compared to 3 years ago, a lot of the low hanging fruit is now gone.
So, what now? Personally I don't know what I will do once I finish up work on GBA. N64 looks like the only approachable system, though maybe its worth it to take another look at SNES. Then there are basic questions like what do we do about disc based systems? What do we do when clock speeds get too high to emulate systems in a cycle accurate way efficiently?
That about sums up my thoughts, anyone else have any thoughts or predictions?
Fixing it was pretty simple, but it's possible I'll need to redo sprite rendering in the future if there are other side effects.
The next thing I'll do is fix the weird behavior involving DMA in ROM area.
It turns out I was not doing rotation and scaling correctly in this case, so I went back and redid that code. With a better understanding how it was supposed to work, the rewrite was more accurate and also quite a bit faster since I got rid of the floating point arithmetic. It gives a couple fps boost to games that use mode 7 stuff. Now the demo works correctly:
I'll probably go back and do the same clean up for rotation and scaling of sprites, might get a small performance boost from that too.
Now that I finally have a Metroid Zero Mission cart I will be testing that game and associated ROM hacks.