Editor, Experienced Forum User, Published Author, Expert player
(3560)
Joined: 11/30/2014
Posts: 2744
Location: US
Looking into it it seems like an IRQ timing issue. I was supposed to look into this with Burai Fighter too but never got around to it, looks like I'll have to move it up on the priority list though if more games are effected.
This severe graphical bug in Recca's title screen after you beat the game (or use a debug/stage select feature to start from e.g. Area 1) still persists:
Still? Was this brought up somewhere before? Are there other bugs you are aware of?
Editor, Experienced Forum User, Published Author, Expert player
(3560)
Joined: 11/30/2014
Posts: 2744
Location: US
No worries take your time :)
Also I have made some progress on reconciling start up state as tested by blargg and others with how start up happens in emulator.
When I corrected start up timing to match NESDEV wiki, (first VBlank ~ 27384 cycles after power ) and sync the warps run it turns out that the game end glitch run also changes. Instead of looking more like the FCEUX run, it almost completely matches what you posted in your console verification. This is good because I was a bit puzzled by why the two results were so different.
The only problem is that it now fails to end the game. So it seems I still have a bit more research to do. Even a single PPU clock changes results quite a bit, so if I can match the console verification test it will be really strong evidence that start up state is correct.
Editor, Experienced Forum User, Published Author, Expert player
(3560)
Joined: 11/30/2014
Posts: 2744
Location: US
Huh? I must be misunderstanding what you said in your previous post then. If the NG test run hit the enemy, doesn't that mean you are already accounting for the DMC glitch?
My understanding of the dump script is that it pulls inputs from the movie file and uses them if there is no lag frame. But it's not the movie file that contains the glitched reads, they only show up on playback. So shouldn't it not matter what the source of the input file is, whether it be FCEUX or BizHawk?
Well either way, here is an NG test run, for me Ryu jumps onto a wall early in level 1 and just kind of sits there. I'll be interested to see what happens on console.
http://tasvideos.org/userfiles/info/34390976874363545
Editor, Experienced Forum User, Published Author, Expert player
(3560)
Joined: 11/30/2014
Posts: 2744
Location: US
Aww so the test rom didn't work after all, too bad, I had high hopes for that one. Thanks for trying anyway. Maybe I will try porting it if it seems simple enough.
Hmmm so the bot was working around the glitch after all. Can you try the published run again now and see if it still syncs? My guess is that it won't but it would be really really helpful to know where it desyncs and if it is consistent. I think we're making progress here!
Editor, Experienced Forum User, Published Author, Expert player
(3560)
Joined: 11/30/2014
Posts: 2744
Location: US
http://tasvideos.org/userfiles/info/34305774732879677
Here is a world 1 run of SMB3 on PRG1, it cmes directly from the current warpless run (which syncs on PRG1 up to the boss.)
Here are the frames in the first 3 levels that NESHawk says a DMC conflict occurs on:
623 , 1000 , 2197 , 2256 , 2327 , 2411 , 2736 , 2747 , 2748 , 2779 , 2791 , 2827 , 3583 , 4345
So if you play this back purely per poll, it should desync very quickly if any hits are different.
And here is another interesting test to try:
http://tasvideos.org/userfiles/info/34305974613284573
This is a ninja Gaiden movie that does nothing but alternater pressing left and right (after moving to roughly the center of the screen. )
On NESHawk this ultimately moves to the right and hits the first enemy at 111 on the timer. It would be very interesting to see what the behaviour is on console.
EDIT: True, I think I know another way to test to DMC behaviour that would be really helpful. The test ROM read_errors_fast prints on screen when there are DMC hits.:
https://github.com/christopherpow/nes-test-roms/tree/master/read_joy3
The problem is that it is inconsistent when using things like everdrive because it doesn't sync to the timer in any way (as demonstrated by feos a couple pages back.) But if it can be dumped to real cart, I think it should have consistent behaviour from power on. Are you able to do this?
Editor, Experienced Forum User, Published Author, Expert player
(3560)
Joined: 11/30/2014
Posts: 2744
Location: US
Wow, that's annoying. Any way to fix this, or to add an onlatch and onclock event to lua similar to LSNES? The amount of reads isn't the only important piece of information - knowing when the data is actually latched is very (and more) important.
I have no idea. That's more of an adelikat or zeromus level question and not really an Alyosha level question. 8D
Ok I'll see what I can put together with prg1. Hopefully by tomorrow I will have something worth testing.
Wow great Battletoads synced! Time to make some Battletoads runs on NESHawk to get some console verifications! Thanks for your testing efforts!
I don't really know what the standards are for adding the console verified tag. Technically the FCEUX run is wrong and obtains the correct result in part from blind luck. But on the other hand all the actual input frames are indeed the same, so it does contain the correct stream of inputs, although every actual frame is not the same.
Editor, Experienced Forum User, Published Author, Expert player
(3560)
Joined: 11/30/2014
Posts: 2744
Location: US
Woah it actually synced? Awesome!
Ok let's try the whole run then:
http://tasvideos.org/userfiles/info/34284092070056771
I guess it's no longer the same run since inputs are changed, but it beats the game with both players, so that seems like a pretty good test of accuracy to me.
I think the times it doesn't sync are probably due to different PPU-CPU alignment, which is known to be pretty much random at poweron and can influence timing. This would make sense with how random the desyncs seem to be.
With this in mind it seems like the game end glitch run should sync, maybe try it a few more times to see if you get good luck? Here is a BizHawk file just in case for comparison, but input frames should match:
http://tasvideos.org/userfiles/info/34284149904082436
I'll try SMB 3 next. The last time I tried all stages it desynced the same way as on console in world 2, so I'm a bit optimistic about that one. But going per poll would be interesting.
1) Does event.oninputpoll poll for input any time latch happens? Or just once per frame?
I'll look into it. EDIT: It's every poll. More specifically, it's called everytime a button is read, so for Ninja Gaiden for example it is 16 times per frame, 17 if there is a DMC hit.
2) Will it be possible to use/read/get subframe input in the future? Not important for these runs, but already shown to be important for some ACE runs that have no emulator with working native support.
Possible? Sure easily. I could hack it in there now as an alternate input method. But it would basically be equivalent to the scripting method used to do the SMB3 run. Natively supported as in working at the same level as input does now, with rerecording and TASStudio and such? I don't think this will happen any time soon.
Editor, Experienced Forum User, Published Author, Expert player
(3560)
Joined: 11/30/2014
Posts: 2744
Location: US
Wow actual progress alright! Of course presumably many RNG values get the enemy going in the desired direction and there is no way to know if we are really matching console, but for now I'll take what I can get!
Well then might as well be bold and try to beat that level 8 boss.
http://tasvideos.org/userfiles/info/34263027362025815
Here is a movie file that does so. If a miracle happens and it syncs through the boss, it should desync at the first checkpoint in level 8. If it doesn't sync, can you maybe get a video clip of the boss fight? It would be good to know what attacks the toads are doing. (Or even just a description would be helpful.)
ALso, my mistake on the game end glitch run. It did require some adjustment to work in BizHawk, just not in terms of input frames. It did need some lag adjustment though. But since the input frames were the same, I'm a bit surprised it desynced on console at that enemy since it syncs there in both emulators. Oh well, I guess there is a bit more to figure out here.
Editor, Experienced Forum User, Published Author, Expert player
(3560)
Joined: 11/30/2014
Posts: 2744
Location: US
Battletoads initializaes all RAM to 00 except for FD and FE. It does this almost immediately after the first VBlank it sees.
FD checks whether the intro has been played through once. If it has, then you can skip it if you hit reset. (The value it uses to check this is 0x28)
FE is not used in any way as far as I can tell.
RNG is initialized with a set seed.
As far as I can tell, the only thing resetting (before the intro plays) does is possibly change the even/odd alignment of the DMA unit relative to code execution, which is known to effect timing enough to change the enemy direction in level 3. Initializing RAM in any way has no effect.
Your present tests seem to indicate that the original alignment was correct. Actually I was scrounging around NESDEV and found some tests that also indicate it was correct before. So I think I'll revert that change.
My only guess is that in your previous tests you happened to do a reset at a point where it would change the DMA alignment, or that on startup there is a small chance of it being changed (since one of the above synced to snakes.)
I'll make a new run with the enemy in level 3 going to the right and see how far it can go. ACtually, it turns out you can get the enemies going back to the left simply by putting one blank frame before the second 'Start' press right at the beginning of the game. So I'll do that and see if it works on console too.
EDIT: http://tasvideos.org/userfiles/info/34230566350048461
Ok here it is. (Don't forget to update your BizHawk build.) This one should get the enemy going back to the left on console. If it does, then I think we are making progress and I'll get the run back up to level 8 or maybe try the new game end glitch run.
EDIT 2: just checked and the newest game end glitch urn syncs on BizHawk without modification, so might want to check that one too since it's short.
Editor, Experienced Forum User, Published Author, Expert player
(3560)
Joined: 11/30/2014
Posts: 2744
Location: US
I found some minor improvements while working on 100%, will post an updated movie file later. So please set this to delayed for now.
http://tasvideos.org/userfiles/info/34236752933750740
please replace the movie file with this one. Thank You.
Editor, Experienced Forum User, Published Author, Expert player
(3560)
Joined: 11/30/2014
Posts: 2744
Location: US
I dumped the FCEUX inputs and compared, and yup they are identical up to the boss.
Well, I'm stumped. 8D
Any ideas? Maybe the run just doesn't sync as reliably as I had hoped.
Any suggestions for other easier / shorter runs to try?
I feel like I'm really close but missing a piece of the puzzle.
Editor, Experienced Forum User, Published Author, Expert player
(3560)
Joined: 11/30/2014
Posts: 2744
Location: US
desyncs on console in Turbo Tunnels? That is where it was desyncing in emulator before I swapped the parity of SPRDMA.
That run contains the exact same input frames as the FCEUX version up to the level 8 boss. The console verification page says the run makes it up that far 5/5 times.
Something must be off here. Can you try the FCEUX version and see what happens to that one? Or compare the input log to see if there are discrepencies?
That is a very strange result.
Editor, Experienced Forum User, Published Author, Expert player
(3560)
Joined: 11/30/2014
Posts: 2744
Location: US
Well, sure it's possible. But I don't know enough of the other parts of BizHawk to do it, and am busy with other things. So I'll add it to the list and save it for later.
(By the way, I tried that chinese translation of TMNT 3 but couldn't make it work. I tried the obvious extension to CHR ROM size but nothing happened, so the game must be expecting something else for CHR mapping but I don't know what.)
Editor, Experienced Forum User, Published Author, Expert player
(3560)
Joined: 11/30/2014
Posts: 2744
Location: US
Nice improvement! This looks much smoother then the previous version (in the parts I watched marked as big improvement.) I still didn't see any corner boosting but admittedly it would only save a few frames over an 1.5 hour movie.
I'm also wonder about the big speed boost you can get while jumping off an enemy while going down hill (as in the published run), was there a reason it wasn't possible in 1-1?
Anyway it seems much better played then before voting yes.
Finally, what is a 'Dolphin Frame'?
Editor, Experienced Forum User, Published Author, Expert player
(3560)
Joined: 11/30/2014
Posts: 2744
Location: US
By adjusting for lag, I mean that if there is a lag frame in BizHawk that is not present in FCEUX, I add in a frame at that point in BizHawk, this isn't necessarily a blank frame, it could have button presses in it. The idea is that every valid input frame in FCEUX will also be an input frame in BizHawk, even there are lag differences. I don't know how the RNG works, but it could be that it only updates on input frames.
I don't know enough about new ppu vvs old ppu to comment on why that Mario run desyncs though.
@True: Here is a Battletoads run (2-p warps) that syncs on the latest BizHawk build through the level 8 boss. I think it's possible that it will sync on console where FCEUX fails. It is identical in terms of input up that boss.
http://tasvideos.org/userfiles/info/34147851839238345
Also, Battletoads does some initializing with this code:
LDA $#0
LDX $#17
STA $4017,X
DEX
.
.repeat until X=0
.
BizHawk and FCEUX report this as an input frame (frame 3) since it is a write to $4016 (also a read from $4016 since it's indexed and the CPU reads before it writes.) But since a 1 was never written, no controller data is actually latched. Does the bot see this as an input frame as well?
Editor, Experienced Forum User, Published Author, Expert player
(3560)
Joined: 11/30/2014
Posts: 2744
Location: US
@True: The Kirby run has a reset on frame 6, how does the bot deal with that? The only difference between the builds is how many lag frames there are at startup I believe. Kirby also used the DCM channel but seems to be more careful then NG about how it uses it.
I'm stumped on Battletoads, need some new info before I can figure out how to fix that one.
EDIT: Ok, so I finally finally finally figured out Battltoads. It turns out this was breaking on SPRDMA timing. It takes either 513 or 514 cycles to complete an even or odd DMA depending starting cycle..
As it turns out, I had my check backwards. Normally this wouldn't matter since you might expect each DMA to roughly randomly take place on an odd or even cycle. However, it turns out most of Battletoads' DMAs were taking place on the same type, which happened to be 514, when this should have been 513. The accumulation of extra cycles eventually added a lag frame at the menus which changes RNG and causes the desync at level 3. Setting the DMA timing correctly gives the right result and the run syncs through level 3.
As a bonus, this fix makes the mysterious behaviour in the second test of scanline/scanline appear correctly. So that's a 2 for 1.
Editor, Experienced Forum User, Published Author, Expert player
(3560)
Joined: 11/30/2014
Posts: 2744
Location: US
Ok, I've got a lot of new info, hopefully this will help make some progress.
Let's start with monopoly since it's the easiest. Monopoly appears to rely on uninitialized RAM. I ran acmlm's run on NESHawk with default RAM settings (the usual FF-00 pattern) and the run does sync. Then I ran it with all start up values set to AE and it desynced after the menus. I haven't tracked down which addresses exactly are at play, but that's what's going on with that one.
I'll try Journey to Silius next.
EDIT: nevermind all that other stuff that was here before about battletoads, turns out my movie file was corrupted somehow. Back to the drawing board
EDIT2: Silius syncs after adjusting for lag, no change of input needed. Will check Ninja Turtles next
EDIT 3: Ninja Turtles does not get the ninja star drop in the Dam level
Editor, Experienced Forum User, Published Author, Expert player
(3560)
Joined: 11/30/2014
Posts: 2744
Location: US
@ true: http://tasvideos.org/userfiles/info/34108792710373640
Ok I tested the most recent Kirby's Adventure run. I had to wait one extra frame before hitting start on the title screen, but other then that it synced through world 1. I truncated the movie to where it desynced in 2-1, so if it makes it there I guess that is a good start.
I also tested Battletoads, 2 players warps, but it de-syncs at the life stealing enemies mid way through level 3. This one I traced back to the title screen, where there is an extra lag frame in BizHawk not present in FCUEX which seems to throw off RNG, changing where the enemies in level 3 go. This seems to come from OAM DMA, which is too short in FCEUX, and the extra cycles it saves over the first 1000 or so frames convert into a saved lag(loading) frame.
Let me know if there are any other games you think it would be useful to test.
I also realized that Ninja Gaiden is effected by the initial conditions of the DMC channel clock and bit counter. The game does nothing special to initialize them, and I had previously only guessed at initial conditions. Since these are constantly going at all times, there can be radically different sync results depending on the iniital state there. But after some tinkering the furthest I got was mid way through the game. another variable to look at though.
EDIT: i also set BizHawk start up back to matching FCEUX again, so now there is no need to add or subtract any frames from an fm2 converted to bk2, this should make testing more convenient and remove a potential source of error. I tested Zelda with this build at least up to getting the step ladder and it synced just fine.
Editor, Experienced Forum User, Published Author, Expert player
(3560)
Joined: 11/30/2014
Posts: 2744
Location: US
^ oh , good timing.
DMC is tested to grueling levels with blargg's test suite, so if something is still wrong I don't know what it is.
The DMC re-read happens in 3 CPU cycles, so maybe it's possible it's not getting picked up by the bot?
Yeah if you are available to do something on IRC let me know where to go, because I do have several things I need clarified before I can make any real headway.
Editor, Experienced Forum User, Published Author, Expert player
(3560)
Joined: 11/30/2014
Posts: 2744
Location: US
@true: Thanks for your efforts so far. That the run syncs at all is baffling. FCEUX doesn't emulate DMC conflicts , so a run made on it for a game with seemingly common ones and weak error correction seems to have little chance of syncing , yet it somehow makes it. Are you able to check ninja gaiden 2? That would be another good test case .
Also I have a suggestion if you have the time . DMC conflicts seem to be pretty common , but would be undetectable if it happened on a frame where ryu is just running left or right with no other inputs. So would it be possible to make the bot stop replaying the file at the first DMC hit? I am very curious where it happens and would be very useful in debugging.
And if there are none at all, well then it's back to the drawing board since I would have no idea what's happening.
EDIT:
for reference, here are all the frames that BizHawk says a DMC conflict occurs on up to desync in level 2. There are about 50, but only 1 happens in a fatal place. So , about 1% of frames have a conflict on them, more or less.
90 , 305 , 497 , 696 , 714 , 733 , 736 , 754 , 767 , 952 , 989
1075 , 1167 , 1267 , 1282 , 1551 , 1606 , 1645 , 1799 , 1816
1983 , 2039 , 2141 , 2190 , 2208 , 2243 , 2252 , 2291 , 2300
2304 , 2352 , 2353 , 2449 , 2457 , 2462 , 3265 , 3336 , 3371
3462 , 3492 , 3563 , 3683 , 3785 , 3804 , 3996 , 4031 , 4139
4189 , 4200 , 4219 , 4265
EDIT2:
true wrote:
I'll write a script to verify the fceux-dumped input stream against bizhawk, and dump the desyncing run from bizhawk and verify against console. I'm not sure what will happen for the former, but I am assuming the run will not (de)sync in the same manner for the latter.
Can you go into more detail on this and how you do this? How is the input stream different from the per-frame inputs? Are you just cataloging what the emulator says $4016 gives at each poll?
