Editor, Experienced Forum User, Published Author, Expert player
(3536)
Joined: 11/30/2014
Posts: 2733
Location: US
thommy3 wrote:
Wow, it's crazy how fast you get these improvements out. Great work.
I've a question though, is an emulator passing all those above tests perfect in the sense that it behaves like real hardware?
Or is it just that he passes all pubished test roms but there might still be differences to real hardware because maybe those roms didn't cover every hardware aspect?
Thanks!
In the case of NesHawk specifically, yes there is a very real difference between the emulation architecture and the real hardware.
NesHawk runs at the the timing level of a PPU tick, when 3 of those ticks go by we run one CPU instruction. A real NES though runs at a timing level one step above this, where a master clock ticks off pulses and the PPU and CPU use their own divders to time execution. In particular the 6502 (or the NES variant in this case) actually runs 2 phases which make up one instruction clock, and this level of hardware detail is not emulated at all by NesHawk. The fact it can still pass all these edge case tests though is simply proof that this level of detail is not really needed almost all of the time.
If you are interested in how things really work, you can check out Visual 2A03 (the NES cpu) or Visual 2C02 (the ppu), pretty neat stuff!
____________________________
I finally got around to working on ppu open bus behaviour. It was pretty simple, now cpu_dummy_writes/cpu_dummy_writes_ppumem and ppu_open_bus/ppu_open_bus both pass. Scratch 2 more off the list.
Editor, Experienced Forum User, Published Author, Expert player
(3536)
Joined: 11/30/2014
Posts: 2733
Location: US
jlun2 wrote:
I got a suggestion: Try running the console verified runs on BizHawk and see if they sync. Most of them were done on FCEUX, but if for whatever reason it does not sync maybe something is wrong?
Good idea! I think I'll give it a try once I get DMA timings worked out. Only games that are very geenrous with timing alignments will sync without that.
Speaking of which, after several failed attempts and hours of debugging, dpcmletterbox/dpcmletterbox now works properly. This is a signifcant improvement to DMC DMA timing so I hope i can make more progress on OAM DMA now.
No where near that bad. The performance hit so far came from code that runs every pixel (3 times per cpu tick) but that is done now. I think only accurate APU emulation remains that will a signifcant negative impact on speed (my guess is another 2-3%.) After that I do plan to go over everything and make obvious optimizations to gain back some of the performance losses, but I doubt I can reach parity with 1.11.6 again in terms of speed.
I don't really understand what feos is suggesting with the cpu refactor, so I'll leave that up to him, seems like a huge undertaking.
zeromus wrote:
Sure thing I installed it, but I can't tell what, if anything, it is doing, how do I use it?
_____________________
I've been testing various games known to be tricky to emulate. The first game that was previously incompatible with BizHawk now works, Fire Hawk!
Curiously this game runs on FCEUX but went to black screen on previous versions of BizHawk.
If anyone knows other games which are just plain incompatible please let me know (I am aware of Time Lord currently) having games to test on really helps.
EDIT: oh, the recent fixes to OAM reads also fixes cpu_dummy_writes/cpu_dummy_writes_oam, so, scratch another one off the list!
Editor, Experienced Forum User, Published Author, Expert player
(3536)
Joined: 11/30/2014
Posts: 2733
Location: US
The version of AtariHawk in BizHawk 1.11.6 contains numerous, and some rather serious, bugs that have since been fixed. In partiuclar a bug in the 6532 causes measurably innacurate timing, which this game uses. This run does not sync in the current dev build (fails at level 1.)
I'm not sure what the rules are regarding this, are runs from dev builds submittable?
Still entertainging game though. Having to play with both joysticks must make it challenging.
Editor, Experienced Forum User, Published Author, Expert player
(3536)
Joined: 11/30/2014
Posts: 2733
Location: US
I really like these technical runs. Impressive result.
Pretty entertaining and good commentary. I'm also curious about those shaky portraits in the credits though, how strange.
Editor, Experienced Forum User, Published Author, Expert player
(3536)
Joined: 11/30/2014
Posts: 2733
Location: US
feos wrote:
What about attempting to speed up the core too?
Well, my current improvements to accuracy slow down the core by a noticable amount. At 400% speed I can run SMB3 on my laptop at about 180 fps on 1.11.6. The current Bizhawk master build can do about 160. My current test build can do about 152.
So... we're kind of going in the wrong direction there 8D
These represent almost entirely ppu changes. So that might also be a good place to look for performance improvements.
EDIT: after spending a long time to find a very small bug, oam_stress/oam_stress now passes.
I'm also making progress on sprdma_and_dmc_dma, I'm hoping maybe by the end of the month those tests will pass.
Editor, Experienced Forum User, Published Author, Expert player
(3536)
Joined: 11/30/2014
Posts: 2733
Location: US
NesHawk now passes:
apu_test/rom_singles/7-dmc_basics
Figuring out what was going on here took some effort but I now feel the implementation is pretty accurate.
Unfortunately it now hangs completely on sprdma_and_dmc_dma. I haven't the slightest idea why. Previously the test resulted in answers off by a factor of about 5, but at least it finished. I'm not sure what to make of this and posted on NesDev looking for some insight. Safe to say it's still a work in progress.
But other games i tested that use DMC still work fine, so prehaps I'm just off by a couple of cycles, sprdma_and_dmc_dma is a very exacting test.
EDIT:
AFter a bit of work filling in some of the undocumented opcodes, I am now able to pass instr_misc/instr_misc
So 4 extra passing tests means we should be tied for 2nd place with MyNes, alright!
Editor, Experienced Forum User, Published Author, Expert player
(3536)
Joined: 11/30/2014
Posts: 2733
Location: US
I greatly improved my code and committed to my fork for testing. So anyone who wants to try it out please do and let me know if anything is amiss. I have tested with several games so far (SMB, SMB3, Battletoads, Micro Machines) and found no deficiancies.
https://github.com/alyosha-tas/BizHawk
This version also passes 2 accuracy tests that previously failed
sprite_overflow_tests/3.Timing
sprite_overflow_tests/4.Obscure
Editor, Experienced Forum User, Published Author, Expert player
(3536)
Joined: 11/30/2014
Posts: 2733
Location: US
I have been (very) slowly working on NESHawk PPU. So far I have rewrote sprite evaluation to take place simultaneously with background generation, as it is with a real NES. Together with proper read behaviour of 2004 during this time, I finally managed to fix those annoying horizontal lines and shaking in Micro Machines.
This is still very early WIP work and generally breaks other things, but the timing is correct so i should be able to start slowly fixing other timing issues and bugs and such.
Right now it also slows down emulation by a noticable factor as well. I don't think my code takes considerably longer to run then the original, so this is the first major issue to be worked out.
Editor, Experienced Forum User, Published Author, Expert player
(3536)
Joined: 11/30/2014
Posts: 2733
Location: US
I happen to notice in Samsara's temp encode that the health never goes down. I have no idea why this might be, different ROM? I just mention it as a note to publishers since health management is a pretty important part of the run.
Editor, Experienced Forum User, Published Author, Expert player
(3536)
Joined: 11/30/2014
Posts: 2733
Location: US
After a lot of staring at the code I finally resolved the high pitch sound bug that effects several games, notably Combat, so the tanks don't squeek anymore.
I also rewrote most of the audio generation code to be closer to cycle accurate, although there are still a few minor bugs to sort out. I'm pretty happy to have gotten this done since it is now much simpler and sounds better.
Editor, Experienced Forum User, Published Author, Expert player
(3536)
Joined: 11/30/2014
Posts: 2733
Location: US
jlun2 wrote:
I'm not sure if this is the right place to post, but the game Star Wars: The Empire Strikes Back seems to not exactly...work. Disregarding the black bars on the side of the screen, on BizHawk 1.11.6, pressing any of the buttons do not move the ship whatsoever.
Given there's only 1 button on the controller I'm not sure what I'm doing wrong. Is it the emulator's fault or the game?
Seems like it's working fine to me. The game expects you to hit 'reset' before you can play it, so make sure you do that, then it plays fine even on 1.11.6.
The black lines are part of the Atari 2600 TIA and are not an error. I usually see them called 'The HMove Comb', it is a side effect of moving objects on screen.
Editor, Experienced Forum User, Published Author, Expert player
(3536)
Joined: 11/30/2014
Posts: 2733
Location: US
Adding the keyboard sounds well beyond my programming ability. I think I'll stick with fixing core bugs, but if you get the framework for it in place I will definitely help putting everything together and sorting out any issues.
I made the initial commits with all necessary bug fixes so the system loads, so it's just waiting for the keyboard to put it all together.
This also makes me think that paddle support would be great to have, but that is probably even harder then the keyboard.
Editor, Experienced Forum User, Published Author, Expert player
(3536)
Joined: 11/30/2014
Posts: 2733
Location: US
So I was asked to look at Compumate as the next big bug to sort out. I had some time today so I dug into it. It turns out there were numerous bugs in both the mapper and core that needed to be sorted out. But, by comparing tracelogs between Stella and BizHawk I was able to sort everything out and get it running pretty quickly.
(the program alternates drawing lines between frames, so it really does look like this if you only look at one frame)
Of course just getting it loaded up is only a small part of the battle, as there is currently no way to send it any meaningful input, but it's a start!
The most important things that came out of this is realizing that console startup needs to have improved accuracy for the compumate to work. The 'game' does not set the stack pointer appropriately before doing basic operations (as most games typicaly do) instead relying on the 6532 start up sequence. This sequence sends a reset signal to the 6507 which sets the stack pointer to FD.
The current code does not utilize this start up sequence instead just setting everything to default, which for the 6507 is stack pointer 0, which unfortunately overlaps TIA addresses, corrupting the return address of a JSR.
Anyway, I hardcoded in an initial stack pointer of FD simulating post reset behaviour, but I am ultimately not very satisfied with this approach and will probably try to use reset correctly, but right now I don't know how this will work with the current machinery.
Editor, Experienced Forum User, Published Author, Expert player
(3536)
Joined: 11/30/2014
Posts: 2733
Location: US
Wow this is fascinating stuff! Seems like it might not come to anything but any step in the right direction is a good thing. I hope this is the start of finding a workable solution.
Editor, Experienced Forum User, Published Author, Expert player
(3536)
Joined: 11/30/2014
Posts: 2733
Location: US
Mothrayas wrote:
If the only thing that would change is the encode, and not any of the run input, there's no reason to update the movie file. Just keep the movie file the way it was originally submitted and published, for authenticity reasons.
Some of the movie files would change due to different timnig. So you'd have to change the movie file to get the updated encode anyway.
I do like jlun2's idea of having an alternate movie file though alongside the original, seems like a good solution to both currency and authenticity. Is there a good way to do this at present ?
Editor, Experienced Forum User, Published Author, Expert player
(3536)
Joined: 11/30/2014
Posts: 2733
Location: US
After many hours trying to figure out the 4A50 mapper and what it was trying to do, and then giving up and looking at how Stella did it, I finally worked out the bugs to get it running.
It turns out there was a rather serious bug in AtariHawk itself where PeekMemory actually pointed to ReadMemory in the system. This caused chaos when I tried to trace log out what was going, as the tracelogger peeks changed banks too when it shouldn't have. Once that got fixed it was only a matter of tracking down where banks weren't being changed properly.
The ultimate problem was just an incorrectly placed closing bracket. :(
Anyway this is still displaying things a bit Different then Stella, so I'll have to see why, but the test program looks perfect and anyway this is the only example game I could find.
EDIT: turns out I just needed to add in a vblank delay in the same manner as the numerous other latch delays. The demo game draws the screen in a unique fashion using vblank and needed this delay to match up the drawing edges. With this inclusion the game draws perfectly, so it looks like 4A50 and the associated bugs are now resolved.
Editor, Experienced Forum User, Published Author, Expert player
(3536)
Joined: 11/30/2014
Posts: 2733
Location: US
Spikestuff wrote:
Also, isn't this topic actually suited for Encoders' corner instead of Sites?
Well my intent was to include updating movie files too, not just encodes. A proper encode of Superman for example would need the Bkm replaced by a bk2 from the next BizHawk release, the run itself being optimal but the emulator used being outdated.
Editor, Experienced Forum User, Published Author, Expert player
(3536)
Joined: 11/30/2014
Posts: 2733
Location: US
Really impressive glitch research. I like seeing how complex games always seem to get broken down bit by bit over time with ever more impressive and elaborate glitches and discoveries. The overlay and audio commentary really helped, and for me at least the run would have been kind of unwatchable without it since I'm not really familiar with the game, but it was fun being able to follow along.
I agree with calls for Stars and newcomer recommneded (only with the commentary and overlay though)
Editor, Experienced Forum User, Published Author, Expert player
(3536)
Joined: 11/30/2014
Posts: 2733
Location: US
I do see the bug in the health bars now though, thanks for pointing it out! Looks like a bug in playfield priority setting, should be pretty simple to fix, I'm surprised it didn't come up before.
EDIT: oh yeah, I had trouble with that file too. mF8_sega.cs was included in the merge, and it shows up properly in the the folders, but for some reason isn't recognized by the project. I don't understand why though.
EDIT:
So it turns out this is a more complicated issue then I originally suspected. What the game is doing is setting playfield priority like this:
A5 04 LDA CXM0FB
85 0A STA CTRLPF
So it's loading A from the collision register that keeps track of collisions between missile 0 and playfield/Ball. Then it uses this to set playfield priority at 0A.
The problem is that the playfield priority bit is bit 2 of CTRLPF, but CXM0FB only uses bits 6 and 7. CXM0FB is a TIA read register, and as far as I know the CPU cannot write to the 2 bit of it (and anyway this would be a write to NUSIZ0 at 04 in the TIA write registers.)
This is all very confusing. I don't see how bit 2 of CXM0FB could be 1, as needed to change the playfield priority in this way, unless there is something unmodelled about the physical arrangement of these registers. Still researching an explanation.
EDIT AGAIN
Ok, apparently this is a well known issue of deciding what to do when there are unused pins on the data bus. Stella's solution is to simply use the last value that was there for each unused pin. It also however gives the option to randomly drive the pins.
Current AtariHawk code implicitly sets the pins to 0, since it's just returning a new byte with the appropriate collision bits set.
Apparently the unused ones are 'usually' in there previous state enough to make the Stella approach more desirable.
A bit sloppy to program something this way, but I'll try to account for it, since there are examples on real hardware that correctly display the health bar.
Ok, so on the left we see the old glitch version where the health bars show through the background. On the right the fixed version where they dont.
This is really just a programming error on the part of the game developer. Certainly he meant to write LDA #04, but forgot the '#' and simply wrote LDA 04 in whatever compiler he was using, a simple and common error. #04 is what is needed to change the playfield priority in the usual way. It is a stroke of luck that Loading CXM0FB doesn't change the bus state (04 being the last fetch and that register only effecting bits 6 and 7.) Any other situation would have made the error obvious.
Ultimately I simply added a bus state variable that is tracked along with everyhting else to fix this. Were it not for such an obscure case I surely would never have even thought of it. Maybe future work would be to make the bus state part of the CPU, but maybe not I'm not sure how practical / useful that would be.
Anyway that was an interesting excercise in bug hunting.
Editor, Experienced Forum User, Published Author, Expert player
(3536)
Joined: 11/30/2014
Posts: 2733
Location: US
^that error with samples should have also been fixed with the last pull request, I even downloaded and compiled from master just to make sure, did you get that error message from a build after the pull request was merged hegyak?
Curiously this game runs on FCEUX but went to black screen on previous versions of BizHawk.
If anyone knows other games which are just plain incompatible please let me know (I am aware of Time Lord currently) having games to test on really helps.
EDIT: oh, the recent fixes to OAM reads also fixes cpu_dummy_writes/cpu_dummy_writes_oam, so, scratch another one off the list!
I'm also making progress on sprdma_and_dmc_dma, I'm hoping maybe by the end of the month those tests will pass.
I have been (very) slowly working on NESHawk PPU. So far I have rewrote sprite evaluation to take place simultaneously with background generation, as it is with a real NES. Together with proper read behaviour of 2004 during this time, I finally managed to fix those annoying horizontal lines and shaking in Micro Machines.
This is still very early WIP work and generally breaks other things, but the timing is correct so i should be able to start slowly fixing other timing issues and bugs and such.
Right now it also slows down emulation by a noticable factor as well. I don't think my code takes considerably longer to run then the original, so this is the first major issue to be worked out.
Of course just getting it loaded up is only a small part of the battle, as there is currently no way to send it any meaningful input, but it's a start!
The most important things that came out of this is realizing that console startup needs to have improved accuracy for the compumate to work. The 'game' does not set the stack pointer appropriately before doing basic operations (as most games typicaly do) instead relying on the 6532 start up sequence. This sequence sends a reset signal to the 6507 which sets the stack pointer to FD.
The current code does not utilize this start up sequence instead just setting everything to default, which for the 6507 is stack pointer 0, which unfortunately overlaps TIA addresses, corrupting the return address of a JSR.
Anyway, I hardcoded in an initial stack pointer of FD simulating post reset behaviour, but I am ultimately not very satisfied with this approach and will probably try to use reset correctly, but right now I don't know how this will work with the current machinery.
It turns out there was a rather serious bug in AtariHawk itself where PeekMemory actually pointed to ReadMemory in the system. This caused chaos when I tried to trace log out what was going, as the tracelogger peeks changed banks too when it shouldn't have. Once that got fixed it was only a matter of tracking down where banks weren't being changed properly.
The ultimate problem was just an incorrectly placed closing bracket. :(
Anyway this is still displaying things a bit Different then Stella, so I'll have to see why, but the test program looks perfect and anyway this is the only example game I could find.
EDIT: turns out I just needed to add in a vblank delay in the same manner as the numerous other latch delays. The demo game draws the screen in a unique fashion using vblank and needed this delay to match up the drawing edges. With this inclusion the game draws perfectly, so it looks like 4A50 and the associated bugs are now resolved.
Ok, so on the left we see the old glitch version where the health bars show through the background. On the right the fixed version where they dont.
This is really just a programming error on the part of the game developer. Certainly he meant to write LDA #04, but forgot the '#' and simply wrote LDA 04 in whatever compiler he was using, a simple and common error. #04 is what is needed to change the playfield priority in the usual way. It is a stroke of luck that Loading CXM0FB doesn't change the bus state (04 being the last fetch and that register only effecting bits 6 and 7.) Any other situation would have made the error obvious.
Ultimately I simply added a bus state variable that is tracked along with everyhting else to fix this. Were it not for such an obscure case I surely would never have even thought of it. Maybe future work would be to make the bus state part of the CPU, but maybe not I'm not sure how practical / useful that would be.
Anyway that was an interesting excercise in bug hunting.