Post subject: New Core Architecture Roadmap
Alyosha
He/Him
Editor, Expert player (3522)
Joined: 11/30/2014
Posts: 2728
Location: US
As I develop more cores for BizHawk, it's increasingly clear where the line is between what can be done in native C# and what can't. And as developing cores for 30+ year old systems has gotten kind of dull for me, I wanted to take some time to think about how I can contribute to more modern systems and where actual gaps are in Bizhawk that need filling. My long term goal is to build an N64 core in C++, but it's too complicated to just drop everything and start doing. I need to get an idea on a simpler system of how to best organize a core in a C++ framework that accounts for both accuracy and performance. The current in-house core architecture for C# is very simple and very robust, I can (and have) added a blank core template in a matter of hours. But, it just won't work for modern systems with higher clock speeds. I need a new approach, but I also want something that's not just a one-off, a new architecture that's re-usable would be ideal. And since I learn nothing from looking at other people's code, I really need to dive in myself. Looking at the current landscape in BizHawk, it seems like the 2 most important cases to consider would be IBM PC and Amiga. IBM would probably be simpler at first, but the IBM landscape is so broad this might be a very deep rabbit hole. It might get the most use though. Amiga would have the advantage of needing an in-house 68000 cpu which could be re-used on a lot of things. But I'm not sure how much those things would actually be used (a lot of people still just use gens even though we have gensplusgx for example.) Also instruction decoding for 68000 is a nightmare, but once that part is done it wouldn't be so bad. I'm open to suggestions here though. Is there a system that has a currently ported core that would be better off with something in-house that has more control? Is there a system I'm overlooking that is more modern then old 8-bit stuff that someone would definitely use? What I don't want though is something like SNES where infinite effort would be expended in PPU details that have nothing to do with overall architecture. Basically I want a roadmap for myself like: Easy system, work out a modern C++ architecture and play around with optimization | | v Medium System, more modern CPU architecture that requires careful planning and optimization. Pipelines, cache misses, etc. but overall system still tractable. | | v N64 If anyone has any input here I would really like to hear it. Thanks!
Sonia
She/Her
Joined: 12/6/2013
Posts: 435
Location: Brazil
Here are the systems currently missing in Bizhawk that I'm looking forward to the most: Arcade Already being worked on by feos. NDS Already being worked on by SuuperW. MSX OpenMSX/BlueMSX could be ported over, but since you want to develop something in-house I guess that's out of question. I remember you once said all the components for a MSX core are already there, and it's just a matter of stitching them together. So maybe this one wouldn't be super hard? NEC PC-9801 Japanese personal computer. Not that many TAS-suitable games, but the ones that do exist are good. Sharp X68000 Another JP personal computer. Plenty of action stuff that's TAS-suitable. A lot of them are Arcade ports, but there are some originals too. That's it. As a side note, I know this has already been said many times, but if you can reserve a bit of time to improve PCEHawk and bring it closer to Mednafen levels of accuracy, it'd be much appreciated. That core is in pretty bad shape.
Editor, Experienced player (817)
Joined: 5/2/2015
Posts: 671
Location: France
+1 for X68000 and PC-98, these are systems that I would definitely use.
Site Admin, Skilled player (1236)
Joined: 4/17/2010
Posts: 11268
Location: RU
Your best bet with IMB PC is just porting PCem (or 86Box). But those have no savestates, so they'd have to be waterboxed. But waterbox build environment has been recently posted and tested, so it's at least now possible to use it again. Cool thing about PCem is that it can run DOS, Windows, Linux, Adobe Flashplayer, JRE8 applications, and whatnot. A pure monster. For Amiga there's a full-blown thing called FS-UAE that's being rapidly developed and kept up-to-date with WinUAE, but has an advantage of being cross platform and hosted by a more friendly dev. And it has savestates. It'd be an interesting task to port given the infrastructure, but probably still easier than PCem. Developing emulation for these from scratch looks like a waste of time to me if instead existing emulators could be improved and reused.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
EZGames69
He/They
Publisher, Reviewer, Expert player (3967)
Joined: 5/29/2017
Posts: 2707
Location: Michigan
Some in the discord server agree that they want PSP support.
[14:15] <feos> WinDOES what DOSn't 12:33:44 PM <Mothrayas> "I got an oof with my game!" Mothrayas Today at 12:22: <Colin> thank you for supporting noble causes such as my feet MemoryTAS Today at 11:55 AM: you wouldn't know beauty if it slapped you in the face with a giant fish [Today at 4:51 PM] Mothrayas: although if you like your own tweets that's the online equivalent of sniffing your own farts and probably tells a lot about you as a person MemoryTAS Today at 7:01 PM: But I exert big staff energy honestly lol Samsara Today at 1:20 PM: wouldn't ACE in a real life TAS just stand for Actually Cease Existing
Alyosha
He/Him
Editor, Expert player (3522)
Joined: 11/30/2014
Posts: 2728
Location: US
Sonia wrote:
Here are the systems currently missing in Bizhawk that I'm looking forward to the most: ... MSX OpenMSX/BlueMSX could be ported over, but since you want to develop something in-house I guess that's out of question. I remember you once said all the components for a MSX core are already there, and it's just a matter of stitching them together. So maybe this one wouldn't be super hard? ... That's it. As a side note, I know this has already been said many times, but if you can reserve a bit of time to improve PCEHawk and bring it closer to Mednafen levels of accuracy, it'd be much appreciated. That core is in pretty bad shape.
xy2_ wrote:
+1 for X68000 and PC-98, these are systems that I would definitely use.
MSX is an excellent suggestion that I completely overlooked. That does sound like a good place to start, I think I'll go for it. Thanks! X68000 might be a good follow on, not sure yet. As for PCE, I just don't have any interest in it. It would be great if someone who was actually interested in the system would try to fix the bugs, it really needs someone who can focus some time on it.
feos wrote:
Developing emulation for these from scratch looks like a waste of time to me if instead existing emulators could be improved and reused.
I wasn't aware that PCem and UAE were so highly developed. The gap between existing and actually being ported is quite large though, but I guess I'll just leave them for someone else.
EZGames69 wrote:
Some in the discord server agree that they want PSP support.
PSP would itself be of the same difficulty as N64, I don't see it happening, at least not form me.
Joined: 9/12/2014
Posts: 536
Location: Waterford, MI
Do you meaning making a PSP core from scratch or using ppsspp? It would make a lot more sense to implement ppsspp into bizhawk rather than from scratch. It has save states, video dumping, and frame advance already included.
Experienced player (630)
Joined: 11/23/2013
Posts: 2208
Location: Guatemala
I would like to see a new Genesis core that could hit cycle accuarcy, like a BlastEm port or something. That's probably going to be difficult to do though.
Here, my YouTube channel: http://www.youtube.com/user/dekutony
Site Admin, Skilled player (1236)
Joined: 4/17/2010
Posts: 11268
Location: RU
dekutony wrote:
I would like to see a new Genesis core that could hit cycle accuarcy, like a BlastEm port or something. That's probably going to be difficult to do though.
You don't like the current Genesis core in bizhawk at all?
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
CoolHandMike
He/Him
Editor, Reviewer, Experienced player (636)
Joined: 3/9/2019
Posts: 582
For PC98 I could translate technical documents and provide some overall support if someone wants to work on those. Installed and tested several thousand disks/hdds for the system couple years ago. PC98 has a lot of unique games and in fact has a really massive library. Some of them are pretty high quality. However similar to Windows there are lots of garbage. PC98 https://www.youtube.com/watch?v=zMNYFBFWHEc Night Slave (mech) https://www.youtube.com/watch?v=v5CDFsdLFO8 Rusty (castlevania like) https://www.youtube.com/watch?v=WhXxU-AB7V8 Totsugeki! Mix https://www.youtube.com/watch?v=XjPDMOgIDb4 Tuned Heart https://www.youtube.com/watch?v=JOEmjTGk8Zw Usagi na Panic https://www.youtube.com/watch?v=KsMk8emA8og Farland Story 2 (rpg) https://www.youtube.com/watch?v=3QBaYmg5cwE Brandish 2 https://www.youtube.com/watch?v=NQdmwjFVHuI Super Dimension Fortress Macross : Skull Leader Complete Pack ------------ X68000 would be awesome to include. Extremely high quality system with almost every game being good to high quality. However the system does have many arcade ports and you might as well play those on the actual arcade emulator. Look for these unique platformers https://www.youtube.com/watch?v=m-P2DYpOMos Aquales (one of the best) https://www.youtube.com/watch?v=MwkKuolGG7Q Akumajou Dracula - advanced version of the original Famicon game https://www.youtube.com/watch?v=bT2gMwbkJao&t=1009s BUSTER https://www.youtube.com/watch?v=3dHoO6UQo2Q Burusu no Fukushuu Etolile Princesse Flappy 2 Gardis Light Genocide 2 Keeper MadStalker Silkroad Legend of Gero Silkroad 2 https://www.youtube.com/watch?v=DoXkhjwClhU Zugya - might not be too tas suitable but I liked it a lot
discord: CoolHandMike#0352
Joined: 8/27/2018
Posts: 5
Personally, I think a working TAS-capable Amiga emulator would be amazing; the workflow in recording games for the system is so long-winded without re-recording. Toni Wilen (long-time author and maintainer of WinUAE) did implement re-recording capability into one of the 2.x releases, but the feature never truly worked, and was only "supported" using a very strict base A500 configuration. I believe he cited problems maintaining cycle-exact timings, which proved problematic due to the custom chipsets and CPU configs, plus it wasn't something he considered a priority to develop anyway. You have multiple possible CPU and memory configurations, OCS/ECS and AGA chipsets and a whole bunch of other stuff as well.
Alyosha
He/Him
Editor, Expert player (3522)
Joined: 11/30/2014
Posts: 2728
Location: US
dekutony wrote:
I would like to see a new Genesis core that could hit cycle accuarcy, like a BlastEm port or something. That's probably going to be difficult to do though.
What utility would you get out of this? Is there something you see that is holding back gensplusgx? I honestly don't see what is holding back that core. Realistically I would probably need to make a basic genesis core anyway in order to test a MC68000 if I go that route, just so I can compare trace logs. Otherwise I would have to blind debug a new CPU core on a new system in an unfamiliar environment and that doesn't sound like a fun time for me. It wouldn't be a production core though. If someone out there wanted to make a 32X core I would commit to making a cycle accurate genesis core with a clean interface to build off of, but I would need a solid commitment.
Al82 wrote:
Personally, I think a working TAS-capable Amiga emulator would be amazing; the workflow in recording games for the system is so long-winded without re-recording.
Even if I did make an Amiga core it would only be a baseline implementation (with easy expandability baked in.) It's just a stepping stone for me, but maybe having a baseline model would inspire someone else to jump in and try expanding it. And I don't mind duplicating effort if necessary, >99% of what I've done in BizHawk was already done somewhere else at least once. For now I'll press ahead with MSX. I've almost got the Z80A core ported to C++.
EZGames69
He/They
Publisher, Reviewer, Expert player (3967)
Joined: 5/29/2017
Posts: 2707
Location: Michigan
Can we also look into porting Mednafen’s PCE core into bizhawk to replace PCEHawk?
[14:15] <feos> WinDOES what DOSn't 12:33:44 PM <Mothrayas> "I got an oof with my game!" Mothrayas Today at 12:22: <Colin> thank you for supporting noble causes such as my feet MemoryTAS Today at 11:55 AM: you wouldn't know beauty if it slapped you in the face with a giant fish [Today at 4:51 PM] Mothrayas: although if you like your own tweets that's the online equivalent of sniffing your own farts and probably tells a lot about you as a person MemoryTAS Today at 7:01 PM: But I exert big staff energy honestly lol Samsara Today at 1:20 PM: wouldn't ACE in a real life TAS just stand for Actually Cease Existing
Alyosha
He/Him
Editor, Expert player (3522)
Joined: 11/30/2014
Posts: 2728
Location: US
EZGames69 wrote:
Can we also look into porting Mednafen’s PCE core into bizhawk to replace PCEHawk?
Not me, maybe feos or SuuperW once they are done now that waterbox can be built. Honestly though, starting from zero, you or anyone other interested person is only like 2 months of relatively simple study away from being 100% capable of fixing the existing core. It's really really not that hard. I would strongly encourage you or any other interested PCE folks to give it a try, you may find it rewarding to get a system working that you are very interested in.
Sonia
She/Her
Joined: 12/6/2013
Posts: 435
Location: Brazil
Alyosha wrote:
Honestly though, starting from zero, you or anyone other interested person is only like 2 months of relatively simple study away from being 100% capable of fixing the existing core. It's really really not that hard. I would strongly encourage you or any other interested PCE folks to give it a try, you may find it rewarding to get a system working that you are very interested in.
Can you give us some links containing the resources/materials necessary for studying then? Because for someone starting from zero like me, I'd appreciate some directions at least. Just information that'd be relevant to fixing the core is enough.
Joined: 9/12/2014
Posts: 536
Location: Waterford, MI
Maybe we should have a vote on desired core? Like poll results? I would really love to see PCem ported to bizhawk. With waterbox there shouldnt be any desync issues. If PPSSPP is easier to integrate, I would vote on that instead.
Alyosha
He/Him
Editor, Expert player (3522)
Joined: 11/30/2014
Posts: 2728
Location: US
InfamousKnight wrote:
Maybe we should have a vote on desired core? Like poll results? I would really love to see PCem ported to bizhawk. With waterbox there shouldnt be any desync issues. If PPSSPP is easier to integrate, I would vote on that instead.
Well I'll be on MSX for at least a month or 2, so there's no rush here. And also it's not really a vote, people just work on things that interest them and seem accomplishable with the amount of time they have.
Sonia wrote:
Can you give us some links containing the resources/materials necessary for studying then? Because for someone starting from zero like me, I'd appreciate some directions at least. Just information that'd be relevant to fixing the core is enough.
Sure! If you are totally new to programming in general, the best place to start is C++ tutorials here: http://www.cplusplus.com/doc/tutorial/ In terms of program complexity, emulator cores are VERY simple. You don't really need to know anything past the Class tutorials there to fix bugs. C++ is a bit different from C#, but aside from syntax things are mostly the same for basic stuff. Once you've gotten comfortable with basic programs, clone BizHawk and start Console.WriteLine()'ing around the PCE core to get an idea of the flow of the system. Experimenting is the best way to learn. For PCE information, the best readily available resource is here: http://www.archaicpixels.com/ It's not the best resource I've ever seen, but it's pretty tractable. Don't get overwhelmed in the details, try to understand this part first: http://www.archaicpixels.com/Memory_Map Staring at the emulator as a whole won't really help you learn too much though. I recommend looking at a simple, concrete problem and seeing if you can track down what's going wrong. This is a good place to start: https://github.com/TASVideos/BizHawk/issues/1580 Splatterhouse is just an ordinary PCE title and doesn't use complicated CD stuff or supergrafx. I am pretty confident you should be able to figure this one out. Alternatively, you can look into this: https://github.com/TASVideos/BizHawk/issues/1231 Crashes are pretty simple to spot in a trace log, should be fairly easy to analyze. Good Luck, you can do it!
Editor
Joined: 6/21/2014
Posts: 17
Location: Bavaria
I was the person who comissioned the Re-Recording feature of WinUAE a few years ago by donating all my Action Replay modules to Toni. Unfortunately as mentioned, it never really worked reliably and I only ever managed to record one game with it. Unfortunately Toni wasnt too interested in that feature. However with Waterbox this shouldnt be a problem anymore now :) I might have someone at hand who could potentially port it :)
Alyosha
He/Him
Editor, Expert player (3522)
Joined: 11/30/2014
Posts: 2728
Location: US
I'm making progress on a new C++ core. I made a game gear version as a proof of concept since it has mostly the same parts as MSX. It was more of struggle to convert things to work in C++ then I thought. Actually most of the problem was due to needing to use a C interface in the DLL. Things would be so much easier if there was a standard C++ interface, but oh well. This was a very worth while exercise because I was able to sort out most of the bugs just by comparison with the existing C# core. As you can see in the screenshot, this core runs around 490 fps on my laptop. This is actually not very good, since this core in C# runs at 310 fps. This is no where near the performance I need, it should be up around 800 like gambatte. So I obviously have a long way to go in terms of optimization. The whole point of going to C++ though is that there are many possibilities to achieve those optimizations, so I'll be starting down that road once all other aspects of the core are done. On the up side, starting from scratch has let me organize the core in very straightforward way that should hopefully be largely reusable as I build more systems. Also some of the more complicated things like memory domains and trace logger are done. The two major thins I have left to add are sound and savestates, but neither of those should be too tough. After that I'll probably rework the CPU core to take advantage of C++ features. The Z80A is really slow, partly because I needed to capture all the signals for spectrum and CPC, but also just because it's poorly optimized. That was partly by design since I wanted maximum clarity in order to easily fix bugs in development, but by now it's quite well verified and I can sacrifice some clarity on the C++ side to go fast.
Alyosha
He/Him
Editor, Expert player (3522)
Joined: 11/30/2014
Posts: 2728
Location: US
I'm making steady progress on the new core. Input, sound, and savestates now work. If this were a production core it would basically be done. Now it's time to optimize. I've already played around with a few things, and was happy to see that things I thought might improve performance actually did. This is very welcome considering how much I banged my head against the wall with C# trying to optimize things. Once I'm comfortable with the level of optimization and can verify that nothing gets broken along the way, I can start the process of turning this into an MSX core. That part shouldn't be too hard hopefully. Having all of this infrastructure and the z80 in place is already at least 80% of the work. For follow on work I'm leaning heavily toward an MC68000 based system. I think this will offer the most utility in the long run just having the cpu available. But I'm still not 100% sure yet.
Fortranm
He/Him
Editor, Experienced player (775)
Joined: 10/19/2013
Posts: 1115
Do you plan to turn this into new cores for SMS and GG as well?
Sonia
She/Her
Joined: 12/6/2013
Posts: 435
Location: Brazil
Alyosha wrote:
For follow on work I'm leaning heavily toward an MC68000 based system. I think this will offer the most utility in the long run just having the cpu available. But I'm still not 100% sure yet.
Well, X68k uses MC68000, so that could be a logical follow up. btw, sorry for the late answer but thanks for all those links. They will certainly prove useful in the future. :)
Alyosha
He/Him
Editor, Expert player (3522)
Joined: 11/30/2014
Posts: 2728
Location: US
Fortranm wrote:
Do you plan to turn this into new cores for SMS and GG as well?
Nope, I don't really see that as being of high utility. After the first big round of optimization in the CPU I'm at about 550 fps. Started from about 495 so not bad, but a long way to go. This came at a fairly substantial cost of clarity in being able to follow the code step by step, as I basically pre-compiled the instruction vectors at the start. This cleans up the execution loop but also adds some complications in some execution paths. Regardless though this is necessary for any more modern core so it was good practice.
Alyosha
He/Him
Editor, Expert player (3522)
Joined: 11/30/2014
Posts: 2728
Location: US
I've converted over to an actual MSX machine now and am getting the bugs worked out. It shouldn't take too much longer to have a reasonably complete MSX1 machine ready for use. I'm not sure how far I'll go into the MSX line up, maybe just the 2.
Alyosha
He/Him
Editor, Expert player (3522)
Joined: 11/30/2014
Posts: 2728
Location: US
I'm able to get in game now on MSX1 with joystick control. There are some audio bugs left to work out and I haven't thoroughly tested savestates yet, but at least for MSX1 this is almost in a TASable state. I could use some help getting a list together of good hashes (SHA1) for BIOS files though, if anyone more familiar with the system knows what the standards are.