Post subject: Dolphin memory engine, a new RAM search made for Dolphin
aldelaro5
He/Him
Joined: 1/8/2014
Posts: 29
Location: Canada, Quebec
Hi everyone, I am pleased to officially announce the first BETA release of a program I have been working for a month and a half:
Dolphin memory engine
Yes, finally, it happened, a RAM search made to track, monitor or edit the Dolphin emulated memory in real time, but this time, it solves a lot of problem that people had with the existing solution which was Cheat Engine, hence the name. To be clear however, this is an EXTERNAL RAM search, it's not integrated within Dolphin because altough doing that would make a lot more things work, I was extremely worried about performance concerns and how....actually weird it would be to even submit it for merge because the reality is, I don't think this is something that would be supported easilly. However, the result isn't far off from what would happen if I went with internal so I kept going. Now, before I explain the advantages of this over CE and its potential, let me give you the important links. To get the sources: https://github.com/aldelaro5/Dolphin-memory-engine To get binaries (the executables without having you to build): Click the link above, but go to the release tab, check the readme first though. This is MIT licensed so feel free to do whatever with these :) With that out of the way, let me describe why this RAM search would replace the use of Cheat Engine in the near future (assuming it supports similar features and everything becuase this is BETA for now). Dolphin communication You should never have to worry about the infamous start address of Dolphin anymore, even after 5.0-3981 which introduced randomnisation of the address, this RAM search will read the mapping information of Dolphin and since we know their mapping (Dolphin is open source after all), it can detect the emulated region and automatically get the start. This is all done under the hood, you NEVER have to worry about this anymore because the program will take this start into consideration when reading or writting memory to Dolphin. This is much easier than CE where you had to not only know it, but also risk having to invalidate your entire table JUST because Dolphin decided to initialise somewhere else. You could use a pointer, but then it would change between revisions and you needed to calculate an offset everytime you add an address, basically, it was horribly inconvenient. It's really simple with this program, you click hook and if Dolphin was running AND had an emulation started.....it's ready to work. Endianness You might have learned this when setting up CE, Dolphin emulates 2 systems that both uses big endian memory but our PC uses little endian memory (at least if it's a x86 or x64 based system). Because of this, you had to add extensions to CE to have the types works. It works fine, it's just a bit annoying to do. This RAM search however, not once you have to worry about it, everything read is converted to little endian so your PC understands and everything written is converted to big endian so Dolphin receives the right memory. MEM2 + MEM1 See the MEM2 text on the top of the Window in the screenshot above? MEM2 is an extra memory region of 64MB that the Wii has on top of MEM1 which is 24MB that both the GameCube and WII uses. Scanning for GameCube games means that you need to restrict to only one range of memory, something CE can already do, you have to set it manually, but you can do it. For Wii games however, you need to scan into 2 separate ranges (the memory in between is invalid), something CE CANNOT do. To solve this problem, this new RAM search allows you to enable it or disable it as well as detect it (the first letter of the game ID tells you the console). Detection works for most of the games, but not for some rare ones so there's a way to manually toggle it as well. What it does is it restrict the range of the scanner accordingly as well as restrict what addresses is considered valid for adding a watch. Basically, you can easilly scan Wii games with its extra region without changing anything for GameCube games except just disabling MEM2 which is much better than having to change the ranges in CE. And finally, the most dreaded problem has finally been solved and it is.... Multi level pointer supports (aka tracking dynamic memory)
Yes, finally, it's possible to track dynamic memory with this new RAM search. The functionallity is exactly like CE, but for Dolphin. To show what I mean, here's a screen shot:
You enter the address of the pointer and its different offsets. By the way, the level represent how deep the pointer is, a level 1 pointer is a simple pointer with an offset, a level 2 pointer is a pointer of a pointer and etc... You could go infinetely, but the chances that you need something crazy like a level 10 pointer are very low. What this does is everytime it reads from it, it will follow the entire pointer path using the standard reads function it does and THEN it will read the memory. Same for writting but in reverse so basically, you have an unbreakable link to the pointer, as long as this pointer is valid (if not, it gives up and show ???). This is huge, this was impossible to do in CE and a lot of the time, I met people who were very disappointed and frustrated to learn that you cannot track dynamic memory, finally at last you can. However.....there's 2 cons with this. 1: it doesn't work for all the games because I am making a technically wrong assumption on the mapping of the game, 75% of the games maps them in a similar manner than how they are layered out phisically, but 25% of them uses custom mapping that unfortunately, I can't deal with. Still, this is in my opinion A LOT better than 0%. 2: It's.....a bit more involved to actually find a path. It's similar from CE, the issue is more that you cannot use ONLY my RAM search, you have to use the dolphin debugger and actually read some PowerPC assembly to figure out the path. However....this is actually not too bad, in fact, here's a quick tutorial. How to find a pointer path using Dolphin's debugger First, read a bit on my thread about using Dolphin's debugger: http://tasvideos.org/forum/viewtopic.php?p=443246#443246 you don't have to read everything, but if you want a short version for this use case, let me tell you: Use AT LEAST Dolphin 5.0-3023, but preferably, get the lattest dev build on their website. To use the debugger, either use a terminal (bash or cmd) and start dolphin with the -d argument or create a shorcut->properties and change the target to have the -d parameter. Then read the section about breakpoints, register view and code view of the thread I linked, you need at least to know how to use these and it's simply too long to describe in this post. After that, add a memory breakpoint to the address you want, either read or write is fine then let the game actually read or write to it and the game should pause itself.
Here, the game paused after I added a breakpoint to the HP address as I just took damage. The interesting part is the instruction it broke, in this shot, it broke with the following instruction: sth r0, 0x0120 (r30) st means store (write), l means load (read) and whatever follows tells you the type (h means halfword or 2 bytes, b means a single byte, w means word so 4 bytes and f means float). If you see letters after that, it's just some parameters that you usually don't care about, but you can google it with the keyword power pc instruction, you should get something. The only other letter you might care is i meaning immediate aka, it loads / writes a constant. The first parameter of all of these is where it reads into / where it writes from so you usually don't necessarly care about it. What you are most interested is the second and third parameter, the third is what base address it loads from / write to while the second is the offset to apply to said address (so if the address is 0x80000000 and the offset is 0x10, it means it's going to laod from / write to 0x80000010). Sometimes, direct numbers will be displayed, but other times, you will see registers like here, we see r30 meaning whatever is curently in r30, to know this, refer to the register view. Here. we see that the game stores 2 bytes located in the address of r0 to the address in r3 + 0x120. It's important you get how to parse these in your head. The next step is to actually scan for a 4 byte hex number that contains that base address, we already have an offset so just take note of it. r30 in this shot was 0x812f72c4 so whatever address the pointer is, its value is 0x812f72c4, this is something the RAM search can query easilly. However, one scan is likely not enough so what you have to do is make the pointer change path (so in this example, laoding another level is enough), find the new address and repeat the memory breakpoint process, this time, you will get a different address, scan for that in the scanner. Narrow down the address this way. There is a chance that the result will be your base address so enter THAT new address in the watch, check "This is a pointer" and enter the first offset you got before, if it was a simple level 1 pointer, congratulation, you found your first pointer which will not break whenever the location change. If however it still changes, you simply redo the process, but with the newer address (so check what reads or writes to the base address you got in the scanner blah blah blah). You may have to do this a couple of times depending on how deep the pointer is, but the good news is usually, pointers path will be similar for the same game, only some offsets may vary. It is an involved, but necessary process to get the path you want, in fact, what CE does for PC games is it handle this automatically, but this isn't something very feasible with an external RAM search and Dolphin provides the tools to allow you to do the exact same process anyway. So if you REALLY want to start to find pointers, I suggest you take some time to learn this process, it's going to be very usefull. Other things I need to repeat this, but this is a BETA release, not everything that I plan to do is there and not everythign works perfectly (mainly about the GUI, I know, it is clunky at times) and it may have some bugs, but the idea is consider this a preview. It works, you could in theory start to use it over CE right away, but just know, it's not STABLE. It does have file saving support however, use the file menu for that, your watch list is going to be saved with a .dmw extension (Dolphin memory watches), but it's just JSON, you can edit it manually using a text editor. I did tested it quite a lot during developpment so I can at least say it should work most of the time. If somethign wrong happens, feel free to talk about it in this thread or private message me (or even tag me on irc/discord, I am always there and I check my stuff), or post an issue in the issues tab of the Github page, I accept bug reports and features request there. The result count will differ from CE for mainly 2 reaosns: 1: I do not use alignement because it's not guaranteed that the GameCube and Wii have aligned address, they support unaligned ones. I might add the option anyway, but for now, I just want to see the impact of disabling it. CE by default enalbles it, but you can disable it. 2: if you are scanning float with the filter bigger than or smaller than, it will give slightly more results than CE, this is because CE scanner has a minor bug where it discards infinite numbers which is technically wrong, you can compare these. Keep this in mind if you plan to compare. Oh and for performance wise, it's similar, however, you might want to watch RAM usage, it can get up to something like 250MB - 300MB if you scan a wii game with MEM2 enabled, this is normal, but as soon as you are done with the scan, click reset, it will free the memory. The usage is 2-3 times lower for MEM1 disabled scan and it's almost nothing without any scan (maybe like max 20MB). To put groups or watches into groups, just drag them into the group. To delete, use the delete key. Rest should be self explanatory like double click to edit. Finally, the signed value scan allows you to take the sign into consideration so that -1 is lower than 1 for example. Only check this if you care about negative values and scan for integers (floating point types always take care of the sign). EDIT: As of version 0.2, it now has a memory viewer, here's a screenshot of it:
Now, time for some FAQ (or at least likely asked question lol) Do you plan to have a memory viewer? Yes, but because of the research I need to do on how to do this and the likely very time consuming process, I decided this will wait after the first beta release. However, rest assured, it is DEFINTELY comming and I am not doing any stable release without one, it's too essential. EDIT: the viewer is DONE, check version 0.2 :) Do you plan to add X? The letter X is likely there alrready......okay serisouly, I am going to make a roadmap soon in the repository, just check a bit after this post and it should answer that. EDIT: I did a roadmap here: https://github.com/aldelaro5/Dolphin-memory-engine/blob/master/Roadmap.md Because of this, I now accept features request using the issues system of the repos, go to the issue tab if you have one, just please, read the roadmap before. How do I build this if I want to submit a pull request? The build instruction are already in the README file in the repository I do welcome pull request, however, right now, since I have so much to do before considering suggestions, they might be delayed / would be easier that I implement it myself if I already planned your idea. Will this work for [insert Dolphin version here]? It normally SHOULD, even those before the ASLR update, you will just notice the start address the program reports would be similar. I haven't tested older versions though so your mileage may vary (if it cannot hook still, it's a bug, please share it with me). Will this work for [insert game name here]? Except if you are tracking pointers and your game uses custom mapping, yeah it totally should. I got unhooked and I didn't press the unhook button, what's going on? Depending on what you did, this may or may not be normal. If Dolphin crash / you killed Dolphin / you stopped an emulation, then yeah it's normal because the emulated RAM when active should ALWAYS be readable and writable so once an error is detected, it will assume Dolphin no longer has its RAM initialised. This only disables the UI so you don't loose your work and you can still save in the meantime. However, if it unhooks on its own while Dolphin is clearly running with a game booted or you click hook and it unhook right after, this is a bug, it should not happen and I would have to investigate what happened with you. I am concerned that the start address might be wrong, what can I do to confirm it? You need at least Dolphin 5.0-765 for this, but start Dolphin with the debugger (refer to my quick pointer tutorial above, but you can pass -l alternatively for JUST the logger). Show the logger and the log configuration tab by going into the view menu. In the log configuration tab, put the log level to info, toggle all logs to off and only check the MI & Memmap one. Clear the log on the log tab and simply boot any game. A line like this should appear: Core/HW/Memmap.cpp:221 I[MI]: Memory system initialized. RAM at 0x7eff68000000 There you go, simply compare this address with the one the RAM search reports you. This should be it. With that ENJOY, I hope you will like this new RAM search and get the potential of it, this project just started and I am already VERY hype myself :D If you hasve any bug reports or issues requests, I welcome issues posted on the Github page, go to the issues tab to post one, I do think you need a Github account however.
Even if it's a sequel, lots of people have to give their all to make a game, but some people think the sequel process happens naturally." - Masahiro Sakurai
Experienced player (703)
Joined: 2/5/2011
Posts: 1417
Location: France
Let's say it again: Thanks for it aldelaro! Makes me really wanna redo a Rayman 3 TAS now that CE hassmle is gone :D
Current: Rayman 3 maybe? idk xD Paused: N64 Rayman 2 (with Funnyhair) GBA SMA 4 : E Reader (With TehSeven) TASVideos is like a quicksand, you get in, but you cannot quit the sand
Active player (432)
Joined: 4/21/2004
Posts: 3516
Location: Stockholm, Sweden
You are a true lifesaver! I tried my best pestering various developers for years to add ram watch but you did it :D Congratulations.
Nitrogenesis wrote:
Guys I come from the DidyKnogRacist communite, and you are all wrong, tihs is the run of the mileniun and everyone who says otherwise dosnt know any bater! I found this run vary ease to masturbate too!!!! Don't fuck with me, I know this game so that mean I'm always right!StupedfackincommunityTASVideoz!!!!!!
Arc wrote:
I enjoyed this movie in which hands firmly gripping a shaft lead to balls deep in multiple holes.
natt wrote:
I don't want to get involved in this discussion, but as a point of fact C# is literally the first goddamn thing on that fucking page you linked did you even fucking read it
Cooljay wrote:
Mayor Haggar and Cody are such nice people for the community. Metro City's hospitals reached an all time new record of incoming patients due to their great efforts :P
aldelaro5
He/Him
Joined: 1/8/2014
Posts: 29
Location: Canada, Quebec
AngerFist wrote:
You are a true lifesaver! I tried my best pestering various developers for years to add ram watch but you did it :D Congratulations.
Thanks, but I just need to clarify something. To note here, this is not OFFICIALLY supported by Dolphin, it's a separate project that I entirely maintain. Having an actual integrated RAM within Dolphin as powerful as this one is, yes possible, but has many performances concerns because I need to call Dolphin functions already super congested because the emulatior is already working hard with them to emulate the game. And it's not just a few calls, it's thousands of them per second, this is definitely going to hit performance. What I am thinking of doing is as this external solution works and will probably serve us well in the future, I want to push it as much as I can. If the project bexomes refined enough, then I could at least attempt it because if you check my code, I separated well enough the communication interface and the working logic so integrating with dolphin will mostly just redefine what reads, writes and translating virtual to physical means. That and I can integrate the UI easilly, Dolphin right now is in the process of migrating to Qt which is what I use for UI. I have actually a lot of reasons to attempt ut, but it will only benefit if I can get reasonable performances. Though, it was already linked in some issues demanding stuff like real time update in the watches so they are aware of the tool, I was even told it might be useful to them for debugging. So yeah, maybe in a distant future, integrated ram search will be a thing, but for now, let's enjoy having a good one in the first place :)
Even if it's a sequel, lots of people have to give their all to make a game, but some people think the sequel process happens naturally." - Masahiro Sakurai
Soig
He/Him
Skilled player (1458)
Joined: 12/4/2010
Posts: 252
It helps me a lot! Scanning speed is so fast. And my computer runs it easily.
aldelaro5
He/Him
Joined: 1/8/2014
Posts: 29
Location: Canada, Quebec
Since this release is so big, I think I should reply here should be worth it: https://github.com/aldelaro5/Dolphin-memory-engine/releases/tag/v0.4-beta Version 0.4 is a HUGE release, updating to it is highly recommended. Eseentially, adds a Cheat table importer as well as handles MEM2 automatically, many convenience features and a TON of bugfixes.
Even if it's a sequel, lots of people have to give their all to make a game, but some people think the sequel process happens naturally." - Masahiro Sakurai
Soig
He/Him
Skilled player (1458)
Joined: 12/4/2010
Posts: 252
So if I want to compare with old record, I need to run several dolphins at the same time. But this tool can only hook the first emulator. Is there any way to help me hook different dolphin?
aldelaro5
He/Him
Joined: 1/8/2014
Posts: 29
Location: Canada, Quebec
Soig wrote:
So if I want to compare with old record, I need to run several dolphins at the same time. But this tool can only hook the first emulator. Is there any way to help me hook different dolphin?
There is not unfortunately, I plan to work on this next February, but not sure if I will add the multiple dolphin option. Though, what do you mean comparing with old records? Do you mean while scanning?
Even if it's a sequel, lots of people have to give their all to make a game, but some people think the sequel process happens naturally." - Masahiro Sakurai
Patashu
He/Him
Joined: 10/2/2005
Posts: 3999
aldelaro5 wrote:
Soig wrote:
So if I want to compare with old record, I need to run several dolphins at the same time. But this tool can only hook the first emulator. Is there any way to help me hook different dolphin?
There is not unfortunately, I plan to work on this next February, but not sure if I will add the multiple dolphin option. Though, what do you mean comparing with old records? Do you mean while scanning?
I think the idea is, you're TASing a game while having an old TAS open to compare to, and you additionally want X/Y/Z/etc RAM addresses for both to compare (so you can see how much further ahead in the world you are on the same frame for example) but without being able to choose which instance of Dolphin you hook you can't do it.
My Chiptune music, made in Famitracker: http://soundcloud.com/patashu My twitch. I stream mostly shmups & rhythm games http://twitch.tv/patashu My youtube, again shmups and rhythm games and misc stuff: http://youtube.com/user/patashu
Soig
He/Him
Skilled player (1458)
Joined: 12/4/2010
Posts: 252
That's what I mean. And I use the same dolphin version as the old record.
aldelaro5
He/Him
Joined: 1/8/2014
Posts: 29
Location: Canada, Quebec
Soig wrote:
That's what I mean. And I use the same dolphin version as the old record.
Hummmm, I never intended this to work on multiple instance like this... TECHNICALLY, I could offer something like an instance switcher, the issue is the only thing I can really get is the different PID from them: it's going to be hard to differentiate them so I don't really know how to do that part. You might have to guess which PID is which Dolphin.
Even if it's a sequel, lots of people have to give their all to make a game, but some people think the sequel process happens naturally." - Masahiro Sakurai
Patashu
He/Him
Joined: 10/2/2005
Posts: 3999
aldelaro5 wrote:
Soig wrote:
That's what I mean. And I use the same dolphin version as the old record.
Hummmm, I never intended this to work on multiple instance like this... TECHNICALLY, I could offer something like an instance switcher, the issue is the only thing I can really get is the different PID from them: it's going to be hard to differentiate them so I don't really know how to do that part. You might have to guess which PID is which Dolphin.
What Visual Studio Debugger does if there's two processes you could attach to is show both processes's PIDs, titles and the date/time they were opened at. That would be enough to differentiate them.
My Chiptune music, made in Famitracker: http://soundcloud.com/patashu My twitch. I stream mostly shmups & rhythm games http://twitch.tv/patashu My youtube, again shmups and rhythm games and misc stuff: http://youtube.com/user/patashu
Soig
He/Him
Skilled player (1458)
Joined: 12/4/2010
Posts: 252
I'll open 2 memory watch tools to watch each dolphin's RAM. First memory tool choose the first PID and second choose another one. It's enough. On the other hand, if I hook a wrong dolphin I don't want, just need to hook another one. List all dolphins I'm running. It's Ok for me. Just like CE.
creaothceann
He/Him
Editor
Joined: 4/7/2005
Posts: 1874
Location: Germany
What if the second instance had a different file name?
Joined: 4/5/2020
Posts: 1
I made an account here to say thank you and to make one suggestion. This is a very useful program. Thank you for your work. I have one suggestion. The HEX values should show like "00000900" instead of "900" when viewing in the watch list, depending on the number bytes.
Dimon12321
He/Him
Active player (478)
Joined: 4/5/2014
Posts: 1120
Location: Ukraine
I'm using Dolphin 5.0-16391. I successfully hooked to a GC game, MMU is off. When I execute scans, no addresses pop up in the list. Is this engine better than Dolphin's Tools -> Cheat Manager -> Start New Cheat Search?
TASing is like making a film: only the best takes are shown in the final movie.