Post by tom_mai78101

Posted: 7/5/2022 9:16 PM
(Edited: 7/5/2022 10:33 PM)

Quote

tom_mai78101

He/Him

Player (100)

Joined: 3/16/2015
Posts: 160

Executive Summary "Variable Disassembler Animation" is a feature where you can watch the Debugger refresh or "animate" and update the registers, RAM addresses, values, and more, by reading all the instructions for each subframe. This feature uses the mGBA core as a proof-of-concept to make Bizhawk have a more fine-grained debugging environment to be able to inspect and pinpoint some uncertain code logic where RNG and other important values are calculated / processed. Details The Debugger is a tool provided with Bizhawk, where you can inspect and observe the assembly instructions and register values of an emulated core an emulated game is running on. However, there are some areas of interests where it's not enough for the user to be able to use the tool and seek out some critical code where the game hides away potential RNG manipulation from being accessed when doing a TAS run. With integration with TAStudio, the Debugger can run and rewind the specific instructions the game uses to calculate the values that is placed in the RAM addresses of interests within subframes (or things that occur within a single frame). The purpose is to allow users wanting to be able to inspect and poke around certain addresses, to be able to search for possibilities to manipulate RNG when making TAS runs with the goal of doing RNG manipulations, so having both the TAStudio and Debugger will help let them see the results more easily. I personally have a strong opinion of not wanting to use the provided Trace Logger. It is useful for dumping the instructions to a file, and we can read the file at once at a particular frame where the code logic happens. However, because there are so many data to parse through, it is often cumbersome to read and it's not easy to jot down notes and figure out where exactly the numbers start to become useful. Goal and What this feature is used for? The goal for me is to use Bizhawk with variable disassembler animation, to help me figure out where exactly the RNG is being calculated in the game, and understand how the process is being done. There is a GBA game called River City Ransom EX, a beat-em-up student deliquents RPG where you rescue the girlfriend of the titular character Ryan, as Alex. Using the trace logger will not help me figure out the solution to this, because the game uses the RNG extensively to determine the CPU behaviors, chance calculations, statistics in the game, and more. I have to use TAStudio to keep the controller input active while the Debugger is in the background animating (or, "emulating while in motion") the instructions until I reached a point where the RNG calculation starts happening in the game. The RNG happens briefly, little enough that the code traces become hidden in the Trace Logger, making it very hard for me to figure out where it's happening. With this new experimental feature, I was able to deduce the ability to manipulate RNG, and has helped another TAS-er, Chamale, in making better optimizations in a TAS run, linked below. #7569: Chamale's GBA River City Ransom EX "alternate final boss" in 10:09.75 Demo

Github Branch https://github.com/tommai78101/BizHawk/tree/experiment-disassembler-animation Known Issues

- I lack the knowledge domain of how Bizhawk actually works under the hood. Requires extensive code refactoring to properly integrate this feature into Bizhawk nicely. - Unsure if this feature is useful for TAS production. - Unable to figure out how to properly calculate lag frames when doing full frame advances while you have instruction advances. - Not a clear picture of how to refactor the Bizhawk and other emulation cores to be able to implement per-instruction advances properly. - If the animation interval (in the Debugger window) is less than 10 milliseconds, TAStudio wouldn't be able to pick up controller inputs properly. - Rendering the updated buffer isn't obvious when animating the instructions. - Unsure if this feature is welcomed by the TAS community. Credits Thanks to feos for suggesting I should make a thread. Thanks to Chamale for collaborating with me to make a pretty optimized TAS run for a game with poorly-documented RNG. Thanks to CasualPokePlayer and YoshiRulz for most of the help on C# programming.

Posted: 7/9/2022 7:48 AM

Quote

feos

Site Admin, Skilled player (1236)

Joined: 4/17/2010
Posts: 11274
Location: RU

Thanks! Please describe actual usage. Like steps of what to do to make use of this, and what would be happening and how it's achieved.

Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.

Posted: 7/9/2022 8:30 PM
(Edited: 7/9/2022 8:31 PM)

Quote

tom_mai78101

He/Him

Player (100)

Joined: 3/16/2015
Posts: 160

Actual Usage I'll be using the GBA game, River City Ransom EX, as an example. I've been using this game for TAS optimization testing/RNG manipulating for a while. Let's say, you (as ALEX, the main character) are about to enter a new area zone, and you wanted to know how the game uses the registers and the game assembly instructions to set the values up, to prepare the data and information it needs to know about the new area's enemy types (Generic Dudes, Frat Boys, Locals, Jocks, etc.) The first thing you would do is to use the TAStudio in Bizhawk to rewind / fast-forward to a frame just before the area transition begins. You would stop your pointer to this current frame, and pause the game. The next step is for you to bring out the Bizhawk Debugger and have it attach to the EmuHawk instance, so that the Debugger is ready to start animating the instructions. The first thing we need to do is to choose the ARM instruction set for our game. Since we are playing this game on a GBA platform, we need to set the mode to ARMv4 (Thumb). In the debugger, you would then choose the animation intervals (or called "Intervals" for short), to say, how many milliseconds the debugger would wait until it advances to the next instruction. 1000 is typically 1 second, so when you set the intervals to be 1000, the Debugger will update the register values, the PC, and finally advance to the next instruction in the disassembler list every second. After the animation interval is set, we then press the Animate button, to let the Debugger begin its "animation". Note that this game has a ton of branch operations (or "opcodes"), so that is why you sometimes see the instructions jump to a different address often. This is 5 milliseconds intervals:

This is me changing the intervals on the fly after I realized I made it too slow:

That is how you would use the Variable Disassembler Animation. You can change how fast or slow the disassembler is animating, be it updating the game logic, or updating the rendering frame. How This is Achieved This requires updating the Bizhawk mGBA core submodule code, to expose the unmanaged code's interface to call directly to the function:

mDebuggerRun()

endrift, the developer of mGBA, mentioned how Bizhawk typically calls on the other function:

mDebuggerRunFrame()

That function would then execute a do...while loop that runs continuously for a duration of 1 frame tick (≈13.333 seconds) invoking the mDebuggerRun() function mentioned above. What I did is I exposed this function interface, and then created boolean flag to check whether the UI button, "Animate", is toggled on. If yes, it goes into a code loop that executes the mDebuggerRun() function per interval, instead of going to the mDebuggerRunFrame(). I then make sure I keep the Debugger information updated after each instruction has been executed/advanced to the next instruction. To do this sort of execution loop, I implemented a simple "Unity C# Coroutines" in Bizhawk. It's a very simple concept of invoking the enumeration of the code, and then yielding the execution to the main UI thread, so that the Debugger UI doesn't become unresponsive while the Debugger is actively executing instructions in the background. If there is anything you wished to know, please feel free to let me know as replies below.

Posted: 7/10/2022 8:30 AM
(Edited: 7/10/2022 8:31 AM)

Quote

feos

Site Admin, Skilled player (1236)

Joined: 4/17/2010
Posts: 11274
Location: RU

I still don't get how making the debugger run by itself makes it easier to track how the needed address got its value than by using a tracelog.

Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.

Posted: 7/10/2022 6:52 PM
(Edited: 7/10/2022 7:09 PM)

Quote

tom_mai78101

He/Him

Player (100)

Joined: 3/16/2015
Posts: 160

feos wrote:

I still don't get how making the debugger run by itself makes it easier to track how the needed address got its value than by using a tracelog.

I shall answer this with another explanation: What makes it better than Trace Logger? Trace Logger is a Bizhawk tool, which is typically paired with TAStudio, that allows you to record all of the assembly instructions executed within certain number of frames (usually 1). Within a single rendering frame, you can obtain a dump of all of the assembly instructions executed, stored in a way where 1 row equals an instruction step in the emulated core. However, the downside of using Trace Logger means, you are not able to use other Bizhawk tools to assist you in pinpointing where exactly in the code that would update a specific RAM address with the value you seek in the registers. With this variable disassembler animation (I'm just going to call this VDA for short from now on), you can choose the animation intervals (between 0 through whatever maximum integer of milliseconds you give it) and watch it cycle through the assembly instructions. While that is going in the background, you can then tell Bizhawk to open up the RAM Search tool, and fill you in on what RAM addresses the core has updated during the assembly animation cycles. If you set the interval to 0, the VDA will know that you wanted to pause the animation, and will be configured to pause the animation until you are ready to continue. This is a temporarily measure, as this part needs more work, so that there is a checkbox to allow pausing/continuing during VDA cycles. Another thing to point out in regards to RAM Search and VDA, is the concept of "subframes RAM searching", or "searching for RAM address changes that occurs within a single frame". Currently, you have the issue in Bizhawk where each time a single frame is advanced, you typically have a bunch of RAM addresses getting updated. This means, after a search, sometimes you need to narrow down the criteria a bit further, do repeated experiements by rewinding and fast-forwarding the frames, poke and freeze the addresses, and sometimes do "binary search and freeze" the addresses in bulk, just to narrow things down. (The unit of time here is "multiple RAM addresses / 1 frame" here.) With VDA, we now shrink the unit of time down to "single RAM address / 1 instruction". With this granularity, you can do RAM searching where the game updates a single RAM address one at a time, allowing you to check the changes on a more granular scale. No more guessing which address is specifically getting updated in between each frame, and no more bulk- poking, freezing and unfreezing addresses to experiment what would occur on the next frame and such. All you see are instructions updating 1 thing at a time. You know how sometimes, when you do a RAM search at a spot between Frame A and Frame B, 1 frame apart from each other, and you see some RAM addresses have more than 100 changes? VDA and RAM Search can handle that for you here. Demonstration Here is me demonstrating quickly how you can pair VDA together with RAM Search. By enabling the RAM Search to search automatically per frame, each time the VDA steps to the next instruction, the RAM search will search and remove RAM addresses that don't fit the search criteria. You can see it in action as the total number of addresses begin to drop as the VDA animates in the background.

When using Trace Logger, there isn't a way for the Trace Logger to dump the RAM address changes and the search criteria used to keep track of which RAM addresses you are looking for. You have to slowly piece back together what all of the instructions and register values are doing from the trace dump, and it takes time to do. But if you have VDA paired with RAM Search, you have an automated way of searching for the RAM addresses and values quickly and easily.

Posted: 7/10/2022 7:32 PM
(Edited: 7/10/2022 7:33 PM)

Quote

feos

Site Admin, Skilled player (1236)

Joined: 4/17/2010
Posts: 11274
Location: RU

Once you know the address you want to understand how it changes, you can use this lua code to see which PC it changes at Download print_disasm.lua

Language: lua
local function print_disasm()
	local pc = emu.getregister("PC") -- or whatever the PC register is called for that core, check using emu.getregisters()
	local dis = emu.disassemble(pc).disasm -- not all cores have this, so you can just print the PC
	print(string.format("%X:  %s", pc, dis))
end

event.onmemorywrite(
	print_disasm,
	0 -- or whatever your address is
)

Depending on the core you may get the PC right before your address is written, but it would still be easy to find in the tracelog for the frame it changes at. And in the tracelog you can go up and see how the value affecting it is changing. You don't have memory view there, but you have affected addresses printed out anyway. I feel you'd be better of just implementing proper breakpoints, then debugger would instantly jump to the location your address is changed on. The problem with the current hawk is that debugger is a tool and tools don't update midframe unless it's a hook like in lua. But you seem to have solved that part. https://tasvideos.org/Bizhawk/LuaFunctions

Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.

Posted: 7/10/2022 7:39 PM

Quote

tom_mai78101

He/Him

Player (100)

Joined: 3/16/2015
Posts: 160

That is an interesting way to customize the Trace Logger. I didn't know about that. The VDA / RAM Search pairing is used when you don't know the RAM address. But I do agree, once you do know the RAM address, customizing the output of the Trace Logger using Lua scripting helps. Thank you for pointing it out. :P Overall, how do you like the proposal?

Posted: 7/11/2022 4:05 PM

Quote

feos

Site Admin, Skilled player (1236)

Joined: 4/17/2010
Posts: 11274
Location: RU

tom_mai78101 wrote:

The VDA / RAM Search pairing is used when you don't know the RAM address.

I don't see a critical problem in finding the address while advancing full frames. It may not always be the RNG address of course. Very often you just see some in-game event happen differently in different scenarios, and you have no idea why. So the approach to that is boiling it down to the visible difference, and then finding the address of what is visible different in those scenarios. Once you know that, you set a hook on writing to that address around that time, and then you do the tracing - when you already know what PC to look for.[/quote]

tom_mai78101 wrote:

Overall, how do you like the proposal?

I like that you cared and you managed to achieve this. You may be interested in checking out this thing https://github.com/TASEmulators/BizHawk/issues/676#issuecomment-407103055 It implements functioning debugger for N64 (see the attached patches), may give you some useful ideas overall.

Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.