If you really need help troubleshooting the disassembly, I would probably ask about it in a different forum, maybe in Tool-assisted laboratory. However, it sounds to me that that might be overkill, and understanding it could take as long as just systematically narrowing down the address (although I guess the disassembly could be useful for more than just that).
Couldn't you unfreeze the addresses a half at a time? (Is it very meticulous to freeze and unfreeze addresses?) If unfreezing 250 of the addresses changes the drop, then obviously the RNG is in that 250. If not, then it's in the other half. By continuing this method, you would cut down the time to find it considerably.
I found some dumping code in the source that can be uncommented to dump all the instructions ran but I can't find it again. I was hoping to modify it to start dumping when it reads the drop rate multiplier and stops dumping when the drop rate multiplier is read a second time (I can manipulate the drop rate so it fails the first time and succeeds the second time which would make sure it doesn't start dumping again on a 3rd check). I can no longer find what I want in the source and I'm having trouble compiling so compiling a hacked version specifically to get a full loop used for drop checks is out of the question.
I tried unfreezing the addresses half at a time but after awhile, it fails because each half would modify the drops in each save state without getting it exactly the same in all the save states. This forces you to unfreeze the addresses one at a time. I wouldn't mind letting a script do it but I'm going to have to dump all the addresses and values to a file for DeSmuMe to work with or find a way to send commands to DeSmuMe from Cheat Engine.
Imagine if we played a game where I think of a random amount of numbers between 1 and 500 and you need to find them all but you don't know how many I'm thinking of. You can only ask me "Is X one of the numbers?" or "Are any between X and Y?" and I can only reply with yes, no, or maybe. Doesn't seem fun does it?
I think dumping all the instructions in a single drop check loop would be more effective because once I sorted out what is what, I would also have pieces of how the RNG works. I really wish I could get this to compile...
Anyways, sorry for taking so long to get this out but here is the lua script for displaying enemy information. There is a lot more than the objects listed there (enemy attacks that must extend beyond their hit box are even objects too) but all this scrambling to get the RNG left it in an outdated state. I use getter/setters to easily organize the offsets for where data is stored in the structs. I have the starting address for each struct displayed so I can easily go to any one with memory view without having to go down the linked list myself.
http://pastebin.com/nbKkMW8Y
If you can figure out how to compile, it sounds like it would fix most of your current problems. Unfortunately I don't know nearly enough about programming to help you with that.
In the meanwhile, you could look into how alien mingling works. It seems like it's affected by randomness. Although the RNG could be different all together from the RNG that affects pin drops, it could still be worth looking into. Also if it's reasonably manipulable, it could be the fastest way to level up pins. Nearby WiFi might help it work better, but I doubt it's required since I got an alien before with no known WiFi available.
Since I'm not engulfed enough with the programmer mindset, I would like to kindly ask you to clarify some things:
By "drop rate", do you mean the game's listed amount? Wouldn't you also be able to set the drop rate at 2 to stop it from dumping a third time?
I wasn't clear about how the game calls for the drop rate. Do you mean "difficultly" as in the actual game difficulty?
And does the game actually call for the drop rate as many times as it is big (calling for the "Should a pin drop?" function 99 times if the drop rate is 99)? Wouldn't that mean there's no true "100%" chance of getting a drop without a specific subroutine to guarantee those drops?
Are you focusing on more than one drop at a time? Shouldn't you be focusing on just one drop for now? I'm probably not understanding something, but anyway, it seems more than one address if not several is responsible for what drops. I'm not sure why that would be unless there's more than one RNG responsible or it's due to some cheat prevention system.
Sorry for not clarifying sooner.
Unless I refer to 'drop rate' in a possessive way (like "a pin's drop rate" or "a noise's drop rate for x pin"), 'drop rate' is the final drop rate multiplier in battles. I just got distracted while rewriting parts of the post, I forgot about that.
Each difficulty has its own item drop but can also drop items from lower difficulties if it fails. If you are on Hard, it will check if the noise will drop the item for Hard. If not, it will check the item for Normal. it continues down the table until there are no more left (fails to drop the item for Easy) or a check says to drop (each enemy can only drop 1 item and no more unless under special circumstances). To answer your previous question, I can set the difficulty to Normal and only have to make sure the item for Normal fails to drop to get only 2 drop checks.
Instead of focusing on 1 drop, I actually have to focus on just 1 noise. I'm playing on Ultimate and I freeze the RNG to where it will always drop the Normal Item. If it drops something other than the Normal Item (including nothing), then one of the addresses for the RNG I want is unfrozen. Focusing on an Easy Item while on Easy so it drops the Easy Item or nothing at all only restricts the amount of possible results from the RNG and make it more difficult to see if anything did change.
The problem with Mingling (and Shut Down) is that you can't skip the PP Counter. If you earned 200 PP, you have to watch that counter go from 0 to 200. You can hold buttons to speed it up but it is barely noticeable. You also have to consider if the time to gain that PP easily will save just as much time. You have to go into the menu, go to Mingle, wait for aliens, save, watch the PP Counter, go back to the menu.
You have to trust me, grinding to level pins will never save the time it costs. The only time is if you are grinding for materials to unlock the last 2 pin slots, get a powerful pin, or the story requires it.
I honestly thought you didn't get pins from lower difficulties. Anyway, now that I know, I assume that the RNG changes between difficulty checks since I was able to get pins from the easy difficulty level even when the noise's normal-level drop rate is higher.
Even if leveling pins is practically useless, I thought it might be easier to figure out the RNG from alien mingling. That's just me being optimistic.
Do you happen to know the addresses that affect the "default" percentage for specific noise drops on the Noise Report screen?
I would also like to use Mingling to find the RNG but it is a lot more random. When I played, I had to leave my DS all night (about 8-12 hours) and only receive 1 or 2 aliens (sometimes 3-4 or none at all if I'm lucky). Maybe it would be easier (it would take less work but a lot more time) but I will need to grab an alien (or more) first to get the addresses I will need. Then I can set up a lua script to monitor these addresses and pause when I grab one. Just restore to an earlier save state, wait until it reaches a few frames before, and I got an save state for comparison. Then I just need to rinse and repeat until I get enough save states. I like that idea so I will give it a shot.
No
I have some news (some good and some bad).
Pin Data during Battle
Bad News:
I started analyzing the linked list containing pin data during battles and discovered that the linked list is also stuffed with gui stuff. The lua script I made to display all the information about the pins (from reboot to uses left) found about 20+ objects in the same list. This value also fluctuates and what gui elements has a home in this list will require additional research that can be put on hold. I was able to find a way to identify specific pins (possibly rebootable pins) but other pins (such as limited uses and such) remain in the dark until somebody decides to sort through the mess and find out which is which.
Good News:
Any pins that don't reboot possibly don't have any interesting information that isn't displayed on screen already.
Mingle
Bad News:
I found the values that tell how many aliens found. Unfortunately, I can't modify these values without the game freezing (and sometimes graphical glitches). I also haven't found the RNG yet.
Good News:
The values that tell how many aliens found can in fact be modified to higher values only after you have encountered an alien at least once (possibly an anti-cheat system). I also found a timer that sets when to check for aliens (and possibly other devices in the area which would explain why it sometimes take awhile). The value is set to 3600 frames (or 1 minute) and when 0, it does it's checks and reset to 3600. The address is 023A35B0 (16 bits) and you can set up a cheat code or something to keep it 0 to check the RNG every frame. With this in mind, my first Mingle can be encountered earlier which brings up the possibility that the RNG only cycles when read. Here is a screenie of my first alien and 3 'forced' aliens by modifying the aliens encountered value up to 4 (notice how I can still mingle with 9 more even though it should be 6 for a maximum of 10). Oh and input does not effect the RNG while Mingling.
Edit: I found a RNG address (only 1 is used) and one (and I believe only) value that affects if you get an alien is the total amount of mingles. I will try to figure out how the 2 addresses are used to determine if you found an alien. The RNG address also changes every frame during battle so it might be the one we want.
Possible RNG Address: 02063760 (32 bit)
Apathos, thanks for suggesting Mingle to find the RNG.
How about closing the DS? It might be a false perception of mine, but I feel like having the DS closed increases my chances of getting an alien sooner. I don't know if the emulator gives that option or also allows you to simulate other DSs to mingle with or other WiFi networks. From what I've been reading, people seem to get more aliens when they're around many WiFi networks (even if the DS isn't connected to any of them).
Another question: Let's say it takes 10 minutes to get an alien normally. If you make the counter check every frame, does it take 10 frames to find the exact same alien? Is it the exact same alien just to be clear when you find an alien that way? What's the fps for this game anyway?
Yet another question: Did you look up any Action Replay codes for this game? Well, I've found some here and here though they're mostly the same. I don't much about how Action Replay works myself, but those codes can directly help you find addresses, right? Also, isn't the main point of "master codes" to first deactivate the anti-cheat system of games? That alone should be useful (I probably should have asked about this sooner).
Take a special note of the debug mode feature. I remember you said that you've messed around with the internal debugger, but I recently found out that there's a secondary debug menu, which I don't know if you're aware of. Once you activate the debugger, you can press Select again to access a little more graphically enhanced and much more mysterious menu. Basically it gives you the access to various letter and number combinations, which you can 'run' by pressing OK. Some of them are various sound or graphic tests. Others seem to crash the game. Maybe you've already looked into it, but for all I know, the RNG could be readily available somewhere in there.
Also maybe through hacking, you can see what happens when you force an unavailable combination (like, idk, "S-2" or something).
Another thing I realized is affected by the RNG is the music that plays at the start of each battle if that's another possible lead.
Thanks for reminding me about the lid. I just tried it and I don't see any obvious effects. The stream of RNG doesn't change and the time between checks didn't change (still every minute). Maybe it accepts a larger range of RNG values but that will take time to test.
I am currently assuming that the game runs at 60 fps (DeSmuME's math for movie time is 60 fps) and the gameplay timer calculates to 60 frames = 1 second. In 10 minutes, the game checks for aliens 10 times. Using a no timer code (surprised no body came up with this but it is another cheat to add to my collection) would only require 10 frames to do those same 10 checks.
I looked up Action Replay codes. TWEWY is where I learned how to make AR codes and I have my personal collection of 'improved' versions. These included holding "R" to receive no PP (very annoying to get tons of shut down PP when you don't need it) or hold "L" to receive x PP, improved food byte management (the simple ones out there causes minor glitches if you don't eat anything), and so forth. I know how the codes work and looked at them all when I started for useful addresses.
I checked out the debug menu as well and the only thing I could find useful was the map scaling.
After reading that, I decided to play around with the RNG in Mingle (I'm considering it a controlled environment) and come across something interesting. Making the RNG 0x00000000 always passes the alien check and doesn't change (reveals there are no external values used in calculating the next value).
I have cracked the lower 16 bits. Here is the process
Shift the 16 bits left (multiply by 2) and any over flowing bits get added back in. For example 4160 becomes 82C0 which becomes 0581 (0580 + 1)
Then add the upper 16 bits and you have your new lower 16 bits.
Cracked upper 16 bits coming (very) soon...
Edit I cracked the upper 16 bits. You just multiply it by 5 and discard the overflow. It seems a little too simple but it works so far.
Edit2 Something isn't working out with the lower 16 bits. My original formula can sometimes be +/- 1 and I'm looking at these right now. I will try to get a final formula later (need sleep).
It's good that you finally found the RNG. I didn't see your edit until after I posted.
By "cracked" do you mean that you can tell what the next number in the cycle will be? As for the lower 16 bits, maybe the upper 16 bits affect it. You could try setting the lower bits to a fixed number at different times (also letting the upper 16 bits be different) and see if the same number always follow it just to verify it's not affected by other addresses.
One guess is maybe the overflow of the upper limit affect that +/- 1 difference.
Yes, I have a lua script that tells me the current value and what the next x amount of values could be.
I did a little more digging with Cheat Engine and each half is read and written to individually. This leads to the possibility that only the lower half is the actually used while the upper half's only purpose is to keep the lower half random.
It seems input has no direct effect on the RNG during battles. It was a bit difficult to keep track because the RNG is read several times every frame. Lucky for me, I only needed to look for the matching upper half in the list of predictions and see how accurate the lower half was. I then ran through the same amount in Mingle and the end results matched up perfectly.
I'm constantly modifying my lua script to do as much of the pencil and paper work for me as possible to make it easier and faster for me to figure out what I'm missing. I tried several possible changes but none worked (perfectly).
EDIT I did some testing and you no longer get the original +/-1 problem if you don't add back the overflow from adding in the upper half. This then leads specific values to be 1 higher than predicted. I believe it has something to do with the upper half. I managed to come with some possible theories but it to find one that works.
When I played TWEWY long ago there was a trick where you could force Aliens to appear if you have access to a Wii. If you have the Wii send a DS demo, TWEWY will count the signal as a alien. I don't know if you still need aliens to appear, but if so I hope this helps.
Regardless, good luck on the run, I'm looking forward to anything put together.
Devices communicating with DS's are Civvies and devices with TWEWY in Mingle Mode are ESP'ers. Both are not random and don't affect Aliens. Thanks for the thought though.
I also found and cracked the RNG mostly (my formula for calculating the next value is missing something but I can't figure out what). I have figured out how the RNG is used to test Item Drops and have a basic idea on how it is used to determine enemy actions.
Everybody, expect a WIP soon. I'm almost done with a serious Day 1.
Edit: Strike that. I found a major game changer that requires me to redo all the battles.
Never mind. After further observation, the game changer is also a game breaker if not used correctly. The game changer I found is that the lag and compensation for lag in the game play timer extends to invulnerability timers as well. The problem is that the invulnerability timers isn't as vulnerable to over compensation like the game play timer. How it can be a game breaker is that if you try to lag it enough to get it to over compensate (if possible), you have to avoid making it lag on 1 or else Neku will be forced to deal damage a frame later (costing you a frame). I will leave this possible frame squeezing method for the next run.
Just so you know, the game play timer over compensates all the time (even if it doesn't lag).
I also learned that passing the puck on the first possible frames will reset the puck. You read right, the person passing the puck doesn't receive the multiplier bonus and the multiplier goes back to x2 (x1 actually but the next person will reset it to the x2 minimum unless you do it again). Stop cheaters and troll TASers with one stone, I can't wait to see what other surprises Square has in store for me (seriously, I'd rather get them over with right now).
How erratic is that behavior? I'm just wondering how different gameplay would be if you focused on in-game time as opposed to real time.
As a last resort, couldn't you just map out the whole RNG cycle? This question isn't meant to be rhetorical; I just want to confirm what I think I know about how RNGs work.
Anyway, I look forward to your WIP.
I have no idea. I haven't bothered with it too much to see if there is an obvious pattern.
I can map out the whole RNG cycle and it would eventually repeat. There are no more than 536,870,912 accessible values. There would of course be values permanently out of reach but to create the entire list of accessible values would still require a massive amount of space (unless you are familiar with rainbow tables). It would be easier to search in real time using the current RNG value. If I'm looking for a specific value (let's say 0xF32732F0), having a program searching through a premade rainbow table would be the fastest. If I'm looking for a range of values (let's say 0x????0000 to 0x????0006 so I can drop an item with a 0.01% chance), it would be faster to have a program search for matching values starting from my current RNG value.
I finished the first partner tutorial battle with Shiki. I tested 2 different strategies (complete focus on the passing the puck and getting in as many attacks as soon as possible). Complete focus on the puck ended up 1 frame faster but after reviewing the movie just now, I noticed something that let me finish 17 frames faster.
I used Shiki's 4 panel combo (available on first possible frame so no manipulation needed) to get the puck to Neku ASAP and Neku only had to wait 5 frames to pass the puck on the first possible frame. When it is Shiki's turn to do her combo finisher, the noise had 35 HP left so it was either wait 8-10 frames (didn't count exactly) and use a 5 panel combo to get the x3 multiplier and deal the final blow. When I reviewed the battle, I noticed Neku had enough time to hit with Pyrokinesis twice before Shiki's combo finisher. I didn't have Neku use Pyrokinesis because it could cause lag and I overlooked how many times Neku could attack (I didn't have save states from before having Shiki wait). In the end, Neku used Pyrokinesis 3 additional times (once after Shiki's combo finisher) and Shiki didn't have to wait and used a 4 panel combo.
Two more battles to go before Day 1 is over. Let's hope luck is in my favor.
Joined: 8/6/2006
Posts: 784
Location: Connecticut, USA
FreezerBurns wrote:
*applause* Nice to see the run finally starting in earnest!
Yes, good luck with this. I feel that there's a lot to show off with this game! I'm also curious as to whether the story will be intact. IIRC you can scroll through text really, really fast.
IIRC you can scroll through text really, really fast.
That's only after you've beaten the game once.
Also Zanoab, have you considered that midnight food trick I mentioned earlier, or will you not need to eat that much food?
The WIP is on hold once again. I did a great job with the first partner tutorial but it put in a place where luck is nearly a worst case scenario. I can get one of the Dixiefrogs on the top screen to switch sides so Shiki can attack both at the same time. Unfortunately, this requires a few seconds to happen and Neku has to remain idle (or use Pyrokinesis to the cycle the RNG and make sure one of the Dixiefrogs get the right RNG value). The luck hates me so much, I can't even get the 2 Dixiefrogs on the bottom screen together in time for Pyrokinesis.
The RNG is now completely cracked, with some previous game hacking experience, a disassembler, Google, and Ilari (deserves a lot of credit for this). I now have the exact method for calculating the next value in the RNG. A straight ARM to Python/Lua didn't work out very well but Ilari managed to get it working after taking a deeper look. I'm putting some finishing touches on my RNG Lua script before I release it which includes allowing it to index RNG values starting where the script started (for better RNG planning).
As for game text, most of the text is visible for a few seconds (if you still can't read that fast, it will be easier for you to pause as necessary). The RNG is never used during conversations which means that I may be able to create a parallel movie that doesn't immediately dismiss text bubbles. The only problem is that each frame I don't dismiss a bubble has a possibility of being regained by reducing lag when the game transitions out of the conversation. I will be doing tests to see if adjusting the lag this way will cause the RNG to desync from the movie. If not, I will also make the movie for encoders that want to encode the movie with text bubbles displayed longer.
As for midnight and food, I don't have an estimate on when I will need it to turn midnight so I can eat more food. There are already major differences from the speed run guide mentioned earlier. First, less non-mandatory battles. Half of the non-mandatory battles are for collecting specific enemy drops and a good amount can be gotten with some luck manipulation before they are needed while the rest will be have to be done because of how scarce the enemies that drop them are. Reduced battle count means longer food consumption and only time can tell when I will be needing midnight to roll around. Some good news is that the RNG is initialized from start up (possibly from the firmware). If we are lucky, food reset is the only thing time has an effect on and I can change the start time in the middle of the run without desync. I will be testing this now as well to make sure.
School also started up so I might not be working on this as much as I want to. I will try to get a Game Resource page up with everything I have some time within the next week in case somebody would like to take over. I'm currently making the route up as I go along right now so I don't mind sitting this run out to let somebody better go on ahead.
I played with luck manipulation and came across some interesting things. Pyrokinesis only causes lag when damaging an enemy. Pyrokinesis helps advance the RNG faster when in use. It is possible to reduce the amount of lag when transitioning to the overworld by manipulating luck to create less NPCs.
This unfortunately means I need to go back to the very first battle (the one where you have to escape) and see if waiting a frame or two could end up saving a frame or two. I will also have to redo every other battle to deal with the new RNG values while playing with Pyrokinesis to try different RNG values to save frames. Fortunately for me (and every other TASer interested in TASing this), a simple save state right before the first loading frame and a lua script that tests every RNG value and records how long the lag lasts can handle the job.
After some testing, everything can be spliced together so it is a matter of lining up the first frame input for a specific part and combine them. I have yet to test splicing movement on the overworld but it should (in theory) not make a difference because there is no lag in the overworld.
Here are my lua scripts that are useful for both TASers and casual players.
I am almost done with the Game Resources page for TWEWY. I should have it up tomorrow if I stop finding new information to add.
The timer for battles does not lag or (over)compensate at all. This makes it a good way to compare times.
Here is my last WIP before starting over. It contains (almost) optimal movement in the overworld and the optimal first partner tutorial battle. Movement in Scramble Crossing is not completely optimal because my tests have shown you can clip the wall to increase your displacement (cut the corner and get pushed out closer to the target) but doing this will cost a few frames to replace the cursor and get moving again. I currently don't have the addresses for Neku's position and the cursor's position (where Neku is moving toward). One of the sets of coordinates are dynamic, I don't remember which and haven't found the pointer for them (yet). Optimizing the second angle was painful and the second cursor could make a better angle if set on a different frame (different camera angle so maybe a perfect angle can be found).
Too bad that WIP isn't encoded, and I guess there's no point in encoding it if you're going to restart anyway.
Outside of technical optimizations, do you have good handle on the route planning, item collection, and other strategies you'll use for the run? Will it basically follow that speedrun guide except with a few less battles?
Also, will you take damage to save time? Is it even possible to not take damage and beat the game?
I've very busy with course work and can't finish up the resource page.
I'm saving the route planning and item collection for after Day 2. I haven't experimented enough with the mechanics of the different psychs to formulate a route. Day 2 forces you to use all the starter pins at least once so after that, I will get a good idea of what each psych is capable of. From there, I could look through all the pins available at each part (including pins that drop from Noise) to get the best damage per 25 frames in most situations. Clothing, optional rewards, and cash also play a role in what pins to use but don't matter until after Day 2.
The only time I can think of where damage would save time are the mandatory Tin Pin Slammer battles. The fastest way would be to use a deck of 1 pin, manipulate the starting position as close to an edge, and lose the battle. But then, that may be more like uses death to save time.
Beat the game without taking damage? Only in my dreams at the moment. It is no problem spending seconds here and there to have Neku avoid getting hit. The only problem is that your partner is a sitting duck (literally). I could counter attack to cancel attacks but that might not work too well later on when almost every enemy you fight has ranged attacks. Unless I figure out how enemy actions are decided, it will be too much work.
Unfortunately, I won't be able to work on this run anymore. If course work lightens up, I might start over and try again but I doubt it. All the useful RAM addresses I have can be found in my lua scripts in case somebody else would like to give it a shot.
Bad News
There is a major emulator bug in Desmume somewhere. A month or two ago, the starting RNG seed when the game starts (frame 0) changed for no reason. I checked my backups and the backups on my other laptop that haven't been touched since the summer and NONE of my movies sync. I encountered this bug during the summer but the starting RNG seed returned to normal after a day so I thought I was getting tired or something. It seems the new RNG seed is here to stay for good this time so I spoke with a developer and unfortunately, he is getting the new RNG seed too. If anybody is able to get the WIP movies I uploaded to sync, please reset Desmume (return to frame 0), make a save state, and upload it. If the RNG seed can spontaneously change, then other areas might also being changing that isn't as noticeable. I also don't want to finish a new movie only to find out that all the work is lost to this bug.
Here is an old WIP for testing: http://www.mediafire.com/?9tgprj0saehtuwc
It should be the test run I encoded and uploaded so I know it used to sync.
Good News
With a little help from a developer, I am able to dump the cpu registers when the RNG function is ran by the game. This means that I can trace back most of the data along with the return register which tells me what code wanted the RNG.
What I found out so far:
Each enemy (only for the bottom screen) stores 8 bytes generated from 2 RNG seeds. Some stuff happens and a little while later, these 8 bytes are modified by 1 RNG seed. This happens for every enemy but only 1 enemy at a time with a few frames in between each. How it is decided when an enemy will have these bytes refreshed and what these 8 bytes are used for is currently unknown.
A piece of code gets 2 RNG seeds every frame and stores it within a linked list I have not explored yet. I suspect this list contains important information about the battle so I will be tracing the head/tail of this linked list later.
Later, I will use this to track the item drop code and hopefully trace it to the location of the drop tables.
Zanoab, I don't know if this works or not, but check your settings.
Firmware: Should be the exact same settings as stored in the movie file. The emulator will not check this for you.
Also, check emulation settings, and see whether "Enable Advanced Bus-Level Timing" should be checked or not.
Zanoab, I don't know if this works or not, but check your settings.
Firmware: Should be the exact same settings as stored in the movie file. The emulator will not check this for you.
Also, check emulation settings, and see whether "Enable Advanced Bus-Level Timing" should be checked or not.
I just double checked my settings and Advanced Bus-Level Timing became unenabled somehow. I enabled it again and the movies work perfectly. I still don't understand how the initial RNG seed changed on me but everything works now. I really don't understand how my settings could have gotten reset without losing my hotkeys. Thanks FractalFusion.
Time to use my upgraded scripts to get new information out of my WIPs.