Pokemon Gen 1, 2, and 3 had save corruption, so why not Gen 4? :)
Game objectives
- Emulator used: BizHawk 2.11.1
- Beat the game as fast as possible
- Use save corruption!
Save Corruption
The Save Medium
Like Gen 3, Gen 4 uses Flash for save data. Flash can only be written to using "commands" against the flash chip.
The main command used is simply the "page write" command. On this Flash chip, a page is 256 bytes. This command allows for writing a stream of bytes within an arbitrary location within a page of Flash. The command itself is combined with the bytes written, so all bytes are sent to the Flash chip in a buffer before the command executes.
Internally, this command is actually two commands in a trenchcoat: it first erases the selected bytes within the page to 0xFFs (as erasing is the only way to set bits), then proceeds to "program" them ("progamming" bytes can only clear bits, not set them).
On hardware, these commands take some time to complete on the Flash chip's side (requiring waiting for completion of the command). However, emulators do not emulate this timing, instead having all commands performed instantly once sent.
These commands are handled by the DS's SDK, so all games have the same internal behavior when it comes to command handling. An interesting quirk within the SDK's handling is that it waits 25ms after executing a command before checking if the command has finished. These commands typically finish much, much sooner than 25ms, resulting in effectively 25ms taken for each page write. This also means emulators having "instant" commands does not matter, as the SDK is going to wait at least 25ms regardless.
Gen 4 has a fairly simple save format, as far as save corruption is concerned. There are two blocks of save data: "normal" save data, and "box" save data. Each block has a footer attached, containing counters, a CRC16, and a "magic number."
There is also a backup save system, where the game has a second pair of save blocks. This effectively creates two save sides, which the game alternates between for each save. The counters in the footer tell the game which side is newer.
This format ends up nearly defeating save corruption itself. Footers are only written at the end, and saving is only going to write to the "older" save, not the "newer" save. The game also will not allow a newer "normal" save to be loaded if its counter does not match the "newer" box save (as a way to prevent cloning box data).
Clearing Save Data
In Gen 3, you could simply abuse clearing save data in order to corrupt the "newer" save and thus force the game to load the "older" save and overwrite the "newer" save, thus allowing for reset as long as it occurred after the point deleting save data corrupted the save.
However, it seems Game Freak realized this, and thus patched this entrypoint. They did this by erasing all save footers before touching the save data. The older save's footer is always erased first here (preventing the older save be force loaded by abusing clearing save data).
However, this does mean you can potentially erase both normal save data's footers, but reset before the newer box footer get erased. This will have the game consider the save entirely corrupted, and thus force a new game.
Note, that in Gen 4, selecting "New Game" with an existing save file will disallow saving. However, it will allow saving when a new game is forced due to save data corruption.
The First Save
In international versions of Diamond/Pearl, and all later Gen 4 games, the first save done since starting a New Game will clear save data footers before performing the save, similar to clearing save data.
However, in all Japanese versions of Diamond/Pearl, this is not done. This oversight thus allows for performing save corruption in these versions, although it is limited to box data and can only be done once within the first save.
The Larvitar with a Bike
Gen 4 encrypts Pokemon data. This is done in a fairly simple manner: each halfword is xor'd against an LCRNG seeded with the encryption key. This ends up resulting in raw "empty Pokemon data" actually look like random data (along with any raw Pokemon data). The game uses the checksum (sum of all decrypted halfwords) as the encryption key.
Like Gen 3, Gen 4 also splits encrypted data into 4 blocks, which are ordered depending on the PID.
By using a specific encryption key and block order, "empty Pokemon data" can be "incorrectly" decrypted into a Pokemon holding a Bike.
However, this itself in theory is unlikely to work. There are only 65536 possible encryption keys and there are 65536 item IDs, with only 1 corresponding with the Bike. Additionally, the Pokemon species must be valid, otherwise the game will crash upon viewing its sprite.
By sheer luck, 1 encryption key and block order happens to produce a Bike: 0xED5A with ??A? block order. This 1 encryption key also produces a valid Pokemon: Larvitar.
The Three Checksums
There are three checksums that need to align for save corruption to work here. The initial Pokemon used must have a checksum of 0xED5A. The Larvitar produced from save corruption must also happen to result in a checksum of 0xED5A (as you cannot grab items off Bad Eggs due to them being Eggs). Finally, the resulting box save CRC16 must compute to 0x5B30 (what an empty box save ends up having).
The first checksum is fairly easy to obtain, given variety of data able to be easily controlled (TID/SID, IVs, met date, OT name). The second one just needs a little bit of this data controlled, as only part of the real Pokemon is saved to create this Larvitar (in this case, only the OT name and met year matter). The final one would mostly depend on the same data the second checksum depends on, along with the PID.
However, a CRC16 isn't just a sum of all data, rather the way it ends up working means simply swapping the order of data can result in different checksums. This particularly matters for the OT name, where there are billions of possible OT names which satisfy the first and second checksums if all 5 chars end up being used, thus guaranteeing a way to collide the CRC16. Ideally, less chars are used, and all of them coming from the first char menu. This TAS ends up using a 3 char name where all chars are on the first char menu.
Tweaking and Voiding
With a Bike, a glitch known as tweaking can be performed. Tweaking can proceed to be used to enter the void, allowing for voiding.
Voiding is used twice in this run: once to get to Jubilife TV, and another to get to the Hall of Fame. The first case is done as it's the only way to skip the tutorial section north of Sandgem. The Hall of Fame voiding needs to done with the Poketch Co building in Jubilife.
Ledge Cancelling
Another glitch known as ledge cancelling is also used. Ledge cancelling involves simply pressing X to open the menu while jumping a ledge, possible in the void where a ledge jump can occur after stopping (you can open the menu as long as you have "stopped"). Normally, there would be no way to actually enter into Jubilife TV from the void. Ledge cancelling allows for bypassing some collision, resulting in being able to enter the building from the outside. Ledge cancelling requires specific ASLR to manipulate a ledge within the void. Manipulating ASLR is actually very similar to Gen 5 initial seeding, and ends up being very easy for a TAS to manipulate as there are only 65 possible ASLR options and 1024 button input combos can be used to manipulate ASLR.
The Actual Run
Initial Seed Manip
RNG is seeded using the following formula:
0xAABBCCCC
AA = Day * Month + Minute + Second
BB = Hour
CCCC = 2 last digits of the year + vblanks since reset
This run uses October 5, 2054 at 9:29:59 AM, with 4039 vblanks since reset. This results in initial seed 0x8A090FFD.
Starter
The starter chosen is Piplup. Piplup has the shortest animation, resulting in the fastest Starly fight and fastest wild Pokemon catch.
Starly Fight
The Starly fight is fairly simple. Piplup has a high attack stat, resulting in being able to 2HKO the Starly even after it uses Growl, resulting in the fastest fight.
Poke Balls
With Poke Balls, another Pokemon can be caught.
A second Pokemon needs to be obtained in order to use the PC. This Pokemon can be any Pokemon, but Bidoof has the shortest cry so it ends up being caught.
Sandgem to Jubilife
The first tweak done puts me in the void from one of the Sandgem houses. With a specific path and a ledge cancel, Jubilife TV can be entered, resulting in exiting to Jubilife.
Jubilife to Hall of Fame
The second tweak done puts me in the void from the Poketech Co building. With a specific path, the Hall of Fame can be entered, beating the game.
Darkman425: It's fun for me to read about how even with more sophisticated game development tools there's ways to still break games in ways the developers tried to but failed to safeguard against, albeit with this specific revision. That information also helped explain what's going on and what specific bits needed to be manipulated to make this possible. Nice work!
Accepting to Standard as a new branch named "save glitch" since it's significantly different from the other publications.
