Experienced Forum User, Published Author, Skilled player
(1020)
Joined: 7/24/2013
Posts: 175
OmnipotentEntity wrote:
MrWint wrote:
Sadly, I found no useful indication in the game's memory that shows beating Red, all there is is one flag that determines whether Red is hidden, which is initially set, unset by beating the elite four and then again set after beating Red.
I would assume that there is a flag or several that show you've beat the Elite Four.
So game completion would simply be flag(s) set shows Elite Four beaten and flag set that Red is hidden.
You probably don't want this as game completion. Simply setting these bits using memory manipulation shouldn't count I guess, whereas skipping the elite four by manipulating Red's bit and then beating Red should. Both of which wouldn't be covered in your definition. Also, I'm not sure if you would want such a strict definition, in the end it's all for the entertainment value anyway.
Sanky wrote:
MrWint wrote:
I looked into the coin case glitch a little, it isn't the position that decides where your jump goes, but the window location of the background tile buffer. The details are somewhat technical, I probably post a more in-depth explanation in the glitch city forum thread when I have more time for it, but basically you get the jump from virtually anywhere on the map, not just New Bark, which makes the glitch far more useful and in particular would make this run faster.
Yeah, I know. I just walked around New Bark until I saw a jump opcode in wram. I wouldn't say it's virtually anywhere, it did take some seeking—around 10 minutes, but I might've been unlucky or just blind.
Also padz looked into it and found a better vector too, iirc.
I posted a description in your glitch city forum thread. It's hard to find positions by running around, since the location you are in doesn't really matter, it's the (relative) path you went that is important.
Experienced Forum User, Published Author, Skilled player
(1020)
Joined: 7/24/2013
Posts: 175
Perdu wrote:
My question was: why are you using the *last* useful value, when you could use any of them? Is it the fastest way to have these values stored in the memory?
I'm using all of the useful values. What I meant was that I'm using the last useful value for the RET instruction. this way I have all the other useful values available for manipulation. If I used an earlier value as ret, I would miss out on som manipulation potential. Also, all these instructions are executed during a single frame, so having a few instructions more or less won't change the number of frames, most of the time the CPU is waiting for VBlank (i.e. a new frame) anyway in order to continue.
feos wrote:
MrWint wrote:
I just noticed that the category of this movie was changed to "glitched". Can anybody tell me why, was there a recent change in policy (my previous movie was not marked as such)? If that is the "glitched" category, is there a "normal any%" or "glitchless" category as well, I thought the categories were "save corruption", "any%" and "151 Pokemon".
Post #355723Post #355818
Changed your previous movie to "glitched" since it heavily corrupts memory and cuts significant amounts of gameplay that can be still seen in the "any%" version.
Thank you for the information. So the categories did get changed, there is a new "any%" category now, that disallows arbitrary code execution/memory manipulation.
Experienced Forum User, Published Author, Skilled player
(1020)
Joined: 7/24/2013
Posts: 175
Regarding the actual submission, I see the same problems that where already mentioned:
The movie is not optimized
The movie does not beat all existing records
The movie does not complete the game
Game completion in GSC is commonly defined by beating Red and not by beating the Elite Four or reaching the credits, since beating Red is the last story element in the game. With glitched endings however, this is a tricky definition, we may need to define more precisely what is required. For example, would it be allowed to skip fighting Red and jump directly to the text when he lost? Would it be allowed to do it outside Mt. Silver, without even seeing Red ever? Sadly, I found no useful indication in the game's memory that shows beating Red, all there is is one flag that determines whether Red is hidden, which is initially set, unset by beating the elite four and then again set after beating Red.
Experienced Forum User, Published Author, Skilled player
(1020)
Joined: 7/24/2013
Posts: 175
I just noticed that the category of this movie was changed to "glitched". Can anybody tell me why, was there a recent change in policy (my previous movie was not marked as such)? If that is the "glitched" category, is there a "normal any%" or "glitchless" category as well, I thought the categories were "save corruption", "any%" and "151 Pokemon".
Experienced Forum User, Published Author, Skilled player
(1020)
Joined: 7/24/2013
Posts: 175
Perdu wrote:
TASVideoAgent wrote:
The EVs and DVs are the last useful information that is stored, so it needs to be the ending of the routine:
Why? Why don't you try to execute those two instructions as soon as possible? The RET will get you back where you want anyway. If I'm right (but I guess I'm not, you have thought way more than me about this), that would save you a lot of trouble. Can't you try to somehow have those values on that squirtle?
I'm not sure whether I understand your question. The data structure that represents a boxed pokemon is fixed, it is made by the programmers, I can't change where each value is stored in memory. All I can do is manipulate the stats of it in a clever way. Some values are easy to manipulate (like the DVs), others can't be manipulated easily (like the Health, it depends on the level and levelling is slow). The last easy-to-use values in this structure are EVs and DVs (see http://datacrystal.romhacking.net/wiki/Pok%C3%A9mon_Red/Blue:RAM_map for a RAM map). The RET will bring me back to where I want, correct, but there is no RET until I manipulate one. And since these are the last useful values, it's the natural place for a RET to go. Using more pokemon would allow for more manipulation potential, but is unnecessary in this case and slower.
Perdu wrote:
TASVideoAgent wrote:
The last point in the list is added because of an undesired side effect that the use of that glitch move can have, which is commonly called the "TMTRAINER effect". It is caused in the following way: Even though the move itself is perfectly stable and harmless, its name is not. There is no name for the move, so the game looks for it past the actual move name list. Where it ends up is determined by the number of end of string symbols (0x50) in the game's RAM at that point, which is highly unpredictable. This random portion of RAM (20 bytes) is then copied to the name buffer and from there to the string buffer. This last copy operation is where the problem occurs: the programmers assumed that each name always ends with an end of string sybmol, so the copy operation continues until it is found. The random move name that is picked from the memory may not have such a symbol, so the game copies far too many bytes, overwriting other important memory areas, glitching out the game and most likely crashing it.
Can't you exploit this like any regular buffer overflow? I.e., trying to erase the return value of current function in order to execute arbitrary code. How is it unpredictable? Is there really no way to predict it?
Yes, in theory you can exploit it like any buffer overflow to manipulate the RAM contents. You cannot, however change the game code. Unlike on home computers, the game's code is not loaded into RAM, but is read and executed directly from ROM. All you can do is mess with the data, which is not as useful. You still need a weak point in the ROM code to redirect the code flow into RAM areas. Also, to use it productively, you would need to know every last bit of the game's assembly, which bytes of data correspond to what and how to manipulate them. There are way easier methods of arbitrary code execution in this game.
Experienced Forum User, Published Author, Skilled player
(1020)
Joined: 7/24/2013
Posts: 175
Sanky wrote:
Hi, I'm the person who figured out and documented the coin case exploit. I still think that using it, werster's glitched speedrun can be beaten, but I haven't gotten around playing with different vectors than the one in New Bark yet (which is obviously undesirable).
I looked into the coin case glitch a little, it isn't the position that decides where your jump goes, but the window location of the background tile buffer. The details are somewhat technical, I probably post a more in-depth explanation in the glitch city forum thread when I have more time for it, but basically you get the jump from virtually anywhere on the map, not just New Bark, which makes the glitch far more useful and in particular would make this run faster.
Patashu wrote:
Ideally, with all the people interested in pokemon gold glitchhunting ATM we'll see some collaboration arise and curbstomp the game into oblivion.
I havn't had much time lately, but as I said, I'm up for improving the Gen II time as well. I'm currently working on the disassembly, as it has proven useful for Gen I and allows for better understanding of game and glitch mechanics (I found both the Inventory underflow glitch and the application of the Transform glitch by actually reading the assembly, and not by randomly playing around). Maybe something interesting comes up in Gen II as well.
Experienced Forum User, Published Author, Skilled player
(1020)
Joined: 7/24/2013
Posts: 175
I already thought about a G/S/C run as well, since there seems to be no run of it (not counting the glitchless run), but I don't know much about the glitches in this generation, yet, and they are not as well explored as in Gen I afaik. Also, it appears to be harder to gather information about it (e.g. I wasn't able to find any reliable information on the glitch used in the video you linked). There will be a lot of research to do first, but I may be up for it in the near future.
Experienced Forum User, Published Author, Skilled player
(1020)
Joined: 7/24/2013
Posts: 175
Right, visiting a Pokécenter won't work, the function that refills the PP there is programmed in a way that can handle it if the first move slot is empty, so nothing would happen.
Experienced Forum User, Published Author, Skilled player
(1020)
Joined: 7/24/2013
Posts: 175
Spikestuff wrote:
"For SGB Pokémon: Blue Version"
Shouldn't there be a field saying GB version
Yes I do know you have the file as:
"MrWint_nosavecorrupt_pokemonblue_280728.bkm (1.25MB bkm movie [GB])"
Just questioning the WIP area.
The confusion is caused by the game selection list, the game is listed as SGB, since is has SGB capabilities. These are not used, however, due to accuracy and performance issues. I can't change the name in that selection box.
Experienced Forum User, Published Author, Skilled player
(1020)
Joined: 7/24/2013
Posts: 175
Alright, I created the updated version of the movie that fixes the routing mistake. The ending is slightly different now, due to different RNG outcomes that cause different encounters, but the setup is exactly the same.
I optimized the encounters even though I saved so much time now that I need to wait before triggering the Transform glitch anyway to progress the in-game time, it would screw with the code execution part otherwise.
Here is the link for the updated movie: http://tasvideos.org/userfiles/info/8907813378020795
I will update the submission comment so it matches the new movie when they are exchanged.
Experienced Forum User, Published Author, Skilled player
(1020)
Joined: 7/24/2013
Posts: 175
mklip2001 wrote:
Spectacular! I'm glad I took your suggestion to avoid reading the submission comments before watching the run. It made all the random fights in the grass so much more WTF-worthy. I'm also amused how un-glitched your stats are when you finish at the Hall of Fame.
Great job. I just have two small questions:
(1) Usually bot-done rerecords are discounted in the total count. Do you have any estimate of how many rerecords you actually did yourself?
(2) This is more off-topic: is your name based on the James Bond villian from Diamonds are Forever? (described at https://en.wikipedia.org/wiki/Mr._Wint_and_Mr._Kidd)
EDIT: Oh, I have one small request. Your Assembly dump in the submission comments takes a whole lot of space. Could you maybe put that text in a tab so that people can choose whether or not to display it all?
ad (1): The answer is 0. Every single frame was done by a computer program, that is why only counting hand-made rerecords would be pointless in this case.
ad (2): indeed, and congratulations on finding that out, not many people have managed to figure that out, even though most did know the novel/movie.
ad EDIT: It already is in a tab that is hidden by default, it just doesn't show up in this forum post, in the actual submission comment it is fine (at least on my end).
Experienced Forum User, Published Author, Skilled player
(1020)
Joined: 7/24/2013
Posts: 175
FractalFusion wrote:
MrWint, is it possible to store Abra as the second boxed Pokemon? It doesn't seem to affect code execution, and it would shorten the hall of fame scene.
Also, how does the bot function? Does it use BizHawk's Lua, or is it something else entirely?
It is not possible to store abra as well, as this will mess up the arbitrary code execution and crash the game. The current box list in the run looks like this:
FA80:01 B2 FF LD BC,#FFB2h
which essentially does nothing (useful).
If I stored Abra, it would look like this:
FA80:02 LD (BC),A
FA81:B2 OR D
FA82:94 SUB H
FA83:FF RST $38
which would crash the game. Having only one stored Pokémon conveniently saves me from this.
The run is generated using a custom computer program (written in Java). It loads the libgambatte library using JNI and works completely independent from BizHawk (which, since it is written in C#, is not as cross-platform as I'd like). I mimic the way BizHawk handles frame timing to make the resulting movies sync. This way I have a better way of controlling the simulation, such as saving game-states into Java objects and setting breakpoints to stop the execution at specific addresses.
Experienced Forum User, Published Author, Skilled player
(1020)
Joined: 7/24/2013
Posts: 175
Toothache wrote:
I noticed in the recent Fire Red TAS, he battles the Lass on Route 3 then walks back to reset her position, avoiding fighting the bug catcher with 4 pokemon (she only has 2). Would that actually save time in this version?
The reason it is faster in FireRed is that it uses Squirtle. Charmander can OHKO the bugs with super effective attacks, which makes it faster than fighting the lass.
Also, I just noticed an obvious and quite embarrassing routing mistake on my part, I'm surprised nobody mentioned it so far. It is obviously way faster to catch Abra in route 24 instead of route 25, saving around 8-10 seconds in my estimation. No other route changes are necessary for it. I'm currently in the process of creating a movie that fixes this issue, we'll see how much it actually saves (if anything at all; I need to luck-manipulate everything from then on again and encounters can be real painful). It'll be finished maybe tomorrow.
I'm not sure about the procedure of updating the movie file, is the update small enough to simply swap the movie files, or do I need to cancel the submission and create a new one?
Experienced Forum User, Published Author, Skilled player
(1020)
Joined: 7/24/2013
Posts: 175
It actually works only in GB mode. It's the same as with my last submission on Pokémon Blue: although the game has SGB capabilities and I'd love to use them, there are no emulators that are accurate, stable or fast enough in order to create such a movie with them.
Experienced Forum User, Published Author, Skilled player
(1020)
Joined: 7/24/2013
Posts: 175
hegyak wrote:
This run also played back on Bizhawk r4878 (That's past 1.5.0).
I didn't even realize there was a new release version of BizHawk. I don't use it for anything other than verifying that I didn't screw up and the run syncs at the end. It is all based on the libgambatte core that is used, which didn't change its timing behavior in a long time (it stayed the same since at least r364, the one used in BizHawk 1.4, up to now), so it is likely to sync up with all recent BizHawk versions.
Experienced Forum User, Published Author, Skilled player
(1020)
Joined: 7/24/2013
Posts: 175
ais523 wrote:
I notice it relies on the uninitialized data for the opponent's 5th and 6th Pokémon starting at 0s. Do Game Boys actually initialize to 0s in practice, or is that an emulator issue? It might be necessary to test on a real Game Boy to find out.
It doesn't. The game itself initializes all values with zeroes upon startup (the function that does it is at ROM address 0x1f54). It is called even on soft-resets, so I have the chance to clear the data, e.g. if Squirtle's data had caused issues.
got4n wrote:
Isn't faster to cancel the charmender evolution?
I tested it briefly, and came to the conclusion that it isn't. The stat boost is too important, without the evolution I can't OHKO 3rd trainer's Ekans, 4th trainer's pidgey, and I would need an additional turn on Onix and an additional crit on Mankey. Also, the difference in stats might actually screw up the arbitrary code execution, havn't tested it, though.
Experienced Forum User, Published Author, Skilled player
(1020)
Joined: 7/24/2013
Posts: 175
got4n wrote:
More rerecords, More exciting, more fun, complete YES.
Actually, it's less rerecords than last time, I guess I knowing where you want to go helps :)
Also, impressive timing, I wonder if you had the time to read or view any of it at all before replying.
Experienced Forum User, Published Author, Skilled player
(1020)
Joined: 7/24/2013
Posts: 175
t3h Icy wrote:
Were there any obstacles with luck that you couldn't overcome and by how much do you think this run could be improved (excluding the speed trade-offs)? The run looks very optimized, but I'm curious how close it could be to perfection seeing that this was made by a program.
Contrary to what you may have read about the D-sum and how hard it is to change it, luck manipulating a sequence of events is not that hard. Even though there are two addresses used for RNG (0xFFD3 and 0xFFD4) that form the infamous D-sum, all that really matters is the value of 0xFFD3. 0xFFD4 is used only in 3 places throughout the whole game:
The high byte of the trainer ID when starting a new game
The species of Pokemon you get on a wild encounter
The initial first trash can in Vermilion Gym
Only the second point listed is interesting to the run, but when encountering a Pokemon you commonly have enough preparation time with no other RNG events to manipulate what you need.
The run itself it rather optimized in terms of luck manipulation. Most of the time there is no delay frame necessary to get the outcome I want, even max damage crits aren't that hard. The entropy you can build up by pressing buttons during text scrolling, use different paths to walk to your destination or sometimes delay certain actions by a frame suffices to always get what you want. Also, the run does not feature really rare events, the rarest event are the two encounters, the bubble miss on poliwag and catching Missingno. All of them have a chance not worse than 1/256. That is not counting the DV manipulation on the starter (which is 1/2048), but you have all the time in the world to prepare that.
It's hard to judge the amount of time lost due to luck manipulation, but I suspect it to be no more than 100 frames total compared to the theoretically best luck manipulation possible.
For comparison, my run is 979 frames faster than the currently published run up until entering Mt. Moon where the routes differ the first time. Note, however, that this is not a fair comparison in terms of luck manipulation quality, since the published run is on SGB, which adds a couple of lag frames every time an SGB packet is sent to change the palette (easily noticable whenever you cross an area transition), so the actual difference is much smaller.
Experienced Forum User, Published Author, Skilled player
(1020)
Joined: 7/24/2013
Posts: 175
FractalFusion wrote:
Looks great, and well done for the extreme optimization by bot. However, there is a rule that says that SGB mode should be used if the game supports it. It would be a shame if this run was rejected based on the rule.
I was aware of the rule you are referring to by the time I chose to use Gambatte, and I tried to explain my reasoning in the submission comments. If it is really going to be an issue, I may be able to rerun the program to create a VBA version that will be on SGB. Using bsnes on the other hand is much too slow to reach this level of luck manipulation, and I rather want to have a well optimized TAS than pretty colors ;)
FractalFusion wrote:
MrWint wrote:
I manipulate the Attack, Special and Speed DVs to be all 15, which is the best they can get.
MrWint, I checked, and Charmander's IVs are F0FE (AtkIV=15, DefIV=0, SpdIV=15, SpcIV=14), so Special IV isn't 15.
You are right! *shame*
The required DVs changed that often with each route change that I obviously forgot about what the requirements were at the end. The Special DV is indeed only needed to be 14, and it turned out to be what I got in the actual run that was produced. I didn't even think of double-checking that, I checked the low trainer ID I got and the Special stat of the Drowzee encounter (both of which have different possible values), but I never checked the DVs. I will edit the submission comment to fix that.
Experienced Forum User, Published Author, Skilled player
(1020)
Joined: 7/24/2013
Posts: 175
I see that the audio samples are the most accurate way of timing a movie and video frames should be kept at regular intervals, but that does not mean the recorded input sequence needs to have equal-distance steps as well. What lsnes currently does is well-suited for creating accurately timed video files, but (and that's my hypothesis) not as well suited for creating accurate input recordings.
You would need to separate video frames from input frames. When creating a video file from an input file, there would be no one-to-one translation of input frames to video frames then. You would basically sample the recording at your desired framerate (based on the number of audio frames that have passed) and use whatever video frame is the most recent for each sampling point in the created video file to make it equidistant even though the input sequence is not. The run-time of a movie is then not only determined by the number of input frames it has, but also by the length of each input frame. BizHawk is indeed not accurate in that regard, as it treats all input frames to have the same length, even though some of them are shorter. On the other hand, the number of "short" frames in a movie is as most as large as the number of times a game messes with the LCD, which is arguably negligible in most instances.
Ilari wrote:
I don't think GB can generate video frames faster than 35112 cycles per frame (but it can generate those slower).
You are probably right about that, the only instance I've seen a video frame being shorter than 35112 audio samples is the very first frame at GB startup, and I'm not too sure about why that is. The reason short input frames occur in e.g. BizHawk is that input frames are capped at 35112 audio samples, and if a longer frame occurs, it is separated into several full frames and one short frame that syncs it up with the VBlanks again.
Warepire wrote:
Except the game boy offered an alternative to polled input which is interrupt driven input, for a game with interrupt driven input that would simply not work as there will never be a "poll" to catch.
You're right, I forgot about that possibility. All of this discussion is only valid for polled input, as interrupt-driven input can't be accurately recorded in any emulator that I'm aware of. It can happen anywhere and would require subframe input capabilites accurate to the CPU cycle.
Experienced Forum User, Published Author, Skilled player
(1020)
Joined: 7/24/2013
Posts: 175
got4n wrote:
didn't seen movie but rerecords seems to be boosted because
65 013 101rerecords is impossible for 133k frames
The number of rerecords is accurate. It required around 80h of computing time to create, resulting in around 200 rerecords per second. These many rerecords were probably not necessary to achieve the level of luck manipulation this run has if it was made by hand.
Experienced Forum User, Published Author, Skilled player
(1020)
Joined: 7/24/2013
Posts: 175
I guess I didn't do a good job in explaining what problem I see with the way lsnes handles input frames, I'll try to give an example:
I recently worked on a TAS of Pokemon Blue (see here), and I wanted to create a computer program to search for favourable RNG outcomes. Pokemon Blue reads inputs exactly once every frame during the VBlank interrupt and stores it in memory. Suppose there is a text box and I want to close it on the first frame possible. I run the emulation until I recognize the assembly operations that evaluate the input to determine closing the text box.
If the frames are aligned at the VBlanks (like in BizHawk), I know exactly that the frame this operation occurs in is the frame I need to press the button to close it. If the frames are misaligned like in lsnes, that doesn't work: depending on whether the operation is in the first or the second half of the frame (sepated by the VBlank), the relevant input frame can be either that frame or the frame before, which is highly undesirable.
To sum up, what happens is that the input of the last frame also influences the next frame, each input is valid for the second half of the frame it is pressed in and the first half of the next frame.
creaothceann is right in that it would be most accurate to record buttons not once a frame, but every time it is read by the game. Since this is during VBlank in the games I know (which aren't many, a more elaborate evaluation might be needed to confirm the hypothesis), separating the frames by VBlanks seems to the a more accurate way to handle frame timing. Making every video frame the same length is less important in my opinion, since it isn't the case on the actual console, either.
Experienced Forum User, Published Author, Skilled player
(1020)
Joined: 7/24/2013
Posts: 175
This section of the forums is the closest I got finding the right place for this topic. Since lsnes is mainly an SNES emulator and this is supposed to be the section for it, I posted it here and not in the GBx section, even though it's about the libgambatte core.
While experimenting with different GBx emulators, I noticed that the frame timing of lsnes is way different from most other emulators, even those who are using the same core (libgambatte). I found the reason in the way lsnes is handling a frame advance:
In lsnes, a frame is forced to be exactly 35112 audio samples long, by running gambatte::GB.runFor in a loop until it is reached (see http://repo.or.cz/w/lsnes.git/blob/refs/heads/master:/src/emulation/gambatte/core.cpp)
In comparison, the Gambatte (SDL) emulator recognizes the VBlank (indicated by the return value) to determine when a video frame has passed. (see https://github.com/sinamas/gambatte/blob/master/gambatte_sdl/src/gambatte_sdl.cpp)
BizHawk is doing something similar to Gambatte, but it ignores the return value under the assumption that a frame can be at most 35112 audio samples long, so a single call of gambatte::GB.runFor ensures to reach the VBlank. (see http://code.google.com/p/bizhawk/source/browse/trunk/BizHawk.Emulation/Consoles/Nintendo/Gameboy/Gambatte.cs)
While both BizHawk and lsnes provide consistent frame timing this way (in the sense of being predictable and reproducable), lsnes timing is completely unrelated to VBlanks. It is true that a video frame is produced every 35112 audio samples as described by the Gambatte documentation, but the ability of GB programs to turn off the LCD messes the count up, making the number of audio samples inbetween video frames longer or shorter. I'm by no means an expert in knowing the GB architecture, but that is how it works afaik.
This causes lsnes to become out of sync with the actual video frames: An lsnes frame then consists of two halves of video frames, having the VBlank somewhere in the middle of the frame. Since the LCD is turned off especially in the initial loading phase of the GB cart, it is out of sync basically all the time.
So what is wrong about being out of sync this way? It largely depends on the method the game uses to read joypad data. If it does it at VBlank (like most games do afaik), the input frames on lsnes can be off. It may even be possible to miss out on input possibilities because of this, when two inputs happen to be in the same lsnes frame, even though they are in different video frames. On the other hand, you might be able to issue inputs that aren't possible on other emulators (they are always possible on console), if it happens the other way round, having two inputs that are in the same video frame separated by the lsnes frames.
I suspect it to be unintentional, i.e., a bug. Changing this woud desync all GB movies, though, and it's not really broken as it is now, just... unintuitive. I guess the acutal question I'm trying to ask here is: what to do about it?
