..or at least some advice, I can see quite a lot of people involved with Sonic submissions.
As some of you may have noticed, I'm currently working on Genesis console verifications. There are games that work right away, there are others that desync even before menus, and there are some that work after some input tweaking. Sonic games are in that third category.
I've started with Sonic1, and it was not syncing, but was quite easy to fix. The reason for those desyncs seems to be that Gens does poor job emulating latencies caused by loading graphics into VRAM, so things go a bit faster in Gens then in the real thing. This could be fixed by inserting some no-input frames in level transitions; main symptom was that Sonic would not do his initial jump at the start of some level (all movement is started with jump in S1 to gain speed, as there is no spindash). So the process was to just insert a frame, play and see how it goes, which was a bit time consuming at the end of the game as there is no easy way to speed up real hardware or load a savestate (had to play the same thing full length each time), but I did it in background while doing other stuff at the PC, so it was ok and worked out in the end.
Now with later Sonic games starting from 2, things become more complicated. The games tend to load stuff mid-level a lot more, and that is causing desyncs, but they can be solved the same old way. However those extra frames seem to affect some objects in the game, like moving platforms, which end up in different positions. See screenshots below:
GhostSonic had the same issue with Sonic2, I'm also getting desync at the same place, haven't checked in detail but it's probably also the same issue.
http://tasvideos.org/forum/viewtopic.php?p=366845#366845
So the question is, if parts of the movie with different object phases would be redone, would remaining levels of the game sync? To put it differently, are Sonic games sensitive to early input changes so much that later levels would desync?
If later levels can be synced, would anyone be willing to rework sections of the movie with different object placements? I think it should be possible to recreate object phases in the emulator by keeping Sonic idle for some frames at the start of the level. I'd just need some TASer to re-run those sections in an optimal way (I have no experience in that area to do it myself), while I could provide a gmv (or bkm) which runs upto the object so it's in hardware-compatible phase.
I've talked about this in the IRC before, but I've found you can sort-of get around the desyncs in Sonic 2 by pausing the game until the universal timer (which controls the flippers on Chemical Plant Zone, amongst various other events) resyncs to what it was on emu. I believe the additional lag introduced on console causes the timer to differ than what it was on emulator, thus when it reaches the boss on CP Act 2, the panels are in a completely different position than what they should be.
I tried S3&K on my setup once, never could get very far there.
You're a hell of a man delving into the Genesis like this.
Well the period of moving platform from my screenshot is 6 seconds or so, I'd have to pause the game (or have Sonic loiter) for around this time which is not going to be fun to watch. That's why I'm hoping I can find someone willing to re-run those sections. This of course assumes remaining movie could still be synced, I'm hoping someone experienced with Sonic runs can comment about that.
As for Sonic 2, if the period is shorter, maybe the pauses could be carefully spread out to not be that noticeable, but I don't know..
I am not remotely a Sonic expert, but I dimly recall hearing that the games are very sync-stable, i.e. changes made in one level should have no impact on actions in a later level. So (assuming I remembered correctly) you shouldn't need to redo the entire run every time a level is fixed.
Pyrel - an open-source rewrite of the Angband roguelike game in Python.
FYI, Sonic 2 actually does not generally load any more stuff mid-level than Sonic 1 does; the exception being Tails and his tails, and some levels having more animated tiles than S1 levels do (but this is always uncompressed art which is just transferred through DMA). S3 and S&K actually load a load more stuff (including decompressing art) a lot of time. Both of the decompression methods used in S3 and S&K are designed to work across multiple frames, but one of them can cause lag on occasion (this one is also used in S1 and S2 to load boss art, capsule and animal art, score tally art, and some art at the start of the level). Either of them can cause some drifting of computation because of something important: not all instruction latencies are correctly emulated by Gens (or Regen, or Kega) because they are based on official Motorola manuals that have since been proven incorrect for some instructions. This can make the resulting DMA transfers to be off by one or more frames, which can further throw off the timing of future decompression and DMAs.
Moreover, Gens also does not emulate other things that are important for timing, such as bus contention (for example, with the z80) -- and even if it did, it would probably be off, as the exact timings are still unknown (but under research). Kega does not fare not much better as far as the z80 goes, because it does not emulate bus requests (which, ironically, Gens does), and this is disastrously bad for S3 and S&K runs because of the way that the driver works.
But that is not all bad news -- the Sonic games tend to be resilient to these timing innacuracies. This is because each zone (S3 and S&K) or act (S1 or S2) is generally independent of each other, with very little information carrying on from one zone/act into subsequent zones/acts. The games also not only tends to use little in the way of randomness, going as far as resetting the PRNG seed at the start of each act (zone in S3 and S&K). Each level has its own unique "universal timer", which is the number of non-lag frames that elapsed from the start of the zone/act, and a set of oscillating numbers that, well, oscillate from the start of the zone/act. And all of this is reset when a zone/act starts, so only in-level delays affect them.
There are a few things that do carry over, and they are annoying. One example are the swinging vines in Angel Island in S3: they run based on the number of frames spent on levels -- any levels -- and is only reset when you do a hard reset.
The other main thing that carries on is what is incorrectly called the "universal timer": the vertical interrupt handler run counter. This counter is incremented whenever the vertical interrupt handler is called, which is once per frame; but this does not happen when display is disabled (for example, because there is a lot of data that is going to be transferred to VRAM by DMA, generally only at the start of levels or when changing game modes) or because interrupts in general, or v-int in particular, are disabled (the latter never happens in unhacked Sonic games after initial boot). In a way, it is the number of frames since power on, except for those frames (lag and non-lag) that were never generated because the VDP wasn't generating the v-int signal.
This counter is used as a weak source of entropy:
Its low bits are used to determine if an event will occur: these event may or may not use the full PRNG for details. Examples: signpost sparkles in S3/S&K (uses PRNG), explosions from bosses in all games (uses PRNG), animals coming from capsules (uses PRNG), starting state of Chemical Plant tipping pipe sections (does not use PRNG), starting state of Wing Fortress platforms (does not use PRNG), and many others.
Directly as a source of entropy: in Sky Sanctuary, the boss reseeds the PRNG using the v-int run counter directly.
(1) is bad because it can throw off the PRNG, and will complicate luck manipulation; but it is not that bad because you generally have to waste less time in pauses to fix the issue (with the exception of the Wing Fortress platforms, where you may have to wait up to 240 frames). (2) is exceptionally bad because you need to either infer what value it currently has, or use progressively longer pauses until you have a sequence of PRNG values that match the result of the TAS.
Other than different timings leading to possible additional lag frames, the v-int run counter is the major source of desynchs in S2, S3 and S&K. It is very important in S1 too, but S1 engine has a lot of additional lag to deal with which make it arguably as important as the v-int run counter.
But where the bad news come is here: that platform is based on the oscillating numbers I mentioned. They do not run when the game is paused. They should be cycling once per (non-lag) frame from the first frame of the zone until the current moment: the function is called only in one place in the entirety of S3&K code, in the main level loop, and before the loop, the numbers are reset to a default state. The function that resets the state of these numbers is also only ever called once in the entirety of S3&K. The function that updates them is called right before the function that updates the ring animation frame, which means they both got called at a different frequency on console than they did on emulator (whether they were more or less frequently called is hard to say with a single screenshot). I am at a loss to explain this difference in frequency, considering these functions should have been called once per non-lag frame.
Hm. How many waiting frames (if any) did you have to add to the flame transition?
Thanks for all the info..
When you say "uses PRNG", do you mean it derives some larger values from those low bits only, or does it also mix in some previous RNG state?
This looks like a showstopper to me. By the time we arrive at Sky Sanctuary (assuming it's possible at all) this "universal timer" will have a lot larger value in console than in Gens, so it's basically impossible to get the value needed by TAS here.
What made you arrive at this conclusion? Both ring animation phase and the platform differ from emulation, but they still may be in sync with each other. Or do I miss something?
2 frames had to be inserted before input starts around Knuckles emerald cutscene.
13 frames inserted before spindash after flame transition to avoid the spindash become useless because of screen lock.
Note that this only counts frames where the game reads inputs, as it's the only thing that microcontroller uses to advance frames.
I've tried messing with Sonic 2 and it seems to be a lost cause too. Inserting frames at the start of the level to attempt to manipulate boss panels affects all other objects too, so the move desyncs earlier.
Here is how the desync looks when there are no input modifications:
Link to video
It seems Sonic jumps a bit too early here, which I don't understand why as even if there was lag earlier in the level, amount of frames between start of that spindash and jump should be the same..
If I delay the jump Sonic hits the boss, but he falls to his death at the left corner because the floor is open (unlike in Gens).
Hm, the draft version of this section was more clear on this point; I wonder what happened to it? Anyway: it is generally used as a "gate": if the low bits fit a given pattern, then an action will be performed which uses the PRNG. A concrete example: animals in S2 capsules are generated when the low 3 bits of the v-int routine are zero; when this happens, the PRNG is called to generate a random number to decide where the animal is going to be placed horizontally (relative to center of capsule) and the PRNG is called again to determine which kind of animal is going to be generated. So a more correct description would be "triggers use of the PRNG".
This one is actually not that bad; the TAS only needs two values for the boss fight, and they are quite easy to get if you pause long enough before the PRNG is seeded. Several versions of the TAS were done by pasting the input from earlier versions of the TAS and adding pauses through TASEditor. The only issue is that you would need to play the entire run from the beginning for each attempt, unless you found a way to obtain the value of the v-int run counter at the start of the level.
notaz wrote:
marzojr wrote:
The function that updates them is called right before the function that updates the ring animation frame, which means they both got called at a different frequency on console than they did on emulator
What made you arrive at this conclusion? Both ring animation phase and the platform differ from emulation, but they still may be in sync with each other. Or do I miss something?
You didn't miss anything, what I wrote was unclear; you are right, the ring animations and the platform are indeed in synch. What I was going on about was that they were apparently emulation was out of synch with console, which didn't make sense: based on the evidence given by the screenshots, there are at least 9 frames of difference between console and emulator in those two screenshots: rings have 4 different sprites that are cycled through, and each sprite lasts for 8 frames; so 9 frames in the best case scenario, but potentially up to 23 frames to go from ring sprite in emulator to console. This is why I asked this question:
marzojr wrote:
Hm. How many waiting frames (if any) did you have to add to the flame transition?
Your response was mostly something I expected:
notaz wrote:
2 frames had to be inserted before input starts around Knuckles emerald cutscene.
13 frames inserted before spindash after flame transition to avoid the spindash become useless because of screen lock.
There is the difference in platform (and ring) positions between emulator and console: 15 frames is almost in the middle of the 9 to 23 interval. And I asked specifically about the transition because I figured out that staying as quiet as possible in it caused up to 20 frames of difference because the game is loading act 2 there. I wasn't expecting the 2 frames before the emerald cutscene, though, but they also make sense -- there is some art loading nearby too.
I need to check a few things, but it may potentially be possible add these frames as pauses -- looking at the code, it may be possible to regain the time lost to these added frames by using pauses to let stuff decompress while the game is paused (but I need to check whether or not it will help at all... only one of the decompression methods allow decompression during pauses, and I think that the other one is used to detect when the transintion is complete, so it might make no difference).
marzojr wrote:
Note that this only counts frames where the game reads inputs, as it's the only thing that microcontroller uses to advance frames.
Unfortunately, those are exactly the kinds of frames that are causing a problem for this platform. For what is worth, the next iteration of the TAS will not use this platform...
Edit: this came through as I wrote the reply:
notaz wrote:
I've tried messing with Sonic 2 and it seems to be a lost cause too. Inserting frames at the start of the level to attempt to manipulate boss panels affects all other objects too, so the move desyncs earlier.
You want to insert frames in the form of a pauses before the 17th second: specifically, before the tipping floor objects are loaded. The number of frames in pauses is more annoying: it can be up to 255 frames... yes, those platforms are one of those annoying things in S2. It may be possible to reduce the amount of work in this case if the approximate difference in v-int run count between emulator and console can be inferred; it may be possible to do that from the video, actually, but I'd need a 60 fps video to compare frame by frame with the run.
I have almost zero experience with Sega emulators, but have you tried Exodus? If it has savestates and a movie feature it might be more useful than Gens and faster than trying the movies on the console.
Exodus is not open source yet, and while it is the most accurate emulator for Genesis, it is not perfect yet, specially the released version: it demonstrably has incorrect timings on several instances (for example, waterlines on Angel Island on S3).
One would also need to figure out the movie format it uses, although I suspect it should be relatively easy.
Joined: 4/17/2010
Posts: 11477
Location: Lake Chargoggagoggmanchauggagoggchaubunagungamaugg
Is gpgx so far off to test on?
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
You want to insert frames in the form of a pauses before the 17th second: specifically, before the tipping floor objects are loaded.
I'll try that..
marzojr wrote:
It may be possible to reduce the amount of work in this case if the approximate difference in v-int run count between emulator and console can be inferred;
Maybe we could use a ROM hack for this, replace ring counter with that, or something. Different drawing routine may affect timing but maybe not too much and it would be enough for debug.. What is the RAM address of that counter?
marzojr wrote:
it may be possible to do that from the video, actually, but I'd need a 60 fps video to compare frame by frame with the run.
Unfortunately I can only do 30fps video..
marzojr wrote:
For what is worth, the next iteration of the TAS will not use this platform...
What would also help is getting rid of excessive input. For example at the Knuckles emerald cutscene, Tails' left input starts way before game accepts input, which got me confused as inserting 10 frames (for example) before input starts made Tails walk left too much and I got even different desyncs. I usually add more frames for a test, and reduce until minimum needed is found, but it didn't work here until I replaced all excessive frames (33) with no-input.
feos wrote:
Is gpgx so far off to test on?
It's somewhere in between of Gens and real hardware. Seems to be closer to Gens according to floor tiles and almost kills CPZ boss in S2, but also desyncs right there.
You want to insert frames in the form of a pauses before the 17th second: specifically, before the tipping floor objects are loaded.
I'll try that..
OK it worked, needed 203 frame long pause. Next desync - Aquatic Ruin 2 animal manipulations, are those hard to fix up? I could instead resync at Casino Night 1 start I guess, like I used to do in S1..
notaz wrote:
marzojr wrote:
It may be possible to reduce the amount of work in this case if the approximate difference in v-int run count between emulator and console can be inferred;
Maybe we could use a ROM hack for this, replace ring counter with that, or something. Different drawing routine may affect timing but maybe not too much and it would be enough for debug.. What is the RAM address of that counter?
I've tried toying with this too, and it seems S2 is optimized to not redraw score/rings until they change, so replacing them is not that useful. I've tried to force redrawing every frame, and that caused things to drift and desync, as expected..
Maybe we could use a ROM hack for this, replace ring counter with that, or something. Different drawing routine may affect timing but maybe not too much and it would be enough for debug.. What is the RAM address of that counter?
The full counter is at $FFFFFE0C; the game generally uses only the byte at $FFFFFE0F for the purposes I mentioned, and generally masks it with a small number one less than a power of two (1, 3, 7, 15, ...).
notaz wrote:
Unfortunately I can only do 30fps video..
To be fair, the 30fps one would allow deducing it within a frame in this case, so I may have been exaggerating...
notaz wrote:
What would also help is getting rid of excessive input. For example at the Knuckles emerald cutscene, Tails' left input starts way before game accepts input, which got me confused as inserting 10 frames (for example) before input starts made Tails walk left too much and I got even different desyncs. I usually add more frames for a test, and reduce until minimum needed is found, but it didn't work here until I replaced all excessive frames (33) with no-input.
That has been reduced too, to "only" about 10 frames in that case; I suppose I could go around removing needless input, but it will delay the TAS even further.
notaz wrote:
feos wrote:
Is gpgx so far off to test on?
It's somewhere in between of Gens and real hardware. Seems to be closer to Gens according to floor tiles and almost kills CPZ boss in S2, but also desyncs right there.
That is interesting to know; I had very little information about gpgx accuracy other than "better than Gens".
notaz wrote:
OK it worked, needed 203 frame long pause. Next desync - Aquatic Ruin 2 animal manipulations, are those hard to fix up? I could instead resync at Casino Night 1 start I guess, like I used to do in S1..
In both S1 and S2, you can fix the manipulation by adding some frames in pauses before the boss starts exploding; specifically, either 3 to 7 frames or 9 to 10 frames in pauses. Both assume that the frames where start is pressed to pause and unpause were added along with blank spaces between them; if you edit frames to turn them into pause/unpause, you need to add from 1 to 7 frames. If nothing else differs (a big if in S1, especially in Labyrinth zone -- lag is liable to be different), the animal manipulation should work once you add the required amount of pauses. If something does differ, and the animal manipulation desynchs after a point, you can add or remove delays right before the desynch point (preferably during an already present pause).
If you are curious: it has to be before the boss starts exploding, as the explosions also use the PRNG for positioning, and they also depend on the v-int run counter in the same way that the animals do.
notaz wrote:
I've tried toying with this too, and it seems S2 is optimized to not redraw score/rings until they change, so replacing them is not that useful. I've tried to force redrawing every frame, and that caused things to drift and desync, as expected..
Considering that only the last byte of the v-int run counter will be relevant in the vast majority of cases, the last byte of the v-int run counter can be displayed in hex this method:
editing the HUD's sprite mappings to have 256 frames, one for each possible hex value; the v-int run counter is used as an index, and the mappings can be done so as to have the same number of sprite pieces as the original mappings, with the same size (to make the VDP happy too, though I doubt it would affect synch).
disable blinking of rings and timer by removing the code responsible.
replace the score/ring/time text with hex digits, so the art does not have to be uploaded to the VDP every frame.
optimize the code so it takes the same number of cycles as the original code, adding dummy code as needed (the differences caused by rings/timer blinking need to be accounted for, but the TAS never comes near enough a time over for the latter to matter).
This adds a lot of data, but this additional data will not cause any additional delays; it can actually be generated through macros in the disassembly. Done this way, nothing will be sent to the VDP which wasn't sent already, the VDP will have to draw the same amount of stuff, and the code will take the same time to run on the 68k side. Basically, it can be done without adding any delays to the game other than things that aren't at all emulated (such as bus contention, memory refresh cycles, etc.). Just give me a few days (as I have finals all over the next week) and I will write this.
I tried the simplest hack for S3K which replaces score with VInt counter (so only updated when score changes), and this is the result:
So the difference seems to be 31.
(also it seems my camera scans bottom-to-top, what's up with that?)
Here is how the desync looks when there are no input modifications:
Link to video
Looks identical to my old video.
Link to video
The link to Exodus is dead. Last I remember, the dev kept debating on how to license his code if he open-sourced.
I tried the simplest hack for S3K which replaces score with VInt counter (so only updated when score changes), and this is the result:
Interesting idea.
notaz wrote:
So the difference seems to be 31.
I take it that this is including the 2+13=15 frames you added? If so, those screenshots would indicate 16 more lag frames than what Gens emulates. Straining your good will, could you check where those frames come from? I would hope it is from before the level starts...
And by the way, these 16 lag frames don't affect the moving platform, vines or ring animations, so this is just plain old curiosity.
notaz wrote:
(also it seems my camera scans bottom-to-top, what's up with that?)
It is not that it scans bottom-to-top; rather, the camera and TV/genesis are neither exactly synchronized nor are likely to be using the same frequency. In this case, your camera likely records at either 30 or 60 fps, while the TV/Genesis is outputting at 60/1.001 fps; thus, your camera is taking snapshots progressively earlier in relation to the TV output, with the effect that you get a rolling dark band going up.
I take it that this is including the 2+13=15 frames you added? If so, those screenshots would indicate 16 more lag frames than what Gens emulates. Straining your good will, could you check where those frames come from? I would hope it is from before the level starts...
Yeah it's with all frames, otherwise the movie would not sync.
So the first time score counter shows it's 7 lag + 2 mine, meaning others come from level transition. Can't check other points using this method as the score counter doesn't update anywhere else.
marzojr wrote:
And by the way, these 16 lag frames don't affect the moving platform, vines or ring animations, so this is just plain old curiosity.
Hmm you mentioned vines using VInt counter before, is that S3 only and not S&K?
I'm having problems with Aquatic Ruin 2 desync, there are 2 pauses before boss fight starts, inserting a frame to any of them, or delaying the pauses themselves doesn't seem to help. Here is how it desyncs, and I couldn't spot any difference after frame inserts:
Link to video
and I wonder what the reason for delaying the last hit?
Joined: 10/6/2011
Posts: 1697
Location: RU · ID · AM
notaz wrote:
and I wonder what the reason for delaying the last hit?
To glitch the boss. I beleive it is the most frequently asked question on all Sonic 2 runs.
If you give the last hit as early as possible, you will then loose time seeing the boss exploding. So, in your attempt the hit is given at exactly proper moment, then happens something weird and the animals glitchimals desync…
S3&A [Amy amy%] improvement (with Evil_3D & kaan55) — currently in SPZ2
my TAS channel · If I ever come into your dream, I’ll be riding an eggship :)
Hmm you mentioned vines using VInt counter before, is that S3 only and not S&K?
The vines in Angel Island actually get updated by the same function that updates the ring animation; so whenever you are in a level and not in a lag frame. The issue with them is that unlike some other things (the oscillating numbers platforms depend on, for instance), they are never reset. This is actually more important for real time speed runners, as they have to hard-reset the game between attempts or the vines will be in the "wrong" place.
notaz wrote:
I'm having problems with Aquatic Ruin 2 desync, there are 2 pauses before boss fight starts, inserting a frame to any of them, or delaying the pauses themselves doesn't seem to help. Here is how it desyncs, and I couldn't spot any difference after frame inserts:
I take you mean the animals at the end? That is a bit mea culpa: I talked all about animal generation and forgot to mention that they use the v-int run counter to determine the direction they go (it has been a while since I manipulated animals in S1 or S2...). Specifically, they use bit 4 of the v-int run counter at the time they land. Meaning you need to add potentially up to 31 frames to get the animal manipulation to synch. The good news is that the pauses to synch animal appearance can be made concurrently with this. You can probably find how many frames to get animals generating at the right time, then add 8 frames at a time until they all turn the right way.
notaz wrote:
and I wonder what the reason for delaying the last hit?
As WST said, it is to glitch the boss: due to a coding error, when you hitting on a specific frame (there are actually two distinct ones for this boss) you cause a double-update to the boss' subroutine counter, making him jump from "exploding" to "blasting away". Some other bosses also have this same coding error, but they cause crashes instead (Oil Ocean boss, Silver Sonic in Death Egg), and so does one badnik (the dinosaur-like badnik in Hill Top).
Ok, thanks for all the info. With that knowledgde I've been able to make a romhack that updates score counter (replaced by VInt counter) whenever animal goes left, which almost never happens normally, except in a few cases like Casino Night, so almost doesn't affect timing. With that it's easy to see how many frames to insert to sync animals now.
Next was Mystic Cave boss, where stone/spike spawns were wrong. Finding the boss routine in S2 disassembly (really good to have it available) and dumping VInt counter solved this again.
Some trivia: the boss dies on VInt counter value 33333 on emulator and 33680 on hardware running my hacked input file.
I'm at Oil Ocean now, need to sync more animals..
Yeah, the disassembly is really useful -- not only for TASing, but also for resynching runs, and, as it turns out, for synching it on console as well. Anyways, only 6 more days until all finals are done and I can make that über hack that displays the low byte of v-int run counter at all frames with no loss of time...
I've finally synced Sonic 2, but unfortunately it's not stable and occasionally desyncs, probably because of some reason mentioned in verification thread. There are also 2 added long pauses, one on Chemical Plant 2 and another in Wing Fortress to sync platforms, so it's rather noticeable and I'm not sure it's worth recording.
I've made Sonic hacks to play this movie and S1 one without any special hardware on real Genesis/MD and made a post at SpritesMinds with hope that some emu author will notice:
http://gendev.spritesmind.net/forum/viewtopic.php?t=1948