I wanted to make this thread discussing the requirements the site has for considering TASes as console verified. (Please note that I am writing this on my phone and wont include links for now, but once I’m at my computer I’ll add some).
Recently, me and TiKevin83 got Super Mario Advance 4 warps to console verify. However there was a bit of adjustments we had to do in order to get it to sync. One of the changes was having to remove the save data creation screen as the time it takes to create save data is non deterministic, which cut about 10-11 seconds from the run. In addition to that, 10 frames had to be added in total in places where loading requires an extra lag frame compared to emulator. So while these adjustments made the run sync on console, we wondered if it could be rejected given the hurdles we had to jump over (Which is unlikely to happen, however it’s hard to tell unless there’s some actual written guidelines to it.)
So I want to discuss what these requirements should be, and see if there’s any objection to anything.
So there are 3 types of verifications I’d like to use as cases:
1. Movies that sync on console without any changes in input.
2. Movies that sync with the same input, but might need a few frames added or deleted to work
3. Movies that were on older emulation but resynced to newest emulation in order to verify.
So first up, case number 1. It’s pretty much impossible for anyone to argue that this case should be excluded. It’s a perfect example of the spirit of console verification, and is the best case scenario for what we would want.
Second up, case number 2. This is where we start to notice issues with emulation not being exact with console. Emulation is not likely to be 100% accurate to console every time, and can usually depend on the game being tested. For Sonic Advance, 1 lag frame was added in order to get the run to sync. A small sacrifice which doesn’t affect anything significant in the input file. In the case of SMA4, 10 frames had to be added in some loading areas to get it to sync, and about 500 frames were removed from the movie file so save data creation did not have to be an issue. I think in these cases, it would still count to include them as console verified, since the adjustments that had to be done were not relevant to the gameplay, but rather just loading. These cases should also have documentation on how to get them to sync, likely in the submission thread.
Then we have the 3rd case. This is where things start to get very sketchy. There are a couple of movies that use older inaccurate emulation that were adjusted to work on console, like Super Mario Land and Trip World. For Trip World, I was able to get the movie to play on GBC in GBA mode, which is required to get GB/C games to console verify. I then added frames to any screen transition which would be an average of about 1-2 frames each. There was no lag during gameplay that I had to adjust, so it was a seamless sync.
However we have a case like Super Mario Land that needs some high hurdles to get to work. For one, bonus levels have to be adjusted in order to get fire flower after each world in the bonus rooms. These are based on what frame you finish the world on, and sometimes require wasting them via pausing and simmilar. The other issue is lag during gameplay. There might be cases where you have to change the inputs slightly in order to keep optimal lag reduction, and when that happens, it breaks the spirit of a true console verification. Since you aren’t testing to see if specific inputs will work on console, and have to make these adjustments in order to work, it might disqualify the verification.
What I suggest is to make the requirements more lenient, due to the nature of emulation and console not always being deterministic or accurate, while at the same time knowing where to draw the line when gameplay inputs have to be changed. I would like to hear some other thoughts from people, especially staff, to see what they think about verifications in general.
[14:15] <feos> WinDOES what DOSn't
12:33:44 PM <Mothrayas> "I got an oof with my game!"
Mothrayas Today at 12:22: <Colin> thank you for supporting noble causes such as my feet
MemoryTAS Today at 11:55 AM: you wouldn't know beauty if it slapped you in the face with a giant fish
[Today at 4:51 PM] Mothrayas: although if you like your own tweets that's the online equivalent of sniffing your own farts and probably tells a lot about you as a person
MemoryTAS Today at 7:01 PM: But I exert big staff energy honestly lol
Samsara Today at 1:20 PM: wouldn't ACE in a real life TAS just stand for Actually Cease Existing
Ambassador, Moderator, Site Developer, Player
(120)
Joined: 3/17/2018
Posts: 348
Location: Holland, MI
It's less even that we want to make the requirements more lenient, and more that there isn't a clear definition anywhere on the site as far as I understand of what makes a movie eligible for the verified flag, which makes the checkmark unclear given the numerous caveats that our efforts require.
Flashing carts to bizhawk expected default save data, inserting/deleting frames to match emulation inaccuracies, completely rewriting movies to work on slower, more accurate emulators, converting frame-based input logs to cycle-based logs for GBI or latch-based logs for NES/SNES over the TAStm32.
Some of these things like exporting the movie to a format that can be verified are critical to the process but actually expose deficiencies in the data in the published input log, it being tied to a specific emulator's concept of frames where the console only cares about controller latches (NES) or cycles (GB/GBC). So the verified movie actually has information that isn't in the published bk2 but only in the combination of that bk2 with the emulator used.
I would propose that for now the checkmark should be used very leniently apart from excessive gameplay modification, in lieu of future site updates to host the console exported movies like GBI movies and .r08s for NES which better tie a movie to being verified/verifiable.
I think that if a particular TAS file does not sync in the actual console without modifications, then it should not be marked as "console-verified" because, well, it isn't. If it doesn't run on the console, then it doesn't, period.
If it can be made to work on the real console by modifying it, eg. by adding or removing some input, then it should be this new TAS file that ought to be published alongside the original, and only this one should get the "console-verified" mark, because this is the TAS file that actually works on the console.
If this modified version which works on the console does not work on the emulator because of deficient emulation, then I think the one that works on the console should be considered the primary version of the TAS, and the one that works on the emulator should be considered an ancillary version of the TAS that's "fixed" to work on the emulator (at least until the emulator is fixed to emulate the console better). I think it would be a bit backwards to consider the file that works on the emulator to be the primary main version, and the one that works on the real console the secondary version, as if it were the emulator that's like the highest authority on how the original game binary should be run, rather than the original console hardware.
Ambassador, Moderator, Site Developer, Player
(120)
Joined: 3/17/2018
Posts: 348
Location: Holland, MI
The logic of that is entirely sound but ultimately with the bk2 format not being usable by GBI or the TAStm32, every movie has some amount of "modification" when console verified, if only just to timestamp the inputs in a different way. And any rule that excluded lag frame insertion would require reevaluating most of the NES/SNES/N64 checkmarks since I'm not sure if that information was even documented consistently.
I prefer strict rules regarding what movies can be considered console verified. If you have to add or subtract frames, then the movie is not console verified, only the input file is. Personally I think of console verification of a movie file as a standardized and ideally reversible process. Take your movie file, run this standard script, use the resulting input file on playback device. If you need to add additional steps in there, like manually editing in frames, or per-game hacks, then you do not have a verified movie.
In practice, this does lead to cases where a movie can be verified according to a standard process but still be emulated incorrectly, as is the case for NES games where only input latches are used, so if lag frames show up in the wrong places it doesn't matter (sometimes) since they are ignored anyway. I believe this is the case for SMB2 at least. But, since the standard process was followed, it can still be considered a verification. This doesn't work for GB/A though, where cycles and not latches are used, so cases like Sonic and Mario Advance which require manual editing, are not verified.
I do agree that there should be a process where if a movie file can be imported into a more accurate emulator where manual editing is not required for console verification then this movie can be placed alongside the original movie in the publication and be considered verified.
EDIT: also if you are able to detect emulator inaccuracies in loading times etc please report them so they can be addressed.
I prefer strict rules regarding what movies can be considered console verified. If you have to add or subtract frames, then the movie is not console verified, only the input file is. Personally I think of console verification of a movie file as a standardized and ideally reversible process. Take your movie file, run this standard script, use the resulting input file on playback device. If you need to add additional steps in there, like manually editing in frames, or per-game hacks, then you do not have a verified movie.
The problem is you can’t expect console to be 100% deterministic, in the case of SMA4, the entire save data screen had to be removed in order to work properly. The time it takes to create is not deterministic and would be impossible to get right. Not only that but if this was the requirement, then the verifications would have to be removed from plenty of other games, such as Sonic Advance, Sonic Advance 2, chip n dale 2 players and 1 player, Pokemon Yellow, anything listed here basically http://tasvideos.org/ConsoleVerificationTests.html. I dont think it should wait until emulation is 100% accurate, because like I said before, emulation will almost never be 100% accurate to console, as console has weird quirks that an emulator would not be able to emulate in a deterministic way, like the save data thing with SMA4.
“Alyosha” wrote:
In practice, this does lead to cases where a movie can be verified according to a standard process but still be emulated incorrectly, as is the case for NES games where only input latches are used, so if lag frames show up in the wrong places it doesn't matter (sometimes) since they are ignored anyway. I believe this is the case for SMB2 at least. But, since the standard process was followed, it can still be considered a verification. This doesn't work for GB/A though, where cycles and not latches are used, so cases like Sonic and Mario Advance which require manual editing, are not verified.
So you think Sonic Advance should have it removed only because it needed one lag frame added, but not say the same thing about NES? I find it really difficult to understand this position.
“Alyosha” wrote:
I do agree that there should be a process where if a movie file can be imported into a more accurate emulator where manual editing is not required for console verification then this movie can be placed alongside the original movie in the publication and be considered verified.
This I am not arguing against, as it fits the case 1 I meantioned above.
“Alyosha” wrote:
EDIT: also if you are able to detect emulator inaccuracies in loading times etc please report them so they can be addressed.
We have, and we always will. Part of the process that I mentioned in Case 2 was to ALWAYS provide documentation on what failed and what was required to get consistent sync. But the goal of console verification should not be a test of how accurate the emulation is, but rather a test of how accurate the gameplay in the TAS is. Do the inputs work on console? Do they produce the same results as on console? It just seems unfair to exclude verification if the only thing is loading time, which can be easily adjusted.
[14:15] <feos> WinDOES what DOSn't
12:33:44 PM <Mothrayas> "I got an oof with my game!"
Mothrayas Today at 12:22: <Colin> thank you for supporting noble causes such as my feet
MemoryTAS Today at 11:55 AM: you wouldn't know beauty if it slapped you in the face with a giant fish
[Today at 4:51 PM] Mothrayas: although if you like your own tweets that's the online equivalent of sniffing your own farts and probably tells a lot about you as a person
MemoryTAS Today at 7:01 PM: But I exert big staff energy honestly lol
Samsara Today at 1:20 PM: wouldn't ACE in a real life TAS just stand for Actually Cease Existing
Would it be helpful to have a tiered system? Like, one strict tier where the input as is (so including any changes necessary to get the input file suited to the NES/GB(C) shenanigans, but excluding lag frames etc.) is console verified, and one looser tier where a certain amount of lag frame edits are allowed. This looser tier would prove that gameplay-wise it is the game being abused and not the emulator used. The stricter tier would also prove any hardware tricks are absolutely legit.
This doesn't say anything about non-deterministic problems, but it might be a start. In any case, I think the rules for console verification should spring from the reasons why you would want to console verify at all, and the only reason I can think to do that is because you distrust the emulator is completely correct.
The problem is you can’t expect console to be 100% deterministic, in the case of SMA4, the entire save data screen had to be removed in order to work properly. The time it takes to create is not deterministic and would be impossible to get right. Not only that but if this was the requirement, then the verifications would have to be removed from plenty of other games, such as Sonic Advance, Sonic Advance 2, chip n dale 2 players and 1 player, Pokemon Yellow, anything listed here basically http://tasvideos.org/ConsoleVerificationTests.html. I dont think it should wait until emulation is 100% accurate, because like I said before, emulation will almost never be 100% accurate to console, as console has weird quirks that an emulator would not be able to emulate in a deterministic way, like the save data thing with SMA4.
So you think Sonic Advance should have it removed only because it needed one lag frame added, but not say the same thing about NES? I find it really difficult to understand this position.
We have, and we always will. Part of the process that I mentioned in Case 2 was to ALWAYS provide documentation on what failed and what was required to get consistent sync. But the goal of console verification should not be a test of how accurate the emulation is, but rather a test of how accurate the gameplay in the TAS is. Do the inputs work on console? Do they produce the same results as on console? It just seems unfair to exclude verification if the only thing is loading time, which can be easily adjusted.
I'd be fine with removing the flag from NES games where extra obscure steps are needed. But, I'm also fine with verification that is accomplished through a consistent process even if the underlying emulation isn't 100% correct. This is because the verification flag is attached to a publication, so it should represent what the movie (combined with the emulator it was made on) can do by themselves.
There are other ways to present console verified input that aren't attached to specific movies, maybe make a page specifically for them. But for the actual flag that gets attached to publications, little to no modifications should be allowed.
Also, loading data is an essential part of what consoles do, I consider it pretty important to get the timing of that right.
Chanoyu wrote:
Would it be helpful to have a tiered system?
At least in my opinion, the movie file is source from which the verification either works or it doesn't, so I don't think any kind of tier is needed.
For SMA4: SMB3, "loading data" is not the problem. The problem is that loading the SRAM medium is nondeterministic between cartridges. It also can degrade in performance over time, so even the same cartridge may not always exhibit the same behavior. There's unlikely to be a perfect solution, and any verification would probably need to be adjusted to the specific cartridge anyway (hence a separate SaveRAM-anchored movie).
In general, I think it would be beneficial if we could more robustly communicate the nature of the console-verification. Adding frames to a movie does demonstrate the emulation for a movie was inaccurate, and we should probably be able to attach extra movie files and explanations for what steps needed to be taken for a movie to be console-verified, and maybe some sort of asterisk on the console-verified checkmark.
But I also think there's a huge difference between being able to verify a movie by adding a few frames to adjust for certain transition screens and not being able to playback on console at all.
I think it's misleading to say a movie was "impossible to console-verify" just because the emulation was slightly inaccurate. Consider how the site judges emulation improvements in regards to movie comparisons:
Movie Rules wrote:
When comparing against a prior movie for faster time, the faster time must come from improved play in the actual game-play segments. For example, gaining time by switching to another version which loads faster, has shorter cut-scenes, or by more optimized usage of the title screen menus is not counted as an actual time improvement. A movie which doesn't have any actual in-game game-play improvements over its published predecessor will not be accepted.
If time is gained from using a more accurate emulator, but game-play hasn't been improved, such a movie will be rejected. However, if improved emulation introduces more lag, extends the cut-scenes, or slows down game-play in some way, yet the actual game-play has improved, such a movie will be considered a valid improvement.
By this metric, we clearly wouldn't consider a movie with the same gameplay made under more accurate emulation valid for another submission. Console-verification can be considered the pinnacle of accurate emulation.
It seems fundamentally inconsistent to say the slightly altered movie can't be accepted as a new submission because it's too similar to the previous movie, but also can't be considered a console-verification for the publication it derives from. In effect, what you're saying is that certain movies cannot be console-verified, not because of anything inherent to the movie itself, but because the emulation at the time the movie was created was slightly inaccurate (probably the case for most movies, except platforms where emulation is extremely refined, but even then there can be complications).
By the way, I am not saying all movies can be console-verified. For instance, movies that manipulate RNG in ways that wouldn't be possible on real-console are un-verifiable. But adding or removing a few blank frames from a movie and getting it to playback on real console has verified the gameplay, which is generally what we consider the essence of the movie in other aspects of the site.
[14:15] <feos> WinDOES what DOSn't
12:33:44 PM <Mothrayas> "I got an oof with my game!"
Mothrayas Today at 12:22: <Colin> thank you for supporting noble causes such as my feet
MemoryTAS Today at 11:55 AM: you wouldn't know beauty if it slapped you in the face with a giant fish
[Today at 4:51 PM] Mothrayas: although if you like your own tweets that's the online equivalent of sniffing your own farts and probably tells a lot about you as a person
MemoryTAS Today at 7:01 PM: But I exert big staff energy honestly lol
Samsara Today at 1:20 PM: wouldn't ACE in a real life TAS just stand for Actually Cease Existing
But I also think there's a huge difference between being able to verify a movie by adding a few frames to adjust for certain transition screens and not being able to playback on console at all.
I don't think so. If you have to modify it, then you aren't verifying the movie. Even if it's only one frame, the solution is to fix the emulator so it works correctly. I disagree with the rules here, and think the emulation environment is an important part of a TAS. I think a movie that works on a NES should be considered superior to one that only works on Not-A-NES.
In the particular case of mario advance, having a movie that creates the SRAM I think would be an ok solution to avoid non-deterministic stuff (although that sounds like something that needs a technical explanation) as there are already standards for movies that start from SRAM, but that would still leave the 10 loading frames inconsistency, which I personally don't agree with excluding and calling it verified.
But I also think there's a huge difference between being able to verify a movie by adding a few frames to adjust for certain transition screens and not being able to playback on console at all.
I don't think so. If you have to modify it, then you aren't verifying the movie. Even if it's only one frame, the solution is to fix the emulator so it works correctly. I disagree with the rules here, and think the emulation environment is an important part of a TAS.
In some sense we're talking past each other: I'm not arguing that changing a few blank frames means the original movie file in such a publication plays back accurately on real-console. That is demonstrably false.
Neither am I arguing that the emulation environment is unimportant to a TAS, as you seem to imply. That's literally the reason we're trying to do runs on console: to evaluate the accuracy of the original movie. Improving emulation is certainly a desirable end-goal. But it's not like a few frames in a movie can instantly fix emulation; it takes time, and documenting these differences is exactly how we arrive at more accurate emulation.
Alyosha wrote:
I think a movie that works on a NES should be considered superior to one that only works on Not-A-NES.
That's exactly what changing a few frames does: it creates a movie that works on an NES. I agree, that movie is superior in accuracy to the original movie it derives from. I would simply argue the gameplay between the two movie files is essentially identical.
Alyosha wrote:
[Marking SMBA4: SMB3 as console-verified] would still leave the 10 loading frames inconsistency, which I personally don't agree with excluding and calling it verified.
I completely agree. We shouldn't be sidestepping the 10-frames being changed, and it should be explicitly acknowledged.
But that leaves a question: We have gotten a movie file that represents more accurate emulation, namely the movie file that has been resynced to real-console. How do we proceed? How do we treat the original publication?
There are many possible answers to this question, and I am curious what you think:
1. Allow the movie file to replace the publication movie
2. Allow the movie file to be a new submission that could obsolete the previous publication.
3. Allow the publication to be marked as console-verified and link to an an explanation of any details (frame alterations, direct console-verified movie file [e.g. GBI format], etc.)
4. Completely ignore that we have an accurate resynced movie file for the purposes of the publication, and relegate that to a forum post on the submission discussion thread.
As I see it, 1 and 2 are unworkable unless the altered movie file syncs on emulators, as the ability to reliably playback movie files is important, and console-verification setups are less accessible to players and judges alike.
But waiting around for emulation to be "fixed" doesn't resolve this either. One reason being our movie rules don't allow movies with identical gameplay to obsolete prior movies (unless you disagree with that rule?)
If our goal is to actually improve emulation, it would be better to improve the ability to communicate the console-verification status of a movie, rather than obscure that by making console-verification completely black and white. Having the site index this is much more convenient than requiring independent efforts to keep track of "verified but somehow not verified" movies.
I basically agree with RetroEdit - ideally we'd console verify the movie itself, but 'a movie/set of instructions very similar to this one has been console verified and here it is' is also useful metadata to store and track, especially as it pertains to improving emulator accuracy in the future (you can take cases that barely don't work and analyze them to see why not)
But that leaves a question: We have gotten a movie file that represents more accurate emulation, namely the movie file that has been resynced to real-console. How do we proceed? How do we treat the original publication?
Both files can still live alongside each other in harmony. The console-verified file can be added to the Published page of links below as a new line. [BizHawk Movie for console (.bk2)]
https://i.imgur.com/vbGY17U.png
But I'm not necessarily a fan of adding the console verified flag since the actual submission itself doesn't do that.
MOD EDIT: Unembedded large image -Mothrayas
Both files can still live alongside each other in harmony. The console-verified file can be added to the Published page of links below as a new line. [BizHawk Movie for console (.bk2)]
[14:15] <feos> WinDOES what DOSn't
12:33:44 PM <Mothrayas> "I got an oof with my game!"
Mothrayas Today at 12:22: <Colin> thank you for supporting noble causes such as my feet
MemoryTAS Today at 11:55 AM: you wouldn't know beauty if it slapped you in the face with a giant fish
[Today at 4:51 PM] Mothrayas: although if you like your own tweets that's the online equivalent of sniffing your own farts and probably tells a lot about you as a person
MemoryTAS Today at 7:01 PM: But I exert big staff energy honestly lol
Samsara Today at 1:20 PM: wouldn't ACE in a real life TAS just stand for Actually Cease Existing
But that leaves a question: We have gotten a movie file that represents more accurate emulation, namely the movie file that has been resynced to real-console. How do we proceed? How do we treat the original publication?
...
But waiting around for emulation to be "fixed" doesn't resolve this either. One reason being our movie rules don't allow movies with identical gameplay to obsolete prior movies (unless you disagree with that rule?)
If our goal is to actually improve emulation, it would be better to improve the ability to communicate the console-verification status of a movie, rather than obscure that by making console-verification completely black and white. Having the site index this is much more convenient than requiring independent efforts to keep track of "verified but somehow not verified" movies.
I think a dedicated forum thread or github is a better place for movies that don't quite work then attaching them to a publication. An issue tracker is a great format for compiling these cases.
I do disagree with the rules requiring gameplay improvements, and think that a console verified version should obsolete an older de-verified one (with the caveat that authorship remains with the original author.)
There is one case I forgot to mention.
Mario Kart 64 is considered console verified, but because you cant press the reset button with TAS inputs, they had to split the movie file on those resets. So it was verified with multiple playback files, where as the emulation uses a single file.
Is it not considered verified if it’s not a single file being played back?
[14:15] <feos> WinDOES what DOSn't
12:33:44 PM <Mothrayas> "I got an oof with my game!"
Mothrayas Today at 12:22: <Colin> thank you for supporting noble causes such as my feet
MemoryTAS Today at 11:55 AM: you wouldn't know beauty if it slapped you in the face with a giant fish
[Today at 4:51 PM] Mothrayas: although if you like your own tweets that's the online equivalent of sniffing your own farts and probably tells a lot about you as a person
MemoryTAS Today at 7:01 PM: But I exert big staff energy honestly lol
Samsara Today at 1:20 PM: wouldn't ACE in a real life TAS just stand for Actually Cease Existing
There is one case I forgot to mention.
Mario Kart 64 is considered console verified, but because you cant press the reset button with TAS inputs, they had to split the movie file on those resets. So it was verified with multiple playback files, where as the emulation uses a single file.
Is it not considered verified if it’s not a single file being played back?
It's not ideal, but my opinion on this is basically that it's ok, since we aren't really concerned with time that the console spends being off. That time is invisible in terms of emulation (as long as no RTC is involved), so nothing was really added, just made convenient, as in just a series of runs strung together by SRAM. Not strictly the same, but I guess there has to be a line of practicality somewhere.
I think that console verification flag should just be that the movie more or less runs on console. Slight modifications are fine. I don't see why it should be treated as a mark of superior accuracy or whatever.
[16:36:31] <Mothrayas> I have to say this argument about robot drug usage is a lot more fun than whatever else we have been doing in the past two+ hours
[16:08:10] <BenLubar> a TAS is just the limit of a segmented speedrun as the segment length approaches zero
Joined: 11/13/2006
Posts: 2794
Location: Northern California
EZGames69 wrote:
So there are 3 types of verifications I’d like to use as cases:
1. Movies that sync on console without any changes in input.
[...]
So first up, case number 1. It’s pretty much impossible for anyone to argue that this case should be excluded. It’s a perfect example of the spirit of console verification, and is the best case scenario for what we would want.
Haha, what? No, we can't add the flag for that. That's ridiculous. Of course this is fine.
2. Movies that sync with the same input, but might need a few frames added or deleted to work
[...]
Second up, case number 2. This is where we start to notice issues with emulation not being exact with console. Emulation is not likely to be 100% accurate to console every time, and can usually depend on the game being tested. For Sonic Advance, 1 lag frame was added in order to get the run to sync. A small sacrifice which doesn’t affect anything significant in the input file. In the case of SMA4, 10 frames had to be added in some loading areas to get it to sync, and about 500 frames were removed from the movie file so save data creation did not have to be an issue. I think in these cases, it would still count to include them as console verified, since the adjustments that had to be done were not relevant to the gameplay, but rather just loading. These cases should also have documentation on how to get them to sync, likely in the submission thread.
Adjusting frames for loading times is perfectly reasonable, if that's all you have to do to get a run to work on console then I see no reason not to call it verified.
3. Movies that were on older emulation but resynced to newest emulation in order to verify.
Ah, this is where it gets dicey.
Then we have the 3rd case. This is where things start to get very sketchy.
Aw heck, you said that already.
There are a couple of movies that use older inaccurate emulation that were adjusted to work on console, like Super Mario Land and Trip World. For Trip World, I was able to get the movie to play on GBC in GBA mode, which is required to get GB/C games to console verify. I then added frames to any screen transition which would be an average of about 1-2 frames each. There was no lag during gameplay that I had to adjust, so it was a seamless sync.
This sounds fine to me, pretty much the same as Case 2 just with a couple of extra caveats that don't matter too much in my eyes.
However we have a case like Super Mario Land that needs some high hurdles to get to work. For one, bonus levels have to be adjusted in order to get fire flower after each world in the bonus rooms. These are based on what frame you finish the world on, and sometimes require wasting them via pausing and simmilar. The other issue is lag during gameplay. There might be cases where you have to change the inputs slightly in order to keep optimal lag reduction, and when that happens, it breaks the spirit of a true console verification. Since you aren’t testing to see if specific inputs will work on console, and have to make these adjustments in order to work, it might disqualify the verification.
Yeah, if you actually have to change the button inputs, that's no longer the original run. The idea is that the submitted run's inputs work on console. To me, adding a blank frame because the console lags where the emulator doesn't is fine. Adding/removing a frame of an A press for any reason isn't, even if it doesn't change how the run looks and only removes a frame of lag. A case could be made that tiny adjustments like that are still okay because the run is 99% accurate to the console, meaning that the emulator is pretty much 99% accurate as well, but it still rubs me the wrong way. Like, don't get me wrong, it's still impressive to see, and it shouldn't discourage anyone from trying to get runs to sync on console. These are arguably the most important cases, since it could potentially show the emu devs what they need to do to make things more accurate. I just think that changing actual button presses disqualifies the run from getting the flag.
TASvideos Admin and acting Senior Judge 💙 | CohostwarmCabin wrote:
You shouldn't need a degree in computer science to get into this hobby.
Moderator, Senior Ambassador, Experienced player
(898)
Joined: 9/14/2008
Posts: 1007
Hi! I have opinions! They're even reasonable and based on extensive personal experience! I'll dispense them below with slightly less eyebrow raising.
The very definition of an emulator is to be like the real thing while not actually being the real thing. An emulator in any capacity is different no matter how accurate your code is or how careful you are with your FPGA clocks. Even if your precious emulator is cycle accurate and who cares about the CPU you have to buy to get it to run I almost guarantee there's still a corner case somewhere.
There's something that hasn't been quite discussed unless I missed it, and it's a glaringly important angle: The MK64 run uses the exact same file that the emulator uses and is thus by definition precisely the same input but differs by literally seconds by the end of the run. If you are arguing that one file type versus another is somehow superior or inferior you're arguing about the wrong thing. The real issue is that the emulator isn't as good as the real hardware and deep down you know it.
Now, we don't kinkglitch shame emulators, consoles, or anyone else here. Not even the consoles are perfect, as we've proven repeatedly by demonstrating that components that are now decades old no longer perform the way they once did (no, *that* pun wasn't deliberate). Even when the consoles were brand new there are cases where the specification that the game designers programmed to failed when the cold hard truth of the laws of nature took their toll. No one is perfect, no not one.
Yes, emulator authors should strive to be better, console maintainers should strive to be better, and TAS authors should strive to be better. We're all about being as close to perfect as we can be here, after all. However, we must face these these inevitable realities: We must be lenient on emulators or no TAS would be acceptable. We must be lenient on consoles or no verification would be acceptable. We must also be lenient on TAS authors or no art they make will be deemed worthy.
I believe the far more productive conversation should be about the level of care that must go in to describing what steps it took to make the console verification happen. The quality and presence of documentation is what matters most here. We specifically need to be documenting deviations - what differed from what in order to get a given sequence of button presses that comprises the core essence of a run to work? Did a particular console need to be used? (I have several SNES consoles that all have different characteristics with some being better for different runs than others, all completely unmodified.) Were non-invasive hardware "nudges" needed, such as biasing voltage or biasing a signal or clock line one way or another? Were more aggressive changes needed, such as replacing a component before the run would sync? Was a flashcart with realtime memory validation allowing for on-the-fly retiming used? Was a hardware clone used and if so what documentation about its accuracy has been released? Was additional software necessary such as a shim layer to intercept inputs? There are a wide range of console types from the 70's through today and each requires different angles. At some point as we start going down this list we might come to a point where we can't stomach the deviations and that's fine - let's cross that bridge when we come to it.
What's obvious at this point is that we will frequently have what I'll call artifacts - something that makes up that core essence of inputs in order that was played in a player piano / piano roll fashion along with various support files such as open source console/game specific code, hardware modification descriptions, offset files for lag adjustment, descriptions of timing methods such as windowing modes, etc. We need it all. Without it, future historians will be robbed of what they need to piece everything back together. TASVideos currently lacks the capacity to handle all of this today and as such I've asked TiKevin83 to assist with an effort to make this possible.
In summary, we should have (reasonable) leniency but we should also have strict quality requirements for documentation. No other path forward ensures what we make now can be understood by future historians.
Thank you for your input DwangoAC (pun not intended)
[14:15] <feos> WinDOES what DOSn't
12:33:44 PM <Mothrayas> "I got an oof with my game!"
Mothrayas Today at 12:22: <Colin> thank you for supporting noble causes such as my feet
MemoryTAS Today at 11:55 AM: you wouldn't know beauty if it slapped you in the face with a giant fish
[Today at 4:51 PM] Mothrayas: although if you like your own tweets that's the online equivalent of sniffing your own farts and probably tells a lot about you as a person
MemoryTAS Today at 7:01 PM: But I exert big staff energy honestly lol
Samsara Today at 1:20 PM: wouldn't ACE in a real life TAS just stand for Actually Cease Existing
What bothers me (not a whole lot, just a bit, but anyway) is that it appears to me (and please correct me if I have misunderstood) that the version of the run that works on the emulator seems to be considered the "main" "primary" "official" version, and this version will only get the "console-verified" badge if it has been demonstrated to run on the original console... even if some minor modifications to the run are required (such as adding some lag frames here and there). In other words, the "fixed" version of the run seems to be considered some kind of secondary optional extra, perhaps provided alongside the actual primary emulator version.
In other words, it's as if the emulator is considered the main authority of how the TAS should work, with an ancillary side variant of it that's fixed to work on the actual console.
Shouldn't it be the exact reverse? I mean in the cases of runs that have been console-verified? Shouldn't it be the console version that's considered the main primary version, with perhaps an ancillary version of the run that has been "fixed" to run on the emulator, if necessary?
One thing where this would make quite a visible difference is if the two versions of the run are of different lengths, in frames. Shouldn't the official time be that of the console, not that of the emulator, if they differ?
Shouldn't it be the exact reverse? I mean in the cases of runs that have been console-verified? Shouldn't it be the console version that's considered the main primary version, with perhaps an ancillary version of the run that has been "fixed" to run on the emulator, if necessary?
TASes on the site do not aim to only work on console.
The other problem is people who want to play back these tases will not know that the main file is actually the console verified one, where it might not sync. It’s just a step that is completely unnecessary and would cause confusion, especially to encoders looking to re-encode older movies.
[14:15] <feos> WinDOES what DOSn't
12:33:44 PM <Mothrayas> "I got an oof with my game!"
Mothrayas Today at 12:22: <Colin> thank you for supporting noble causes such as my feet
MemoryTAS Today at 11:55 AM: you wouldn't know beauty if it slapped you in the face with a giant fish
[Today at 4:51 PM] Mothrayas: although if you like your own tweets that's the online equivalent of sniffing your own farts and probably tells a lot about you as a person
MemoryTAS Today at 7:01 PM: But I exert big staff energy honestly lol
Samsara Today at 1:20 PM: wouldn't ACE in a real life TAS just stand for Actually Cease Existing
In other words, it's as if the emulator is considered the main authority of how the TAS should work, with an ancillary side variant of it that's fixed to work on the actual console.
Shouldn't it be the exact reverse? I mean in the cases of runs that have been console-verified? Shouldn't it be the console version that's considered the main primary version, with perhaps an ancillary version of the run that has been "fixed" to run on the emulator, if necessary?
I was thinking about this as well. I think it really depends on your perspective.
For the site, I think submission movies that work on emulators are needed as a matter of procedure. Judges (and anyone else curious) can't easily verify that something works on real console; it's infinitely more convenient to have movie files that work on an emulator. Once a movie is published, the site's general operations is that the movie itself doesn't get modified (maybe it does in certain cases; I'm not sure?) In any case, one of the principles of the site seems to be having movies that are accessible to anyone for playing and analyzing.
However, for the purposes of comparing movies and determining which ones are faster, console-verification and the related testing that goes alongside it can show whether improvements should be considered valid (or a previous version considered invalid in some way). Our rules already address cases where new submissions that "seem" to take more time can obsolete older movies by merit of the older movie have inaccurate emulation (often times less lag frames). This can extend beyond lag frames, and console-verification could be considered a final step in understanding how to compare movies.
