Why do the Celeste and F-Zero GX runs not on TASVideos yet?
In F-Zero GX's case, I'm assuming it's because only a few levels were made, but the Celeste runs were clearly complete, so I'm quite confused there.
Moderator, Senior Ambassador, Experienced player
(898)
Joined: 9/14/2008
Posts: 1007
SmashManiac wrote:
Why do the Celeste and F-Zero GX runs not on TASVideos yet? ...the Celeste runs were clearly complete, so I'm quite confused there.
I have struggled greatly with attempting to get content from GDQ's submitted on TASVideos. There is a general resistance amongst site staff to allowing anything "new" and Celeste falls in this category. The Celeste TAS was created using a rerecording framework designed specifically for the game and there is no way the runs can be submitted using the current site code. This continuing unwillingness to allow content that does not fall within the confines of traditional runs, i.e. anything that isn't one of the existing accepted emulators, limits the content that can be highlighted on the site. The only format for this kind of content on the site is Wiki pages and forum posts per various conversations with site administrators but that is a very poor, second-class experience both for those posting the content and those attempting to find and view it.
Thus we find ourselves in the same situation as in years past - it is highly desirable from a notable contingent of site visitors to have this content represented shoulder-to-shoulder alongside traditional TAS runs. The lack of a way to do this has led to a growing schism where content like Celeste is pushed off the site and onto other platforms. I am part of site staff and would love to see changes here but it will take dedicated site coding to resolve the issues I've identified above along with a willingness to allow more diverse content on the site and I'm afraid I don't see any changes on either front coming from the administrators on staff. Yes, I'm repeating myself a bit by saying the same things in different ways but it's because it's very important that everyone understands why we're here. I have no solutions beyond hoping the prevailing winds shift.
Supporting custom rerecording workflows that only work on the per-game basis is not only extra coding, but also a whole new platform that needs to be verified against all sorts of abuse. In emulators, we can guarantee that no abuse happens. In internal replay formats, like Doom, not so much. Some games use replay systems that are absolutely unacceptable for us due to what they allow internally.
Right now we all hope that Linux games become TASable with libTAS.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
I have struggled greatly with attempting to get content from GDQ's submitted on TASVideos. There is a general resistance amongst site staff to allowing anything "new" and Celeste falls in this category.
Consider how strict of a stance sites like speedrun.com take on cheating, and justly so. In unassisted speedrunning cheating can take many forms, such as submitting a segmented run to an unsegmented category, using a spliced video (ie. snippets from different runs have been deceptively edited together to give the impression of one single unsegmented run), surreptitiously using tool-assistance, and so on. Infamous examples of such things exist plentiful (some even having been published for years before having been discovered as having been cheated), and sites like speedrun.com go to great lengths to catch and deter such things from happening.
TASvideos also has very strict standards of legitimacy for runs. Runs cannot, for instance, abuse emulation errors to do things not possible with the original game running on the original console, the game's code must not be modified using emulator features, and so on and so forth.
When a TASing toolset is game-specific, created specifically for one single game, and sometimes perhaps even requiring modification of said game in order to work, it opens a can of worms. How can we know that the end result is legit, using the above criteria? How can we know that what's happening in the run is indeed achievable with the unmodified game, running on unmodified hardware, using normal input?
Of course the runs shown at GDQ are always legit (I think it's pretty safe to make this assumption, given the reputation of the people involved), but accepting a game-specific TASing environment may in some cases open the possibility of more unscrupulous people to submit cheated runs for it.
Perhaps the toolset could be verified as making cheating impossible (assuming that's the case), but that would be quite a lot of work for the site administrators. And this work would have to be repeated for each such toolset that may be created. Just for one single game.
One could argue that perhaps just publish the run showcased at GDQ, but don't accept any further submissions for it. However, I would say that accepting a run made with a game-specific toolset, on the basis of the reputation of the people who made it, and then closing further submissions for that game, with perhaps the exception of a small group of "trusted" people, would sound strange, unusual, inconsistent and perhaps even a bit elitistic.
Thanks a lot for the explanation dwangoAC!
feos wrote:
Supporting custom rerecording workflows that only work on the per-game basis is not only extra coding, but also a whole new platform that needs to be verified against all sorts of abuse.
I agree, but preventing such submissions because they require additional code analysis seems a bit extreme to me. Just my two cents.
I agree, but preventing such submissions because they require additional code analysis seems a bit extreme to me. Just my two cents.
Please list games that have 100% abuse-proof rerecording system that we explicitly prevent submissions of.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Moderator, Senior Ambassador, Experienced player
(898)
Joined: 9/14/2008
Posts: 1007
We're quickly getting to the point where this conversation needs to be split off to a different thread, but I think we can be mature enough to avoid another Todd Rogers. Will we be 100% successful every single time we encounter a new emulation platform? Certainly not, as my own experiences with TI-83 tests show a case where an emulator core was deemed not accurate enough. Does that mean we shouldn't try? I don't think so.
I believe it is necessary to expand the role of an arm's length / third-party reviewer to allow other experimental rerecording frameworks to be accepted (of which I'm roughly speaking an author of one in the form of the NetHack rerecording framework although that one will never be put to the test as the run syncs on jpc-rr and I'm digressing terribly). I'm obviously biased in every possible way but I cannot conceive of a situation so dire that we can't at least try.
Let's talk about Celeste itself for a moment - we have full code we can look at, a patch file, and even support from the developers. It's as if this was handed to us on a silver platter. Why, tell me, is it literally impossible to at least be open to the idea that someone on the site could review the work that was done? We will never know, because as it stands now there is no submission method whatsoever to allow such a thing to be attempted.
There is a chicken and egg problem here - feos (rightly) makes the assumption that based on what we know now we don't have the ability to rigorously verify new rerecording frameworks, a valid assumption based on a lack of prior cases. We have no prior cases because the submission process does not allow anything but already accepted platforms. We thus are caught in a loop where we will never know how the community would respond.
I'm nearly blue in the face from saying this, but we have got to stop being so elite or we will diminish in importance in time as platforms move on and as every old platform is exhausted of games worth TAS'ing. A quick chat with link_7777 and the TASMania project shows we still have a few more years but it's only a matter of time before there aren't any interesting improvements or new games to work with with the classic consoles. We need to start making the decisions now that allow us to at least make it possible to consider newer PC games. I'm in no way downplaying the amount of work that will be required, don't get me wrong here, but to never start is to stagnate.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Keep in mind, there is already a non-standard game specific format accepted on the site: Doom Demos. It's not that far of a stretch to do something similar with other custom TASing engines.
[16:36:31] <Mothrayas> I have to say this argument about robot drug usage is a lot more fun than whatever else we have been doing in the past two+ hours
[16:08:10] <BenLubar> a TAS is just the limit of a segmented speedrun as the segment length approaches zero
Now that it's properly split, here's the deal. We need people who understand the internals of Celeste tas tools well to describe its tas framework in details that are relevant to validity and abuse-proofing.
The game itself should be intact and impossible to affect through this workflow in any way other than via input methods it normally provides to the user. Savestates should be stable enough to provide for heavy usage without problems. Video and audio output should allow lossless dumping.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Moderator, Senior Ambassador, Experienced player
(898)
Joined: 9/14/2008
Posts: 1007
feos wrote:
Again, who is saying No explicitly?
The site itself is antagonistic (or at least passively unhelpful) by not facilitating any possible method whatsoever to submit non-standardized content. I recognize there have been endless and fruitless arguments about names for content like this and phrases like demo tier now have a terrible name as a result. I could care less what we call it, let's just please open up the submission form to allow for "Other" as the platform and remove restrictions on filetype matching and see what happens. This is obviously both the wrong way and the wrong place for this conversation so I'll stop now.
I could care less what we call it, let's just please open up the submission form to allow for "Other" as the platform and remove restrictions on filetype matching and see what happens.
As a judge, I'm not downloading any submissions files that I do not know if I can trust. Period. Having some sort of workflow for accepting new TAS frameworks, sure go ahead. Having me have to guess as to whether I'm about to download a virus? No thanks.
EDIT: Having some sort of separate submission form in order to facilitate new TAS frameworks, even if they are game-specific, I can accept something like that getting introduced. It might even require a new team of people to evaluate each framework. That I can see actually happening.
What I cannot accept is just opening submissions to everything without a plan in place whatsoever, which is exactly what you're advocating right now. That would be absolute chaos.
[16:36:31] <Mothrayas> I have to say this argument about robot drug usage is a lot more fun than whatever else we have been doing in the past two+ hours
[16:08:10] <BenLubar> a TAS is just the limit of a segmented speedrun as the segment length approaches zero
Just so it doesn't die unanswered.
dwangoAC wrote:
The site itself is antagonistic (or at least passively unhelpful) by not facilitating any possible method whatsoever to submit non-standardized content.
The site has no will, it only does what we tell it to do. If we want to tell it to do something, it'd require a clear, robust, abuse-resistant, beneficial plan, people that would come up with it, people that would use the result, and people that would evaluate and judge the works people send us according to that plan.
dwangoAC wrote:
I recognize there have been endless and fruitless arguments about names for content like this and phrases like demo tier now have a terrible name as a result.
What exactly do you mean by "content like this"?
dwangoAC wrote:
I could care less what we call it, let's just please open up the submission form to allow for "Other" as the platform and remove restrictions on filetype matching and see what happens.
We already know what happens.
1000 submissions of cat pictures
100 submissions of clearly illegal pictures
10 submissions of subtly illegal pictures
1 submission of a torrent file that seeds 50TB of porn (it'd take a lawyer to check how much of that porn is illegal, and in which countries)
1 virus
1 archive containing 1000000000 viruses
1 bitcoin miner
1 elaborate exploit that turns tasvideos.org into a pro-ISIS site
10 legit movie submissions
10 sloppy movie submissions
10 movie submissions that subtly break the rules
10000 movie submissions that obviously break the rules
1 movie submission worth a star
1 textual description of a Half Life 3 TAS
1 replay file that is sent as instructions to make a nuclear bomb
1 instruction to make a nuclear bomb that is sent as a replay file
1 Celeste TAS replay file
100000 replays of novel internal game-based rerecording frameworks that do not sync
100 replays of novel internal game-based rerecording frameworks that do sync, but use in-built cheats
1 replay of a novel internal game-based rerecording framework that lasts for 40 days of pure gameplay
1 replay of a novel internal game-based rerecording framework that lasts for 400 days of pure gameplay, and that's a movie of a 3D game that only runs at 8K and only syncs if a person uses a 16-core 67.2GHz CPU, 80GB of RAM, Voodoo 2 GPU, and runs a Chinese edition of MacOS with a VM running Windows 10 on Mondays
1 submission of a sofa
1 submission of Singularity
All judges that quit the site
All publishers that quit the site
All tasers that quit the site
All admins that quit the site
All abusers in the world that join the site because it offers EVERYTHING now, and we have no one to check and reject
So it'd be tons of fun everyone here is fully looking forward to.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Let's talk about Celeste itself for a moment - we have full code we can look at, a patch file, and even support from the developers.
Does "patch file" in this case mean that the game itself is being modified?
One of the core principles of TASing is that the game itself ought to be unmodified. Once you start modifying the game itself, it opens yet another huge can of worms with regards to the validity of runs.
A modified game could be used to create the keypress input data for the game, but said input data ought to be, at least in principle, usable with the original unmodified game to replicate the run via controller input alone.
Somebody mentioned Doom as a huge exception to this whole principle of no game-specific TAS environments. And it is indeed quite unusual.
It is my understanding (and please correct me if I'm wrong), that Doom's own replay files do not store keypresses and replicate the run by feeding these keypresses to the game. Instead, they contain, essentially, data of the type "object X is at place Y at time Z" which (if that's indeed the case) could perfectly well be abused to create "runs" that are impossible to replicate when playing normally (eg. the player just jumping from one end of the level to the other in one frame, just because the replay file says that at the next frame the player's position is at that other end of the level).
What games other than Celeste have similar TAS tools currently? It would be good to have a list somewhere to have more reference points.
Currently, the site's prospects for Windows / PC TASing do not look great, other than libTAS perhaps. In the absence of any good submittable general PC TASing tool, we need to look outward more.
Obviously, given the problems mentioned in the thread already, allowing any *.* rando file to be submitted is not gonna happen, but in the case of a game like Celeste, where the source code of the tools is openly available, there should be a way to get it on the site.
We should try to define some kind of process by which game-specific TAS tools can be vetted, similar to how emulators are vetted currently. I'm not sure what the exact requirements should be, but I expect they wouldn't stray too far from what is already required of emulators (open source, only keypresses in files, etc.).
I completely agree with dwangoAC's points on the matter. We would absolutely be richer for having games like Celeste published on the site. Going forward, there will be even more games with TAS tools developed specifically for them, and the site should not be left in the dust while we guffaw about having added rerecording support for the Oric-1 to Bizhawk.
"Memory wrote:
As a judge, I'm not downloading any submissions files that I do not know if I can trust. Period. Having some sort of workflow for accepting new TAS frameworks, sure go ahead. Having me have to guess as to whether I'm about to download a virus? No thanks.
In fairness, vulnerabilities can exist for regular emulator movie files too.
feos wrote:
We already know what happens.
1000 submissions of cat pictures
100 submissions of clearly illegal pictures
[...]
I seriously don't think there will be that many custom game submissions that they couldn't be handled one at a time.
I seriously don't think there will be that many custom game submissions they couldn't be handled one at a time.
I don't quite understand this sentence.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Ah, the post isn't talking about custom game submissions, it's talking about the downsides of using "anything goes" to introduce verifiable, stable, usable game-specific tas frameworks.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Ah, the post isn't talking about custom game submissions, it's talking about the downsides of using "anything goes" to introduce verifiable, stable, usable game-specific tas frameworks.
Ah, I should've read your post better then. My bad.
As a former encoder and publisher, I believe that a completely open source project is essential to anything suitable for TASVideos. I say this not from idealistic or moral grounds but from practical grounds. Syncing, judging, encoding, and publishing runs is incredibly difficult. The open source nature of the emulators we support has allowed our community to improve many of these emulators long after the original authors abandoned them.
I advocated Doom demos for TASVideos because I felt it could be held to the same standard. It is completely open source, and has a long history of support of many of the useful goals of TASing, including years of research and fixes to determinism. I myself added some video dumping code to PRBoom+ before I came here, which I would not have been able to do without the code.
I don't see add-on tools for individual games like worms or towerfall meeting this standard unless the entire game is open source. 5, 10, years from now everything will be derelict and abandoned and it will be near impossible for users that want to play the runs themselves to do so, or for new encoders to reencode to the newest standards. Closed source games makes the question of cheating much more difficult as well.
Yeah we're not even starting to talk about crazy kinds of abuse like officially releasing a new version of the game that does secret shenanigans with such external replay systems that break the validity.
Emulation is the fundamentally right way of doing this kind of thing. Internal replay systems are worse, but tolerable. External replay systems are bad. But in order to determine just how bad (or not bad) they are, we need actual devs of those to talk to.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
but in the case of a game like Celeste, where the source code of the tools is openly available, there should be a way to get it on the site.
Note that the source code being available isn't some kind of magic bullet that will automatically erase all possibility of invalid runs.
Even small programs can become quickly quite complex, and software bugs are infamously common even in small open source projects. Checking even a small program for bugs is a rather large endeavor, even for a team of people.
Does this mean that all the officially accepted emulators have been 100% checked for bugs, and are guaranteed to be completely bug-free, and emulate the hardware with 100% accuracy in every possible situation? Of course not. But the line has to be drawn somewhere, if we want TASes at all. The advantage of generic console emulators is that they are being used to TAS literally hundreds of games, and many of these TASes are being verified with the actual original hardware, so this gives a very good degree of confidence that the emulators are as accurate as can be reasonably expected. The longer the emulator has been in use, and the more games have been TASed and verified with them, the more confidence it gives.
However, if we have a TASing environment that's specific to one single game, the situation is much more difficult. After all, we only have one specific game as a "test" of the "accuracy" of these tools. Sure, some people could go through the source code of these tools and try to find bugs and estimate how legit the data it produces is, but with so little data to test (usually only one run for one game), things can be easily missed. It's a bit comparable to a scientific test with a ridiculously small sample size.
And this would have to be repeated for any future single-game-specific TAS tools.
Somebody mentioned Doom as a huge exception to this whole principle of no game-specific TAS environments. And it is indeed quite unusual.
It is my understanding (and please correct me if I'm wrong), that Doom's own replay files do not store keypresses and replicate the run by feeding these keypresses to the game. Instead, they contain, essentially, data of the type "object X is at place Y at time Z" which (if that's indeed the case) could perfectly well be abused to create "runs" that are impossible to replicate when playing normally (eg. the player just jumping from one end of the level to the other in one frame, just because the replay file says that at the next frame the player's position is at that other end of the level).
I agree, it is unusual.
However, you are wrong about Doom replay files. They store attempted movement in forward/backward and sideway components, turning and attempted use of buttons and weapons. The movement data is in the same format as that which the interface feeds to the engine, after interpreting mouse and keyboard input.
You are probably mixing it up with demos from Quake and its derivatives, which do store information as "object X is at place Y", which is why they can never be accepted here.
Moderator, Senior Ambassador, Experienced player
(898)
Joined: 9/14/2008
Posts: 1007
So I haven't replied recently because I wanted to think through a few things. There's been some really good feedback to some of the positions I've taken and I needed time to consider. For now, I think I'd like to focus only on what it might take to get Celeste on the site as an example of what we might need to do and ignore any possible volume of other submissions for other games as it doesn't seem likely we'll have a massive influx right now anyway. I think I was wrong to assume that.
Mothrayas has convinced me that the existing site parser isn't too terrible to modify by hand for these scenarios and I'm on board with that, although I still see no harm in allowing a user to fill in fields and have a third-party validate that they're accurate like a referee. I understand that we're currently biased by our inadequate number of judges and the massive backlog on the workbench but that's not a forever thing. My goal is to pull more people to TASVideos and that growth would hopefully come with fresh judges but I recognize how foreign it sounds to say we could in the future support this kind of thing.
As the President of the North Bay Linux Users' Group and an advocate for open source software I heartily agree with the comments above - only open-source code should be considered, to allow the method of rerecording to be analyzed for cheating. No judge should be expected to run some random executable.
I'd like to thank everyone who participated in the conversation thus far for your thoughts and thank you for challenging my perceptions on some things. I'm willing to change my perspective and opinions even if I still have a strong hope that we can expand past the existing console focused frameworks to support more interesting things. I'm seriously interested in the libTAS project and I'd like to better understand it so that's my next area of interest for when I have free time (I have little of that right now). I think libTAS could be huge if it pans out and I'm looking forward to what happens.
