Today I took notice of the most recent scandal in speedrunning, about a guy who faked dozens of GTA runs by simply changing the configuration values of the vehicles' speed so that he could drive faster.
The method was hilariously simple, and easily detectable. By simply going frame by frame, one could check that his speed was impossibly high. Nevertheless, he was only caught because there were graphical changes which arose suspicion.
That outlines some issues I think the speedrunning community simply ignores. Basically, their verification process is an IQ test, in the sense that it tests the cheater's intelligence to not get caught.
Aside from obvious ways of cheating like splicing segmented runs and editing the audio to cover up the splice, streaming emulator footage that was pre-recorded and act like you're playing it, and using scripts in PC games, there are more elaborate ways.
Some popular runs like Pokémon and Mega Man X, for example, have some strong RNG dependence. If someone tweaked the ROM to increase the probability of a favorable outcome, for example a damage range in Pokémon or a favorable boss pattern in MMX, they could get a significant advantage, even playing live on Twitch, on a real console, and the run could not be distinguished from a real, lucky one, by looking at the footage.
Let me know what you guys think about this. People sometimes get thousands of dollars in donations for performing runs live. Should this behavior be considered fraud? Is it time to only accept records performed on environments controlled by a third party?
Joined: 8/15/2005
Posts: 1934
Location: Mullsjö, Sweden
I've been following the "cheating" discussions in speedrunning for many years, and given many example of ways of cheating that you can't detect, even if you are sitting next to the person playing the game. There simply isn't a way to prevent everything no matter the verification method.
This is where TASes are superior, as they are impossible to cheat, as the input file is provided and has to sync through.
But yeah, there isn't really much to do about this...
With my thousands of hours on Awesomenauts, I've run into subtle cheaters every once in a while. They would temper with their connection or straight up hack their stats (movement speed) sometimes very noticeably, other times only subtly. At some point, the devs implemented proof anti-cheating technology and I haven't encountered any cheater ever since.
On the topic of speedrunning, I used to think, as long as the footage looks like it was played on console, it must be legit. But we are in times where you can homebrew your games and play on an emulator on your TV and make your footage look like it was on a real console. Indeed there is a chance of people starting to cheat subtly. Speedrunning on emulator (on PC) also becomes more and more common and with that, the possibility of ROM hacking, scripting, pre-recording your run, etc.
On the topic of money, I have learned that it doesn't matter what you do or how hard you work. People will only give you money if you are a sympathetic and entertaining personality who streams consistently (a certain amount, predictable content, regularly). I have spent half my life investigating and contributing to speedrunning and glitch discoveries - as have some others here. But I haven't seen any cent. Money is pretty unfair, without even taking cheating into consideration.
Is it time to only accept records performed on environments controlled by a third party?
Possibly. Are you thinking about something that uses checksums to verify that the ROM is unmodified?
Not really. I was thinking about a place controlled by a referee where runners would go and stream the runs with the equipment they're provided. The logistics costs seem enormous, though.
By the way, checksums would be of little help. You can just modify the ROM and change the values of useless bytes to make the sum match the previous value. Checksums protect against accidental modifications, not malicious ones. A cryptographic hash like SHA-2 would be better, but as long as you're in the cheater's computer, he can simply rig the program that does the checking.
We would also need another extra referee to control that referee.
You're just fucking stupid, everyone hates you, sorry to tell you the truth. no one likes you, you're someone pretentious and TASes only to be on speed game, but don't have any hope, you won't get there.
speed demos archive used to be the site that tried to do what you suggest, pwn3r, but nobody goes to that site anymore because watching runs is what makes cheating gets discovered and that site is a 2000s site in terms of coding and rules,
speedrun.com already has lots of mods and known players to check the runs in games that matter, and since speedrunning is and probably will always be niche, no use in going any further than SRC does
TAS i'm interested:
megaman series: mmbn1 all chips, mmx3 any% psx glitched
fighting games with speed goals in general
well, after mailing, there was a verifying process, but they had a less people to spot stuff, as they kept the runs hidden before the actual publishing, and that's one of the reasons why the site had some pretty famous cheated runs like TSA's zelda runs.
We also evolved knowledge about the games, so it's not just the process, but the popularity of games, the knowledge built upon the years and the streaming technology that made watching runs easy and acessible
TAS i'm interested:
megaman series: mmbn1 all chips, mmx3 any% psx glitched
fighting games with speed goals in general
There simply isn't a way to prevent everything no matter the verification method.
This is where TASes are superior, as they are impossible to cheat, as the input file is provided and has to sync through.
One way to make speedruns that are verified to not be cheated would be to make them in games that record input files at the same time you do your run. If that input file plays back identically on other people's systems, then the run itself is legitimate (at least in terms of not being run on a modified version of the game logic - they could still be pretending to play back a TAS, but that is hopefully easier to detect, especially if, say, the run was streamed with handcam). Such games include Doom and Portal.
This is where TASes are superior, as they are impossible to cheat, as the input file is provided and has to sync through.
I think there is one very rare exception (that has probably become so rare as to be practically non-existent nowadays, but could theoretically still happen): Abusing faulty emulation to achieve things that are not possible in the real console. Such as abusing glitches that happen only on emulators, because of imperfect emulation.
It's up to a judge, or somebody, to find out if some glitch, technique or other advantageous thing happening in the run is due to imperfect emulation or not, and not replicable in the real console. That can be extraordinarily hard to corroborate sometimes (especially if the difference is just a few frames, but even if it allows major skips, especially with less popular games because there's less interest and less scrutiny with them.)
And the funny thing about this is that the "cheating" may be completely inadvertent and unintentional. The TASer was simply testing the game to see if he could find something, and stumbled across this technique, without realizing that it was an emulator-only thing. How could he even realize it?
Aside from obvious ways of cheating like splicing segmented runs and editing the audio to cover up the splice, streaming emulator footage that was pre-recorded and act like you're playing it, and using scripts in PC games, there are more elaborate ways.
I don't think there is any reasonable and feasible way to deter all forms of cheating. Heck, cheating could theoretically happen even at events like the Games Done Quick marathons (if I understand correctly, runners usually bring the game they are going to run themselves. As you note, it would be theoretically possible to tamper with the game code and somehow get it into the original cartridge. Wouldn't be completely trivial, of course, as it would probably require some hardware mods inside the cartridge, but it would be theoretically possible. If the changes are subtle enough, such as making one particular random event non-random, nobody would notice. People would just think that the runner got lucky in that one instance.) Perhaps the only way to be 100% sure that this kind of cheating doesn't happen is if GDQ always provided its own copy of the game that's being run, and doesn't allow the runner anywhere near the console. I think even this could be a bit unreasonable, especially with less common games.
I suppose that, in the end, cheating is most damaging with games that are very popular among speedrunners (such as certain Mario, Metroid and Zelda games, and so on), but on the other hand scrutiny with these games is much, much higher. If a complete nobody would suddenly get into the top-5 list or something, out of nowhere, the run would certainly be scrutinized to death. Most of the top runners of a given game are usually very known long-time popular speedrunners who usually eg. have a strong presence on twitch.tv. It would be highly unusual if one of these people would suddenly try to cheat (but, of course, also their submissions always go through the verification process, but I'm assuming there's a level of trust still involved in that the submission might not be scrutinized with such a fine-tooth comb as a surprise submission by a completely unknown person.)
If the changes are subtle enough, such as making one particular random event non-random, nobody would notice. People would just think that the runner got lucky in that one instance.)
It wouldn't work that well if runner stream his attempts. It will be noticed after some tries.
There are almost no way to hack the game in such way that it wouldn't be noticed, but you can so easily make emulator movie that will be guaranteed to be undetectable. Or you can make lua script that helps you with hard tricks. You can make so lua script only work if you push some button and then you can make any number of attempts including failed attempts.
Prerecorded and luascripted runs are way to go to cheat on speedrun.com It's cheap, easy and completely 1000% undetectable. Here is some WRs nes runs that we will never know if they are real or not LUL
https://www.speedrun.com/Battle_City/run/mrqqd5dyhttps://www.speedrun.com/Jackal/run/1zx1enkyhttps://www.speedrun.com/pizzacats/run/7z0nlpom
Joined: 1/24/2018
Posts: 251
Location: Stafford, NY
Koh1fds wrote:
Here is my list of WR nes runs that we will never know if they are real or not LUL
Maybe we could figure that out by carefully replicating the run in TAStudio and seeing if anything that would seem like cheating showed up in the process? (E.g. luck manipulation being necessary, a Lua script being necessary to replicate the run, etc.)
c-square wrote:
Yes, standard runs are needed and very appreciated here too
Dylon Stejakoski wrote:
Me and the boys starting over our games of choice for the infinityieth time in a row because of just-found optimizations
^ Why I don't have any submissions despite being on the forums for years now...
Here is my list of WR nes runs that we will never know if they are real or not LUL
Maybe we could figure that out by carefully replicating the run in TAStudio and seeing if anything that would seem like cheating showed up in the process? (E.g. luck manipulation being necessary, a Lua script being necessary to replicate the run, etc.)
For example, Pizza Cats have pretty damm hard quick kill on every boss. There is no way to know if it was segmented emulator movie or if it was lua scripted (you can make lua script that provides necessary intputs when you press button on keyboard) or if it was a real run. There are absolutely no way to tell if it's real or not because there is no difference in this particular example.
edit: all those 3 example runs are extremely hard to beat, but absurdly easily to fake on emulator fceux
edit2:
Here is example of such lua script:
https://i.imgur.com/CWH2u78.png
You just go to cutscene before boss fight then hold K on keyboard = boom! quick kill! EZ
Want to make some failed runs? Boom! Just don't press K LUL Easy, cheap, undetectable! Here you go
Perhaps the only way to be 100% sure that this kind of cheating doesn't happen is if GDQ always provided its own copy of the game that's being run, and doesn't allow the runner anywhere near the console. I think even this could be a bit unreasonable, especially with less common games.
Well, 100% is a bit strong. The person could always bribe the judge, but this looks like an effective measure for the popular games, which are easy to find.
As you note, it would be theoretically possible to tamper with the game code and somehow get it into the original cartridge. Wouldn't be completely trivial, of course, as it would probably require some hardware mods inside the cartridge, but it would be theoretically possible.
That's not impractical at all. Older consoles like the SNES have no security measures, one can simply make a custom cartridge with the modified ROM and try to pass it off as the real thing. The security measures in the PSX are very bad, and modchipped ones are easy to obtain. I also remember I had a GameShark CD for PSX games.
You just loaded the gameshark CD, input the codes by hand or from the memory card and after that open the lid and place the game that would boot up with the hack. There you have it, you can run a modded game without tampering with the ROM. Imagine how easy it is to hide this on stream!
It isn't hard to launch modified game on real console. It's pretty damn hard to make useful modification and stay undetectable. Making random events not random are like the only thing that could work. Anything else = not that much. What you can do? Reduce enemies / boss hp = easy to detect. Increase character speed = also easy to detect. Increase your HP = easy to detect.
Why to do it and take a risk of being detected when you can always submit your segmented emulator movie without putting any effort and stay completely undetectable? LUL
Well, Koh1fds, I don't think it's useful to discuss whether the easy of detection will discourage people from using modified games. There was a case where that happened already, the question is what can be done about it. Even the criterion of "mutual trust" was useless. The person found guilty was a respectable member of the community.
Is it easily detectable? For us, who are used to runs and who are looking in retrospect, very easily. But take a look at the whole context. Is there a "speedrunning Ombudsman" who can force you to answer questions or else be banned? How many people actually have the skills to check game footage and look for scripts or mods? Even in this forum, with our highly technical runs, it's difficult to find people who can do that. In that case, a trivial cheat stood for a long time, and took an investigation of one year to get to the facts, and in the end the whole community took a hit on their reputation.
I had the unpleasant experience of witnessing an administrative investigation of someone who was accused of embezzling donation funds. The evidence was obvious to everyone, no need to be an accountant. I've seen things I never thought I would, retaliation against whistleblowers, attempts to obstruct the investigation, etc. In the end the information was leaked on a facebook group. The admins even tried to censor all the posts discussing the fact but failed miserably. After the leak, the person was dismissed immediately . Despite the recommendation of filing criminal charges to recover the money (the value embezzled was bigger than my savings account), the directors decided to simply release a note saying the organization was "in debt" and did not disclose fully what happened, and the guy didn't even have to return the money. This prompted mass resignations and other organizations withdrawing support.
I've been on some streams at the time when people get world records. They can gain 200+ subscribers, and I'm not even counting resubs and other donations. I know that there was some GDQs had controversy surrounding the finances of an organization they were asking donations for. Trust me, in the context that some misguided fool decides to stream a fake run, there's a serious possibility that everyone will get into serious trouble with the law and the institution will take serious permanent damage. This is an extremely serious matter, just saying that the run is legit because emulators are banned may not be enough!
I just want to say that it's absurd to accept FCEUX, BizHawk, VirtuaNES, Snes9x, Gens Re-recording etc runs, because it's TOO EASY to cheat on those emulators. Speedrun.com administration could make their own emulator and it could log everything. Not only rom's checksum, but also what controller was used, buttons binds, even what programms was launched on PC at that time etc. Button binds are also important, because unfair button binds are also a problem. For example in Ninja Gaiden 1 you can bind Down + B on the same button and that would make slash canceling extremely easy. Honestly, i think even just banning fceux, bizhawk, etc would make cheating much more difficult.
I don't really care about GDQ. I don't think that quality of the run on GDQ actually matters. You can play as sloppy as you want. There is no need to show wr run on GDQ. So what's to point to cheat on GDQ?
If the changes are subtle enough, such as making one particular random event non-random, nobody would notice. People would just think that the runner got lucky in that one instance.)
It wouldn't work that well if runner stream his attempts. It will be noticed after some tries.
In that example I was specifically talking about how someone could hypothetically cheat at a GDQ marathon.
So what's to point to cheat on GDQ?
I didn't give that example with the idea that it would be somehow beneficial or reasonable for someone to try to cheat at GDQ. The idea behind that example was to demonstrate how difficult it can be to detect smart cheating: Even at a public event like GDQ, where you control next to nothing of what's happening, with hundreds of eyes watching in-situ what you are doing, seeing that you are playing a real console in real-time, and you cannot tamper with the console itself or what it displays, it would still be possible to cheat, and nobody would notice. That's how hard it is to eradicate cheating 100%.
I just want to say that it's absurd to accept FCEUX, BizHawk, VirtuaNES, Snes9x, Gens Re-recording etc runs, because it's TOO EASY to cheat on those emulators.
I am trying to bring up the issue when the cheating is done by modding, with or without emulators. If this rerecording issue affects you more, it would be a good idea to contact SRC people. I am a bit surprised that runs on these emulators are accepted, though. At least for the games I follow, these emus are banned. Well, they usually accept some version without rerecording tools, but they're open source and it's simple to modify them. Like I said before, the verification is an IQ test. Only idiots will fail it.
Koh1fds wrote:
I don't really care about GDQ. I don't think that quality of the run on GDQ actually matters. You can play as sloppy as you want. There is no need to show wr run on GDQ. So what's to point to cheat on GDQ?
Although I've never seen an instance of cheating at GDQ, I know a YI runner once played back a TAS during another marathon, getting WR. He later got caught for playing back another run, and people investigated his previous runs and found it out. To be honest, I have yet to understand why people are so inclined to do wrong stuff. Why do they embezzle things when everything ends up on an accounting log? Why do they steal things with cameras filming the place? I don't know the answer to these questions, only the fact that they do it not as rarely as I would expect.
My point is: if someone eventually does it, everyone gets in a lot of trouble, so it's necessary to prepare for this.
Why do they steal things with cameras filming the place?
Reminds me of the numerous examples of people cheating in Magic the Gathering pro tournaments (which have relatively large cash prizes), on camera. They know perfectly well they are on camera, yet they still decide to cheat and hope they aren't caught.
