TODO: Rewrite this page for the new site.
Guidelines for judges
With great power comes great responsibility.
This page summarizes and details thoughts the judges should adhere to when watching and judging submissions, and communicating with their authors.
Table of contents
- Guidelines for judges
- Judging
- More reading
Sum-up
- TAS content is the main focus of our site, so treat TASers with care, inspire and encourage them with your wording and decisions.
- Satisfy the audience's expectations. Fellow staff members will help you build a bridge between desired and practical.
- Value quality above quantity.
- Keep the number of different branches per game balanced. A run for a proposed new branch for a game should offer clear differences relative to previously published runs of that game.
- Don't rush your judgment, let discussions play out. When necessary, start new discussions about the submission so users generate more feedback. Take your time and ponder everything relevant to the submission.
- Hard work should have a reasonable chance of being published.
There are no laws that work in all possible situations
Each case is slightly different, and a judge should be able to use multiple perspectives when judging a movie, both short-term and long-term consequences to:
- site content quality;
- authors' feelings;
- audience's trust in the site staff;
- site's maintainability;
- the rules of TAS.
Judging
Watch the entire movie from the input file!
It is important that the judge verify the authenticity of the input file itself, especially with the increase of authors/users encoding and uploading submissions to streaming media sites.
The input file should sync properly on an approved version of a rerecording emulator. Being an approved version means it can be downloaded from the official website as a packaged release. Development builds don't count as such (unless it's a messy case like Dolphin). Any settings necessary for sync should be documented in the submission properly (edit the submission to include this information if necessary). The settings chosen must be considered legal, and not used to hack the game in some way. If the submission does not sync on at least one official release, it must be rejected (unless there is an unavoidable exception). If it syncs for someone other than the author, it is still valid (so that games/emulators that require similar OS/computer specs to be deterministic have a chance).
The input file should complete the game (or achieve the goal stated in the submission text).
It should not contain any blank input at the end (trailing frames recorded with no input), because it unreasonably increases the movie's total time.
- Note that DOOM replay files require blank input at the end for the purpose of encoding, but the movie file that is being published should not have that.
It should fulfill the claimed goals of the author. I.e., if it is a Super Mario Bros. "walkathon", it should not press B in any level where doing so would allow the player to run.
It should not start from reset, SRAM, or soft-reset unless, of course, that is necessary for the goal choice (such as to unlock a character, or take advantage of newgame+, for instance). Of course this depends on the goal choice being worthwhile too.
Act consistent with the message of the site
- Treat submitters with the same respect you would like to be treated with when you make a submission.
- Reject movies that break the movie rules.
- Judge movies in concordance to the guidelines.
- Ensure movies consist of superior play that we expect, by ensuring expected techniques are utilized, such as those from our common tricks and our boss fighting guide.
If a Judge is not familiar with a game, they should do basic research to ensure that they understand the game mechanics, and ask the audience for details they are missing. Read the game manual if needed and if it is available. If something does not look right in a movie, check it yourself to be sure. However, Judges do not need to thoroughly in-depth test a game as if they themselves were the one trying to TAS it. If it looks good on the surface, and the movie appears to consist of superior play which outperforms casual play and any records elsewhere, it's good enough for TASVideos.
Be fair
A judge has the greatest control over the content of this website in the long run.
All judges must act towards the goal of having an encouraging and rewarding atmosphere for both the players and the audience. You must be fair towards both.
It is fairness towards the audience when judges disqualify worse submissions and qualify the better ones.
- Too many bad publications turn the audience away.
- Too few publications turn the audience away (and possibly the players too).
The players must have chances of getting their movie published.
- Don't demand them do laboursome optimizations that can't actually be noticed under normal viewing conditions [1]. For example, if a jump performed one frame late still moves the character along its purported trajectory with no interruptions, it is a negligible flaw. If, however, a late jump bumps the character into an obstacle, thus interrupting its movement, it's an easily noticeable mistake.
- For rejected submissions, refer to the relevant guidelines that would improve their chances of being accepted the next time. If possible, point out some specific mistakes.
Every newbie could be the next superstar. Their first submission might be rejected, but don't destroy their self-esteem. Reject with reason, but only in the necessary amount.
[1]: Normal viewing conditions refer to real-time playback with no additional indicators to judge the player's performance; i. e., like most people watch these videos.
Notice to new players: Experienced TAS makers often possess a great observation skill for typical mistakes in TAS movies. Mistakes unnoticeable to you might still be noticed by experienced TAS players and the judges. (But of course, we don't know and see everything.)
Notice to new players: Experienced TAS makers often possess a great observation skill for typical mistakes in TAS movies. Mistakes unnoticeable to you might still be noticed by experienced TAS players and the judges. (But of course, we don't know and see everything.)
Be thorough
Uncover, evaluate and address as many aspects of a given run as possible. This allows for decisions that could serve as a long-term reference or set important precedents in complicated situations. If you miss some of the aspects that might have influenced the final decision, your decision will be imperfect.
Make sure the movie is optimal and not sloppy
This is mostly covered by the Guidelines, but whenever you see something that looks like it has potential for improvement, either try it out yourself or ask the author to address it. Note that the run isn't required to be completely unimprovable, it just should not be sloppy. If it uses sub-optimal strategies all over the place, even if it's not obvious for an untrained eye, but which becomes clear when you actually try to improve it, then it should be rejected for sub-optimality.
Always check the runs against existing records, which might be present on other sites (for example, Speedrun.com, since its goal is to document the latest RTA records). If the author did poor research, help them find a better approach.
Judges however do not have to try to TAS the game themselves and try to uncover every technique and secret. The primary goal is to ensure the basics are well covered (such as those in GameResources/CommonTricks and GameResources/BossFightingGuide), and that the player didn't miss out on something the game obviously offers (such as being able to charge a weapon, or basic movement options such as running, hopping, or high jumps).
Use your and others' experience
Past decisions can always give an insight if the case is complicated. So refer to the decisions that you know of, to their key points, and to what can be assumed based on them. Ask other judges and the senior judge for opinions for nuances that are possibly outside your scope.
However, note that past mistakes don't necessarily justify future ones, so do not use precedents blindly. Moreover, when considering a precedent to base your decision on, always ask others to make sure it's not a misjudgment in the first place, but a reliable valid decision.
Review future possibilities for obsoletion, rejection, and branching
Check the previous submissions for a given game. They may contain decisions on how some future situations for that game should be handled. In some cases, when an exception is being allowed, you may also need to set such regulations for future decisions. Keep in mind though, that as time passes, previous regulations may be rendered obsolete or irrelevant, so examine the situation carefully and apply such regulations wisely.
Consider the audience's feedback
Viewers that are experienced in the game you're judging can expand your outlook even further by providing good food for thought. Always analyze their opinions and talk to them in order to get valuable information from them.
Make an elaborate judging note
The research that you've done should be available in the form of reasoning and conclusions based on it. Address in your decision all the aspects that you've accounted for, to make them clear for the author, for the audience, and for the judges of the future submissions that might need to refer to your decision.
Try to make sure your judgment notes are readable. You can use wiki markup to add sections or tables if required. If a diagram is needed, post the necessary image. If your judgment note is really long, make sure there's a summary for those who don't want to read the whole thing.
If there's anything important a publisher needs to know, mention it, and make sure whoever attempts to publish it is aware of it.
Classes and goals
For moons, it’s important to determine what the overall feedback is.
- Read the thread, determine the ratio of viewers that enjoyed the movie. See to what extent viewers liked or disliked a movie.
- Factor in the submission votes. The Moons class can only be used if the feedback is positive by a strong majority. 80% yes votes suggest a strong majority.
If a movie is obsoleting a Starred movie, a judge can accept straight to Stars assuming the entertainment value has not decreased significantly.
When judging the run's goals, offer some insights to publishers regarding what kinds of goals this run represents, to help them figure out what to put in the branch label. This is a good practice, because after having judged the run, you have already done some research that can help publishers. Their job is assigning accurate labels, and at times conditions are very complex, so collaboration helps to prevent mistakes and having to fix them afterwards.
Improvements and obsoletions
- Improvements to published movies should meet the Movie Rules, even if the published movie doesn't (such as use of correct ROM, cheat codes, etc.).
- Avoid meaningless publications. It may turn the audience away when a large improvement is possible, yet each small incremental change with no visible differences in gameplay finds its way onto the main page. If the submitted movie is clearly improvable as well, it should (usually) be rejected.
- Small improvements have and will be published, but only in a situation where it seems reasonable that only those small optimizations are left. If larger known improvements aren't implemented, it may be grounds for rejection.
- While it is expected that the new run should use all tricks and techniques known at the time, it is not uncommon for new time-saving techniques to be found during the later stages of making a run. Ideally, the run should be restarted or edited to allow for inclusion of these new discoveries, however, if restarting will be especially time consuming, exceptions can be made to this rule per judge's discretion.
- If there are no known improvements to demand, and a new submission happens to have the same length as a published movie, it may obsolete it if it's significantly more entertaining.
- Sometimes what is claimed to be an improvement may necessitate a new branch, such as when a huge time-saving glitch is involved (see SMW2), or unusual goal choice is exhibited (as in Princess-only SMB2 run, whose first iteration beat then-current any%).
- Sometimes an improvement necessitates a double obsoletion.
- This may occur when a new submission aims for a goal in a certain branch (for instance, any%) but also achieves primary goals of other branches (such as low% or 100%) better than their respective published movies (see Super Metroid glitched any%). Another notable example is when a 2-player movie, generally considered to be more entertaining than a 1-player version, also outperforms it time-wise (see Battletoads).
- Cross-branch, and even cross-class obsoletions can happen, if the movies really are considered similar enough by the audience and the judges, and the new one is better.
- Cross-platform obsoletions, on the other hand, are not done unless it's about deliberately identical ports (as happens with modern console games).
- A new movie may involve emulation accuracy improvements.
- Always ensure the new movie has gameplay improvements. If the new timesavers can be applied to the old movie, they count as improvements. If new timesavers only become possible due to accuracy improvements, they are also valid. But if the only difference is in slightly different timing of otherwise identical events, it's not an improvement, but only a resynchronization.
- For movies that are just resyncs on a more accurate emulator, if accuracy improvements are outstanding, for example, the new movie now can be replayed on console, such a movie can be added as a secondary movie file to the existing publication, but its submission must be rejected (or canceled).
- Sometimes an improvement may result in an entire branch getting superseded by a bigger branch.
- If there is a movie that fully contains gameplay of another movie, satisfies its goals, and builds more content and goals upon them, the smaller movie is considered superseded by the bigger one. Make sure that gameplay is necessarily contained if both branches are done optimally. See #6059: ViGadeomes's A2600 Private Eye in 29:52.99.
- Superseding may happen in reverse, when a bigger branch is already published and a smaller one gets submitted. If the superseding criteria are satisfied, the smaller submission is rejected. See #5854: ThunderAxe31's GBC Toki Tori in 42:20.21.
Major skip glitches
Major skip glitches (sometimes called game-breaking glitches) require additional confirmation that it's actually a developer oversight, and not a cheat code or a password that's subtly entered and looks like a glitched skip.
The most certain way to determine this is reading what programming code the game is executing, but there are some trends among cheat code scenarios that can make this detection easier:
- Input that works specifically during pause screens, menu screens, or other non-gameplay-based input
- Buttons have effects that have nothing to do with what the button normally does in gameplay or in menus
- Doing certain steps a number of times, or entering certain button combinations ("jump 10 times then press B+Select")
- Easy and consistent to replicate if you know the steps
- Increments or decrements memory values in consistent steps, especially if not related to normal gameplay functions (see also point 2)
Game completion point
Glitched endings
Runs that incorporate "game end glitch" techniques need certain verification on whether the ending really occurs. It can be done by comparing how the game acts after it ended normally, with how it acts after it was glitch-ended. Missing some critical ending routines would mean it was not really completed.
Input length
There are two basic opinions as to what kind of TAS ending is better:
- Minimize input length, the game should reach a stage where it completes itself, no matter how long it takes.
- Game must be brought to a certain completion state as soon as possible, after which no normal game input can prevent the game from completing, even when this lengthens the movie file length considerably.
The author can tweak their movie's ending as a stylistic choice, even after it was accepted (but before it is published). However, a run looking to obsolete a published run needs to do more than just tweak the movie's ending, as we aim to avoid meaningless publications. A run with real improvements is free to use a different ending than previous movies did.
Post-completion input
Sometimes extra input is required after the game ending sequence has started, so it could fully proceed to the last ending screen. It is preferred to contain such input in the submitted movie, but there are conditions under which such extra input can be excluded, and only applied during encoding.
The author may forget to mention that this extra input is required, to include it in the main movie, and to provide a secondary movie containing it. To prevent publishing movies and encodes that lack some part of the ending, always manually ensure the game has fully ended. For example, after the game reaches some screen that does not seem to automatically fade out nor lead to the game start again, save a state and try pressing a few buttons to see if it's possible to proceed further (switch to recording if needed).
If there is indeed extra input required by the ending sequence, but not provided by the author, ask them to either include it in the original movie, or to post a secondary one that has this input, so that movie could be used for encoding.
Endless games
Games without a clear ending which keep repeating can still have a defined completion point according to our rules. The mechanics of the game must be reviewed to ensure that the author(s) chose a valid ending point which conforms with our rules.
If the author(s) chose an earlier ending point from the list we offer, and another which shows off more of the game is also reasonably viable (the movie won't go on for days), the judge should ask the author(s) if they'd like to play more and provide an updated movie. If the movie is accepted for publication with an earlier ending point, the judgment message should include remarks specifying which completion point was reached, and identify what other more complete options are viable which would be accepted to obsolete the movie in question.
Hacks
By nature, hacks have different criteria for judgment compared to regular games.
- A hack must be judged for its entertainment value as its own separate game but also in context to the original game.
- Too many hacks of the same game engine causes the same issues as too many categories of the same game.
- Quality of the hack should come into play as well. Hacks where only sprites are altered do not make for quality hacks. By contrast, a good hack alters the levels, the physics, the sprites, expands the overall game play, and even combines elements of other games to the point it doesn't feel like the same game engine.
- Popularity of the hack needs to be considered.
Be open
Collect opinions
The audience has voice. If they don't use it, encourage them to use it.
Read what other people think of the submission.
- If nobody says anything, try to ignite discussion.
- This is especially important if the submission does something which could be controversial once people realize it.
- If someone offers an interesting viewpoint regarding a submission, try to cultivate it.
- Minority opinions in a discussion are important too, and should be weighed according to their validity.
Be impartial
Judge movies, not players.
Be verbose
Inform people of your actions, your thoughts and everything regarding a particular submission.
Try to express yourself in an encouraging tone, even if you are going to reject a submission.
Editing the submissions
- Apply the Editor Guidelines to your editions. Specifically:
- Do not delete text written by others. (But you can reformat it for readability.)
- Use clear, easy to read markup and language.
- Attribute clearly.
When you add a comment to the submission, your markup should look like this for example, so that it links to your profile
[user:your username]: Rejected in favor of a [828S|faster movie].
(
828S (an example) stands for a link to submission number 828.)
Replacing submission files
Judges have the ability to replace a submission file. This can be done when the submitter finds a small improvement to their submission; it is usually better to incorporate it immediately rather than cancel and submit a new movie.
Check that the movie still satisfies its stated goals and that the movie file is good. Make sure it still syncs.
Use this feature only for small improvements and not major changes. Anything that could possibly invalidate the voting results should definitely be a new submission (such as a controversial trick, a major improvement, or anything that might affect the entertainment value of the movie).
Procedures
There is a minimum time a submission can be in queue before a verdict can be made (currently 72 hours). Prior to this time a submission can be set to "judging underway" or "needs more info".
Use the "judging underway" feature, and use it sooner rather than later. If you decide to handle a submission set this first so other judges know it is already being taken care of.
Let the conversation play out. The audience has a say, so let them say it. If a thread is very active still (many posts in a short time period), let it sit until the conversation dies down a bit.
Give a chance for rebuttal. Try to indicate the verdict you are leaning to in a post, or ask questions rather than make statements ("Why did you do x?" instead of "x looks improvable"). Give the author a sufficient chance to respond to assumptions made about the movie.
Avoid saying it "looks" improvable. Instead, take the time to show that it IS improvable. Show an existing WIP, or make a short one of your own. Document some specifics rather than making blanket statements impossible to prove.
When you're done
Even after you've finished judging a movie, a judge's job doesn't necessarily stop there.
- Communicate with viewers who have questions regarding your judgment.
- Communicate with the publisher and assist them as necessary.
- Communicate with other judges who need to judge a different submission for the game if they have any questions.
- If it was revealed you made a mistake in your judgment, first of all, don't beat yourself up, everyone makes mistakes.
- Any mistake other than those which resulted in a published movie is fixable.
- If the movie was already published, speak to the other staff, maybe someone has a creative solution. If not, it happens, revel in your ability to be human.
- Learn from your mistakes for next time.
- Any mistake other than those which resulted in a published movie is fixable.
More reading
- For notable/long judgments: Notable Judgments/Longest Judgment Notes
- For movie analysis: Movie Rules, Guidelines.
- For editing: Editor guidelines, Text formatting rules.
- For movie editing: Movie Tag Guidelines.
If you're also a publisher, please read the Publisher guidelines as well.