Joined: 4/17/2010
Posts: 11473
Location: Lake Chargoggagoggmanchauggagoggchaubunagungamaugg
Rerecording emulators have been tracking rerecord numbers (how many times you loaded a movie savestate) for a long time, and they've become a standard stat for TASVideos publications. There were only a few exceptions:
Movies where a brute-force kinda tool was used (usually called a bot), for which we used to exclude botted rerecords from the main count
Movies where the real rerecord count is not known, and we set it to 0, which the site parses as "unknown"
Movies where the rerecord count doesn't reflect the real picture, but was left as is
Movies where the rerecord count was set to some arbitrary value
Over the years, some questions about how rerecords are tracked and used on the site have accumulated.
In an ideal scenario, no rerecords are lost or bloated, and a well done movie usually has a rerecord count that is bigger than the frame count. Which usually makes people who love statistics happy, because they can appreciate the work that went into the movie. But...
Sometimes low rerecord count is used as a reason to question the movie, without actually looking into its optimality, and sometimes authors are even asked questions about why the number is so low.
Rerecords cannot reflect research and development behind the scenes
Some people work with several movies at once, and their rerecords don't even reflect their actual TAS work (unless they explicitly sum everything up)
Regardless of research and development, rerecords can be set to an arbitrary number, which makes them a completely meaningless stat in those cases
Even legitimately high rerecord counts won't tell a lot about actual movie quality, because to understand it one has to understand the underlying challenge, which in turn requires facing the same problems as the author did. And that may be impossible because the ground work has already been done, or if the author is good at writing, it can be described in the submission text
So we started wondering if it still makes a whole lot of sense to keep tracking rerecords on the site the way we've been doing it all along, or maybe something is worth updating. For example I wouldn't want low rerecord count to be used as an argument against the movie, or even authors anticipating it and explaining in advance why it's low. Movie quality should be assessed by actual gameplay. On the other hand, making rerecord display optional on the site (or even removing it entirely) would imply more dev work, and maybe some people actually want to show this stat about their movie.
So I'm asking for opinions, and I got an idea myself. What if we hide rerecords on the site and don't use it in publications, unless the author has provided it in the submission description?
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Joined: 10/12/2011
Posts: 6435
Location: The land down under.
I personally like seeing the rerecord counts, like it's nice to see it cleanly in the arrangement it has in the Publications.
Like it's been part of Publications since the beginning of the site, so like it would be something weirdly miss, at least to me.
It's also a very useful thing to gauge the quality of work behind a TAS, something feos points out as a detractor.
In writing that yes, there's proof where there are TASers who can make something with an extremely high time + low rerecord count and still be extremely optimal.
However, to me it also helps quickly explain something that can validate a "oh so that's why the author spammed shooting, and was blatantly missing targets instead of being accurate this is something that needs to be brought up and inform the author to take their time in the sections".
I don't want to personally always download a usermovie, check for the issues manually and get something out of that.
It's a good thing to have if there's something amiss in someone's work, and it's extremely helpful to at least give pointers to newcomers who do come back swinging back with something that they put the work into.
And even when it's a low rerecord count with a high time, you can still get an "oh, wow" element out of it in the reactions of those that watched the Publication.
This is either going to be straightforward or diverse in the opinions, but I'm against removing it,
I've always liked seeing that back when I was watching the TASes before I joined almost 12 years ago, and I still personally like seeing it.
To me it would just be a void if it was gone.
WebNations/Sabih wrote:
+fsvgm777 never censoring anything.
Disables Comments and Ratings for the YouTube account.Something better for yourself and also others.
Ambassador, Moderator, Site Developer, Player
(154)
Joined: 3/17/2018
Posts: 357
Location: Holland, MI
I agree with Alyosha, I think keeping rerecord count would attribute relevance or utility to it that it really doesn't have because of the number of problems listed. Even a basic heuristic for a new TAS like "ratio of rerecords to frames" ignoring all of the other problems like hand bk2 edits and external botting is broken by the fact that a game could run at half or a quarter of the system framerate internally and would have no value across different games. If people really want to keep it around, there should be a warning icon posted with the label explaining its deficiencies.
Regarding rerecord counts in submissions:
Spikestuff makes an excellent point regarding rerecord counts being used to validate/explain why a poor/high quality movie is poor/high quality. And this particular use of rerecord counts (assuming they are accurate in the first place) is a very valid reason to (at minimum) maintain keeping the rerecord count in emulator input files.
Yet, I'm inclined to agree more with feos regarding listing of rerecord counts; in that we limit publication of rerecord count only to movies where the author explicitly requests it.
Specifically, listing rerecord count in submissions provides potential viewers an opportunity to pre-determine whether or not they think a submission will be of poor or high quality. This anticipated quality level may then bias their actual viewing of the run and cause them to deem the run either as having more/less quality than they would otherwise have done had they not started with a preconceived expectation. This could occur in a couple different ways depending on what the initial expectation is.
1) RR count is low and quality is assumed poor by the prospective viewer; who then deems a truly poor quality run as even worse than they would have otherwise, or a good quality as significantly higher than they would have otherwise because it significantly exceeded their expectation. Beyond this, it's also possible that a low rerecord count may bias a potential viewer's expectations so severely that they may choose not to even watch a run that they may otherwise enjoy and/or be impressed by.
2) RR count is high and quality is assumed good by the prospective viewer; who then deems a truly good quality run as even better than they would have otherwise, or a poor quality as downright awful because it didn't come close to their expectation.
Bottom line, rerecord counts should never be used to set one's expectations for a submission. They should ONLY be used, as mentioned above, to explain why a movie is the quality that it is. Rerecord count alone should never be used as a reason to NOT watch a movie.Regarding inclusion of rerecord counts on official publication encodes:
As published runs have already been deemed of high-enough quality to be published in the first place, including the rerecord count here doesn't really serve much purpose, in my opinion. I can't really see any particular pro to having it listed in the encode. I can however see some potential cons:
1) A high/very high rerecord count may potentially detract other TASers from trying to beat the current publication. This could theoretically limit discoveries from being found because no-one looked into the game further.
2) A low rerecord count may unfortunately dupe a TASer (particularly newer TASers) into thinking that not much work was needed for a current publication and that they may be able to beat it easily. Then when they actually get into the process of TASing the game, they become discouraged for not being able to beat something they expected would be easy to beat.
3) Having rerecord count listed on publications provides for a method of competition that really shouldn't be a method of competition on our site. Some TASers may want to have the highest count, others may want to have the lowest. I recall someone a while ago in the discord server effectively boasting about making a TAS with legitimately zero rerecords because they seemed to think that was something to be proud about. (Not that there's anything wrong with being proud of any legitimate accomplishment, but it was a perfect example of the wrong view of our site). Our site is based primarily on optimization quality with a secondary emphasis on entertainment...not on how simply/complexly one achieves that goal.
Regarding rerecord counts in general:
I primarily don't put much stock in the values presented due to issues of accuracy/tracking including (but not limited to): emulator crashes, using multiple project files instead of branches within one project file, and manual editing ability of the input file itself.
A rerecord count is only truly comparable to another when it's regarding the same game, and then only when both movies are being authored by TASers who hold equal knowledge of the game prior to starting the construction of the TAS. Any difference in knowledge will impact what one TASer does over the other and may result in more/less rerecords due to said knowledge. The discrepancy between rerecord counts then wouldn't be a fair comparison of actual work put into the TAS.
Comparing rerecord counts of one game to another makes no sense whatsoever. For one game, 10,000 rerecords may be quite a lot; while for a different game, 100,000 rerecords may be miniscule.
Low rerecord counts don't mean poor quality. If a game is simple enough to understand from an optimization standpoint, it may not take many rerecords to produce a truly optimized run (Duck Hunt: Game A). Likewise, high rerecord counts don't mean high quality, they just mean stuff changed a lot. It's absolutely possible to spend thousands upon thousands of rerecords optimizing a sub-optimal route.
All this shows, in my opinion, that the value presented by rerecord count only holds true meaning after a movie has already been watched. And only then as a data point aiding in clarification on why the resulting movie is the quality that it is. For this reason, I feel that if someone truly feels they need to know the rerecord count before watching a run, the impetus should be on them to look it up from the input file. We shouldn't blatantly provide it ahead of time.
TL:DR I agree with feos. Going forward, we should eliminate public display of rerecord count on submissions and publications' encodes unless explicitly requested by the author in their submission notes. I don't feel it's necessary to retroactively change all current/obsolete publications (not that we couldn't...someone may just be really bored someday and want something to do).
All that said: If rerecord counts are maintained, given the inherent issues I have with the statistic, I see no good reason to continue truncating botted rerecord counts. To me, a rerecord is a rerecord regardless of how that change was produced. So if we are going to keep them around. I feel we should stop automatically reducing botted runs to 0 rerecords (or any other arbitrary number just because that's what the author believes they did manually).
I don't feel there is any definitive purpose of rerecords.
We could say that it allows us to quickly identify movies that are likely low quality. Yet, actually it can't unless we look at other data. But that other data would already tell us the answer regardless of the rerecord count. For example, movies that are improvements on past publications. In some cases, but not all, the rerecord count is going to be very low. Does that mean that the movie is low quality or that very little effort was used? Impossible to say with just the rerecord number.
We could say that in light of some witnessed behavior in the movie, that the rerecords offers a quick explanation of that behavior. But does it really? We don't know which sections of the movie the rerecords are from. Did the author struggle in one part which turned out great, and then simply missed/overlooked time save in another part? Was the author just highly skilled or knowledgeable about the mechanics of the particular game? Or were they completely clueless? We can't know for certain with just the total number of rerecords.
It doesn't appear to me that rerecords are a good indicator of anything. At best, it can create speculation, doubt, or suspicion.
In considering what to say on this topic, I tried to calculate some statistics of movies on TASVideos (using the site's API). It was incredibly difficult to make any reasonable comparison based just on the rerecord value. To me, there's no obvious correlation between rerecords and anything else about a movie.
For example, here are two movies. Both are in the Stars class, both from the same console, of similar length, and both were highly rated by viewers as entertaining. Yet, when we compare the number of frames to the number of rerecords, they are entirely different:
What do the rerecord values tell us about these movies? Which one is better than the other? Which one had more effort and work put into it? The answers to these questions are not obvious.
Here's two more movies. Again both Stars, same console, both fairly long length, highly rated by viewers. Yet again, the difference in rerecords vs frames is significantly different. Additionally, compared to the two Genesis movies above, the rerecord vs frame differences are fairly similar despite these movies being drastically longer.
I like it a lot. I would be very sad if it was removed.
There's not a sort by variable for it though, which makes me sad for sniping low rerecord counts. Usually it's pretty accurate and pretty accurate for quality depending on the movie.
Obviously all things have to be taken with a grain of salt and determine the quality of the movie alone based on it would be silly.
I will say it's outdated as hell, the system itself. It comes from a time of basic TASing. Doesn't seem perfect with TAStudio.
Apparently it adds a count every time BizHawk generates a new green zone, but whatever I'm doing in TAStudio feels like it should yield a much higher number.
Published TASes: #1, #2, #3, #4, #5, #6, #7, #8, #9, #10, #11, #12Please consider voting for me as Rookie TASer Of 2023 - Voting is in December 2023My rule is quality TASes over quantity TASes... unless I'm bored.
I don't feel there is any definitive purpose of rerecords.
We could say that it allows us to quickly identify movies that are likely low quality. Yet, actually it can't unless we look at other data. But that other data would already tell us the answer regardless of the rerecord count. For example, movies that are improvements on past publications. In some cases, but not all, the rerecord count is going to be very low. Does that mean that the movie is low quality or that very little effort was used? Impossible to say with just the rerecord number.
We could say that in light of some witnessed behavior in the movie, that the rerecords offers a quick explanation of that behavior. But does it really? We don't know which sections of the movie the rerecords are from. Did the author struggle in one part which turned out great, and then simply missed/overlooked time save in another part? Was the author just highly skilled or knowledgeable about the mechanics of the particular game? Or were they completely clueless? We can't know for certain with just the total number of rerecords.
It doesn't appear to me that rerecords are a good indicator of anything. At best, it can create speculation, doubt, or suspicion.
In considering what to say on this topic, I tried to calculate some statistics of movies on TASVideos (using the site's API). It was incredibly difficult to make any reasonable comparison based just on the rerecord value. To me, there's no obvious correlation between rerecords and anything else about a movie.
For example, here are two movies. Both are in the Stars class, both from the same console, of similar length, and both were highly rated by viewers as entertaining. Yet, when we compare the number of frames to the number of rerecords, they are entirely different:
What do the rerecord values tell us about these movies? Which one is better than the other? Which one had more effort and work put into it? The answers to these questions are not obvious.
Here's two more movies. Again both Stars, same console, both fairly long length, highly rated by viewers. Yet again, the difference in rerecords vs frames is significantly different. Additionally, compared to the two Genesis movies above, the rerecord vs frame differences are fairly similar despite these movies being drastically longer.
What do the rerecord values tell us about these? Anything? Nothing? Who knows. Only a deeper study of the movie will give us answers.
Zelda (this era) is a much simpler game, with easier movement. That's definitely reflected in the count.
Honestly I don't removal of anything is a light decision, and I would lean towards something not being removed that is near useless of some people find enjoyment in it and it also has no negative effects. I think the potential negative effects of removing it are greater than any perceived negative effect of keeping it, that is the best way of saying it
Published TASes: #1, #2, #3, #4, #5, #6, #7, #8, #9, #10, #11, #12Please consider voting for me as Rookie TASer Of 2023 - Voting is in December 2023My rule is quality TASes over quantity TASes... unless I'm bored.
Zelda (this era) is a much simpler game, with easier movement. That's definitely reflected in the count.
Reflected in what way? The linked example movie required more rerecords than frames, which would seem like the opposite of what you said. If the movement was easy, then why were there more rerecords? And more importantly, how do you know that the rerecords were because of the movement and not something else?
OtakuTAS wrote:
it also has no negative effects
But it does have negative effects, as feos pointed out:
feos wrote:
Sometimes low rerecord count is used as a reason to question the movie, without actually looking into its optimality, and sometimes authors are even asked questions about why the number is so low.
Rerecords cannot reflect research and development behind the scenes
Some people work with several movies at once, and their rerecords don't even reflect their actual TAS work (unless they explicitly sum everything up)
Regardless of research and development, rerecords can be set to an arbitrary number, which makes them a completely meaningless stat in those cases
Even legitimately high rerecord counts won't tell a lot about actual movie quality, because to understand it one has to understand the underlying challenge, which in turn requires facing the same problems as the author did. And that may be impossible because the ground work has already been done, or if the author is good at writing, it can be described in the submission text
Joined: 11/14/2014
Posts: 927
Location: South Pole, True Land Down Under
I have to admit, I'm liking the arguments here, but all I can offer is this:
I do a TON of BOTing, which bloats the re-record count; however, it was true effort in searching for the end result. With an upcoming release, my new BOTing will now require that I manually change the value to be 16 times that of a single instance....since I have moved into a new type of speedy BOTing. (Multi-Instance)
As for manual effort, re-records on my part do show my effort. As time goes by, my re-record count is getting lower and lower...only because I have improved over the years and don't need to do as much experimenting to figure things out.
Now...I can be fine with it being left for public viewing, but I am finding myself not looking at it anymore.
I recently discovered that if you haven't reached a level of frustration with TASing any game, then you haven't done your due diligence.
----
SOYZA: Are you playing a game?
NYMX: I'm not playing a game, I'm TASing.
SOYZA: Oh...so its not a game...Its for real?
----
Anybody got a Quantum computer I can borrow for 20 minutes?
Nevermind...eien's 64 core machine will do. :)
----
BOTing will be the end of all games. --NYMX
Zelda (this era) is a much simpler game, with easier movement. That's definitely reflected in the count.
Reflected in what way? The linked example movie required more rerecords than frames, which would seem like the opposite of what you said. If the movement was easy, then why were there more rerecords? And more importantly, how do you know that the rerecords were because of the movement and not something else?
OtakuTAS wrote:
it also has no negative effects
But it does have negative effects, as feos pointed out:
feos wrote:
Sometimes low rerecord count is used as a reason to question the movie, without actually looking into its optimality, and sometimes authors are even asked questions about why the number is so low.
Rerecords cannot reflect research and development behind the scenes
Some people work with several movies at once, and their rerecords don't even reflect their actual TAS work (unless they explicitly sum everything up)
Regardless of research and development, rerecords can be set to an arbitrary number, which makes them a completely meaningless stat in those cases
Even legitimately high rerecord counts won't tell a lot about actual movie quality, because to understand it one has to understand the underlying challenge, which in turn requires facing the same problems as the author did. And that may be impossible because the ground work has already been done, or if the author is good at writing, it can be described in the submission text
At the end of the day everything that feos said is a people issue, users incorrectly placing too much weight and judgement on the count and being brash if they're voicing those concerns publicly without other evidence of a rushed run.
The number should be taken with a grain of salt, but that does not mean it should be removed because a few do not do so.
Published TASes: #1, #2, #3, #4, #5, #6, #7, #8, #9, #10, #11, #12Please consider voting for me as Rookie TASer Of 2023 - Voting is in December 2023My rule is quality TASes over quantity TASes... unless I'm bored.
