adelikat
He/Him
Emulator Coder, Site Developer, Site Owner, Expert player (3585)
Joined: 11/3/2004
Posts: 4738
Location: Tennessee
Currently our movie rating consists of two parts: Entertainment Rating and Technical Rating. Let's talk about technical ratings. Many (most?) people have used this rating value incorrectly historically. They use it as a measure of how close to optimal they think the movie is. When used this way, it is a way inflated, and mostly meaningless value. Most movies get a 7 or higher. Simpler movies tend to get high marks because it seems clear they would be hard to improve. I rated many movies this way, and it was clear to me, this rating has almost no value. At some point, I changed the way I rate to the original intent, which is to rate it based on how many "techniques" are demonstrated in the movie. Things such as obvious luck manipulation, route planning, use of glitches, etc. This seems like a more useful value, at first. But after rating a number of movies I found something obvious. Movies that had more aspects to it like this, I tended to like more, for obvious reasons. TASes that have a variety of TAS techniques are generally more entertaining. Go figure. So technical rating, in this context, seems to have no value either. It is just a value strongly correlated with entertainment value. Let's talk about the cons. 1) It adds complexity and confusion to the user, making it less likely for them to participate. Participation in movie ratings is what we want most as it drives engagement and gives us values we can't do interesting things with. 2) Site complexity. We have the technical burden of maintaining the data and the code surrounding the receiving, displaying, and calculating of this value. Not to mention the performance costs of having to calculate this value frequently, and further code complexities of caching that calculation to mitigate that cost. Given the cons and limited value of this value, it has been my conclusion for awhile now that this value should be removed, and leaving a single value that represents the overall impression by the user.
It's hard to look this good. My TAS projects
Samsara
She/They
Senior Judge, Site Admin, Expert player (2121)
Joined: 11/13/2006
Posts: 2793
Location: Northern California
Murder it. Tech may have worked in the very early days of the community, when this was essentially just a niche hobby and everyone was more or less involved in every aspect, but as things grew it became more and more clear that people just come here because TASes are fun to watch. Evaluating the "tech rating" of a TAS is not something worth doing to most of the community these days, especially when the meaning of "technical quality" is far more subjective than even individual entertainment. Evaluating a movie's tech rating is quite literally what I do for the site, too, even down to the original intent of what tech rating was supposed to be. Movies that forego these techniques will get rejected if using them is faster, no matter how simple or complex the game is otherwise, so in that case every single published movie has needed to meet a minimum standard of tech quality, which to me makes a community rating entirely redundant. There's no inherent value to any tech rating either, in my opinion. What is a low tech rating from the community supposed to imply? Is it meant to imply that the Judge made a mistake? Is it meant to imply that the TASer didn't try hard enough? Is it meant to imply that the game itself is just bad? Any of those could be true for any TAS, but a simple 1-10 rating would never tell you that just by looking. On the opposite end, what is a high tech rating supposed to imply? Agreement with the Judge's decision to accept it? An indicator that the community recognizes a specific instance of hard work on a run? Or is it just a 10 because it's a game end glitch run and people are like "wow, how technical"? Again, any of these could apply to any TAS, and the truth wouldn't immediately be clear. Worst of all, what the hell is a middling 5 or a 6 rating supposed to represent? The "you tried" star? tl;dr it bad, pls delete
TASvideos Admin and acting Senior Judge 💙 | Cohost
warmCabin wrote:
You shouldn't need a degree in computer science to get into this hobby.
Editor, Skilled player (1404)
Joined: 3/31/2010
Posts: 2086
adelikat wrote:
We have the technical burden of maintaining the data and the code surrounding the receiving, displaying, and calculating of this value. Not to mention the performance costs of having to calculate this value frequently, and further code complexities of caching that calculation to mitigate that cost.
I would honestly be surprised it this was a significant factor compared to everything else the site has to do. Anyway, meh. I don't really rate movies to begin with because I don't think I'm able to fairly express a complex opinion about a movie with a simple number. While there is clearly the problem of subjectivity for the entertainment rating, the technical rating is not much better. How can I compare two movies against each other and quantify their quality fairly if I know the games differently well? Obviously I'm gonna give a higher rating to the game I know well because I have a better understanding of what had to go into making the TAS, regardless of how much effort or ingenuity actually went into the runs. I also agree that the definition of the technical rating itself is vague and left to the interpretation of the user. Frankly, yeet it. Streamlining the rating system probably makes it easier for everyone.
Darkman425
He/They
Editor, Judge, Skilled player (1065)
Joined: 9/19/2021
Posts: 242
Location: Texas
As a note I've watched videos here for at least a decade before registering and I had no clue that there was a split of entertainment and technical ratings. Take that as you will. The whole technical rating is hard to really get a grasp for a viewer. Sometimes the tech isn't obvious and requires reading the submission notes to get. Not everyone's going to have time to read those notes so that makes it trickier to determine. If a technique isn't something visually obvious, exactly how would that affect the tech rating? Also there's the problem of asking the viewer what techniques make something more "technical" than others when there are situations where that can be a debatable point. I say it's best to leave technical ratings behind. If the a run got published on the site, that already implies that the tech behind the movie is good enough for the standards of the site already. You probably don't need to have the viewers put in the work that's been done already.
Switch friend code: SW-2632-3851-3712
TiKevin83
He/Him
Ambassador, Moderator, Site Developer, Player (119)
Joined: 3/17/2018
Posts: 348
Location: Holland, MI
I agree with removing technical ratings. It's somewhat redundant with the judging process except that the judge actually goes through and checks technical quality to some extent so their pass/fail has much more weight. A viewer or reader of the submission text has no clue whether anything happening has technical value towards some crazy glitch or something (the published Yellow NSC TAS has to -intentionally jump back over a ledge- to keep a glitch preserved) and so may think a movie is suboptimal at face value, it's not something that can really be rated just from a viewing audience.
Experienced player (994)
Joined: 1/9/2011
Posts: 227
I like having more options of being able to critique a movie, but I agree that there’s no particular value in the technical rating. I can’t recall a single movie that we’ve wanted published because it was so technically complex. Entertainment has been the driving factor in most of the site changes, so I agree that technical rating could go away.
adelikat
He/Him
Emulator Coder, Site Developer, Site Owner, Expert player (3585)
Joined: 11/3/2004
Posts: 4738
Location: Tennessee
Wow, I expected the poll to be a little less one sided :) Since it seems pretty clear we are going to do this, let's talk about the migration. Rather than just remove the technical ratings, I was thinking that we calculate the "average" of the user's rating and replace entertainment with that. The formula being: entertainment+entertainment+tech / 3 I think this helps give a closer value to the overall rating someone might have given. "Boring but impressive, 3e, 9t" I don't think would translate to a person just rating it just a 3 under the new system? Opinions?
It's hard to look this good. My TAS projects
Noxxa
They/Them
Moderator, Expert player (4138)
Joined: 8/14/2009
Posts: 4083
Location: The Netherlands
The site already used to maintain a 13:7 ratio between entertainment and tech in order to calculate the overall rating for every movie. If every user rating is merged to the same ratio, then it would keep every movie's overall rating (mostly) the same. That should be the way to go, imo.
http://www.youtube.com/Noxxa <dwangoAC> This is a TAS (...). Not suitable for all audiences. May cause undesirable side-effects. May contain emulator abuse. Emulator may be abusive. This product contains glitches known to the state of California to cause egg defects. <Masterjun> I'm just a guy arranging bits in a sequence which could potentially amuse other people looking at these bits <adelikat> In Oregon Trail, I sacrificed my own family to save time. In Star trek, I killed helpless comrades in escape pods to save time. Here, I kill my allies to save time. I think I need help.
Site Admin, Skilled player (1236)
Joined: 4/17/2010
Posts: 11268
Location: RU
Noxxa wrote:
The site already used to maintain a 13:7 ratio between entertainment and tech in order to calculate the overall rating for every movie. If every user rating is merged to the same ratio, then it would keep every movie's overall rating (mostly) the same. That should be the way to go, imo.
Agreed.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
adelikat
He/Him
Emulator Coder, Site Developer, Site Owner, Expert player (3585)
Joined: 11/3/2004
Posts: 4738
Location: Tennessee
Tech ratings are GONE! I also replaced all entertainment ratings with what would have been the calculated average rating. Overall ratings of movies are very similar to before, only differing slightly due to rounding differences. Thanks to everyone that participated in this conversation :)
It's hard to look this good. My TAS projects
Arc
Editor, Experienced player (768)
Joined: 3/8/2004
Posts: 534
Location: Arizona
adelikat wrote:
I also replaced all entertainment ratings with what would have been the calculated average rating.
I have no idea why you thought this was a good thing to implement. You wiped out every entertainment rating with almost no time given to discuss the idea, and you buried it within a topic that was supposed to be about simply removing the tech rating. You created the thread arguing that tech rating has no value, but then you used this supposedly worthless tech rating to calculate a phony "average" (weighted) rating. I did not post in this thread earlier because I did not object to removing the tech rating. But I stated in two (2) previous posts that I wanted to be able to sort by entertainment rating, which is a clear indication that entertainment rating matters to me. Of course those posts did not receive any sort of acknowledgement. This "average" rating does not reflect how I intended to rate movies. You are well aware that I have seen and rated every published movie. This is a graph of my entertainment ratings: I maintain a nearly perfect bell curve. Whereas this "average" rating turns my entertainment ratings into a jumbled mess. The movies that I intended to rate the lowest are now at some random point in the bottom 100. I am shocked (not really) that the staff would fail to take another person's perspective and consider that each user put a considerable amount of thought into every entertainment rating that they gave. Very, very disrespectful.
Noxxa
They/Them
Moderator, Expert player (4138)
Joined: 8/14/2009
Posts: 4083
Location: The Netherlands
The point of removing tech rating was never to make entertainment rating the sole measure of rating TASes. TASes can have merits other than being entertaining, and ratings should be able to reflect that as well. For example, I would rate [4567] NES Super Mario Bros. 3 "game end glitch" by OnehundredthCoin in 00:00.22 low in entertainment rating due to lack of entertaining gameplay, but on the whole, it is a high-quality and notable achievement, so I would give it a higher overall rating. Not doing so would be like giving Citizen Kane a low score on IMDB because it isn't a comedy. So ultimately, having a singular rating system isn't really compatible with either the notions of entertainment or technical rating, so we can't provide compatibility for either. What we did do, was to have compatibility (as much as was possible) with the overall ratings for TASes as they existed previously. It's unfortunate that this ruined your system of entertainment rating, but entertainment ratings themselves technically now also don't exist anymore, so it is what it is.
http://www.youtube.com/Noxxa <dwangoAC> This is a TAS (...). Not suitable for all audiences. May cause undesirable side-effects. May contain emulator abuse. Emulator may be abusive. This product contains glitches known to the state of California to cause egg defects. <Masterjun> I'm just a guy arranging bits in a sequence which could potentially amuse other people looking at these bits <adelikat> In Oregon Trail, I sacrificed my own family to save time. In Star trek, I killed helpless comrades in escape pods to save time. Here, I kill my allies to save time. I think I need help.
Arc
Editor, Experienced player (768)
Joined: 3/8/2004
Posts: 534
Location: Arizona
Noxxa wrote:
The point of removing tech rating was never to make entertainment rating the sole measure of rating TASes. TASes can have merits other than being entertaining, and ratings should be able to reflect that as well.
The thread was not framed that way. It was purely about getting rid of the technical rating. The poll asks "Should we remove technical rating?" It does not ask about merging entertainment and tech into a single rating based on a formula. There was no indication that entertainment ratings would be altered until the bait-and-switch merger idea was hastily done by the staff. The staff callously overwrote the ratings users actually gave—evidently the site staff know better than the rest of us what we "really" intended with our ratings. These are not users' ratings anymore. This seems like something that is not just morally wrong but possibly a legal issue; for example, the site is now falsely claiming that "Arc" gave the Discworld II movie a "3" rating when I never did—that movie is a "0" for me. This is false authorship attribution—literary forgery. These ratings are your creation, but falsely attributed to my name, against my will.
Noxxa
They/Them
Moderator, Expert player (4138)
Joined: 8/14/2009
Posts: 4083
Location: The Netherlands
The formula was already well-established, it is how movie ratings (as they are shown on each publication page) are calculated to begin with. It is not a new invention. If someone for example gave a movie an 1 in entertainment but 10 in tech, then that effectively translated to an overall rating of approximately 4.2 (because of the 13:7 split). What this change did, is make the given overall rating explicit. If parts of the rating were removed without the remaining numbers getting compensated, then numbers would be far more skewed. The 1 entertainment/10 tech example from before would turn an overall rating of 4 into an overall rating of 1. And that would be simply wrong. The rating number you see now is not called "entertainment rating". It does not represent what you filled in as entertainment rating, nor does it claim to. It represents the overall rating you gave to the movie, when you filled in all the applicable numbers.
http://www.youtube.com/Noxxa <dwangoAC> This is a TAS (...). Not suitable for all audiences. May cause undesirable side-effects. May contain emulator abuse. Emulator may be abusive. This product contains glitches known to the state of California to cause egg defects. <Masterjun> I'm just a guy arranging bits in a sequence which could potentially amuse other people looking at these bits <adelikat> In Oregon Trail, I sacrificed my own family to save time. In Star trek, I killed helpless comrades in escape pods to save time. Here, I kill my allies to save time. I think I need help.
Player (26)
Joined: 8/29/2011
Posts: 1206
Location: Amsterdam
adelikat wrote:
Given the cons and limited value of this value, it has been my conclusion for awhile now that this value should be removed, and leaving a single value that represents the overall impression by the user.
Good call. I am reminded of dating site OkCupid, that used to have a feature where you could rate other users on appearance and personality. And after some time they checked their data, and it turned out that almost everybody used both ratings for "how much do I like this person". The two were almost 1:1 linearly correlated, so there was no point in having two separate axes for a single user; and so they removed it. The same principle applies here: it adds complexity for no practical benefit. And I'm really glad that this site is willing to analyze its own "sacred cows" and (re)consider if they're worth keeping. Just because something "has always been done like that" doesn't automatically mean that we need to keep doing it that way in perpetuity.
Arc
Editor, Experienced player (768)
Joined: 3/8/2004
Posts: 534
Location: Arizona
Noxxa wrote:
It represents the overall rating you gave to the movie, when you filled in all the applicable numbers.
I don't agree that I ever gave an overall rating. I authored an entertainment rating. I authored a technical rating. The site calculated a number from there. The problem is the principle that the site staff have granted themselves the power to edit users' content, while still attributing authorship to the user, i.e. forgery. It is just as wrong as if staff were stealth-editing the text of users' forum posts. THE SOLUTION would be to have an opt-in/opt-out choice, where each user could individually decide whether to (a) keep their original entertainment rating as their singular rating or (b) accept the weighted calculation as their singular rating.