The above graphic would give the impression that the tas has exactly average ratings,
No, for the reason I mentioned above. The average rating on a five-point scale is expected to be above four
You are still missing the point completely. I am not talking about the subjective meaning of "average", as used in colloquial language when judging a work of art. Nor am I talking about people's behavior when they estimate and rate such things. I am not talking about how the results should be subjectively interpreted from a psychological perspective in terms of the quality of the work. I am not talking about "four stars means that the work is meh". You seem to still be clinging to your complete misinterpretation of my original question "how would you interpret this graph?" I was not asking for a psychoanalysis of the mentality of the people who have rated the work, or an essay on typical human behavior.
I was talking about how the graphic visually misleading, giving the impression that the ratings are exactly half-way through the scale, when in reality they are significantly lower than half-way. Forget "average", since you seem to have so many problems in understanding what that word means. Think of the more mundane "half-way between lowest and highest" concept instead.
The reason for that is that the first star is extraneous, always lit, and makes the graphic misleading.
The reason is that you think of a five-point scale as 0 through 4, whereas most people think of a five-point scale as 1 through 5. That's why the latter is the standard and the former is not.
But the problem is that when you display the results using five stars, it leads to a misleading result. People don't think that "oh, the range is 1 to 5? That means I should ignore the first star in the image and just look at the four remaining ones." They will look at the entire image and see that the highlighted portion is covering exactly half of it, and thus come to the intuitive but wrong conclusion that the ratings are also likewise completely evenly split, when that's very far from the truth.
Too bad this forum doesn't have a "report for personal attacks" button. How about we get back to the actual discussion instead of flaming people?
Games are basically math with a visual representation of this math, that's why I make the scripts, to re-see games as math.
My things:
YouTube, GitHub, Pastebin, Twitter
Personally, I'd prefer a rating system that, rather than 1 to 10, or any "1 to x", is instead balanced around zero. "Extremely boring", "Boring", "Mildly boring", "Neutral", "Mildly fun", "Fun", and "Extremely fun", or something to that extent.
However, I'm not posting to talk about the "honesty" or flaws of any particular rating system. A few days ago, I've been following on the discussions of problems involving how to get people to rate movies (I lurked this thread). Back in the "good ol' days", people rated a lot. Now? Not so much. I don't care for a topic on trying to get the best possible rating system that is unambiguously good on all fronts, as it will give no benefit if there is no one using it.
It can be worth discussing the flaws of a rating system, perhaps even in its own topic, but the issue of getting votes and the issue of a more honest form of rating are separate, whatever relation it has. I want things to see more on how to draw in votes. And I don't want "just create a new thread" for an answer when this is the very thread I should be seeing that in. (The post starting this discussion did begin by pointing out this is a tangent to the prior discussion)
Consider this post to be a request to split this thread into separate topics. I don't exactly want to stop one or another, they're both good discussion points, but these are, in my view, two distinct things where posts involving one do not advance the other appreciably.
Joined: 4/7/2015
Posts: 330
Location: Porto Alegre, RS, Brazil
FatRatKnight wrote:
I've been following on the discussions of problems involving how to get people to rate movies (I lurked this thread). Back in the "good ol' days", people rated a lot. Now? Not so much.
For sure there are other reasons for this, but I guess some simple changes could help people rate more, like giving more space for things that are more important in the movie page, like this:
Games are basically math with a visual representation of this math, that's why I make the scripts, to re-see games as math.
My things:
YouTube, GitHub, Pastebin, Twitter
One simple change, which wouldn't require modifying the backend almost at all, would be to embed the 0-10 dropdown menu into the "rate this movie" line in the movie description, and add some javascript magic to make it update (either whenever it's changed, or when a button besides the dropdown menu is clicked) without going to a different page. The decimal part menu could simple be left out (and assumed to be 0 in this case).
I think this has been suggested in the past by people, and I think it would perhaps be the solution that requires the least amount of meddling with the backend code, and thus perhaps the easiest to implement.
I'm in the process of watching and rating every active movie.
1841 total active movies.
553 active movies rated as of today.
(695 obsolete movies rated.)
Which will happen first: I reach 100% or the rating system gets revamped?
(My goal is to reach 100% by 31 December 2018.)
I'm in the process of watching and rating every active movie.
1841 total active movies.
553 active movies rated as of today.
(695 obsolete movies rated.)
Which will happen first: I reach 100% or the rating system gets revamped?
(My goal is to reach 100% by 31 December 2018.)
I rate tech based on how much improvement I think is possible. 10.0 means a game has been pushed to its limit. What would be the point of either ent or tech ratings if they were inherent qualities of a game (e.g. if everyone gave Atari games a 1 tech rating)? For both entertainment and tech, I think I should rate what the author has some control over.
Super Adventure Island seems fairly straightforward, and so I'm surprised it's that flawed, but I'll lower its tech rating. Giving so many high tech ratings may appear unfair to movies like [3358] GBC Pokémon: Yellow Version "arbitrary code execution" by MrWint in 05:48.28. But in those cases, the technical skill is a direct part of the entertainment, and so they get both high entertainment and technical ratings.
I rate tech based on how much improvement I think is possible. 10.0 means a game has been pushed to its limit.
It's not my place to tell people how they should rate, or interpret the meaning of the ratings, but in my opinion it's not necessary to be able to rate a run with 10.0 on either entertainment or technical quality.
I don't think "technical quality" should mean solely "how frame-perfect the run is" (and thus if the run is demonstrably frame-perfect and cannot be improved further, it should get a straight 10.0). It ought to encompass more than that. A lot more.
It's impossible to prove, or even know, if a run is frame-perfect. Or how frame-perfect it is. Therefore such an interpretation would be rather nonsensical. Nobody would be able to give an objective rating, because nobody can know how frame-perfect it is. (It may well even be something that's mathematically impossible to prove.)
I was one of the people, or even the person, who proposed and implemented the original entertainment&technical ratings (although Bisqwit also made a big chunk of that original server code as well. There might have been more people involved, at least in the idea part, but it has been so many years now that I can't remember all the details anymore.) My original idea for "technical quality" was not "how frame-perfect it is". Rather, it was more like "what kind of techniques the run uses, and how well it uses them."
If a run looks technically cool, it should receive a higher technical score than a run that doesn't look as cool. An example of a run that, to me, looks extremely cool in terms of technique, is the Megaman run.
The Super Mario Bros run is one that has gotten technically better over time. Not because it has become faster, but because more techniques are being used. (For example, when the flagpole glitch was found, it upped up the technical coolness of the run, for instance.)
Sometimes technical quality can only be discovered by reading the background material of the run (ie. its submission text). Sometimes the amount of technical work put into the run can be enormous, and is only visible through this text. And that ought to contribute to the score as well.
Is "how technically cool the run looks" a subjective thing to judge? Yes. And that's completely fine! It's supposed to be! Just because it says "technical" doesn't mean it can't be subjective. It's about technique, not about (frame) perfection. (Of course perfection counts, but not as the sole thing.)
In this way, not all games lend themselves to the highest technical score. Perhaps the game is just too linear, too straightforward, too simple, and with too few bugs and glitches to exploit. The run is probably very boring, not because the author made a bad job, but because the game itself is boring when it comes to speedrunning it. You shouldn't feel bad giving it a bad entertainment and/or technical score. It's not the fault of the author, but it was just a bad game choice.
Just my opinion.
Joined: 12/28/2013
Posts: 396
Location: Rio de Janeiro, Brasil
I agree with Warp. Just like most games don't have the potential of a 10 entertainment rating, many also don't have the potential of a 10 technical rating, even if you literally can't save a single frame.
But anyway, nice job rating every movie, Arc. Even if I disagree on your parameters, the distribution shows you were probably consistent with them.
We have Voting Guidelines for understanding better how submission votes and movie ratings should be applied. In particular, I want to note the following paragraph, as it's relevant to the current discussion:
Voting Guidelines wrote:
Note that not all games are suitable for a TAS with perfect technical rating, similarly to how not all games are suitable for a perfect entertainment rating. Some games simply don't lend themselves for extensive technical achievements (eg. if they are too simple or straightforward, with no route planning, exploitable bugs, etc.).
Warp wrote that paragraph on the voting guidelines page. It sounds too much like games have an inherent, objective technical value when these are supposed to be subjective user ratings. You have to believe that any movie can be 0-10 in either category, as long as the user can explain why they honestly feel that way. You may disagree, and that's why you have your own rating. Dragster is the best example of a very simple game that has been extensively examined to exhaust its limits, and so I gave it a 10 tech.
I think that "cool techniques" belongs more to the entertainment category. Zipping in Mega Man adds greatly to the entertainment, but it also clearly saves time, so it contributes to tech in that way. We don't know what the perfect time is for most movies, but tech rating shows appreciation for the effort that the author put into trying to perfect the movie. Tech rating comes from not just how optimized the TAS looks but also the details on the submission page and forum discussion.
Another factor is comparing movies today with the very early movies. I know what it was like to make movies in early 2004. They were often completed in days or even hours, utilizing little more than the save-state feature. Those are the kinds of movies that I would give a 0 tech rating. Whereas most movies made today have relatively far greater effort put into them, and so most of them earn a high technical rating.
Dragster is the best example of a very simple game that has been extensively examined to exhaust its limits, and so I gave it a 10 tech.
I agree that the run deserves a high technical score. But not because it can't be improved (that's part of it, but only a part), but because of the sheer amount of work involved in background research, and proving that the time is optimal. That work alone is worthy of a high technical score.
I would rate it low on entertainment. After all, not much happens in the run that's extraordinarily entertaining. (I would say that the "entertainment" score would more reflect how enjoyable the run is to watch assuming you know nothing about the background work, the amount of work put into creating the run, or even the game itself. Simply whether watching it out of the blue, as a fan of speedruns, is enjoyable.)
So it's a good example of high technical rating, low entertainment rating.
I've been talking and wondering why exactly people haven't been rating as much. I would like for people that don't rate post-publication to explain why.
[16:36:31] <Mothrayas> I have to say this argument about robot drug usage is a lot more fun than whatever else we have been doing in the past two+ hours
[16:08:10] <BenLubar> a TAS is just the limit of a segmented speedrun as the segment length approaches zero
For me personally, most of my energy spent watching and giving an opinion on a movie is when it’s in submissions. Once it’s actually published I don’t usually bother giving it a rating right away (unless it’s a movie I really like or a game I really like).
Typically when I decide to spend some time rating a bunch of movies I go through this list: http://tasvideos.org/MovieStatistics/LeastRated.html. I feel that this list shows the types of movies that most desperately need ratings (i’m also salty because a majority of my movies end up on this list, so i dont want others to end up there either). Unfortunately the list is filled with a ton of movies that I’ve already rated so there’s not really anything I can do about it.
So generally I dont usually do rating as soon as movies are published, but rather in chunks based off of lists like these.
[14:15] <feos> WinDOES what DOSn't
12:33:44 PM <Mothrayas> "I got an oof with my game!"
Mothrayas Today at 12:22: <Colin> thank you for supporting noble causes such as my feet
MemoryTAS Today at 11:55 AM: you wouldn't know beauty if it slapped you in the face with a giant fish
[Today at 4:51 PM] Mothrayas: although if you like your own tweets that's the online equivalent of sniffing your own farts and probably tells a lot about you as a person
MemoryTAS Today at 7:01 PM: But I exert big staff energy honestly lol
Samsara Today at 1:20 PM: wouldn't ACE in a real life TAS just stand for Actually Cease Existing
Joined: 8/14/2009
Posts: 4083
Location: The Netherlands
Memory wrote:
I've been talking and wondering why exactly people haven't been rating as much. I would like for people that don't rate post-publication to explain why.
This question has been asked a lot of times, and usually it comes down to the rating form being too much hidden away, requiring too many clicks to use, and just being over-complicated in general.
Since it's also detached from the submission process, usually when people watch a submission they often don't come back to the publication again for the same movie, and hence skip over the opportunity to rate it.
Nothing has changed of this since the last time it was asked, either, as we haven't had any site development on that end.
http://www.youtube.com/Noxxa
<dwangoAC> This is a TAS (...). Not suitable for all audiences. May cause undesirable side-effects. May contain emulator abuse. Emulator may be abusive. This product contains glitches known to the state of California to cause egg defects.
<Masterjun> I'm just a guy arranging bits in a sequence which could potentially amuse other people looking at these bits
<adelikat> In Oregon Trail, I sacrificed my own family to save time. In Star trek, I killed helpless comrades in escape pods to save time. Here, I kill my allies to save time. I think I need help.
I think even rating things can be confusing. Like any time I ask someone to rate a movie, they sometimes get confused at what rating to give it. Like they understand the single digit ratings like 5, 6, 7, but the decimals are confusing too, like what makes somthing earn a 5.7 instead of a 5.8?
Imo it would make things simpler if we kept the decimals at like .0 or .5 for those who cant decide between a 6 or 7, and go for 6.5 instead.
[14:15] <feos> WinDOES what DOSn't
12:33:44 PM <Mothrayas> "I got an oof with my game!"
Mothrayas Today at 12:22: <Colin> thank you for supporting noble causes such as my feet
MemoryTAS Today at 11:55 AM: you wouldn't know beauty if it slapped you in the face with a giant fish
[Today at 4:51 PM] Mothrayas: although if you like your own tweets that's the online equivalent of sniffing your own farts and probably tells a lot about you as a person
MemoryTAS Today at 7:01 PM: But I exert big staff energy honestly lol
Samsara Today at 1:20 PM: wouldn't ACE in a real life TAS just stand for Actually Cease Existing
I've been talking and wondering why exactly people haven't been rating as much. I would like for people that don't rate post-publication to explain why.
I enjoy creating TAS content much more than I enjoy simply watching TAS movies. Therefore, I'd rather spend my free time creating new TAS content or updating an old one than watching runs of already published games simply to rate them.
With how little attention is paid by members in regards to doing the rating post publication, it does beg the question of how important post-publication ratings are in the first place.
Who actually cares and pays attention to these ratings other than those doing the ratings? (legitimate non-rhetorical question)
Frankly, I don't.
If I'm interested in watching a particular movie, a low rating will not discourage me from watching it. Likewise, if I have little to no interest in watching a particular movie, a high rating won't miraculously make me suddenly decide to watch it. My limited viewing of TASes is based on game interest not movie rating.
For that matter, I don't really care much how my own movies are rated (pre or post publication). If they are good enough for publication, I'm satisfied. Even when one gets rejected, I can still feel a sense of accomplishment for completing the project.
Who actually cares and pays attention to these ratings other than those doing the ratings? (legitimate non-rhetorical question)
It's one of the factors used in determining tiering. In fact it's pretty much the only factor post publication that can result in a tier change. Of course you could argue about how much tiering matters, but it very much affects whether or not certain TASes can be published here at the very least.
EDIT: I would not be against the idea that ratings should be phased out and other methods be used to determine tier.
Ratings are also used in calculating player's points but your mileage may vary on whether or not those matter.
[16:36:31] <Mothrayas> I have to say this argument about robot drug usage is a lot more fun than whatever else we have been doing in the past two+ hours
[16:08:10] <BenLubar> a TAS is just the limit of a segmented speedrun as the segment length approaches zero
Who actually cares and pays attention to these ratings other than those doing the ratings? (legitimate non-rhetorical question)
It's one of the factors used in determining tiering. In fact it's pretty much the only factor post publication that can result in a tier change. Of course you could argue about how much tiering matters, but it very much affects whether or not certain TASes can be published here at the very least.
EDIT: I would not be against the idea that ratings should be phased out and other methods be used to determine tier.
Ratings are also used in calculating player's points but your mileage may vary on whether or not those matter.
I mostly understand what the ratings are used for regarding the site. But if so few are concerned with the ratings to begin with, should we really be using them to determine these things on the site?
Disclaimer: I don't have any other suggestions for tier changes. But I don't like the idea that ratings affect player points. I personally feel player points (as a quantitative value) should be based more on the quantity of currently published content that person has produced, not on how others qualitatively perceive the content that's been produced; I have no problem with losing points due to obsoletion.
Minor Side Note: You're accurate in assuming that the player points don't mean a whole lot to me either.
This is my distribution of entertainment ratings. You may disagree with individual ratings, but as a group the ratings are a normal distribution.
Most of the highest rated movies are Zelda, Metroid, Mario, Castlevania, Sonic, and Mega Man. But here are my top 5 "underrated" movies (excluding my own creations):
5. [2608] GC Crazy Taxi by solarplex in 05:53.18. It really is a crazy taxi.
4. [3389] SMS Golden Axe Warrior by zoboner in 30:49.10. It's like Zelda on the SMS.
3. [3158] GBA Spider-Man 3 by arandomgameTASer in 13:44.01. Fast and dazzling movement.
2. [3334] NES The Goonies II by McBobX in 10:53.64. Surprising high-speed glitching.
1. [3006] GB Pac-In-Time by slamo in 27:31.45. It's easily the best Game Boy movie.
I also created the Quasars, a list of the 100 most kino TASes.
