Honestly I really dislike the very concept of a technical rating to begin with. Ultimately it either relies on the quality and thoroughness of the submission text if it is to include techniques that are not visible via watching or it simply tells how many techniques were visible via watching which will probably reflect in entertainment score regardless and doesn't seem that useful to me. Pretty much all other forms of media only have one rating metric: entertainment. When you rate a movie on say IMDb, you don't get offered a second rating for cinematography. It's just not needed.
It's more like traditional figure skating rating system, which rates the performance based on three categories: Technical merit, required elements, and presentation.
In this case it's just technical merit and entertainment..
This question has been asked a lot of times, and usually it comes down to the rating form being too much hidden away, requiring too many clicks to use, and just being over-complicated in general.
When that part of the backend code was developed, Ajax techniques did already exist to a large extent, but they were still rather new. The implementation unfortunately uses very old-fashioned web forms from the 90's.
A more convenient implementation would be just some kind of rating scale (which is visible and editable if you are logged in) directly on the movie's description box which you can simply click at any time to change, and it will update the rating in real-time without having to explicitly submit anything.
A rating scale between 0 and 10, with no decimals, would probably be just fine. (In retrospect it might have been a mistake to add the decimal part.)
But somebody would need to implement that...
Technical could mean many things. What about runs that are extremely time consuming but boil down to brute force approach to find really fast sections? It may only use a single trick but may be dozens of hours. But should another run that uses some assembly information that only took the author a couple minutes to acquire get a higher rating?
Entertainment is also poorly explained. Look no further than Masterjun's Mario 3 run. The video itself contains next to no game play but you had people rating wildly differently for different reasons. Although I think this is the better of the two criteria.
Since 99.9% of people misunderstand what "technical rating" means, it might be a good idea to invent a new name for it. These 99.9% of people seem to think that it's a synonym for "frame perfection", and therefore every single game could theoretically have a 10 for technical rating. And no amount of explanations about how that's not what it's supposed to mean seems to help. I think the only thing that would help is changing the name of the rating category.
Don't think of it as a techniCAL rating, but as a techniQUE rating. Subtle but big difference at the same time.
It's quite a subjective "coolness" rating. How many impressive speedrunning techniques are being used (both visually on screen, and in the background work that was necessary to make the run). Don't think of it has clocking someone's time in a 100-meter sprint race, but the techniques used in a skateboarding half-pipe competition.
Not all games lend themselves for a perfect technical rating in the exact same way that they may not lend themselves for a perfect entertainment rating. Maybe they are too straightforward and there just isn't much there to show any cool TASing techniques and knowledge.
And it's completely OK to be subjective about it. If you personally think that it wasn't technically very impressive, that's fine. Just like with the entertainment rating.
Which, by the way, IMO should be more about how enjoyable the run was to watch independently of the technique aspect. Again, highly subjective and personal, but that's just fine.
I think you typoed there.
An interesting version of the problem could be posed by using an infinite amount of hoses/resistors (although using the hoses version is more misleading because it doesn't immediately become apparent what formula you need to use):
Suppose one hose can fill the container in 1 hour. Another hose can fill it in 4 hours. A third hose in 9 hours. A fourth hose in 16 hours, and so on and so forth (going through the square numbers). How long does it take to fill the container if you use this infinite amount of hoses all at the same time?
Which reminds me of: What's the answer to the problem posed in xkcd #356?
Isn't his proposal to simply change the text of the answers?
No; see the first post of this thread.
I did.
DrD2k9 wrote:
Implementation (At least for the second suggestion)
If this would indeed be an approach desirable to the community...How difficult would it be to implement this on the site?
I recognize that the coding for the workbench polls would need changed. The submission list page shouldn't need much change as it could maintain the vote percentages (using much the same calculation as is currently utilized) by the following variable conversions:
(old question answers) = (new question answers)
Yes = Moons
No = Vault
Meh = I Don't Care
This would still offer 3 entertainment level choices in the poll while yielding a general % entertainment value that would theoretically be more accurate than results generated by the current polling method.
If that suggested solution is not simply changing the text of the answers, what exactly am I missing here?
Well, I'm all in favor of temporarily changing the forum to test things; but then we should really be testing DrD2k9's proposal that started the thread, rather than Warp's; because the former is more comprehensive.
Isn't his proposal to simply change the text of the answers?
I don't really know how you would measure the "success" of that test.
I think though, we have historical evidence that suggests it won't be better than what we have now.
How much work would it be to at least try it for a while and see how it pans out? If it turns out that everybody's just giving 5 stars to everything, I'll admit I was wrong.
I got it: The problem can be posited like this:
"A water hose can fill a container in 3 hours. Another water hose can fill it in 5 hours. How long does it take to fill the container if both hoses are used at the same time?"
This can be extended to any number of hoses, and I'm assuming the resistors-in-parallel formula can be used (even though I haven't thought about how to prove that). It's also probably conceptually closer to the resistors scenario.
More coincidences. I presented elsewhere the simple problem that I did here before: "It takes Mark 3 hours to walk from town A to town B, and it takes Julia 5 hours to walk from town B to town A. Assuming they start walking at the same time, and walk at a constant speed, how long does it take for them to meet?"
After the answers I noted that the generic formula for the answer, (X*Y)/(X+Y), works for any pair of times X and Y.
Somebody noted that that's actually the formula for the total resistance of two resistors in parallel.
We couldn't figure out if that's just amazing coincidence, or if there's a correlation between the two. I guessed that it's not coincidence.
Then I started wondering if the generic equation for parallel resistors, eg. for n resistors in parallel: 1/R=1/R1+1/R2+1/R3+...+1/Rn could be posed as a similar problem as above.
If, however, the poll is "Rate this run", with a grade of 0 to 5, it becomes extremely clear that this isn't asking whether to publish the run or not.
Why would that be the case? It's pretty easy to jump to the conclusion that 0-2 means don't publish, and 3-5 means do publish. Just like how e.g. a college test might map 0%-59% to "you fail" and 60%-100% to "you pass".
I honestly think you are reaching now.
Why would anybody think "0-2 means don't publish"?
If you are so worried about it, use literal star images for the poll answer (with zero stars being a possibility).
It is my understanding that the poll is treated this way, at least in part, due to the ambiguity of the results; this ambiguity stemming from the fact that some users answer the question regarding degree of entertainment, while others answer the question as if the movie in question should be published or not.
My suggestion is exactly a proposal for a solution to that. It makes it clear that this is a popularity vote, and gives nuance to the degree of likeability that can be expressed in the poll itself even without an additional post to the thread.
To clarify what I mean: The current "yes/no" poll question may give people the wrong impression that this is a "should this be published?" poll. (It really doesn't matter that the poll question is "did you like this run?" People don't really read that, and instead read a "yes/no" answer and draw conclusions from that.)
If, however, the poll is "Rate this run", with a grade of 0 to 5, it becomes extremely clear that this isn't asking whether to publish the run or not. There is no binary option potentially causing confusion to the user as to what exactly it means. It becomes much clearer that the poll is asking how much they liked the run.
And I'm suggesting 0 to 5 because IMO it gives enough nuance for most people in expressing how much they liked it, but not too many possible values as to become overwhelming and meaningless. 0 to 5 is, IMO, in that sweet spot between too few and too many.
As someone who's historically had trouble figuring out how to rate things, there is a big problem with a 5-star rating system: It's arbitrary.
So? All opinions are arbitrary. "Yes/no/meh" is arbitrary. Whether to publish something on Vault or Moons is ultimately arbitrary and up to opinion. You cannot remove arbitrariness from this. Arguing that it's "arbitrary" is like arguing that "it's on a web page". Of course it is. So what?
Unless you specify and explain benchmarks and criteria to earn each of the stars
You don't need to specify anything. It's a popularity vote. It's asking for people's opinion on how much they liked it. You cannot quantify that mathematically.
A second problem is strategic voting.
That can never be solved, and only few people engage in it. It doesn't make much sense to worry so much about it.
In short, you're trying to solve the problem of 'the workbench poll isn't clear enough and people use it to vote strategically' by exacerbrating both problems.
That's not the problem. Read the original post.
DrD2k9 wrote:
It has been brought up a few times recently (roughly in the last year or so) that the workbench entertainment polls can be (and in some cases are) essentially ignored by a judge when determining publication tier, especially if explanation/content in the thread itself is contrary to the poll results. There are even times that the poll may be essentially ignored simply because there is little to no discussion in the forum at all.
[...]
It is my understanding that the poll is treated this way, at least in part, due to the ambiguity of the results; this ambiguity stemming from the fact that some users answer the question regarding degree of entertainment, while others answer the question as if the movie in question should be published or not.
My suggestion is exactly a proposal for a solution to that. It makes it clear that this is a popularity vote, and gives nuance to the degree of likeability that can be expressed in the poll itself even without an additional post to the thread.
I honestly don't see a problem with my suggestion.
Frankly I see your suggestion as less clear than the current situation, so I don't see how that would help.
Can you elaborate on that?
5-star rating systems are literally everywhere. You would need to be a complete hermit to have never encountered such a thing. If you encounter this:
Rate this run:
[ ] 0 [ ] 1 [ ] 2 [ ] 3 [ ] 4 [ ] 5
what exactly is there to be confused about? Tick one of the boxes, and that's it. What's unclear about that?
I don't think it's such a huge problem as you make it sound like.
Some people will tick that 5 if they want the run published on Moons, but I do believe that they are the minority of people who vote in these submission polls. I think the rest can be safely given the option to express their opinion with more nuance than just "yes/no/meh".
I think it would be valuable information for the judges. If a run has only 4's and 5's, it ought to be a clear indication that it's very well liked. If the majority of the votes are 2's and 3's, with perhaps a couple of outlying 5's, it may be an indication that this is a Vault publication.
I honestly don't see a problem with my suggestion.
Perhaps you linked to the wrong post, because it doesn't answer my question in any way.
Mothrayas wrote:
The guideline in question states votes are only one of multiple factors to take into account when judging tiers. It does not say "80% = instant moons". It gives it a decent case, of course, but it is not conclusive or guaranteed.
Mothrayas wrote:
The idea of the voting question (as it is currently) is to ask about entertainment. Optimization or publishability are determined by the judge and not by votes. The problem is that, regardless of what you ask of the viewers, they will vote their own things which will on occasion not align with the actual question being asked. That is something that cannot be changed (we tried, and basically failed). People still vote "yes for Vault" nearly three years after tiers have been introduced and the voting question changed accordingly.
This also goes for movie ratings - no matter how exactly we want to define 'technical' rating etc., people will invent their own qualifiers and vote according to their own ideas. Even for entertainment rating, which everyone can get at least a decent idea of, their metrics can still be all over the place. Anyhow, there is another topic about this.
I don't see anything here that would answer the question of what's wrong with my suggestion of making the poll question to be:
Rate this run:
[ ] 0, [ ] 1, [ ] 2, [ ] 3, [ ] 4, [ ] 5
I agree with this tbh. There’s nothing about it that strikes me as super entertaining or super impressive.
There's nothing "super entertaining" about Cola Kid, either.
I don't think starred runs should by necessity be "super entertaining" or "super impressive". They should be representative of what TASing is about. Of course being "super" helps with that a lot, but I wouldn't say it's an absolute necessary requirement.
I'm not sure why we insist on having exactly one SMS game starred, but if there must be one, Golden Axe Warrior strikes me as the better pick.
I think the idea is diversity. Diversity in the types of runs, in the types of games, and even on the types of consoles. We wouldn't want 20 Super Mario and 20 Zelda runs in the starred list (no matter how awesome all of them might be).
I have another idea. How about the poll question being like:
Rate this run:
[ ] 0, [ ] 1, [ ] 2, [ ] 3, [ ] 4, [ ] 5
(The question could also be eg. "how much did you like this run?")
I don't think the poll requires any more explanation than that. Also, that kind of poll would be less likely to be confused with "should this be published?" (which I feel the current one resembles a bit too much.)
Yes, this would conceptually overlap with movie ratings, and some people would suggest them to be merged (and IIRC this was tried at one point and it didn't really pan out), but I really think that a submission poll should both have more nuance than just 3 options, but doesn't need as much nuance as the current rating system. A rank from 0 to 5 ought to be more than enough for people to express a justifiable opinion with enough nuance.
I have sporadically objected to the notion that Vault be considered some kind of garbage dump where undeserving "boring" runs are shoved into if they aren't "good enough". Some/many people seem to have the attitude that's it's pretty much effectively a sign of shame if the run is relegated to Vault and doesn't get enough recognition to get the privilege to get into Moons.
In other words, the Vault tier has a really bad reputation.
It just occurred to me that the poll question probably helps reinforcing this notion, which is a shame.
Unfortunately I don't know what the solution to this is (even assuming it could be solved).
