This actually makes sense only if your monitor has just the right gamma :)
On my laptop's LCD, if you watch it from an upwards angle, it starts looking right, but if I watch it straight, not so.
Yes, but in general it's a far better ideal than the other two. (Besides, it was a quick hack and not meant to account for anything other than color brightness.)
For me it seems to be mainly 4 horizontal bars (black, medium, light, white), though I don't think that's an issue of this so much as crappy vision.
Joined: 11/18/2006
Posts: 2426
Location: Back where I belong
After a few months of using the new system, I have to admit that it is deeply flawed. Much like communism, the rating system works great in theory, but utterly fails when it is applied in the real world. Many borderline submissions are artificially bolstered by people who are seemingly unable to make intelligent ratings, such as shown here and here. The ratings of recently published movies have become incredibly inflated, and have rendered the entire idea of ratings nearly useless (many people would argue that the technical rating was useless beforehand, but that's already been covered). One recent example is adelikat's Earnest Evans, which jumped by nearly 2 full points.
As I was the one who proposed the new system, I figured that I would be the best person to say that it simply isn't working out well. While the yes/no/meh system also had its flaws, I don't believe this system is any better, and has caused the ratings system to become even less reliable and helpful. I don't know what the best solution would be, but as of right now, I'm starting to think that removing the polls altogether and making people voice their opinions through a post would be the best. But seeing as how I proposed the current system, I may not be the best person to listen to at the moment.
I don't think that any system we implement will be perfect, but the current system has been too polluted by non-discriminatory raters to be of any use. Any ideas on how to make the submission discussion more productive and less crappy?
Emulator Coder, Site Developer, Site Owner, Expert player
(3572)
Joined: 11/3/2004
Posts: 4754
Location: Tennessee
I also was a big proponent of the new rating system, and after seeing it in action, I'd say it was a failure.
I'd like to see a return to the yes/no/meh system. I know a lot of people who have quit voting/contributing on submissions once this was removed. That is the exact opposite of what we want.
I'd like to keep the ability to rate submissions, however. It could be kept hidden and not apart of the submission process, but it would be nice to rate a movie when I watch it and not have to remember later when it is published. I only suggest this because it seems easy now that the ratings have been implemented. If it is hassle, it probably isn't worth it.
Emulator Coder, Site Developer, Site Owner, Expert player
(3572)
Joined: 11/3/2004
Posts: 4754
Location: Tennessee
Another thought would be to weight peoples ratings based on their average rating. So if someone rated all 10's, then a 10 is an average rating and would be weighted as a 5.
Then the average rating would truly reflect which movies people thought were awesome.
But still, this doesn't take into account the issue that we have turned away a lot of good/verteran TASers & viewers with the change away from a voting system. Turning away these community members doesn't sound like progress.
I know a lot of people who have quit voting/contributing on submissions once this was removed. That is the exact opposite of what we want.
I guess I would likely be one of those persons. After all, the votes that are currently cast aren't anonymous like yes/meh/no were (at least I haven't found a way to do so). Thus after publication the runners could come back to me in a "you only voted 6.2 instead of 6.2, I don't like you anymore" kind of way which I don't like/want. So before getting myself in such a pickle, I rather avoid voting at all.
Joined: 6/13/2006
Posts: 3300
Location: Massachussetts, USA
I thought there was an old option in user forum settings where you could hide your ratings in published movies; is that feature gone now?
I agree with adelikat; the "calculate average" thingie seems like a good way to fix the problem. I've noticed a large number of users constantly giving high ratings to everything they see.
I know a lot of people who have quit voting/contributing on submissions once this was removed.
*raises hand*
adelikat wrote:
Another thought would be to weight peoples ratings based on their average rating. So if someone rated all 10's, then a 10 is an average rating and would be weighted as a 5.
Hot or Not does that, yet there is still a whole shitload of girls with 9+ ratings. Many people just can't avoid to say the magic word "awesome", apparently.
Comicalflop wrote:
I thought there was an old option in user forum settings where you could hide your ratings in published movies; is that feature gone now?
Emulator Coder, Site Developer, Site Owner, Expert player
(3572)
Joined: 11/3/2004
Posts: 4754
Location: Tennessee
I didn't realize nineko & shiny were in that group. That makes about 5 people I know of that were really good about watching all submissions that no longer vote. And the average vote count for the lesser known games is about 5-10, so 5 is certainly a significant % of viewers.
I've been less active in the workbench forum since the switch, because whenever I vote, I feel I should post and explain. And it's a lot easier to support "yes" or "no" than "8.4/9.7" or "3.2/5.0"
How fleeting are all human passions compared with the massive continuity of ducks.
Another thought would be to weight peoples ratings based on their average rating. So if someone rated all 10's, then a 10 is an average rating and would be weighted as a 5.
While I agree that this is a good idea, doing it exactly like that probably wouldn't be good as it would make some people who constantly rate 10/10 drag down highly rated movies.
Oh, and if the big problem here is the inflation in the average score (which I think) then it would probably not help to just make the ratings of the submission hidden until it is published. Because the main reason that the ratings have been inflated is just that more people have started to vote on movies, many of whom don't find use for numbers lower than 9, and hiding the score would not change that.
Anyway, I'm all for change. Optimally I think that the judging should be based solely on what people say about the movie, but it greatly increases the work for the judges and less popular movies would likely end up with little to no useful comments at all. So I guess that leaves us with the yes/no/meh system or some new idea that someone might come up with.
I didn't realize nineko & shiny were in that group.
I voted a bunch of times when the new system was implemented, but I quickly stopped doing so.
As for suggestions about how the voting system should be, what about if we go with mmbossman's rubric? Each submission can be equipped with a 3x3 grid (or 5x5 to add more resolution, though I would avoid that). Then, for counting purposes, there are two feasible ways: either count the individual votes like a normal poll, or calculate the weighted average and display the position on a graph. Here is a quick mockup:
1) the voting options
2) the results in a poll fashion
3) the results in a graphic fashion
Emulator Coder, Site Developer, Site Owner, Expert player
(3572)
Joined: 11/3/2004
Posts: 4754
Location: Tennessee
Cpadolf wrote:
While I agree that this is a good idea, doing it exactly like that probably wouldn't be good as it would make some people who constantly rate 10/10 drag down highly rated movies.
I'm fine with that detail. Because the consequence is that movies with high ratings truely deserve it. In order to be the best, more people have to give you a rating higher than they normally would give. To get those top ratings, your movie really has to be the best of the best.
Oh, and if the big problem here is the inflation in the average score (which I think) then it would probably not help to just make the ratings of the submission hidden until it is published. Because the main reason that the ratings have been inflated is just that more people have started to vote on movies, many of whom don't find use for numbers lower than 9, and hiding the score would not change that.
What I meant by hiding them was to take them away from the voting process. Meaning judges and viewers aren't using the ratings in their decision making. It would then just be a convenience for those who like to rate publications. You can rate before the publication so you don't have to remember later.
'm fine with that detail. Because the consequence is that movies with high ratings truely deserve it. In order to be the best, more people have to give you a rating higher than they normally would give. To get those top ratings, your movie really has to be the best of the best.
Well I still think that it'd be better if you didn't affect the average at all if you voted whatever your exact average is. Like say you have a very good average and then 3 people who always vote 10/10 votes and it drags your average down, wouldn't it be preferable in that case that the average stays the same?
Emulator Coder, Site Developer, Site Owner, Expert player
(3572)
Joined: 11/3/2004
Posts: 4754
Location: Tennessee
Cpadolf wrote:
[and then 3 people who always vote 10/10 votes and it drags your average down, wouldn't it be preferable in that case that the average stays the same?
I honestly don't think so. Those 3 people gave your movie the same rating they would give any movie. Essentially they are calling it average without meaning to. While it might pull your rating number down, I think it would be pulling it to down to a more realistic number. And depending on how many movies these people rate, all movies are being pulled down equally so it isn't really hurting your movie rating in comparison to other movies.
EDIT: And I would rather their ratings deflate rather than inflate ratings. The only way a 10/10 could not effect the rating would be to make their rating not count. This strategy has been done to one member in the past, but it requires human interaction rather than being automatic. Also, it is less subject to abuse (maybe we don't want someone deciding who's votes should count).
Joined: 11/22/2004
Posts: 1468
Location: Rotterdam, The Netherlands
nineko wrote:
adelikat wrote:
I didn't realize nineko & shiny were in that group.
I voted a bunch of times when the new system was implemented, but I quickly stopped doing so.
As for suggestions about how the voting system should be, what about if we go with mmbossman's rubric? Each submission can be equipped with a 3x3 grid (or 5x5 to add more resolution, though I would avoid that). Then, for counting purposes, there are two feasible ways: either count the individual votes like a normal poll, or calculate the weighted average and display the position on a graph. Here is a quick mockup:
1) the voting options
2) the results in a poll fashion
3) the results in a graphic fashion
This idea wins the geek award. I kinda like it, even if it may not be all that practical.
As for me, I don't care too much about the rating of movies, as I don't think it is of that much use to the average visitor.
I've found myself never looking at the ratings. I don't think anybody really does, for three reasons:
Most people are probably not return visitors; so they won't understand the concept of speedrunning, meaning the rating of said concept has no meaning either.
Most people probably make their pick based on the game rather than the quality of the run. The game is the only relevant piece of information to them (other than perhaps the completion time) because they cannot interpret any of the other information, let alone a complex multi-variable rating system.
The completion time is already sort of a rating system. People remember how long it took to beat a game, compare that to the run's time, and then conclude it must be very good. Although this is not exactly the same as an actual rating system, it works.
Having the rating information is interesting, but I don't think it is very important information to have on the site.
Another thought would be to weight peoples ratings based on their average rating. So if someone rated all 10's, then a 10 is an average rating and would be weighted as a 5.
Then the average rating would truly reflect which movies people thought were awesome.
I say this idea is very good. Moreso, it should have been implemented since the start, since it apparently fixes the inflation problem very efficiently. People will have to give out lower ratings to make their high votes more significant; at the same time, people who already utilize a good spread of votes in their rating won't have to worry about anything. BUT! There is always a possible backlash of crappy voters go and rate any old, bad, or otherwise non-significant submissions with 0/0 just to make the other votes count. I could say I'm sure it's still better than what we have now, though, but that's just me. I can't tell if it will actually be better yet.
Warp wrote:
Edit: I think I understand now: It's my avatar, isn't it? It makes me look angry.
I was wondering: Could it be that some people who vote on submissions don't actually fully realize that the submission voting and movie rating have been unified? Maybe they have the notion that when they rate on the submission forum, they are voting for the submission to be published or rejected. I see how this would induce some people to vote 10/10 when they want a movie to be published.
Likewise, when someone goes to the actual rating page of a movie and rates it, he most probably fully realizes that he is now expressing an opinion of the quality of an already-published movie, and that his rating doesn't really affect anything, which can make him rate more rationally and conservatively. In this sense the yes/no submission vote was more practical (and didn't bias the ratings).
It really makes sense that while being able to rate at the submission forum sounds handy and practical, it induces this kind of bias. It just tempts people to vote 10/10 to get the movie published. Maybe it would indeed be better that only published movies were rated.
This gave me an idea for another suggestion: How about separating the submission ratings and the published movie ratings? Basically the submission ratings can be used by the voters to express their opinion and the judges to overview the general opinion, but then these ratings would be reset when the movie is published. Sure, it may seem an annoyance to have to rate movies twice, but if the current system doesn't work, it could be an improvement.
So in practice rather than having a yes/no/meh vote, you have a 0.0-10.0 vote, which is only for expressing your opinion on whether the submission should be accepted. If you want to actually rate the movie, wait for it to be published and rate it.
Btw, I think the idea of biasing a user's ratings by the average of them has merit. Every person has his own concept of what "average" means (to some it's 5.0, to others it's 7.5, yet to others it's something different even if they don't realize it themselves). OTOH it could be difficult to make this system fair.
Didn't read the last couple replies too carefully, sorry, but what sort of formula has been suggested for the weighting of people's average rating? Again, sorry if I just skimmed over it.
Another thought would be to weight peoples ratings based on their average rating. So if someone rated all 10's, then a 10 is an average rating and would be weighted as a 5.
Then the average rating would truly reflect which movies people thought were awesome.
So what if I don't like watching crappy movies on obscure, unsuitable games, and hence never have to vote low? Averaging a user's votes is only valid if the user watched an standard distributed set of movies.
It's also doesn't hurt to have our top movies actually display a 9.x on the front page. Promotion and stuff.
If we wish to avoid abuse, we could employ a rating bias metric: Calculate how close a user's ratings were to the public opinion, i.e. the difference between his rating and the total average.
For example, I rated 8 on movie A and 4 on movie B, while the average rating is 6 on A 4 on B. My opinion is on average ((8-6) + (4-4))/2 = 1 above the popular opinion. I'm a high-voter, reduce all my ratings by 1 point.
(there are some corner cases that need to be taken into account to avoid unwanted side-effects, though. Like users with only 1 vote.)
Of course, any form of normalization can be manipulated by placing a few strategic votes to shift your average. But I feel that my suggestion will give better results than just setting the rating's mean to 5.
That being said, I have an unnatural fear of voting/rating systems. Not knowing whether a 7 or 8 is fair (or rather 7.3?), I often prefer not to vote at all, to prevent giving a "false" vote. Choosing between Yes and No was easy, and for the few cases where it wasn't, there was still a friendly Meh.
1) the voting options
2) the results in a poll fashion
3) the results in a graphic fashion
