There has been a recent thread about movie rating that was about its visual representation and interface. This thread is about calculating the values.
As of today, a movie's rating is calculated as an average of the individual users' rating votes. Furthermore, each user's vote is algorithmically assigned with certain weight values based on the three-level scale:
— staff members' votes have a weight fixed to 1;
— banned users and users who have repeatedly abused the system have a weight fixed to a very low value;
— normal users (players, watchers, editors).
Normal users have additional cumulative criteria that factor into the final weight, driving it up to 1 as well:
— having a published movie (firsthand experience counts);
— number of posts (involvement with community counts too).
This system gets the job done, but doesn't cope very well with low vote amounts and heavy bias (in other words, fanboy and troll votes). So, let's gather improvement ideas that would make our movie rating less prone to abuse and more favorable towards seasoned critics, experienced players, and other TAS connoisseurs. Hopefully this will make somebody else than math nerds happier in some way. :D
I propose elaborating on the principle of weighing.
1. Ratings could factor in the mean values of each voter's personal rating history and the amount of movies watched (you can see these bits of statistics right now in our forum profiles under the "How this user rates movies"). User A who watches and rates a lot of movies and assigns to them the full spectrum of votes from 0 to 10 is expected to have a better idea of a movie's merits than user B who only votes on a few selected movies and tends towards giving out 10/10 indiscriminately. This doesn't mean that user B's 10/10 vote will drag the rating down as if it were, say, 5/5, but it does mean this vote will be more easily offset by a lower vote when calculating the average.
This measure alone will rather efficiently deal with lurkers registered to namevote a handful of familiar games or TASers, as well as diminish other kinds of abuse, since one would have to involve themself with the community and acquire themself with different kinds of movies in order to give more substance to their votes.
2. Authors' votes for their own movies, at least on the entertainment side, should have considerably lower weight due to the heavy bias involved. This bias is normal and expected; usually players have a lot of fun working on a TAS, which translates into the vote if they decide to cast it, but this entertainment is largely based on the process of TASing and not purely the result as would be expected from the rest of the viewers.
3. Instead of assigning the highest weight to staff members by default, it would seem fairer to do that to highly experienced players, as defined by player points (perhaps 1000 and above?). People with expertise in backend coding or publishing can't be expected to have the same, let alone higher, expertise in TASes as players actively making them. Besides, most of the high-rank players are judges anyway.
What are your ideas?
Warp wrote:
Edit: I think I understand now: It's my avatar, isn't it? It makes me look angry.
I'm glad this is being discussed, and those sound like some good ideas.
On another note, for a while I've been thinking that votes should be separated into votes of people that have played the game before versus people that haven't. I've seen videos on the site that are voted low, but I still have gotten a lot of enjoyment out of them since I'm familiar with the game. Also, It'd be nice to know what videos are entertaining to watch even if you haven't played the game. There are some high scoring videos on the site that I just don't enjoy because I'm not familiar enough with the game, but some are really great regardless.
On another note, for a while I've been thinking that votes should be separated into votes of people that have played the game before versus people that haven't. I've seen videos on the site that are voted low, but I still have gotten a lot of enjoyment out of them since I'm familiar with the game. Also, It'd be nice to know what videos are entertaining to watch even if you haven't played the game. There are some high scoring videos on the site that I just don't enjoy because I'm not familiar enough with the game, but some are really great regardless.
When most videos on this site already only have from 1-5 votes I do not think segmenting that value will help in the rating at all.
moozooh wrote:
1. Ratings could factor in the mean values of each voter's personal rating history and the amount of movies watched (you can see these bits of statistics right now in our forum profiles under the "How this user rates movies"). User A who watches and rates a lot of movies and assigns to them the full spectrum of votes from 0 to 10 is expected to have a better idea of a movie's merits than user B who only votes on a few selected movies and tends towards giving out 10/10 indiscriminately. This doesn't mean that user B's 10/10 vote will drag the rating down as if it were, say, 5/5, but it does mean this vote will be more easily offset by a lower vote when calculating the average.
This measure alone will rather efficiently deal with lurkers registered to namevote a handful of familiar games or TASers, as well as diminish other kinds of abuse, since one would have to involve themself with the community and acquire themself with different kinds of movies in order to give more substance to their votes.
2. Authors' votes for their own movies, at least on the entertainment side, should have considerably lower weight due to the heavy bias involved. This bias is normal and expected; usually players have a lot of fun working on a TAS, which translates into the vote if they decide to cast it, but this entertainment is largely based on the process of TASing and not purely the result as would be expected from the rest of the viewers.
3. Instead of assigning the highest weight to staff members by default, it would seem fairer to do that to highly experienced players, as defined by player points (perhaps 1000 and above?). People with expertise in backend coding or publishing can't be expected to have the same, let alone higher, expertise in TASes as players actively making them. Besides, most of the high-rank players are judges anyway.
I like number 1. It follows along the already established rules of weighing user votes and it eliminates what could be referred to as the Legend of Zelda Ocarina of Time voters who either are relatively new hardcore Zelda fanboys who consider any OoT run to be the greatest TAS on earth, and also accounts for those who vote only on the run to give it a bad review.
I am sort of surprised that it hasn't been implemented to prevent authors from rating and voting on their own runs. I am completely for this rule or even putting in place a way to prevent the rating from happening at all because voting on your own movie is the least biased opinion on its entertainment.
I also agree with point 3, but less so than number 1. I disagree with this point because there are many members of this site who contribute greatly and have a good eye for quality but do not have a published TAS. mklip2001 has never submitted a run but his opinion of what is a good TAS should hold way more weight than my opinion even though I have multiple published runs. He just has a better eye for quality even though he hasn't submitted a run yet.
Overall I agree with this idea and would add, as I have said before, that the staff should add a separate "rating" rank to the users to incentivize rating so that more people do so eliminating the movies with only 1 or 2 ratings.
Before anything is done with #1, rating needs to be made far less of a hassle. As is, i don't even bother rating anything if it it's not really great, because it just requires way too much effort (and goddamn loading 2 extra pages can take forever sometimes).
Otherwise all of this makes sense, pending how each thing would actually be weighed.
Also, i just noticed the lowest rating i've given was for my own movie :(
Joined: 4/17/2010
Posts: 11475
Location: Lake Chargoggagoggmanchauggagoggchaubunagungamaugg
I believe giving only high ratings has nothing to do with weight of the vote. Someone may really care only about awesome runs and rate them only. But have a good eye on everything. I mean, he doesn't want to give low rates to the movie he can't watch.
Also, about author's rates. The author not just remembers how he tased the game, but he knows the things that average viewer can miss during watching. The author knows all these, but other people may need to watch it several times in a row to notice. So, he really may give a reasonable entertainment rating.
What we reay need to do I think is improving our voting guidelines, reminding the important factors that help to give objective rates. See my page.
And as for players voters, I think the higher the score, the more weight the rate shall have. And at some point it shall overcome 1. Probably after 1000. And players with score around 100 definitely not have such a trained eye.
PS: The rating shall not ever depend on mood, only on real feeling about the amount of the work and on the resulting impression. And if someone votes by mood only, we shall work out a method to minimize that abuse. That;s why we shall set different weight for differently scored players.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
1. Ratings could factor in the mean values of each voter's personal rating history and the amount of movies watched (you can see these bits of statistics right now in our forum profiles under the "How this user rates movies"). User A who watches and rates a lot of movies and assigns to them the full spectrum of votes from 0 to 10 is expected to have a better idea of a movie's merits than user B who only votes on a few selected movies and tends towards giving out 10/10 indiscriminately. This doesn't mean that user B's 10/10 vote will drag the rating down as if it were, say, 5/5, but it does mean this vote will be more easily offset by a lower vote when calculating the average.
If this were implemented and public knowledge, it would be trivial to abuse: Just give 10/10 to a dozen of movies you like, and 0/0 to a dozen random movies. You have a couple of dozen rates (more than enough to get a full weight) and a full spread. In fact, this makes it only worse because now some innocent movies are being thrown some zeros they don't deserve, just to abuse the system.
(If an even more elaborate algorithm were implemented, which checks not only the range of ratings, but also their distribution, then it's probably enough to give some 5/5 ratings to a dozen of random movies more.)
rog wrote:
Before anything is done with #1, rating needs to be made far less of a hassle. As is, i don't even bother rating anything if it it's not really great, because it just requires way too much effort (and goddamn loading 2 extra pages can take forever sometimes).
Could you be a bit more specific what exactly makes the current design so much of a "hassle"?
When you wrote that post, you certainly had to load like a half dozen pages total. If your internet connection is so poor that you can't load a couple of pages, I think the problem is in your end.
Putting the rating drop-down menus in the movie's main info box would only save clicking one link, and would clutter the already-cramped info box even more than it is. (And it would be useless clutter in the vast majority of cases because 99% of the time you just watch the info box, you don't rate.)
I suggest using the median of all votes instead of the average. It's robuster and should be easy to implement.
AzumaK wrote: I swear my 1 year old daughter's favorite TASVideo is your R4MI run :3
xxNKxx wrote: ok thanks handsome feos :D
Help improving TASVideos!
On another note, for a while I've been thinking that votes should be separated into votes of people that have played the game before versus people that haven't. I've seen videos on the site that are voted low, but I still have gotten a lot of enjoyment out of them since I'm familiar with the game. Also, It'd be nice to know what videos are entertaining to watch even if you haven't played the game.
I like this idea. Separation definitely won't happen for the reason goofydylan8 stated above, but just for the sake of interesting statistics I would like such a tick to be implemented someday. For many people this would require going over their rating though, so it should be at the very least accessible from the personal rating page, allowing to select multiple entries at a time.
goofydylan8 wrote:
I also agree with point 3, but less so than number 1. I disagree with this point because there are many members of this site who contribute greatly and have a good eye for quality but do not have a published TAS.
You misunderstood. Currently the staff members have a fixed value of 1 by default; in other words, they are the reference group for the rating system. What I propose here is shifting the reference from the staff to the experienced players, which won't hurt either as there is a significant overlap between the two. For those with no published movies this measure alone will make no difference whatsoever.
feos wrote:
I believe giving only high ratings has nothing to do with weight of the vote. Someone may really care only about awesome runs and rate them only. But have a good eye on everything. I mean, he doesn't want to give low rates to the movie he can't watch.
When combined with low participation, this general approach skews the rating of already popular movies towards the upper limit so much it makes it completely pointless, and that is the effect I would like to avoid in the first place. If it's only the already high-rated movies you care about, they can do just as well without the reinforcement that is your vote.
feos wrote:
What we reay need to do I think is improving our voting guidelines, reminding the important factors that help to give objective rates. See my page.
[...]
PS: The rating shall not ever depend on mood, only on real feeling about the amount of the work and on the resulting impression. And if someone votes by mood only, we shall work out a method to minimize that abuse. That;s why we shall set different weight for differently scored players.
The problem with this is that no matter what you think people "shall" be doing, they will still do as they please; they already do that with the current system anyway. Instead of trying to influence that, it's better to adapt the system in a way that accounts for that.
Warp wrote:
If this were implemented and public knowledge, it would be trivial to abuse: Just give 10/10 to a dozen of movies you like, and 0/0 to a dozen random movies. You have a couple of dozen rates (more than enough to get a full weight) and a full spread. In fact, this makes it only worse because now some innocent movies are being thrown some zeros they don't deserve, just to abuse the system.
Fair point, but don't forget that the overall number of votes also factors into weight distribution. As such one would have to rate a lot of movies with zeros to maximize their weight, and that shouldn't be hard to track. You're probably right that the exact formula shouldn't be public knowledge, though.
Warp wrote:
Could you be a bit more specific what exactly makes the current design so much of a "hassle"?
If you haven't noticed, the site has been awfully slow. Sometimes—more often than not when I actually need something—it may take several minutes to load the page over the countless "internal server errors".
Patashu wrote:
Could you be a bit more specific what exactly makes the current design so much of a "hassle"?
If you haven't noticed, the site has been awfully slow. Sometimes—more often than not when I actually need something—it may take several minutes to load the page over the countless "internal server errors".
Changing the rating system design is not going to help that problem. It would just be fighting the symptoms rather than the root cause.
If that's the only reason to change the current design, that's a really bad one.
Changing the rating system design is not going to help that problem. It would just be fighting the symptoms rather than the root cause.
If that's the only reason to change the current design, that's a really bad one.
Reducing the number of pages that the site has to generate would reduce the load on the server, thus giving it more spare cycles to generate other pages. It's only an incremental improvement, sure, but it wouldn't be pointless.
Pyrel - an open-source rewrite of the Angband roguelike game in Python.
Could you be a bit more specific what exactly makes the current design so much of a "hassle"?
It takes a hundred thousand clicks and a million page loads to rate a movie. And/or it's 2 page loads and 10 clicks. It's one of those.
If your internet connection is so poor that you can't load a couple of pages, I think the problem is in your end.
Have you just not noticed the almost constant problems on tasvideos? 500 errors galore, and 10+ second loads happen all the time.
this general approach skews the rating of already popular movies towards the upper limit so much it makes it completely pointless,
Popular games should be rated higher. We have a lot of people around here who seem to think it's impossible to enjoy a tas of a game they've never played, which although is not true, it definitely tends to be more entertaining to watch a tas if you've played the game before, for obvious reasons. More people will find tases of popular games entertaining solely because they've played them before, and thus they should be rated higher.
If your internet connection is so poor that you can't load a couple of pages, I think the problem is in your end.
Have you just not noticed the almost constant problems on tasvideos? 500 errors galore, and 10+ second loads happen all the time.
And changing the design of the rating system fixes this problem how, exactly?
Derakon wrote:
Reducing the number of pages that the site has to generate would reduce the load on the server, thus giving it more spare cycles to generate other pages. It's only an incremental improvement, sure, but it wouldn't be pointless.
I'm getting mixed signals here. People complain that they don't rate because it's so many clicks away. A design change is suggested that would ostensibly make them rate more often.
Exactly how does this help reduce the traffic? Wouldn't it be the exact opposite?
(Besides, I don't think the problems the server is having is due to ratings. It's not like the server receives dozens of ratings per second.)
I'm getting mixed signals here. People complain that they don't rate because it's so many clicks away. A design change is suggested that would ostensibly make them rate more often.
Exactly how does this help reduce the traffic? Wouldn't it be the exact opposite?
Imagine that there's a little widget next to each movie that lets you rate it. When you use the widget, it sends a message to the server to update the rating for user "Warp" and movie 1701 to be 8.2, say. The server's extra load under this scenario is:
1) Generating the widgets for each movie on the page. This should be very cheap; the HTML is trivial (a single function definition and some straightforward HTML for each movie) and the extra database query needed to load the user's current ratings (so the widget remembers what you've rated already) should be a nearly-trivial join of movies vs. user ratings.
2) Handling rating requests; this doesn't even require a page to be loaded, just a database update to be performed; since there's no need for it to be done quickly, it can take as long as it likes.
You'd have to do many ratings for this to exceed the cost of doing a single rating under the existing scheme, where rating a movie requires generating two entire pages.
Pyrel - an open-source rewrite of the Angband roguelike game in Python.
If your internet connection is so poor that you can't load a couple of pages, I think the problem is in your end.
Have you just not noticed the almost constant problems on tasvideos? 500 errors galore, and 10+ second loads happen all the time.
And changing the design of the rating system fixes this problem how, exactly?
Derakon wrote:
Reducing the number of pages that the site has to generate would reduce the load on the server, thus giving it more spare cycles to generate other pages. It's only an incremental improvement, sure, but it wouldn't be pointless.
I'm getting mixed signals here. People complain that they don't rate because it's so many clicks away. A design change is suggested that would ostensibly make them rate more often.
Exactly how does this help reduce the traffic? Wouldn't it be the exact opposite?
(Besides, I don't think the problems the server is having is due to ratings. It's not like the server receives dozens of ratings per second.)
Uh, it wouldn't affect the site problems (at least not in any significant way). Why would it? I mentioned the slow loading times and frequent 500 errors because they are another obstacle to rating.
Joined: 4/17/2010
Posts: 11475
Location: Lake Chargoggagoggmanchauggagoggchaubunagungamaugg
I suggest the following calculation for a rate weight.
(Player points + Postcount) / 1000. Round the fraction up to tenths. By this system, my rate weight is 1.6, and adelikat's is 8.5 hehe. If the numbers look too big, just fix the formula. But I believe it indicates both aspects of being involved into the art of TASing. Players know the technical stuff, posters understand entertainment.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
I'm strictly against uncapped weighing, as it provides some people with unreasonable swaying power. In my opinion the difference between lowest unpenalized class's and the reference class's rating weight should never exceed 3:1 or so.
Also, I think post count or registration date shouldn't count towards weight upon reaching some sensible threshold (100 posts/1 year respectively, or something like that).
Warp wrote:
Edit: I think I understand now: It's my avatar, isn't it? It makes me look angry.
Joined: 4/17/2010
Posts: 11475
Location: Lake Chargoggagoggmanchauggagoggchaubunagungamaugg
This just needs some notes added to my formula, but what do you think about the concept compared to Aglar having the same weight as Velitha (nothing personal)?
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Had rating been institutionalized the same way judging is, I would trust that Aglar is more qualified to evaluate a TAS. But that's why we have at most a dozen judges in a community of thousands, and judges have way more responsibility than normal users: they actually need to watch a lot of movies that other users can safely skip or postpone until much later, they need to properly explain their opinion when they judge, they are expected to be eloquent and moderate in their expressions. Granting benefits without such responsibility would lead to elitism and unfair stratification.
Right now anybody here, even a freshly registered user, can make a clever point and deliver it better than players with dozens of published submissions under their belt. It is important that it remains that way both on the forum and in the rating, but forum posting is a system based on free expression that makes it nearly impervious to abuse. In other words, you can't persuade people verbally without certain effort. Rating system, however, is impersonal and requires no conscious analysis on both ends. The intention of the changes I am proposing aren't to penalize some users and/or give advantage to others, but to make the calculation self-regulating and better protected from low participation issues as well as perfunctory/malevolent behavior.
Warp wrote:
Edit: I think I understand now: It's my avatar, isn't it? It makes me look angry.
Joined: 4/17/2010
Posts: 11475
Location: Lake Chargoggagoggmanchauggagoggchaubunagungamaugg
I thought my proposition is dedicated to minimize the infair use of rating powers. Let's say, I am a publisher and give everything 0-4 rating because of stupid things. But I have no tasing experience myself and my rates are insanely mood-dependent. My system would minimize my harm. Current can not.
TAS experience IS a factor, and extremely important one. If you have solid formula suggestion to implement right now voice it or just repeat. It would also be good to try to consider mine, developing the whole thing.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
I thought my proposition is dedicated to minimize the infair use of rating powers. Let's say, I am a publisher and give everything 0-4 rating because of stupid things. But I have no tasing experience myself and my rates are insanely mood-dependent. My system would minimize my harm. Current can not.
Your system works if you assume that TASers (those with currently published movies in particular) are always fair, unbiased, and their votes never depend on mood, jealousy or other stupid things. Unfortunately, none of these assumptions are true. In all the significant cases of abuse on rating and voting systems thus far the culprits have been TASers. Yes, all of them.
In other words, your system solves one problem but introduces another.
feos wrote:
TAS experience IS a factor, and extremely important one.
I'm pretty sure I never argued otherwise; read the first post again.
Again, what I am against is stratification that would result from active players having significant weight advantage over the rest of the community. It is completely unneeded and only serves ego purposes. For instance, my own credibility in this community is pretty much entirely based on my forum posts. I know how to TAS, and some members here have seen the extent—but I'm easily bored/frustrated by the process and end up choosing not to spend my very limited time on it. I expect that there are many people in the same boat as myself, but if they'd been here long enough and seen enough movies (or made them long enough ago that they'd all gotten obsolete), why should they be put at such a disadvantage? Surely you can't say they can't fairly evaluate a movie?
It should be possible for everybody to eventually attain the same (or at least very similar) default rating weight experienced players have. It will be scaled back by personal vote distribution anyway.
Warp wrote:
Edit: I think I understand now: It's my avatar, isn't it? It makes me look angry.
Offtopic but I didn't want to make another forum for this one question.
Is it possible for, at the very least, the author of a run to view the rating that a movie has before it hits the 3 ratings required? Currently 4 of my 8 movies have either 1 or 2 ratings and so I have no idea what each of them are rated. I know it doesn't matter in the long run but it is slightly annoying not knowing.
