1 2
Posted: 10/29/2012 5:56 PM
Post subject: Adjustments to the player point/forum rank system
Quote
Baxter
Skilled player (1402)
Joined: 5/31/2004
Posts: 1821
TAS of 2013 SNES TAS of 2013 Speedy TAS of 2007 Innovative TAS of 2007
Due to future site changes regarding the publishing of movies that previously would have been rejected due to being a "bad game choice", as discussed in this thread, it may be necessary to change the current formula that is used to determine the number of player points. If you are already familiar with this player point formula that I am talking about, please skip ahead to the part below the === line. Otherwise, keep on reading. You may have noticed the number in parenthesis right after certain people's forum rank. These are the number of "player points" this person has. If you have a published TAS at this site, the rating of this TAS determines the number of player points you get for this TAS. The total number of player points determines your forum rank. Details about the forum rank system can be found here (this page will be edited if the system changes) or [url= http://tasvideos.org/forum/viewtopic.php?p=179250#179250]here[/url] (this page won't change and describes the system used as of this post). Here is a short description of the outlined formula to determine player points, outlined on those pages: numberofplayerpoints = (movierating^2.5) * (numberofauthors^-0.5) This formula has two aspects: 1) The number 2.5 makes sure that for instance one movie that was rated an 8 provides more player points than 2 movies rated a 4. 2) The number -0.5 makes sure that you will get fewer player points if the TAS has more authors. For a two-player TAS, you get roughtly 0.71 times the points you would have gotten had it been a single author TAS. =================================================== There are two reasons (but feel free to add others) why we should consider adjusting the current formula: (1) The number of ratings is not a factor in the current formula. (A movie that has an average rating of a 9 with 30 ratings gives the same number of player points as a movie with an average rating of a 9 with only 2 ratings.) (2) The addition of published "vault" movies. (Should a vault movie and a "regular" movie provide the same number of player points if they have the same rating?) Note that (1) and (2) may be related. This will be discussed later when looking at a possible answer to (2), but lets first look at (1): How should the number of ratings affect the player points you get for that TAS? There are a few things that may sound reasonable: #1 Since an average rating of a 9 with 30 ratings should be more valuable than a 9 with 2 ratings, one would think that more ratings should mean a higher number of player points given. Especially runs with very few ratings should be penalized, as the rating may not be very representative. #2 If a TAS with a certain number of ratings gives a representative rating, it should get about the same number of player points as it gets now. #3 To avoid deflation of player points due to TASes with few ratings getting less player points, TASes with more than the representative number of ratings should get slightly more player points than currently. #1, #2 and #3 can be taken into account by changing this part of the current formula: (movierating)^2.5 to: (movierating)^(A- (B(A-2.5)/X) ) "X" is the number of ratings that the movie received "B" is the number or ratings that is deemed to provide a representative rating of the TAS. If X=B, the formula reduces to (movierating)^2.5 which means you will get the same score as before. If X is lower than B, you will get less player points than before for this movie. A good value of B may be between 10 and 15. "A" is a number that influences what fraction of the player points you get for X<B and how many player points you get when your TAS has a large number of ratings. If X is significantly higher than B, the formula reduces to (movierating)^A. "A" should be higher than 2.5, but probably not higher than 2.6. Now lets look at problem (2), the addition of vault movies. As pointed out in a post by goofydylan8, it may be easier to create a ton of vault movies and get more player points than spending your time trying to perfect a star movie. Effort and high rated movies should be awarded, which is why a single movie with an 8 rating provides more player points than two movies with a 4 rating, but should this scale be adjusted to account for vault movies? Should a different change be made to account for this? One option would be to just multiply the player points that a vault movie would give by a factor lower than 1. Nice and easy solution which prevents the potantial problem that goofydylan8 raised. On the other hand, if a solution to (1) is implemented, where TASes with more ratings get more points, does this not already solve the problem? Vault TASes are bound to get fewer ratings and lower ratings. Both have a significant effect on the player points... do we really need another factor lowering the points for vault movies? You can also wonder why a vault movie that gets the same number of ratings resulting in the same average rating as a non-vault movie deserves less player points. This discussion will hopefully generate some posts with opinions on: 1. When taking the number of ratings into account when calculating the player points, what features should such a formula have? Are the ones (#1, #2 and #3) listed in this post good? Any objections or additions? 2. What do you think about the concept alternate formula that I posted (in bold the bold font), which takes the number of ratings into account? Is is good, do you know some tweaks that could improve it, or do you have a completely different suggestion? 3. How should the player point formula deal with vault movies? Will taking number of ratings into account be enough, or are additional measures required to prevent the issue that goofydylan8 raised? Also, if you have some suggestions/ideas for the player points/forum ranks system in general, those are of course also welcome!
http://tasvideos.org/Baxter.html
Posted: 10/29/2012 6:54 PM
Quote
RachelB
She/Her
Player (127)
Joined: 12/3/2011
Posts: 1579
The number of ratings should definitely play a role, at least for movies with very few ratings. I don't think we need to scale it up for movies with huge numbers of ratings, but a couple of ratings is obviously not going to be representative of what most people might think of it. Movies with less than probably 10 votes or so should probably get a penalty. Above that though, i don't think anything needs to be done. A movie that attracts huge numbers of raters is probably already going to get a very high rating anyway, and thus gets more points on its own.
do we really need another factor lowering the points for vault movies? You can also wonder why a vault movie that gets the same number of ratings resulting in the same average rating as a non-vault movie deserves less player points.
As i mentioned in irc, i strongly disagree with this. Bad movies will already get less points, we don't need to penalize good speed runs that don't happen to get a star. And if bad movies are still getting too many points, then we should tweak the formula so movies with very low ratings get fewer points. I am fine with giving a bonus for stars, but moons and vault should be equal.
Posted: 10/29/2012 8:15 PM
Post subject: Keep it simple!
Quote
dwangoAC
Moderator, Senior Ambassador, Experienced player (898)
Joined: 9/14/2008
Posts: 1007
Exotic platforms TASer of 2015 Exotic platforms TASer of 2013
Baxter wrote:
On the other hand, if a solution to (1) is implemented, where TASes with more ratings get more points, does this not already solve the problem? Vault TASes are bound to get fewer ratings and lower ratings. Both have a significant effect on the player points... do we really need another factor lowering the points for vault movies? You can also wonder why a vault movie that gets the same number of ratings resulting in the same average rating as a non-vault movie deserves less player points.
This is definitely a case where I think it should be kept as simple as possible without causing problems. I personally feel that we probably don't need another factor lowering the points for vault movies, especially when a vault movie is boarderline for crossing into a higher tier. Either way, the solution should be as simple as possible in my opinion. Thanks for thinking through the options, Baxter! A.C. ******
I was laid off in May 2023 and could use support via Patreon or onetime donations as I work on TASBot Re: and TASBot HD. I'm dwangoAC, part of the senior staff of TASVideos as the Senior Ambassador and BDFL of the TASBot community; I post TAS content on YouTube.com/dwangoAC based on livestreams from Twitch.tv/dwangoAC.
Posted: 10/29/2012 8:39 PM
Quote
Derakon
Joined: 7/2/2007
Posts: 3960
If people like movies, why should we care what tier they're in? Good movies are good movies.
Pyrel - an open-source rewrite of the Angband roguelike game in Python.
Posted: 10/29/2012 9:14 PM
Quote
Baxter
Skilled player (1402)
Joined: 5/31/2004
Posts: 1821
TAS of 2013 SNES TAS of 2013 Speedy TAS of 2007 Innovative TAS of 2007
rog wrote:
Movies with less than probably 10 votes or so should probably get a penalty. Above that though, i don't think anything needs to be done.
Do you have some intuitions regarding this penalty? As in, what percentage of the normal points should a movie with only 1 rating get? And movies with 2 ratings? etc :)
http://tasvideos.org/Baxter.html
Posted: 10/30/2012 4:07 AM
Quote
feos
Site Admin, Skilled player (1235)
Joined: 4/17/2010
Posts: 11264
Location: RU
NES TAS of 2011
If a vault movie gets the same rating the non-vault one has, placing the former in vault must be revised.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Posted: 10/30/2012 4:41 AM
Quote
jlun2
Skilled player (1705)
Joined: 9/17/2009
Posts: 4952
Location: ̶C̶a̶n̶a̶d̶a̶ "Kanatah"
GBA TASer of 2010
feos wrote:
If a vault movie gets the same rating the non-vault one has, placing the former in vault must be revised.
You can't really trust ratings, since (some) people can simply put 10/10 on anything that interest them.
LLCoolDave wrote:
This seems to be competing for the "Worst looking TAS of 2011" Award.
Posted: 10/30/2012 9:28 AM
Quote
AnS
Emulator Coder, Experienced player (723)
Joined: 2/23/2006
Posts: 682
I agree with Derakon and feos, ratings should not depend on tier. If a (vault) movie suddenly gets many high votes then it simply means the judge made a mistake by not considering the movie entertaining. And I don't think the formula should minimize the audience impact.
rog wrote:
Movies with less than probably 10 votes or so should probably get a penalty. Above that though, i don't think anything needs to be done.
Defining limits by constant numbers is asking for trouble. The number should be calculated on the fly, to account for audience growing. Maybe it should be median of the number of votes for all published movies.
Posted: 10/30/2012 10:04 AM
Quote
GMan
Joined: 4/7/2008
Posts: 117
The current and suggested methods do seem a bit ad hoc, I'm interested in sampling some data to run some tests, is there any easily available? That is, is there a dump of data containing movie votes (note: not the final average score, but the votes themselves; I think using the average might not be the best strategy for this), movie author count, and resulting score?
Posted: 10/30/2012 10:25 AM
Quote
Baxter
Skilled player (1402)
Joined: 5/31/2004
Posts: 1821
TAS of 2013 SNES TAS of 2013 Speedy TAS of 2007 Innovative TAS of 2007
GMan wrote:
The current and suggested methods do seem a bit ad hoc, I'm interested in sampling some data to run some tests, is there any easily available?
At the moment, data like that is not easily available. Unless Ilari wips something up.
GMan wrote:
That is, is there a dump of data containing movie votes (note: not the final average score, but the votes themselves; I think using the average might not be the best strategy for this), movie author count, and resulting score?
You can see the individual ratings by clicking the number of votes. Some people have chosen not to show their votes publicly, so you won't be able to get the complete data you'd like probably.
http://tasvideos.org/Baxter.html
Posted: 10/30/2012 3:02 PM
Quote
feos
Site Admin, Skilled player (1235)
Joined: 4/17/2010
Posts: 11264
Location: RU
NES TAS of 2011
I think the tier (at least for vault) must be decided by a judge as he accepts the movie. And the future placing must depend on rating: whether it is a vault movie or a regular one (moons/stars are still assigned by trained monkeys starmen). So we seem to need a rating borderline to consider movies vault worthy (6.0?). Also, for each tier we may have a thread for complaints and suggestions, moving the run here or there if the audience disagrees with the current placing. That is: at the moment of publication, place the run to a certain tier, then allow it to navigate on its own, then if people ask, sticky it to some tier. And I have some suggestion on how to improve the validity of ratings, as in, true weighting. Entertainment: for anyone above lurker or newbie, must be waighted as 1.0. Anyone can have a good taste on entertainment, net depending on anything at all. We just need to make sure it's not a troll. But making 5 posts isn't enough to be considered non-abusive, so I suggest weight newbies' votes as 0.5 or so. Probably, 0.3 for lurkers, 0.6 for newbies and 1.0 for the rest. Technical: we need to be sure you understand the aspects of tasing to weight the ratings you give well. Let's take the average rating as a factor (both entertainment and technical rating the rater himself has) - but that's for tasers! And for non-tasers we can choose something different as a factor. I keep suggesting rating posts themselves (+/-). For forum members it might also be a rank factor. We shall invent something to keep the average post rating of a person within 0.0-1.0 to match the factor the tasers would have. For example, mklip's posts are so helpful his rating would be extremely solid. Actually, rating the posts would help us to make sure the person is sane. Probably, tasers may have 1.0 + average reting. I dunno. You say how you feel about improving the weighting system.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Posted: 10/30/2012 8:42 PM
Quote
Baxter
Skilled player (1402)
Joined: 5/31/2004
Posts: 1821
TAS of 2013 SNES TAS of 2013 Speedy TAS of 2007 Innovative TAS of 2007
feos wrote:
I think the tier (at least for vault) must be decided by a judge as he accepts the movie. And the future placing must depend on rating: whether it is a vault movie or a regular one (moons/stars are still assigned by trained monkeys starmen).
While I do think that it should be possible to switch a movie's tier in hindsight, I don't think this should (solely) depend on the ratings it receives. Whether or not the ratings of some users should weight heavier on the final rating than the ratings of other users is a different topic, that does not directly bear on the player point/rank system. (Unless some user's ratings would not count fully to the total number of ratings maybe...)
http://tasvideos.org/Baxter.html
Posted: 10/30/2012 10:25 PM
Quote
GMan
Joined: 4/7/2008
Posts: 117
Baxter wrote:
At the moment, data like that is not easily available. Unless Ilari wips something up. You can see the individual ratings by clicking the number of votes. Some people have chosen not to show their votes publicly, so you won't be able to get the complete data you'd like probably.
Yeah, I just ended up grabbing data manually (the public data anyway) from a select few games. To make my understanding clear, we have (in utmost generality): A set of movies with a set of votes in [0, 100] (divided by ten for display) for a set of different categories, with a set of authors? (In the current case, categories are "Entertainment" and "Tech Quality".) And there's potentially going to be a tier. We want to, for each author across all movies, generate some score based off all this information? I'll post something later tonight with what I have so far. It's more sweeping than just changing the score calculation, though, and would also effect the calculation of a movie's final score as well.
Posted: 11/1/2012 11:35 AM
Quote
GMan
Joined: 4/7/2008
Posts: 117
This was quite a bit of fun. I ended up writing a C++11 program for demonstration, and source for it can be found on my blog (off-site to save space on this post). I wouldn't read it until you read this post, though. But as such, I omitted the math formulas from the post and instead provided links to the wiki pages describing it, as my code covers it all anyway (obviously). Anyway, I'm mostly dealing with issue #1 in this post. I have ideas about #2 as well, but they are a bit dependent of the outcome of my other post from the tier proposal thread. So to avoid discussing things that may not turn out, I'll wait to post that stuff. Plus this is already quite lengthy. So the problem is that a movie's ranking is, ultimately, an average of all the votes. (I'm going to keep referring to "a movie's ranking", but we all know there's actually two ratings: the entertainment rating and the tech quality rating. For now, just assume one; we can figure out how to combine the two into one later.) As noted by Baxter, an average of 9 from a single vote is a lot less meaningful than an average of 9 from a million. Luckily this is a "solved" problem, and what we need are confidence intervals. What I'm going to describe is effectively an analog to the article "How Not To Sort By Average Rating", an idea made popular when Randall of xkcd pushed hard for it be to the sorting method used by comments on Reddit. We cannot use the algorithm directly because our votes are two scales from 0-100, rather than a single yes-or-no vote, but we can still use the confidence interval idea. So to start, let's describe our overall goal. What we're really trying to calculate is the 'true' mean (average). That is, if every single possible viewer voted, we could take the mean and with 100% certainty say "this is the mean", because there couldn't possibly be another vote to throw it off. The problem is that not every possible voter casts a vote, so we have some uncertainty. Warning: This post is about to get a bit mathy, skip to the text graphs to get a more intuitive idea if you don't care and just want pretty pictures. We're going to calculate the 95% confidence interval of the mean vote. Basically, a 95% confidence interval says this: "there's a 5% chance that the 'true' mean lies outside of this calculated interval, but otherwise we're 95% sure it'll end up in here." I chose 95% because it's exceedingly common, and for our sample size (often not too large), asking for 99% or higher just ends up including most of the possible voting range, making the interval rather uninformative. The important part is this: as more votes are cast, the size of the confidence interval becomes smaller, because the confidence in the sample data's accuracy is higher. We'll see how to turn this nice feature into a final score later. To calculate a confidence interval, we have to make an assumption about the distribution of votes. For those that don't know, a distribution specifies a probability to each possible outcome of a random experiment. Height is a very common example, and it can be modeled with a normal distribution. This means that there is an average height, where most people's height is, then it tailors off as you leave this average height (see picture on the normal distribution wiki page). The normal distribution is very common across many measurements, and it's no different for votes. We can see individual votes as "guesses" to the true mean: there's going to be a single concentration of votes around an average, with the frequency of more deviant votes lowering as they become more extreme. Luckily for us, calculating the confidence interval from a normal distribution is easy, as is calculating the parameters for a normal distribution from sample data (votes). Here's an example to make this concrete (all text graphs generated by the aforementioned C++11 program, with 1 million samples). These are the actual votes for Super Mario 64 (all votes are, for the duration of these examples, only from the publically-visible Entertainment column): 
SM64
|                                                                                          #          | 4
|                                                                                          #          | 3.9
|                                                                                          #          | 3.8
|                                                                                          #          | 3.7
|                                                                                          #          | 3.6
|                                                                                          #          | 3.5
|                                                                                          #          | 3.4
|                                                                                          #          | 3.3
|                                                                                          #          | 3.2
|                                                                                          #          | 3.1
|                                                                                          #          | 3
|                                                                                          #          | 2.9
|                                                                                          #          | 2.8
|                                                                                          #          | 2.7
|                                                                                          #          | 2.6
|                                                                                          #          | 2.5
|                                                                                          #          | 2.4
|                                                                                          #          | 2.3
|                                                                                          #          | 2.2
|                                                                                          #          | 2.1
|                                                                                        # #  # #   ##| 2
|                                                                                        # #  # #   ##| 1.9
|                                                                                        # #  # #   ##| 1.8
|                                                                                        # #  # #   ##| 1.7
|                                                                                        # #  # #   ##| 1.6
|                                                                                        # #  # #   ##| 1.5
|                                                                                        # #  # #   ##| 1.4
|                                                                                        # #  # #   ##| 1.3
|                                                                                        # #  # #   ##| 1.2
|                                                                                        # #  # #   ##| 1.1
|                                                                 #    #            # #  # ## # #   ##| 1
|                                                                 #    #            # #  # ## # #   ##| 0.9
|                                                                 #    #            # #  # ## # #   ##| 0.8
|                                                                 #    #            # #  # ## # #   ##| 0.7
|                                                                 #    #            # #  # ## # #   ##| 0.6
|                                                                 #    #            # #  # ## # #   ##| 0.5
|                                                                 #    #            # #  # ## # #   ##| 0.4
|                                                                 #    #            # #  # ## # #   ##| 0.3
|                                                                 #    #            # #  # ## # #   ##| 0.2
|                                                                 #    #            # #  # ## # #   ##| 0.1
|#####################################################################################################| 0
+-----------------------------------------------------------------------------------------------------+------
 |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |
 0.0  0.5  1.0  1.5  2.0  2.5  3.0  3.5  4.0  4.5  5.0  5.5  6.0  6.5  7.0  7.5  8.0  8.5  9.0  9.5  10.0
We can calculate the parameters for a normal distribution of this with just a few values. First, we need the mean. This is easy and already done on the site. Second, we need the variance and deviation (the deviation is just the square root of variance). The variance has more than one way of calculation, but I chose a fairly simple bias-corrected method. Next comes the critical bit. We calculate the standard error. This is the deviation divided by the square root of the sample count (note: this is where sample count starts to come into play!). Luckily for us, because our voting population is fairly small, sometimes we get a fairly significant (>5%) sample size! Assuming a voting population of 100†, it only takes 5 votes to meet this criteria. This means it's worthwhile to factor in finite population correction (FPC). What this does is account for the fact that as the sample count (vote count) nears the total population size (number of voters), our confidence increases towards 100%. Once it reaches 100%, we no longer have a sample but a census, and the sample mean is the true mean. We just multiply our initial standard error by FPC to get the true error. (This is something we're lucky to be able to take advantage of. Consider a site like Reddit where only a tiny fraction of the entire user base will vote on a comment.) Last, we need the z-score for the 97.5 percentile point of a normal distribution. (This is the number of standard deviations away from the mean that 95% of the values lie). It's not trivial to calculate, but it's constant and the value is approximately 1.959963984540. So now we can calculate our 95% confidence interval. Take the mean vote score and subtract from it the error multiplied by the quantile to get the lower bound, and instead add this product to the mean to get the upper bound. Now since we know the interval can never go below 0 or above 10, clamp it if necessary. Ta-da! We have our interval. This interval has a 95% chance of containing the true value (though there is a 5% chance our sample mislead us!). Note this has the desired property of being dependent on the number of votes cast. From the SM64 votes above, the resulting normal distribution is thus, where #'s indicate votes within the confidence interval and *'s indicate votes outside the interval: 
SM64 Normal Distribution
|                                                                                          #          | 45179
|                                                                                       #####         | 44049.5
|                                                                                       #####         | 42920
|                                                                                      #######        | 41790.6
|                                                                                      ########       | 40661.1
|                                                                                     #########       | 39531.6
|                                                                                     #########*      | 38402.2
|                                                                                    *#########*      | 37272.7
|                                                                                    *#########**     | 36143.2
|                                                                                   **#########**     | 35013.7
|                                                                                   **#########**     | 33884.2
|                                                                                   **#########***    | 32754.8
|                                                                                  ***#########***    | 31625.3
|                                                                                  ***#########****   | 30495.8
|                                                                                  ***#########****   | 29366.4
|                                                                                 ****#########****   | 28236.9
|                                                                                 ****#########*****  | 27107.4
|                                                                                *****#########*****  | 25977.9
|                                                                                *****#########*****  | 24848.5
|                                                                                *****#########****** | 23719
|                                                                               ******#########****** | 22589.5
|                                                                               ******#########****** | 21460
|                                                                               ******#########*******| 20330.5
|                                                                              *******#########*******| 19201.1
|                                                                              *******#########*******| 18071.6
|                                                                             ********#########*******| 16942.1
|                                                                             ********#########*******| 15812.6
|                                                                            *********#########*******| 14683.2
|                                                                            *********#########*******| 13553.7
|                                                                            *********#########*******| 12424.2
|                                                                           **********#########*******| 11294.8
|                                                                          ***********#########*******| 10165.3
|                                                                          ***********#########*******| 9035.8
|                                                                         ************#########*******| 7906.33
|                                                                        *************#########*******| 6776.85
|                                                                        *************#########*******| 5647.38
|                                                                       **************#########*******| 4517.9
|                                                                      ***************#########*******| 3388.42
|                                                                    *****************#########*******| 2258.95
|                                                                  *******************#########*******| 1129.48
|*************************************************************************************#########*******| 0
+-----------------------------------------------------------------------------------------------------+------
 |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |
 0.0  0.5  1.0  1.5  2.0  2.5  3.0  3.5  4.0  4.5  5.0  5.5  6.0  6.5  7.0  7.5  8.0  8.5  9.0  9.5  10.0

Count: 19 Sum: 170.4 Mean: 8.96842 Variance: 0.877774
Confidence (95%): [8.58737, 9.34948] (Range: 0.76211)
Note the calculated numbers displayed under the graph. To demonstrate how the interval size changes depending on the sample size, here's the exact same calculation except only half of the SM64 votes are used: 
SM64 (half sample) Normal Distribution
|                                                                                          #          | 45084
|                                                                                        ###          | 43956.9
|                                                                                       ######        | 42829.8
|                                                                                       ######        | 41702.7
|                                                                                      ########       | 40575.6
|                                                                                     #########       | 39448.5
|                                                                                     ##########      | 38321.4
|                                                                                    ###########      | 37194.3
|                                                                                    ############     | 36067.2
|                                                                                    ############     | 34940.1
|                                                                                   #############*    | 33813
|                                                                                   #############*    | 32685.9
|                                                                                   #############*    | 31558.8
|                                                                                  *#############**   | 30431.7
|                                                                                  *#############**   | 29304.6
|                                                                                 **#############**   | 28177.5
|                                                                                 **#############***  | 27050.4
|                                                                                 **#############***  | 25923.3
|                                                                                ***#############**** | 24796.2
|                                                                                ***#############**** | 23669.1
|                                                                               ****#############**** | 22542
|                                                                               ****#############*****| 21414.9
|                                                                               ****#############*****| 20287.8
|                                                                              *****#############*****| 19160.7
|                                                                              *****#############*****| 18033.6
|                                                                             ******#############*****| 16906.5
|                                                                             ******#############*****| 15779.4
|                                                                             ******#############*****| 14652.3
|                                                                            *******#############*****| 13525.2
|                                                                            *******#############*****| 12398.1
|                                                                           ********#############*****| 11271
|                                                                           ********#############*****| 10143.9
|                                                                          *********#############*****| 9016.8
|                                                                         **********#############*****| 7889.7
|                                                                         **********#############*****| 6762.6
|                                                                        ***********#############*****| 5635.5
|                                                                       ************#############*****| 4508.4
|                                                                      *************#############*****| 3381.3
|                                                                    ***************#############*****| 2254.2
|                                                                  *****************#############*****| 1127.1
|***********************************************************************************#############*****| 0
+-----------------------------------------------------------------------------------------------------+------
 |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |
 0.0  0.5  1.0  1.5  2.0  2.5  3.0  3.5  4.0  4.5  5.0  5.5  6.0  6.5  7.0  7.5  8.0  8.5  9.0  9.5  10.0

Count: 9 Sum: 80.9 Mean: 8.98889 Variance: 0.891852
Confidence (95%): [8.39736, 9.58042] (Range: 1.18306)
As you can see, lower samples imply a larger range. The final step is to turn this interval into a single value. The linked articles on Reddit's comment ranking simply take the lower bound of the confidence interval, and this works extremely well. The reason is that fewer votes will bias the lower bound to a lower score, which solves our original problem: a few votes end up contributing less than many votes, even when the average is the same, because the confidence interval will be wider. The justification for this is approach simple as well. The lower bound says: "I'm 95% certain your true mean won't be lower than this, but get more votes and I'll let you know! But until then, this is quite fair of a rank to get because I'm 5% sure I'm not overinflating your true mean; in the rare case it's wrong, you're welcome for the bonus." In practice low-voted movies won't get punished that much, but enough to knock near or equal means away from each other. (At most a single point: 7->6, for example.) Also, remember that because our population is finite, as the number of votes reaches the total number of voters, the error value tends towards zero. At some point, with every vote accounted for, the error is zero and the lower bound and the mean coincide, giving the true mean. So for SM64, the final score would be (roughly, as I don't know the private votes): 8.58737, which is a difference of -0.381055 from the simple mean. Note that every score on the site will go down slightly as the number of votes gets taken into account. This is okay: we only care about the relative ordering, and the number won't vary in practice that much at all. (Keep in mind that for ranking and calculations for problem #2 these need to store all the decimal places; truncated/rounding to one decimal for display is sensible.) Usage-wise, just note that a single vote is not enough to calculate variance, and two votes can give a very meaningless answer unless the two votes happen to be close to each other. The site already requires three votes before it calculates a rating though, so that's good. Turning two scores into one will need to be discussed after the tiers thing settles down. I think a movie in an "entertainment-based" category should get most of its score from the Entertainment rating, while a movie in a "technical-based" category should get most of its score from the Tech Quality rating. It might be worthwhile to investigate interval arithmetic (something I'm not as familiar with) for this task. And that's it. What essentially comes down to a few multiplications and a couple square roots gives us a very meaningful and theoretically justified score (not just ad hoc tweaking). Here are some more example plots and data: 
SMB 3
|                                                                                          #          | 7
|                                                                                          #          | 6.825
|                                                                                          #          | 6.65
|                                                                                          #          | 6.475
|                                                                                          #          | 6.3
|                                                                                          #          | 6.125
|                                                                                          #          | 5.95
|                                                                                          #          | 5.775
|                                                                                          #          | 5.6
|                                                                                          #          | 5.425
|                                                                                          #          | 5.25
|                                                                                          #          | 5.075
|                                                                                          #          | 4.9
|                                                                                          #          | 4.725
|                                                                                          #          | 4.55
|                                                                                          #          | 4.375
|                                                                                          #          | 4.2
|                                                                                          #          | 4.025
|                                                                                #         #    #     | 3.85
|                                                                                #         #    #     | 3.675
|                                                                                #         #    #     | 3.5
|                                                                                #         #    #     | 3.325
|                                                                                #         #    #     | 3.15
|                                                                                #         #    #    #| 2.975
|                                                                                #         #    #    #| 2.8
|                                                                                #         #    #    #| 2.625
|                                                                                #         #    #    #| 2.45
|                                                                                #         #    #    #| 2.275
|                                                                                #         #    #    #| 2.1
|                                                                                #    ##   # # ##    #| 1.925
|                                                                                #    ##   # # ##    #| 1.75
|                                                                                #    ##   # # ##    #| 1.575
|                                                                                #    ##   # # ##    #| 1.4
|                                                                                #    ##   # # ##    #| 1.225
|                                                                                #    ##   # # ##    #| 1.05
|                                                       #              # #       #    ## ### # ## ## #| 0.875
|                                                       #              # #       #    ## ### # ## ## #| 0.7
|                                                       #              # #       #    ## ### # ## ## #| 0.525
|                                                       #              # #       #    ## ### # ## ## #| 0.35
|                                                       #              # #       #    ## ### # ## ## #| 0.175
|#####################################################################################################| 0
+-----------------------------------------------------------------------------------------------------+------
 |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |
 0.0  0.5  1.0  1.5  2.0  2.5  3.0  3.5  4.0  4.5  5.0  5.5  6.0  6.5  7.0  7.5  8.0  8.5  9.0  9.5  10.0
SMB 3 Normal Distribution
|                                                                                        #            | 43343
|                                                                                      ####           | 42259.4
|                                                                                     ######          | 41175.8
|                                                                                    *######*         | 40092.3
|                                                                                    *######*         | 39008.7
|                                                                                   **######**        | 37925.1
|                                                                                   **######***       | 36841.5
|                                                                                  ***######***       | 35758
|                                                                                  ***######***       | 34674.4
|                                                                                  ***######****      | 33590.8
|                                                                                 ****######****      | 32507.2
|                                                                                 ****######*****     | 31423.7
|                                                                                *****######*****     | 30340.1
|                                                                                *****######*****     | 29256.5
|                                                                                *****######******    | 28173
|                                                                               ******######******    | 27089.4
|                                                                               ******######*******   | 26005.8
|                                                                               ******######*******   | 24922.2
|                                                                              *******######*******   | 23838.7
|                                                                              *******######********  | 22755.1
|                                                                             ********######********  | 21671.5
|                                                                             ********######********* | 20587.9
|                                                                             ********######********* | 19504.3
|                                                                            *********######********* | 18420.8
|                                                                            *********######**********| 17337.2
|                                                                           **********######**********| 16253.6
|                                                                           **********######**********| 15170
|                                                                           **********######**********| 14086.5
|                                                                          ***********######**********| 13002.9
|                                                                          ***********######**********| 11919.3
|                                                                         ************######**********| 10835.8
|                                                                         ************######**********| 9752.17
|                                                                        *************######**********| 8668.6
|                                                                       **************######**********| 7585.03
|                                                                      ***************######**********| 6501.45
|                                                                      ***************######**********| 5417.88
|                                                                     ****************######**********| 4334.3
|                                                                   ******************######**********| 3250.72
|                                                                  *******************######**********| 2167.15
|                                                               **********************######**********| 1083.58
|*************************************************************************************######**********| 0
+-----------------------------------------------------------------------------------------------------+------
 |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |
 0.0  0.5  1.0  1.5  2.0  2.5  3.0  3.5  4.0  4.5  5.0  5.5  6.0  6.5  7.0  7.5  8.0  8.5  9.0  9.5  10.0

Count: 33 Sum: 291.3 Mean: 8.82727 Variance: 0.917633
Confidence (95%): [8.5584, 9.09614] (Range: 0.537744)
Final score as lower bound of confidence range: 8.5584
Current score on site: 8.82727
Difference: -0.268872
SMW
|                                                                                          #          | 21
|                                                                                          #          | 20.475
|                                                                                          #          | 19.95
|                                                                                          #          | 19.425
|                                                                                          #          | 18.9
|                                                                                          #          | 18.375
|                                                                                          #         #| 17.85
|                                                                                          #         #| 17.325
|                                                                                          #         #| 16.8
|                                                                                          #         #| 16.275
|                                                                                          #         #| 15.75
|                                                                                          #         #| 15.225
|                                                                                          #         #| 14.7
|                                                                                          #         #| 14.175
|                                                                                          #         #| 13.65
|                                                                                          #         #| 13.125
|                                                                                          #         #| 12.6
|                                                                                          #         #| 12.075
|                                                                                          #         #| 11.55
|                                                                                          #         #| 11.025
|                                                                                          #         #| 10.5
|                                                                                          #         #| 9.975
|                                                                                          #         #| 9.45
|                                                                                          #         #| 8.925
|                                                                                          #         #| 8.4
|                                                                                          #         #| 7.875
|                                                                                          #         #| 7.35
|                                                                                          #         #| 6.825
|                                                                                          #         #| 6.3
|                                                                                          #         #| 5.775
|                                                                                          #         #| 5.25
|                                                                                #         #         #| 4.725
|                                                                                #         #         #| 4.2
|                                                                                #         #         #| 3.675
|                                                                                #         #         #| 3.15
|                                                                                #         #         #| 2.625
|                                                                                #         #         #| 2.1
|                                                                                #    #    #    #    #| 1.575
|                                                                                #    #    #    #    #| 1.05
|                                        #         #    #          #   #         #  # #    ###  # #  #| 0.525
|#####################################################################################################| 0
+-----------------------------------------------------------------------------------------------------+------
 |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |
 0.0  0.5  1.0  1.5  2.0  2.5  3.0  3.5  4.0  4.5  5.0  5.5  6.0  6.5  7.0  7.5  8.0  8.5  9.0  9.5  10.0
SMW Normal Distribution
|                                                                                       #             | 25105
|                                                                                     **#####*        | 24477.4
|                                                                                    ***#####***      | 23849.8
|                                                                                   ****#####****     | 23222.1
|                                                                                  *****#####*****    | 22594.5
|                                                                                 ******#####******   | 21966.9
|                                                                                *******#####******   | 21339.2
|                                                                                *******#####*******  | 20711.6
|                                                                               ********#####******** | 20084
|                                                                              *********#####*********| 19456.4
|                                                                             **********#####*********| 18828.8
|                                                                            ***********#####*********| 18201.1
|                                                                            ***********#####*********| 17573.5
|                                                                           ************#####*********| 16945.9
|                                                                           ************#####*********| 16318.2
|                                                                          *************#####*********| 15690.6
|                                                                         **************#####*********| 15063
|                                                                         **************#####*********| 14435.4
|                                                                        ***************#####*********| 13807.8
|                                                                       ****************#####*********| 13180.1
|                                                                       ****************#####*********| 12552.5
|                                                                      *****************#####*********| 11924.9
|                                                                     ******************#####*********| 11297.2
|                                                                     ******************#####*********| 10669.6
|                                                                    *******************#####*********| 10042
|                                                                   ********************#####*********| 9414.38
|                                                                  *********************#####*********| 8786.75
|                                                                  *********************#####*********| 8159.12
|                                                                 **********************#####*********| 7531.5
|                                                                ***********************#####*********| 6903.88
|                                                               ************************#####*********| 6276.25
|                                                              *************************#####*********| 5648.62
|                                                             **************************#####*********| 5021
|                                                            ***************************#####*********| 4393.38
|                                                           ****************************#####*********| 3765.75
|                                                         ******************************#####*********| 3138.12
|                                                       ********************************#####*********| 2510.5
|                                                      *********************************#####*********| 1882.87
|                                                  *************************************#####*********| 1255.25
|                                              *****************************************#####*********| 627.625
|***************************************************************************************#####*********| 0
+-----------------------------------------------------------------------------------------------------+------
 |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |
 0.0  0.5  1.0  1.5  2.0  2.5  3.0  3.5  4.0  4.5  5.0  5.5  6.0  6.5  7.0  7.5  8.0  8.5  9.0  9.5  10.0

Count: 57 Sum: 509.4 Mean: 8.93684 Variance: 1.58221
Confidence (95%): [8.72163, 9.15205] (Range: 0.430417)
Final score as lower bound of confidence range: 8.72163
Current score on site: 8.93684
Difference: -0.215208
Fortified Zone
|                                                                   #       # #                       | 1
|                                                                   #       # #                       | 0.975
|                                                                   #       # #                       | 0.95
|                                                                   #       # #                       | 0.925
|                                                                   #       # #                       | 0.9
|                                                                   #       # #                       | 0.875
|                                                                   #       # #                       | 0.85
|                                                                   #       # #                       | 0.825
|                                                                   #       # #                       | 0.8
|                                                                   #       # #                       | 0.775
|                                                                   #       # #                       | 0.75
|                                                                   #       # #                       | 0.725
|                                                                   #       # #                       | 0.7
|                                                                   #       # #                       | 0.675
|                                                                   #       # #                       | 0.65
|                                                                   #       # #                       | 0.625
|                                                                   #       # #                       | 0.6
|                                                                   #       # #                       | 0.575
|                                                                   #       # #                       | 0.55
|                                                                   #       # #                       | 0.525
|                                                                   #       # #                       | 0.5
|                                                                   #       # #                       | 0.475
|                                                                   #       # #                       | 0.45
|                                                                   #       # #                       | 0.425
|                                                                   #       # #                       | 0.4
|                                                                   #       # #                       | 0.375
|                                                                   #       # #                       | 0.35
|                                                                   #       # #                       | 0.325
|                                                                   #       # #                       | 0.3
|                                                                   #       # #                       | 0.275
|                                                                   #       # #                       | 0.25
|                                                                   #       # #                       | 0.225
|                                                                   #       # #                       | 0.2
|                                                                   #       # #                       | 0.175
|                                                                   #       # #                       | 0.15
|                                                                   #       # #                       | 0.125
|                                                                   #       # #                       | 0.1
|                                                                   #       # #                       | 0.075
|                                                                   #       # #                       | 0.05
|                                                                   #       # #                       | 0.025
|#####################################################################################################| 0
+-----------------------------------------------------------------------------------------------------+------
 |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |
 0.0  0.5  1.0  1.5  2.0  2.5  3.0  3.5  4.0  4.5  5.0  5.5  6.0  6.5  7.0  7.5  8.0  8.5  9.0  9.5  10.0
Fortified Zone Normal Distribution
|                                                                         #                           | 105660
|                                                                        ##                           | 103018
|                                                                        ##                           | 100377
|                                                                       ####                          | 97735.5
|                                                                       ####                          | 95094
|                                                                       ####                          | 92452.5
|                                                                       ####                          | 89811
|                                                                       ####                          | 87169.5
|                                                                      ######                         | 84528
|                                                                      ######                         | 81886.5
|                                                                      ######                         | 79245
|                                                                      ######                         | 76603.5
|                                                                      ######                         | 73962
|                                                                      ######                         | 71320.5
|                                                                     ########                        | 68679
|                                                                     ########                        | 66037.5
|                                                                     ########                        | 63396
|                                                                     ########                        | 60754.5
|                                                                     ########                        | 58113
|                                                                     ########                        | 55471.5
|                                                                     ########                        | 52830
|                                                                    ##########                       | 50188.5
|                                                                    ##########                       | 47547
|                                                                    ##########                       | 44905.5
|                                                                    ##########                       | 42264
|                                                                    ##########                       | 39622.5
|                                                                    ##########                       | 36981
|                                                                   ############                      | 34339.5
|                                                                   ############                      | 31698
|                                                                   ############                      | 29056.5
|                                                                   ############                      | 26415
|                                                                   ############                      | 23773.5
|                                                                  ##############                     | 21132
|                                                                  ##############                     | 18490.5
|                                                                  ##############                     | 15849
|                                                                 *##############*                    | 13207.5
|                                                                 *##############*                    | 10566
|                                                                **##############**                   | 7924.5
|                                                                **##############**                   | 5283
|                                                               ***##############***                  | 2641.5
|******************************************************************##############*********************| 0
+-----------------------------------------------------------------------------------------------------+------
 |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |
 0.0  0.5  1.0  1.5  2.0  2.5  3.0  3.5  4.0  4.5  5.0  5.5  6.0  6.5  7.0  7.5  8.0  8.5  9.0  9.5  10.0

Count: 3 Sum: 21.9 Mean: 7.3 Variance: 0.373333
Confidence (95%): [6.61561, 7.98439] (Range: 1.36878)
Final score as lower bound of confidence range: 6.61561
Current score on site: 7.3
Difference: -0.684391
Addams Family
|                         #              #    #         #      #                                      | 1
|                         #              #    #         #      #                                      | 0.975
|                         #              #    #         #      #                                      | 0.95
|                         #              #    #         #      #                                      | 0.925
|                         #              #    #         #      #                                      | 0.9
|                         #              #    #         #      #                                      | 0.875
|                         #              #    #         #      #                                      | 0.85
|                         #              #    #         #      #                                      | 0.825
|                         #              #    #         #      #                                      | 0.8
|                         #              #    #         #      #                                      | 0.775
|                         #              #    #         #      #                                      | 0.75
|                         #              #    #         #      #                                      | 0.725
|                         #              #    #         #      #                                      | 0.7
|                         #              #    #         #      #                                      | 0.675
|                         #              #    #         #      #                                      | 0.65
|                         #              #    #         #      #                                      | 0.625
|                         #              #    #         #      #                                      | 0.6
|                         #              #    #         #      #                                      | 0.575
|                         #              #    #         #      #                                      | 0.55
|                         #              #    #         #      #                                      | 0.525
|                         #              #    #         #      #                                      | 0.5
|                         #              #    #         #      #                                      | 0.475
|                         #              #    #         #      #                                      | 0.45
|                         #              #    #         #      #                                      | 0.425
|                         #              #    #         #      #                                      | 0.4
|                         #              #    #         #      #                                      | 0.375
|                         #              #    #         #      #                                      | 0.35
|                         #              #    #         #      #                                      | 0.325
|                         #              #    #         #      #                                      | 0.3
|                         #              #    #         #      #                                      | 0.275
|                         #              #    #         #      #                                      | 0.25
|                         #              #    #         #      #                                      | 0.225
|                         #              #    #         #      #                                      | 0.2
|                         #              #    #         #      #                                      | 0.175
|                         #              #    #         #      #                                      | 0.15
|                         #              #    #         #      #                                      | 0.125
|                         #              #    #         #      #                                      | 0.1
|                         #              #    #         #      #                                      | 0.075
|                         #              #    #         #      #                                      | 0.05
|                         #              #    #         #      #                                      | 0.025
|#####################################################################################################| 0
+-----------------------------------------------------------------------------------------------------+------
 |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |
 0.0  0.5  1.0  1.5  2.0  2.5  3.0  3.5  4.0  4.5  5.0  5.5  6.0  6.5  7.0  7.5  8.0  8.5  9.0  9.5  10.0
Addams Family Normal Distribution
|                                                #                                                    | 17243
|                                       # #########                                                   | 16837.9
|                                       #############                                                 | 16432.8
|                                    #################                                                | 16027.6
|                                   ####################                                              | 15622.5
|                                  #######################                                            | 15217.4
|                                 #########################                                           | 14812.2
|                               *###########################                                          | 14407.1
|                               *###########################*                                         | 14002
|                              **###########################***                                       | 13596.9
|                             ***###########################****                                      | 13191.8
|                            ****###########################****                                      | 12786.6
|                           *****###########################******                                    | 12381.5
|                          ******###########################******                                    | 11976.4
|                         *******###########################*******                                   | 11571.2
|                        ********###########################********                                  | 11166.1
|                       *********###########################*********                                 | 10761
|                      **********###########################*********                                 | 10355.9
|                     ***********###########################***********                               | 9950.75
|                    ************###########################************                              | 9545.62
|                   *************###########################************                              | 9140.5
|                  **************###########################**************                            | 8735.38
|                 ***************###########################**************                            | 8330.25
|                ****************###########################***************                           | 7925.13
|               *****************###########################****************                          | 7520
|               *****************###########################*****************                         | 7114.88
|              ******************###########################******************                        | 6709.75
|            ********************###########################********************                      | 6304.62
|            ********************###########################********************                      | 5899.5
|          **********************###########################*********************                     | 5494.38
|         ***********************###########################**********************                    | 5089.25
|        ************************###########################************************                  | 4684.12
|      **************************###########################*************************                 | 4279
|     ***************************###########################**************************                | 3873.88
|    ****************************###########################****************************              | 3468.75
|  ******************************###########################*****************************             | 3063.62
|********************************###########################*******************************           | 2658.5
|********************************###########################*********************************         | 2253.37
|********************************###########################***********************************       | 1848.25
|********************************###########################**************************************    | 1443.13
|********************************###########################******************************************| 1038
+-----------------------------------------------------------------------------------------------------+------
 |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |
 0.0  0.5  1.0  1.5  2.0  2.5  3.0  3.5  4.0  4.5  5.0  5.5  6.0  6.5  7.0  7.5  8.0  8.5  9.0  9.5  10.0

Count: 5 Sum: 22.7 Mean: 4.54 Variance: 2.32343
Confidence (95%): [3.2312, 5.8488] (Range: 2.61759)
Final score as lower bound of confidence range: 3.2312
Current score on site: 4.54
Difference: -1.3088
The program contains additional tests with manually constructed data. Thanks for reading. ----- †There could be some argument about what this finite voter population should be. It can either be the number of people who have cast at least one vote (are voters), or the number of people that could cast a vote (could be voters). Right now the most rated movie has 89 votes, so for my tests I assumed (using the former metric) that the total voting population is 100. If we go with the former, to my knowledge that means the number of registered forum members is the count, nearer to 5030. With this metric, it takes 230 votes to reach 5%, and considering the number of lurkers there are this seems useless to me. Either way though, a new calculation must be done for each movie every time this population count increases, and I assume the former increases less often (thinking about server load now). Another implementation strategy is to have "gates". These are just multiples of 50 (for example), and each time a gate is reached the population count increases by 50 and the gate is increased by 50. This allows the FPC to near 1 on highly-rated movies, but avoids constant server load whenever the population increases due to happenstance. (Huge final note: there are other and potentially better ways to do everything I described: maybe a credible interval would work better, or another distribution better fits the data. I think, though, that the chances that a normal distribution being insufficient are extremely small and not worth the computational effort required to move to a hypothetically better model. What I've presented is fairly easy to implement.)
Posted: 11/1/2012 4:41 PM
Quote
feos
Site Admin, Skilled player (1235)
Joined: 4/17/2010
Posts: 11264
Location: RU
NES TAS of 2011
Starring superb runs can inspire TASers if a star would add score for a run (double it?)
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Posted: 11/1/2012 8:27 PM
Quote
CoolKirby
Editor, Experienced player (608)
Joined: 11/8/2010
Posts: 4012
Exotic platforms TASer of 2014 NES TAS of 2013
That looks great, GMan! I read through the whole post and understood a good part of the math. Using confidence intervals to calculate ratings, taking number of ratings into account, it all sounds like a solid new rating system. If I could Yes vote it, I would.
Posted: 11/2/2012 3:59 AM
Quote
feos
Site Admin, Skilled player (1235)
Joined: 4/17/2010
Posts: 11264
Location: RU
NES TAS of 2011
I thought about what would be the score factor for belonging to tiers. I came up with this: Make Stars double the score from the movie for all authors. For Moons make it 1.5. For Demo and Regular movies - the same, and for Vault - 0.5 (regardless of being rather low already). This way the Vault movies would just fill the content, and showcase movies really fill the score. Do more star/moon movies, get more score.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Posted: 11/2/2012 4:12 AM
Quote
GMan
Joined: 4/7/2008
Posts: 117
If I understand what would get Star versus what would get Vault, that would mean, for example, the fastest 0-star SM64 run could be worth four times less than a slightly slower version that was more entertaining. I understand the importance of entertainment, but that seems a bit wrong. I think to do this at all we need to get some examples and see how the rankings change, and decide if that change is better or worse. For example, I agree with you conceptually, but maybe the number groupings should be smaller, like 1.2 for Star, 1.1 for Moon, 1.0 for Regular, and 0.9 for Vault. (But we should probably wait until the tier's dust settles before decided how to do things. There are only three tiers now, for example, not four.)
Posted: 11/2/2012 1:40 PM
Quote
marzojr
He/Him
Experienced player (748)
Joined: 9/29/2008
Posts: 964
Location: 🇫🇷 France
Sega TASer of 2014 Sega TAS of 2014 Sega TASer of 2010 Sega TAS of 2010
Hm. How about this: split weights for entertainment and technical ratings depending on tier. Regular tier has weight 1 for both; star has higher weights for both; moon has higher weight for entertainment and lower weight for technical; vault has higher weight for technical and lower weight for entertainment. This would mean that each tier values its primary trait(s) more.
Marzo Junior
Posted: 11/2/2012 2:21 PM
(Edited: 11/2/2012 3:08 PM)
Quote
feos
Site Admin, Skilled player (1235)
Joined: 4/17/2010
Posts: 11264
Location: RU
NES TAS of 2011
GMan: I've made 2 star-worthy runs, and 2 regular runs. Don't ask the difference of effort and time spent on the first two comparing to other two. Making the run really fantastic contributes MORE to the art. Read here, why we must encourage TASers to put MORE effort and time (also teaching HOW to do it). marzojr: We decided to raise another thread to discuss weight if I got it right. EDIT: after AnS' explanation on semi-autoimatic tiers we really can choose the splitted weighting trick!
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Posted: 11/2/2012 2:28 PM
Quote
Baxter
Skilled player (1402)
Joined: 5/31/2004
Posts: 1821
TAS of 2013 SNES TAS of 2013 Speedy TAS of 2007 Innovative TAS of 2007
Gman wrote:
Anyway, I'm mostly dealing with issue #1 in this post.
You failed to explicitly mention what exactly you are trying to do. Issue (1) in my post refers to taking the number of ratings into account when converting the ratings to player points. Are you trying to change the way the average is calculated which incorporates the number of votes in such a way that the old player point formula can be used? Would the old player point formula still need adjustments? Also, the average is currently calculated I think by giving the entertainment rating a weight of 2/3 and the technical rating a weight of 1/3. Did you account for this? These ratings can be very different for certain movies; everyone agreeing that the movie is quite a technical achievement, but completely disagreeing on the entertainment value. This also makes you wonder how good the assumption of a normal distribution is.
GMan wrote:
So to start, let's describe our overall goal. What we're really trying to calculate is the 'true' mean (average). That is, if every single possible viewer voted, we could take the mean and with 100% certainty say "this is the mean", because there couldn't possibly be another vote to throw it off. The problem is that not every possible voter casts a vote, so we have some uncertainty.
I also don't know if this is a good assumption. I for instance would never watch a 4 hour rpg TAS of a game I haven't played, so I would never rate such a movie. If I had to, I would probably rate it extremely low. People watch and rate the movies they are in some way interested in, so the ratings that you do get are in no way a good representation of what the distribution of ratings if everyone were to vote, and finding out/approximating what everyone would vote (including people who have never player the game/have no interest in it) does not seem like a good goal then. Note that, as feos pointed out in the post above, discussing weight of entertainment/technical ratings or the way to calculate the final rating in general given the individual ratings is slightly off-topic.
http://tasvideos.org/Baxter.html
Posted: 11/2/2012 4:06 PM
Quote
Baxter
Skilled player (1402)
Joined: 5/31/2004
Posts: 1821
TAS of 2013 SNES TAS of 2013 Speedy TAS of 2007 Innovative TAS of 2007
These last few posts were about point (2) in the first post, fixing it by multiplying the player points the different tiers general by different numbers. The first few comments were against this, arguing that unentertaining movies will get little player points anyway, and that it could be unfair to movies near the line vault/moon. Those who argue for such a factor I ask again, is a system that favors high ratings with lots of votes not enough to distribute player points fairly? Why not? Sure, people must be encouraged to make high quality TASes with great entertainment value, but is the audience's response, the placement in a certain tear etc not enough as a motivation? =============================== On the other hand, I would like to know the following: A average rating of a 6.3 currently gives about 100 player points. Just going by intuitions, how many player points would this 6.3 actually be worth if you know that it consisted of: 1 rating 2 ratings 5 ratings 10 ratings 20 ratings 50 ratings 100 ratings Do you think this should be linear, like 1 rating, 10 points, 2 ratings, 20 points... 10 ratings 100 points, and cut it off there. Or do you think it should start low, but more up faster as in: 1 rating 8 points, 2 ratings 30 points, 5 ratings 70 points, 10 ratings 90 points, 15 ratings 100 points, 50 ratings 110 points. Any posts of this type would be appreciated.
http://tasvideos.org/Baxter.html
Posted: 11/2/2012 4:18 PM
Quote
feos
Site Admin, Skilled player (1235)
Joined: 4/17/2010
Posts: 11264
Location: RU
NES TAS of 2011
What's the dispersion of amount of raters in total? Is there any sort of sats about the most rated runs? We could see the average of those runs and decide the value for such cases and for few rates, seeing WHICH runs attract more raters. But speaking of much raters, sometimes I see the runs that get lower rates than they actually deserve, but there're too many raters, so I can't change the situation with my single "true" rating. There's a problem of people not caring too much (we need to expand the gidelines list of what to pay attention to when rating).
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Posted: 11/2/2012 4:29 PM
Quote
Baxter
Skilled player (1402)
Joined: 5/31/2004
Posts: 1821
TAS of 2013 SNES TAS of 2013 Speedy TAS of 2007 Innovative TAS of 2007
feos wrote:
What's the dispersion of amount of raters in total? Is there any sort of sats about the most rated runs? We could see the average of those runs and decide the value for such cases and for few rates, seeing WHICH runs attract more raters.
I don't think there is. There are of course the moviestatistics, but it only lists the top 10 most rated movies. I thought Ilari said the other day that the average number of ratings per movie was about ~15?
feos wrote:
But speaking of much raters, sometimes I see the runs that get lower rates than they actually deserve, but there're too many raters, so I can't change the situation with my single "true" rating. There's a problem of people not caring too much (we need to expand the gidelines list of what to pay attention to when rating).
We already have a page for rating guidelines and voting guidelines. If you think that something is missing there, then you can always add it... I wonder if that will really solve your problem though. I don't think that you should base your rating on what other people rated though... if people were to view all of your ratings, they'd want to see what you like compared to other movies. If you base your ratings per movie on other people's rating to steer the average in the direction you want, then your personal ratings page is not very informative anymore.
http://tasvideos.org/Baxter.html
Posted: 11/2/2012 5:27 PM
(Edited: 11/2/2012 6:51 PM)
Quote
GMan
Joined: 4/7/2008
Posts: 117
Baxter wrote:
You failed to explicitly mention what exactly you are trying to do. Issue (1) in my post refers to taking the number of ratings into account when converting the ratings to player points. Are you trying to change the way the average is calculated which incorporates the number of votes in such a way that the old player point formula can be used? Would the old player point formula still need adjustments?
I'm trying to change the way movie ratings are calculated to be more meaningfully determined and less ad hoc. This means factoring in sample size from the start, which in turn happens to affect your problem #1. So the answer is sure: the old formula could be used with the new movie rating system and your old formula suddenly automatically takes into account sample size. But I wasn't suggested it has to be either-or, you could improve both. But factoring the sample size into both is unnecessary if it's already done in a previous step. You could (I don't know and would be willing to look) probably do the player points calculations on the intervals themselves, rank them by size, then use that ranking as an interpolation value from the lower bound to higher bound of the interval as the final score. Players with many higher-confidence scores would be rewarded.
Baxter wrote:
Also, the average is currently calculated I think by giving the entertainment rating a weight of 2/3 and the technical rating a weight of 1/3. Did you account for this? These ratings can be very different for certain movies; everyone agreeing that the movie is quite a technical achievement, but completely disagreeing on the entertainment value. This also makes you wonder how good the assumption of a normal distribution is.
I clearly specified I was considering only one column for exposition. You would calculate the confidence interval for both columns separately and then combine them in some fashion (i.e. could be the same as you just described), which I also said I was holding off from discussing because the teirs system is still being discussed and it would be a waste to bring it in. (To be honest, this part becomes entirely subjective so I don't have much interest in it. It's the more objective rating calculations I think need improving.)
Baxter wrote:
I also don't know if this is a good assumption. I for instance would never watch a 4 hour rpg TAS of a game I haven't played, so I would never rate such a movie. If I had to, I would probably rate it extremely low. People watch and rate the movies they are in some way interested in, so the ratings that you do get are in no way a good representation of what the distribution of ratings if everyone were to vote, and finding out/approximating what everyone would vote (including people who have never player the game/have no interest in it) does not seem like a good goal then.
I understand what you're saying, but you're reading into what I said backwards. Or put another way, your problem solves itself. The point is that the movies on the site are ultimately capable of being voted on, and if you watch it and hate it and rate it 1.5, then that's an accurate sample of either "how a voter on the site liked it" or "how a watcher of this movie liked it". My model assumes movie ratings should attempt to be the mean of all voters, yours wants to be the mean of all watchers. But unless you force all voters to be watchers, our systems are one and the same. (In other words, what my system calculates is "how all watchers liked it", which as the number of voters tends to the number of watchers becomes "how all voters liked it". This is no different than it is now. Your argument applies to the current system too, after all, since it's just an unadjusted mean.)
Baxter wrote:
Note that, as feos pointed out in the post above, discussing weight of entertainment/technical ratings or the way to calculate the final rating in general given the individual ratings is slightly off-topic.
Fair enough, but I figured if we're going to talk about redoing player points and discussing movie ratings, this was quite a suitable topic to bring it up.
Baxter wrote:
A average rating of a 6.3 currently gives about 100 player points. Just going by intuitions, how many player points would this 6.3 actually be worth if you know that it consisted of
There's no need for this approach, this is a solved problem in statistics, and I already outlined it. You're just ad hoc approximating the standard error. EDIT: Actually Baxter, I mislead you and perhaps misunderstood where you were coming from, and was only giving you half of what you were asking for. I think that's where the confusion is. Sorry about that, let me recollect my thoughts and I'll post and update.
1 2