Posted: 12/17/2009 8:43 PM
Post subject: Confidence (cf:ce) formula
Quote
sgrunt
He/Him
Emulator Coder, Former player
Joined: 10/28/2007
Posts: 1360
Location: The dark horror in the back of your mind
I've noticed a discrepancy between what the submissions queue claims is the formula for calculating the confidence (cf:ce) percentage and the formula the site actually seems to use. After some experimentation, I've concluded that the following formula is in use, and I'm posting it here for reference (both my own and everyone else's): cf:ce = floor(100% * min( 1, ((abs(yes - no) / (yes + no + meh)) + 0.7 * (yes + no + meh / 12)^1.1 )/(1.7)) (This should actually be readable for those people whose browsers can't render MathML, by the way. If there's interest, I can render this formula in TeX or something similar so that it's even more readable.) EDIT: EDIT2: The old picture was hard to read, so I enlarged it.
Posted: 12/18/2009 10:22 AM
Post subject: Re: Confidence (cf:ce) formula
Quote
Bisqwit
Editor, Active player (296)
Joined: 3/8/2004
Posts: 7469
Location: Arzareth
sgrunt wrote:
cf:ce = floor(100% * min( 1, ((abs(yes - no) / (yes + no + meh)) + 0.7 * (yes + no + meh / 12)^1.1 )/(1.7))
Is that which you posted, not in fact the precisely same formula as indicated on the page, except for the floor() wrapping it? This is the formula shown on the submission page (rendered by Firefox, acquired by using the "?xml=1" parameter to the page (rewritten its url using queue.cgi to get around the cache) and manually fixing the broken xhtml introduced in the advertisements): EDIT: Oh, the min(1,x) in the upper right corner. Are you sure it is not there? EDIT2: Hmm yes, that min() is indeed removed. And there is indeed an outer min(). The floor() is done implicitly by sprintf using the %d modifier. Good job.
Posted: 12/18/2009 4:47 PM
Quote
sgrunt
He/Him
Emulator Coder, Former player
Joined: 10/28/2007
Posts: 1360
Location: The dark horror in the back of your mind
The shuffled-around min()s have the interesting effect of the submission being guaranteed to hit 100% confidence eventually, even if the voting is heavily divided. With the formula as written on the queue, an evenly split vote would only ever reach 41% confidence, and could only ever reach 100% if the voting was completely in one direction (it could get arbitrarily close to 100%, but never reach it). Which is preferable? With the currently implemented formula, 100% is reached with at most 27 votes, and as few as 12 if the votes all go one way. I can see arguments going either way - after 27 votes the judges should have a very clear picture of what the viewer feedback on the submission is; however, if the voting is very heavily divided, we have no way of knowing what the average viewer will think of the submission. Have there been any major controversies over this in the past?
Posted: 12/18/2009 5:55 PM
Quote
Derakon
Joined: 7/2/2007
Posts: 3960
How much do the judges pay attention to the voting, anyway? It's always been my impression that it's the comments (and, of course, the judge's own opinion on watching the work) that matter the most.
Pyrel - an open-source rewrite of the Angband roguelike game in Python.
Posted: 12/18/2009 6:23 PM
Quote
alden
Player (120)
Joined: 2/11/2007
Posts: 1522
I always thought that that was there just as a reminder that we're pedantic nerds :D
I make a comic with no image files and you should read it. While there is a lower class, I am in it, and while there is a criminal element I am of it, and while there is a soul in prison, I am not free. -Eugene Debs
Posted: 12/19/2009 5:42 PM
Quote
Nach
Emulator Coder
Joined: 3/9/2004
Posts: 4588
Location: In his lab studying psychology to find new ways to torture TASers and forumers
Derakon wrote:
How much do the judges pay attention to the voting, anyway? It's always been my impression that it's the comments (and, of course, the judge's own opinion on watching the work) that matter the most.
I only use the votes if I'm not sure. If I see the video is terrible, violates rules, or there's no place for it, I'll reject it. If I see it's amazing (based on entertainment choices, how well the runs plays the game, comparison to other published runs of the game), I'll publish it even if I see some people didn't like it. If I thought the video could be done better, but I know there's a lot of work involved to do so, and it was somewhat interesting as is, then I'll look at the votes and see if the people want it or not. Edit: However, when I'm looking at the submission queue filled with unknown to me games, and want to judge something, it's nice to see a confidence rating along with vote count on the side to know which ones are ready to be judged.
Warning: Opinions expressed by Nach or others in this post do not necessarily reflect the views, opinions, or position of Nach himself on the matter(s) being discussed therein.
Posted: 3/21/2010 3:28 AM
Quote
sgrunt
He/Him
Emulator Coder, Former player
Joined: 10/28/2007
Posts: 1360
Location: The dark horror in the back of your mind
Out of curiosity, how many people (if any) find this useful for anything these days? We are considering doing away with it.
Posted: 3/21/2010 4:35 AM
Quote
Flygon
Joined: 11/4/2007
Posts: 1772
Location: Australia, Victoria
I always treated voting as less of a Judge related thing and more of an Encoder related thing, I see no reason to do away with it (Unless you mean the confidence rating... in which case, I never really paid much attention to it unless it's a fairly extreme number, usually below 70%). It's a rough guide but it provides a quick reference for what should be on my encoding priority list... assuming it isn't empty, anyway.
Posted: 3/21/2010 4:36 AM
Quote
sgrunt
He/Him
Emulator Coder, Former player
Joined: 10/28/2007
Posts: 1360
Location: The dark horror in the back of your mind
Flygon wrote:
Unless you mean the confidence rating... in which case, I never really paid much attention to it unless it's a fairly extreme number, usually below 70%).
Look at the thread title... I do mean confidence rating. :-P