Experienced Forum User, Published Author, Experienced player
(822)
Joined: 11/18/2006
Posts: 2426
Location: Back where I belong
Zurreco wrote:
By your logic, everyone at the Olympic Diving events should be getting perfect 10s in every dive because the other 7 billion people in the world are probably a lot worse at diving.
And by your logic, there should be as many published movies receiving total scores of 1 as there are movies receiving scores of 9. But that would seem to argue for acceptance of all runs.
There are 24 total publications out of more than 500 that currently have a rating of below 5. If you figure into the equation that most of the rejected runs would receive below 5 ratings were they to be published, there's your missing half of the spectrum. The fact that we chose to limit our acceptance of runs to higher than average quality does not mean that there should be as many sub 5 runs as above 5 runs.
Zurreco wrote:
e: <Zurreco> think of the workbench as 'where we rate things above or below 5' and the ratings section as 'where we rate things 1-10 based on the fact that they were already published'
So this would give us two flawed ratings systems, instead of one that covers all the movies submitted?
Experienced Forum User, Published Author, Experienced player
(822)
Joined: 11/18/2006
Posts: 2426
Location: Back where I belong
Zurreco wrote:
As I have stated before: if we can extrapolate that 5 is not held as an average value for both Entertainment and Optimization, the ratings system is inherently flawed and should be abandoned.
The "average" rating system for published movies is probably not going to be equal to 5, because we accept higher quality and higher entertainment runs to be published. However, if all the submissions in the past two years were rated, I would find it very surprising to see any sort of large deviation away from an average score of 5. The system I described allows us to vote lowly for submissions that deserve it, and accept submissions that meet our higher standards. So if you look at the ratings of published movies only, then you're really missing half the picture (ie the ratings of the gruefood, which don't exist [yet]).
Experienced Forum User, Published Author, Experienced player
(822)
Joined: 11/18/2006
Posts: 2426
Location: Back where I belong
Quietust wrote:
Count me in for another vote on multi-category ratings. I'm partial to the box plot method of representing them, though min/max/median would also work.
Yeah, I'm not sure whether having the quartile system is especially needed, when most people would simply care about what the average is, and what the high and low ratings are. But the specifics can be ironed out in the future, if this is shown to be a popular choice.
Experienced Forum User, Published Author, Experienced player
(822)
Joined: 11/18/2006
Posts: 2426
Location: Back where I belong
After talking things over with some people, I've come up with an idea that would be my ideal method for fixing this dilemma. Firstly, I voted for the last option: "No poll, but ratings, but individual ratings are kept secret until judging."
I think that implementing ratings are a very neat way of tying two parts of the site together, and increasing usability of both systems. However:
1) I don't like the idea of having only an average score being displayed, because I do not think that a simple average is a good enough measure for all the information we'd be getting
and
2) Nor do I like the idea of having everyone's rating be displayed anonymously (similar to how we used to with yes/no/meh votes). Displaying everyone's ratings anonymously would continue the witch-hunts, and would become very cluttered in large/popular submissions.
So, my suggestion is this (Warning: Statistics 101 revisisted): Use a Box Plot (also known as a box and whiskers plot). For those of you who do not know and don't care to click the handy link, a box plot is a simple graphical representation of a set of data, that makes it very clear what the lowest vote is, the 1st quartile, the median, the 3rd quartile, and what the highest vote is. Example
0 1 2 3 4 5 6 7 8 9 10
|-------[====|====]--|
The line furthest to the left is the lowest score. The first "[" is the 1st quartile, the middle "|" is the median, the "]" is the third quartile, and the farthest right "|" is the highest vote.
Three of these graphs could be used, one for entertainment scores, one for tech scores, and one for overall scores. It would allow everyone, judges and viewers alike, to see if a run has received high entertainment, but low technical scores, or high technical scores, but lower entertainment scores (with the overall score still being used as a good indicator as to whether the public thinks the submission warrants publication.)
If the quartile system seems to be superfluous information, a similar type plot showing the low score, the mean (average), and high score would work as equally well, in my opinion.
So, the points I'm trying to make with this suggestion
1) It incorporates a system already in place on the site, which brings more continuity between the published movies and the forums
2) It provides an anonymous vote system where the lowest/highest vote for a specific category is displayed, but could possibly lead to a decrease in witch-hunts (the person who voted 1 in entertainment could also have voted 9 in tech, after all).
3) It provides everyone an equal representation of the data available, without giving judges extra knowledge.
4) It provides a much greater wealth of information that yes/no/meh votes, in that a run with a high entertainment score and low technical score will most likely need to be redone for publication, whereas a run with a high technical score but a low entertainment score will likely not be accepted due to poor game choice.
5) The viewers will still be able to voice their opinion of their run before, with less of a temptation to simply post what they vote ("I voted 6/7" without giving a reason seems to be more of an incomplete post than "I voted yes" without giving a reason). This will hopefully increase the signal:noise ratio.
Experienced Forum User, Published Author, Experienced player
(822)
Joined: 11/18/2006
Posts: 2426
Location: Back where I belong
You could also argue that people who don't live and breath video games wouldn't have any idea what "4th generation" means.
Until I became involved with this site, I would have picked the Genesis as a 3rd generation console (I only really knew about the Atari 2600 and NES before that point). I also didn't know that the Genesis was also known as the Megadrive. "4th generation" uses no descriptive terms to help describe what is included in the subforum, and conveys very little helpful information (other than it is one more than 3rd generation).
Experienced Forum User, Published Author, Experienced player
(822)
Joined: 11/18/2006
Posts: 2426
Location: Back where I belong
BoltR wrote:
On the topic of the judges are complaining about having too much to read. You have no room to complain. This is what you signed up for. YOU are the quality control, not the people on the forum. YOU are judges because you are suppose to have a good eye and watch the runs and take up questionable TASing with the author. YOU are suppose to sift through the posts to see if the there is something wrong with the run in the posts. YOU are the hidden face of the front page and what gets published. YOU are the 'voice' of Bisqwit. Why do you think there have been so few people to do it? It's a lot of work and Bisqwit puts a lot of trust in those people. If you don't like this, or think it's too much work, maybe you should reconsider your desire to be a judge.
Just because I can live without something does not mean that I should want to live without it. I like having a dishwasher so that I don't have to do dishes by hand. I like having a car so that I don't have to walk everywhere. I like having votes to prioritize my time spent watching movies. Votes are a convenience. Votes are helpful to me. And that is why I like having some sort of poll around. It makes life easier for me.
I also dislike having you up on your soapbox tell me and others how to judge, when you were (and continue to be) very inactive in that roll. Since the Wiki based system was set up, you have 65 edits, very few (<5) of which happen to be judgements. Just because you have the title doesn't make you an active participant. You want to initiate change. I'm not against that. But please, if you're going to suggest a change in a system that you don't use, at least get some opinions from people who do use it before suggesting it be removed entirely.
I am glad that this has gotten more people talking and passionate about the state of the site. However, I think that it could have been implemented in a much better way.
Experienced Forum User, Published Author, Experienced player
(822)
Joined: 11/18/2006
Posts: 2426
Location: Back where I belong
AKA wrote:
I will admit that it could be abused with 0's because you don't like the game or the author and 10's for vice versa, or because you don't know any better. Hopefully most people on the forum will be mature enough and give sensible ratings.
Or people could give unusually low scores because the author didn't aim for fastest time, and instead aimed for entertainment.
Experienced Forum User, Published Author, Experienced player
(822)
Joined: 11/18/2006
Posts: 2426
Location: Back where I belong
Yeah, I saw that. You don't exactly give much support to your argument though. Care to actually back it up with some sort of facts, logic, or examples?
Experienced Forum User, Published Author, Experienced player
(822)
Joined: 11/18/2006
Posts: 2426
Location: Back where I belong
Apo123 wrote:
stickyman05 wrote:
JESUS CHRIST GET RID OF THE TEXT! SOOOO UGLY!
(about as ugly as all my yelling and such in this post :/ )
Or just have a check for if you've posted in the topic. If you've posted, no text at the top.
Umm... no?
To clarify, the text appears in all the submissions that I have posted in, voted on, or both.
EDIT: Oh, wait... I misunderstood that you were making a suggestion, not that what you were saying was how the text behaves right now. I'll sit down and shut up now.
Experienced Forum User, Published Author, Experienced player
(822)
Joined: 11/18/2006
Posts: 2426
Location: Back where I belong
You could just post which two options you find equally acceptable, and then vote for one or the other. This whole process was supposed to help people rely less on polls, and more on posting their thoughts anyway.
Experienced Forum User, Published Author, Experienced player
(822)
Joined: 11/18/2006
Posts: 2426
Location: Back where I belong
moozooh wrote:
Gah, you and your american internets!
:D
Granted, I didn't take into account other languages, but English isn't only spoken in the US, after all. And it happens to be the only language I know... so yeah. Damn my american internets.
Experienced Forum User, Published Author, Experienced player
(822)
Joined: 11/18/2006
Posts: 2426
Location: Back where I belong
adelikat wrote:
How about a rating system where individual ratings are hidden until publication. At that point, only those who choose would have their ratings public (as it is now).
This would:
1) address the issue in option D (which is winning the poll currently)
2) give more accurate information to judges
3) give more control to the audience and less to the judges which I am in favor of if it comes with 2)
I change my vote to this.
One less for "the way it was" and one more for the ratings option.
I also like this better than any of the options currently suggested. It would prevent off-topic discussion stemming from "no" voting, would result in greater information being made available to those making the decisions, and would help contribute to the ratings of the movie post-publication.
I change my vote to this idea (as long as ratings are not locked upon input).
Experienced Forum User, Published Author, Experienced player
(822)
Joined: 11/18/2006
Posts: 2426
Location: Back where I belong
nineko wrote:
And what about adding "Game Gear" to the Master System forum?
Roughly 98% of game gear games are direct ports of SMS games, and there has yet to be a submission of a game gear exclusive. I don't see any problem with changing the listing once there is such a publication, though.
Experienced Forum User, Published Author, Experienced player
(822)
Joined: 11/18/2006
Posts: 2426
Location: Back where I belong
adelikat wrote:
Let's say someone votes no on a popular submission. Then it is published and the votes are displayed. At that point, wouldn't someone then demand to know who voted no?
I would do exactly what I've done in the past few threads where this has already come up: Tell them to stop whining, because their run got published, which was the end goal anyway.
adelikat wrote:
Also, if voting results are removed from the public, the total number of votes should at least be displayed. People use the voting stats as a means of selecting which submissions to watch.
Experienced Forum User, Published Author, Experienced player
(822)
Joined: 11/18/2006
Posts: 2426
Location: Back where I belong
Bisqwit wrote:
Another idea: Re-enable voting, but make poll results viewable only to judges.
Plus:
-- Avoids the witch hunt issue; no particular vote is going to catch attention.
-- Judges get useful numbers.
Minus:
-- People WANT to see poll results in order to be motivated at voting.
-- Authors still need to see positive feedback to get gratification from their work.
----
[Edit by Bisqwit]: Added the issue Baxter pointed out.
I agree with this idea, and would suggest that the votes could be made known after a decision has been reached (either publication or rejection).
Experienced Forum User, Published Author, Experienced player
(822)
Joined: 11/18/2006
Posts: 2426
Location: Back where I belong
I personally think it's a bad idea to remove the votes, for many reasons.
1) It makes things harder for the judges. Now, instead of having to not only watch a 3+ hour movie, I have to read through 10+ pages of responses, and then try to assimilate all those into one opinion that would best describe the thoughts of the group as a whole? Sorry, that just makes it harder for me to want to watch a long movie that I know probably won't be entertaining to me in the least, but should probably be published anyway.
2) It makes it harder for judges to see which runs are nearly guaranteed publication status (15 yes votes, 1 meh, 1 no), versus which runs I may need to get more opinions on (8 yes, 2 meh, 2 no). Removing the voting system takes away a lot of the ease of keeping the submission queue under control.
Bisqwit wrote:
since the number of submissions compared to the number of judges is so much more manageable now than it used to be, the numbers are not really needed now.
This would be true, if all the judges were active. I have been the most active judge in the past couple of months (which is no problem), but before that, adelikat and Truncated were the only judges who actively kept the queue to manageable levels. Removing the votes will certainly not make our jobs any easier, and people already piss and moan about runs staying unaccepted when they've been on the workbench less than one month. I'm looking at you, OoT.
3) It removes yet another "fun factor" from the site. First the stars were removed because Bisqwit didn't want to offend anyone (please, bring them back in some form), and now we have the voting removed, which I think was one of the more fun things about the site. People got excited when they could finally vote and authors like to see a lot of yes votes for their movies. Who cares if it's an ego boost? This site is meant to be fun, not to be all for the greater good (comrade).
4) We lose the opinion of many people who may not be fluent in English. If they are embarrassed to write out a response because they know it may not convey their actual thoughts, they now have no other way to show their approval/disapproval of the run. As much as I like to encourage people to provide reasons for their votes, I do no think it should be a mandatory requirement.
5) The voting system was removed because BoltR suggested it. I heavily disagree with this, not because it was BoltR to suggest it, but because it was done without asking for the opinion of anyone else. Experimentation is good, but removing a major part of the site due to one person's suggestion smacks of favoritism (and if it doesn't, I strongly suggest to replace the voting system).
6) Whether you scoff at it or not, peer pressure on the internet does exist, because it is human nature to be liked, and providing an opinion that clearly differs from the majority can lead to ridicule (looking at you again, OoT). Yes, when someone votes, they should provide their reasoning. But when a no vote already gets ostracized because it's different than the crowd, can you really expect people to become even more active? I have a feeling that, instead of increasing the commentary, this will simply act as a catalyst to decrease negative commentary on all but the most horrible runs. So what we'll be left with is a handful of people who say they somewhat like a run, and then get pissed when it gets rejected for being a crap-tastic game (although this already happens anyway). In any case, I think it will reduce the amount of negative criticism of all runs.
I will probably think of a few more reasons, but this is my list of arguments against removing the voting system.
Experienced Forum User, Published Author, Experienced player
(822)
Joined: 11/18/2006
Posts: 2426
Location: Back where I belong
Warp wrote:
mmbossman wrote:
A solution like that would open up a very, very large can of worms, especially when it comes to unpublishing other runs.
I'm getting really tired of this current trend of shooting down all possible improvement suggestions with either a slippery slope argumentation or the tired "it would only cause disagreements" argument.
I disagree. Just because one video is unpublished will *not* open any can of worms. If someone requests unpublishing something else "because that Zelda run was also unpublished" you simply answer to him "that was an exception", and if he protests, just ignore him.
Again: Since when has this website converted into a nazi regime where every rule is absolutely strict and no exceptions are allowed, ever?
You act like you're the first one to suggest unpublication. You're not. It's been brought up multiple times before, and been shot down before, by the administrator of the site. I, for one, am actually in favor of it for various games, but unless you can convince Bisqwit that it should be a viable option, you're just reviving old arguments.
I especially dislike this line:
I disagree. Just because one video is unpublished will *not* open any can of worms. If someone requests unpublishing something else "because that Zelda run was also unpublished" you simply answer to him "that was an exception", and if he protests, just ignore him.
We should ignore him, unless of course, it's you? And if I were to ignore you, you'd just start complaining about the Nazi regime here? Great system there, Warp.
