I think mmbossman's suggestion is a great one, and a no brainer here. The previous votes indicate that people are clearly in favor of having less information displayed. This addresses that issue, while avoiding a "You guys can't behave so I will remove this feature" kind of approach, AND it increases the amount of information available for judging. I also think that rating the movies will increase the amount of involvement a voter feels when watching submissions. In addition, it improves another aspect of the site (movie ratings). A win/win/win/win if you ask me.
(Obviously I voted for this option).
Exactly my opinion... it's by far the best option that could possibly get a majority ;)
Count me in for another vote on multi-category ratings. I'm partial to the box plot method of representing them, though min/max/median would also work.
* Quietust, QMT Productions
P.S. If you don't get this note, let me know and I'll write you another
Joined: 11/18/2006
Posts: 2426
Location: Back where I belong
Quietust wrote:
Count me in for another vote on multi-category ratings. I'm partial to the box plot method of representing them, though min/max/median would also work.
Yeah, I'm not sure whether having the quartile system is especially needed, when most people would simply care about what the average is, and what the high and low ratings are. But the specifics can be ironed out in the future, if this is shown to be a popular choice.
Well you got me to switch from "no voting at all" to that awesome quartile craziness. My only suggestion would be to iron out the definition of "tech" as it was discussed recently that the definition of "close to perfection" is kind of ambiguous.
Well you got me to switch from "no voting at all" to that awesome quartile craziness. My only suggestion would be to iron out the definition of "tech" as it was discussed recently that the definition of "close to perfection" is kind of ambiguous.
I can only point to the rating thread (and hope people support my latest suggestions there ;)
Joined: 8/1/2004
Posts: 2687
Location: Seattle, WA
As I have stated before: if we can extrapolate that 5 is not held as an average value for both Entertainment and Optimization, the ratings system is inherently flawed and should be abandoned.
Joined: 11/18/2006
Posts: 2426
Location: Back where I belong
Zurreco wrote:
As I have stated before: if we can extrapolate that 5 is not held as an average value for both Entertainment and Optimization, the ratings system is inherently flawed and should be abandoned.
The "average" rating system for published movies is probably not going to be equal to 5, because we accept higher quality and higher entertainment runs to be published. However, if all the submissions in the past two years were rated, I would find it very surprising to see any sort of large deviation away from an average score of 5. The system I described allows us to vote lowly for submissions that deserve it, and accept submissions that meet our higher standards. So if you look at the ratings of published movies only, then you're really missing half the picture (ie the ratings of the gruefood, which don't exist [yet]).
As I have stated before: if we can extrapolate that 5 is not held as an average value for both Entertainment and Optimization, the ratings system is inherently flawed and should be abandoned.
Or, it just suggests that most TAS's are above average in quality.
Even if we take 5 to be a sort of 'base' quality, it doesn't follow that it should be the median for everything. I think there are 2 ways of using ratings - to reflect upon 'absolute quality' or 'relative quality'.
If reflecting upon 'absolute' quality, folk might ask themselves, 'how much did I enjoy this?', simply trying to give a 5 to something they felt impartial about and so on.
If considering 'relative quality', then folk would try to ensure that the mean or median rating they handed out was 5. How many people do this though? And even if they do, maybe they rate relative to more than only TASs - perhaps considering how much they enjoyed the TAS relative to a speedrun or a movie, TV show or whatever.
As long as folk EITHER rate along similar lines as many others OR rate a significant portion of movies, there isn't any problem with this as the relative scores between TASs will still be meaningful.
I'm just some random guy. Don't let my words get you riled - I have my opinions but they're only mine.
Joined: 8/1/2004
Posts: 2687
Location: Seattle, WA
Bezman wrote:
Or, it just suggests that most TAS's are above average in quality.
If "most" are "above average," the average needs to be shifted. The whole idea of a relative scale is that 49.99% are above the average, 49.99% are below the average, and a few select items are right on the average.
mmbossman wrote:
The "average" rating system for published movies is probably not going to be equal to 5, because we accept higher quality and higher entertainment runs to be published. However, if all the submissions in the past two years were rated, I would find it very surprising to see any sort of large deviation away from an average score of 5. The system I described allows us to vote lowly for submissions that deserve it, and accept submissions that meet our higher standards. So if you look at the ratings of published movies only, then you're really missing half the picture (ie the ratings of the gruefood, which don't exist [yet]).
5 as average means that the average should be 5. Otherwise, the ratings system isn't relative. There is a reason that the ratings are applied to publications and not submissions: they are supposed to be indicative of site content, not theoretical site content. If we had intended to factor submissions and/or gruefood in to the ratings system, then they would have been rate-able from the get go.
By your logic, everyone at the Olympic Diving events should be getting perfect 10s in every dive because the other 7 billion people in the world are probably a lot worse at diving.
e: <Zurreco> think of the workbench as 'where we rate things above or below 5' and the ratings section as 'where we rate things 1-10 based on the fact that they were already published'
Joined: 11/18/2006
Posts: 2426
Location: Back where I belong
Zurreco wrote:
By your logic, everyone at the Olympic Diving events should be getting perfect 10s in every dive because the other 7 billion people in the world are probably a lot worse at diving.
And by your logic, there should be as many published movies receiving total scores of 1 as there are movies receiving scores of 9. But that would seem to argue for acceptance of all runs.
There are 24 total publications out of more than 500 that currently have a rating of below 5. If you figure into the equation that most of the rejected runs would receive below 5 ratings were they to be published, there's your missing half of the spectrum. The fact that we chose to limit our acceptance of runs to higher than average quality does not mean that there should be as many sub 5 runs as above 5 runs.
Zurreco wrote:
e: <Zurreco> think of the workbench as 'where we rate things above or below 5' and the ratings section as 'where we rate things 1-10 based on the fact that they were already published'
So this would give us two flawed ratings systems, instead of one that covers all the movies submitted?
Joined: 8/1/2004
Posts: 2687
Location: Seattle, WA
No, that was a simplified argument to get rid of your bullshit 'only publish things above 5' concept.
The whole idea that everything we publish is above average negates the concept of an average in the first place. If we are to assume that every run published is above a 5 already, why even give the option to vote less than 5?
Joined: 11/18/2006
Posts: 2426
Location: Back where I belong
Zurreco wrote:
No, that was a simplified argument to get rid of your bullshit 'only publish things above 5' concept.
The whole idea that everything we publish is above average negates the concept of an average in the first place. If we are to assume that every run published is above a 5 already, why even give the option to vote less than 5?
As we discussed over IRC, we give the option to vote lower than 5 on published submissions because some movies slip through the cracks, and get published when they shouldn't, just like some movies that end up in the grue probably should have been published. Such is life. I know you don't agree with me, so we'll just agree to disagree.
Psychologically it'd be near impossible to convince people that 5 represents average when they watch videos of games they love being played in an amazing way. The votes of the judges at the Olympics are inherently critical, it's their job... the votes of the average TAS viewer is naturally going to come from a different perspective, one that will probably feel like the number 5 is far too low for a run of average quality.
Mathematically I absolutely agree with your point, but I don't think it would last long if it WAS the standard, then you'd have the guys in charge just constantly having to remind everyone to make their scores lower than it feels like they should be. Seems like an unnecessary chore for a semantic issue.
If "most" are "above average," the average needs to be shifted. The whole idea of a relative scale is that 49.99% are above the average, 49.99% are below the average, and a few select items are right on the average.
As I tried to indicate, not everyone uses a 'relative' rating system - some use an 'absolute' rating system.
In reality, most folk use something in between the two - taking into consideration other things out there but ultimately giving a '4' to things they deem 'slightly unenjoyable' and so on.
Zurreco wrote:
The whole idea that everything we publish is above average negates the concept of an average in the first place. If we are to assume that every run published is above a 5 already, why even give the option to vote less than 5?
Some people may disagree with a published movie's worth. The scale exists for folk to express their opinion.
Is there actually any problem if not everyone uses the entire scale? I can understand that it's frustrating when your ideals don't match up with everyone else's, but maybe consider taking a step back - is it really an issue?
What would be your ideal situation? Having everyone order the movies they've seen from best to worst? (only then is a rating truly relative...)
Also bear in mind that 'casual' folk are more likely to watch movies they're interested in - those same ones that they'd rate highly.
I'm just some random guy. Don't let my words get you riled - I have my opinions but they're only mine.
I think that if you want your rating system to be reliable, then you have to curve peoples' rankings based on their past history. Not everyone rates every movie; is it fair if, say, the only people who rate $obscure_but_awesome_game are unusually harsh raters, while those who rate $popular_game_that_makes_bad_TASes practice rampant grade inflation? So you look at each person's past rating history, and use that to adjust the range. If a given person only ever uses the range 8-10, then you rebalance their 8 to a 1, 9 to 5, and 10 to 10, for example.
Obviously this is hideously flawed for a number of reasons (not least of which is the number of people who only rate one or two movies), which leads me to the conclusion that we're just reading way too much into the ratings.
Pyrel - an open-source rewrite of the Angband roguelike game in Python.
(1) Posts debating the votes:
- I think posts wondering why people voted in a certain way are perfectly legit, and very much on topic of discussing the run. I agree that posts like "OMG, 10 yes votes already" or "a yes votes 2 minutes after the run is submitted" aren't useful at all, but I don't think the majority of the posts is like this. Discussing the reasons for what was voted, and if the votes should indicate publication or rejection seems perfectly fine to me.
I agree with you, however in practice that's not how it's been working. Someone votes no on a popular movie, and a few people complain and call the person out. No one responds, or they respond with "I didn't find it entertaining" and a discussion is launched into the true meaning of the sentence telling you what to vote.
If someone makes a comment about why the movie is bad and there is discussion about it great, but that is separate from the voting system, and they would have likely done it anyways without it.
Baxter wrote:
(2)Posts stating a personal vote:
- I don't see any problem with this. If it states a no-vote, than it's perfect, since especially those need posts attached to them. If it states a yes-vote, then it's fine too I think. People might know the author, and want him to know that they personally enjoyed the run, rather than being just one of the pile of ananymous yes votes. Most of the "I voted yes"-posts contain something like "great movie, yes vote", which gives the author of the run a feeling that his hard work is being appreciated, I think this is a good thing (ok... preferably the reasons why it is a great run should be stated... but I think even without it, it's fine).
Basically you touch on what I want to say at the end. Beyond the fact that there are many yes votes that don't really mention the run, the ones that say "great movie, yes vote" are actually less helpful than you think. If someone votes yes, it's safe to assume that they though it was a great movie. Unless of course they make a post along the lines of, "Although I voted yes, it's a pretty weak one. I think <x> could be improved." Which again, is much more helpful and quite likely would have been posted anyways.
It's also more helpful if someone says "Great movie, I really enjoyed when you <x>", as it gives the judge something to look for.
Baxter wrote:
Well, obviously I disagree with your statement that it's a bad thing if people would start posting their ratings (although obviously better with explanation of course). And you have to agree, someone just saying "8 entertainment, 5 technical", is more informative of someone just saying "yes vote". I also think this system encourages voting, which will be a good thing for the statistics (more ratings = better). I personally wouldn't mind these votes being public before publication, as it's nice to see what people voted (two 7's or a 5 and a 9 are quite different)... but I know this will never be done... so I'm just in favor of an average, and the amount of voters being public before publishing).
As we discussed in IRC, this comment came about because we took different interpretations to the poll. You assumed the judge wouldn't be able to see the ratings until it was judged, I assumed they would. Making such a post if the judge can already see the ratings is pretty useless. If someone wants to rate it anonymously (which I don't agree with, but whatever), then they wouldn't be posting their rating in the thread anyways.
Acheron86 wrote:
I'm still a bit confused as to what was so wrong with the original system. As long as there are forums, people will bicker about something, votes or no votes. How exactly does removing the voting system solve the problem?
There wasn't anything really, beyond the fact that there were so few judges that we couldn't keep up with the tide of submissions. Voting was then added to help pick out the quality runs first so they didn't sit around for literally years. There was alot more poor submissions back then.
Acheron86 wrote:
It seems like you're basically saying that whether or not I enjoyed a video isn't really relevant if I can't write out an essay describing it.
I'm not saying it's irrelevant without an essay, I'm saying a comment like "Why did you jump at the end of level 3?" is infinitely more helpful than "LOL YES VOTE BEST MOVIE EVER!!!!". Making a comment on the run itself to say what you didn't like, or even what you did is what I am looking for. Pick something at random if you need to.
Acheron86 wrote:
So if there's a voting system, the judges will ignore our posts and only look at the numbers? Forgive my ignorance, but where has this been a problem?
No, as I said, if there is a post saying "First yes vote!" that is no help. Because the poll tells the judge that you voted yes. You are providing no helpful insight into the run, and just repeating information that can be seen elsewhere. Judges are suppose to be fair and impartial, playing favourites in voting goes against that.
mmbossman wrote:
New suggestion
This is a good suggestion too, maybe the best so far. I'll think on it more during my day at work and see if I can come up with any flaws/improvements.
For your point 2), that is why people were suggesting keeping it hidden to members until the judges had finished with it. That said, I don't like the idea of anonymous voting on the submissions any more than you do. If people aren't willing to stand up to their vote, should their vote really matter?
On an unrelated note. Baring a few less than stellar posts off the start, that new submission is doing pretty well without votes if you ask me. (Though i'm sure after I post this, the childishness will attack to try and prove their point)
Joined: 11/18/2006
Posts: 2426
Location: Back where I belong
BoltR wrote:
On an unrelated note. Baring a few less than stellar posts off the start, that new submission is doing pretty well without votes if you ask me. (Though i'm sure after I post this, the childishness will attack to try and prove their point)
[childish] The old one did pretty well too, even with a poll [/childish]
Interestingly, the main argument from that thread stemmed from a "meh" vote, but quickly moved on to attacking a players opinion of the improvement, which seems to be the type of argument you were advocating earlier in the thread. Or maybe I'm misunderstanding.
That's not really what I mean. I was thinking people making junk posts to try and throw the thread off course ;)
Edit: Sort of mmbossman. However, after they discuss the meh vote, it does sort of degenerate into a discussion about meh vs no because of personal preference. How the the voting system doesn't require knowledge of the game because it's not SDA and other pointless bickering. Even you yourself are telling people to stop arguing by the end of the thread.
And then Hoe sums up my point pretty well:
Hoe wrote:
What the fuck happened to this site? I have not been around in a while, but what I always liked about it, is that it was a programmatic dissection of things which honestly didn't need it, just as a hobby in peoples free time -- people who enjoyed it got together. It wasn't gamefaqs where people would get at each others throats over pointless crap. I, and no one else, really give a shit if you did not enjoy the game, get over yourself. I'm not 14 years old anymore, sorry if you are (even if not physically).
I'd like to suggest a combination between option 3 and option 10.
Ratings, with median visible to all users, and more specific information visible to judges.
I also have the suggestion that people who post on the workbench forum have the option to make the post anonymous, so they feel more free to speak their precise opinions.
How fleeting are all human passions compared with the massive continuity of ducks.
(2)Posts stating a personal vote:
- I don't see any problem with this. If it states a no-vote, than it's perfect, since especially those need posts attached to them. If it states a yes-vote, then it's fine too I think. People might know the author, and want him to know that they personally enjoyed the run, rather than being just one of the pile of ananymous yes votes. Most of the "I voted yes"-posts contain something like "great movie, yes vote", which gives the author of the run a feeling that his hard work is being appreciated, I think this is a good thing (ok... preferably the reasons why it is a great run should be stated... but I think even without it, it's fine).
Basically you touch on what I want to say at the end. Beyond the fact that there are many yes votes that don't really mention the run, the ones that say "great movie, yes vote" are actually less helpful than you think. If someone votes yes, it's safe to assume that they though it was a great movie. Unless of course they make a post along the lines of, "Although I voted yes, it's a pretty weak one. I think <x> could be improved." Which again, is much more helpful and quite likely would have been posted anyways.
It's also more helpful if someone says "Great movie, I really enjoyed when you <x>", as it gives the judge something to look for.
You didn't adress my main point that an author is getting appreciation from a "great movie, yes vote", which is vital to the authors motivation to TAS possibly.
