The Problem
It has been brought up a few times recently (roughly in the last year or so) that the workbench entertainment polls can be (and in some cases are) essentially ignored by a judge when determining publication tier, especially if explanation/content in the thread itself is contrary to the poll results. There are even times that the poll may be essentially ignored simply because there is little to no discussion in the forum at all. Side Note 1: We need to stop equating lack of interest with low entertainment. Just because a movie is sparsely viewed or commented on, doesn't mean that the movie itself is not entertaining to those who do chose to watch it.
Basically this means that a poll vote which is not also explained in the forum thread holds little to no value. This essentially makes the polling a (mostly) futile feature of the site in the first place. If the thread trumps the poll anyway, why not just eliminate the poll and rely solely on thread comments?
The primary problem with this approach is that it allows for ignoring the held opinions of some simply because they chose not to explain their viewpoint via words in the forum thread. It's as if we reward explained opinion more than held opinion; but all opinions are equally valid.
Side note 2: I'm not trying to say discussion of opinions isn't also valuable. Discussion can absolutely help to understand or even change opinions, but lack of discussion shouldn't be used to diminish the value of someones held opinion.
It is my understanding that the poll is treated this way, at least in part, due to the ambiguity of the results; this ambiguity stemming from the fact that some users answer the question regarding degree of entertainment, while others answer the question as if the movie in question should be published or not.
I'm aware that various suggestions have been made regarding changing the poll question/answers. I'm not going to pretend that I remember all of them, nor do I have time to go searching through the forums for all the various suggestions. That acknowledged, I apologize if the following has been asked/suggested before.
Two Suggestions/Questions1) Should the poll be outright eliminated given its minimal value that can be relatively easily trumped by comments? (I do not support this approach!)2) Would it be possible to make the workbench poll a two-question response that attempts to answer both aspects of how the current question is treated?
Essentially have two questions that need answered to submit a poll vote:
1) Should this movie be published (does it adhere to the rules as you understand them)?
Yes
No
2) If published, in which tier should it be published?
Moons
Vault
I Don't Care
This approach would yield a better understanding of how/why people are voting the way they are on submissions (regardless of whether or not they choose to comment with text in the discussion). This would restore greater value to the the results themselves and the whole endeavor of polling the workbench submissions in the first place.
It would eliminate the ambiguity (as described above) of the current poll:
The first question obviously answers whether or not the viewer feels the movie if valid for publication. The second question establishes the degree of entertainment value in the movie. Side Note 3: I did not include Star tier in the second question as it's an individually curated (while open to suggestion) tier. I suppose it could be included in the poll, but I wouldn't recommend it.Implementation (At least for the second suggestion)
If this would indeed be an approach desirable to the community...How difficult would it be to implement this on the site?
I recognize that the coding for the workbench polls would need changed. The submission list page shouldn't need much change as it could maintain the vote percentages (using much the same calculation as is currently utilized) by the following variable conversions:
(old question answers) = (new question answers)
Yes = Moons
No = Vault
Meh = I Don't Care
This would still offer 3 entertainment level choices in the poll while yielding a general % entertainment value that would theoretically be more accurate than results generated by the current polling method.
What are people's thoughts on this?Note to site staff/moderators: I realize this may have been better categorized under Site -> Open Forum: Feature Requests, but I made a new topic in hopes of obtaining more attention/feedback. If it absolutely needs moved/merged, please do so.
I agree that the workbench poll really should be split up into at least 2 questions.
There are a lot of times where I see a run on the workbench which is pretty boring for me to actually watch, but is extremely optimized and a good example of TASing. However, by convention I vote yes on the workbench poll for these, even though I'm not very entertained by them.
Even if I believe that the quality of a run lends itself to publication in moons or vault tier, that doesn't necessarily mean that I found the run very entertaining. Even for movies of popular games which get accepted into moons, I still sometimes run into the issue of not wanting to vote no because I think that the movie should be published (but I don't necessarily find it entertaining).
On the other hand, there are also occasionally runs submitted to the workbench which have some unoptimal or sloppy parts that should prevent them from being published. However, when I watch the movie itself, I am entertained by it anyway. However, I feel obliged to vote no on the workbench poll, because me voting yes would imply to some that I believe that the movie is optimized enough that it should be published, when in reality, all I want to say is that I was entertained by watching the movie.
I would actually go one step further than what DrD2k9 purposed and say that the workbench poll should be broken down into 3 smaller questions, which are as follows:
1. Do you believe that this movie should be published?
a. Yes
b. No
c. I'm not sure
2. (A question that only appears if the user answers yes to the first question)
What tier do you feel that this movie belongs in?
a. Vault
b. Moons
c. Stars
d. I'm not sure.
3. Were you entertained by the movie?
a. Yes
b. No.
c. Meh.
This would generate an unambiguous assessment of how entertaining users find movies on the workbench and how strongly users feel that a given movie is optimized or entertaining enough to deserve being published. Thus, all of the ambiguity of the current system could be easily ended.
I would also like to clarify why I have a separate question for entertainment and what tier a movie belongs in, since some may think that this is redundant. With certain movies, a user may not find them to be personally entertaining, but may still be so impressed by the TAS (or entertained with the background work that went into making the TAS) that the user may want the movie to be in moons or stars tier despite not being particularly entertained by watching the TAS.
A good example of this that came up recently was the Super Mario Brothers 3 TAS that was just published. Some people thought that the background work that went into the TAS was so impressive that the movie deserved a spot in either moons or stars tier, despite not finding the run itself entertaining.
Changing the workbench poll would allow users to express these opinions in a clearer and more nuanced manner, which would allow the person judging the submission to better understand how exactly the audience has reacted to the movie. After that, the judge could use their experience and knowledge of the rules of the website and the quality of the movie to decide what should best be done with the movie given the audience's perception of the movie.
I would also like to clarify why I have a separate question for entertainment and what tier a movie belongs in, since some may think that this is redundant. With certain movies, a user may not find them to be personally entertaining, but may still be so impressed by the TAS (or entertained with the background work that went into making the TAS) that the user may want the movie to be in moons or stars tier despite not being particularly entertained by watching the TAS.
The reason most would consider your latter two questions redundant is because, the very difference between vault and moon tier is based on entertainment....not impressiveness of the TAS technical level.
Vault is the appropriate landing place for runs that aren't entertaining even when they possess technically impressive achievements.
As others have mentioned before (Warp is commonly one of them) a run being accepted to vault should not be looked at as a negative thing. Unfortunately, many in the community seem to treat the Vault as a collection place for 'bad' TASes. This isn't the case. The vault is the landing place for unentertaining TASes whether they are technically impressive or not. Bad TASes don't get accepted to begin with.
If someone is claiming that they didn't find a particular run entertaining but then say it should be published to a tier other than the vault, that individual doesn't understand that the site's tier system is first and foremost entertainment based.
I would also like to clarify why I have a separate question for entertainment and what tier a movie belongs in, since some may think that this is redundant. With certain movies, a user may not find them to be personally entertaining, but may still be so impressed by the TAS (or entertained with the background work that went into making the TAS) that the user may want the movie to be in moons or stars tier despite not being particularly entertained by watching the TAS.
The reason most would consider your latter two questions redundant is because, the very difference between vault and moon tier is based on entertainment....not impressiveness of the TAS technical level.
While what you're saying is technically true, the way that people vote might not always reflect the way that the tiers are set up. Having different options to clarify what people really think about the movies makes the most sense, since this would remove any ambiguity.
Like I said, some of the people who voted in the SMB 3 TAS thought that the run should go in moons, despite not being entertained by the TAS itself. Bearing that in mind, its worth considering if there's other runs on the site that people have voted "yes" to the site question for because they believed that the movie should be in moons (despite not being very entertaining).
1) Should this movie be published (does it adhere to the rules as you understand them)?
Yes
No
I'd like to note that this question would not contribute to the actual publication process. It's explicitly the judge's role to determine this, not the viewer.
Also Judges don't literally ignore the polls (at least I don't think we're supposed to), it's just one of the smallest factors in determining tier.
EDIT: Tbh I prefer when people vote honestly as to how they feel about entertainment of a movie. If you are not entertained by a movie, vote that you are not entertained by the movie. Don't vote yes instead because you think the movie should go to Moons. It makes our job a lot harder when people overthink things because that results in TASes getting moved between tiers.
[16:36:31] <Mothrayas> I have to say this argument about robot drug usage is a lot more fun than whatever else we have been doing in the past two+ hours
[16:08:10] <BenLubar> a TAS is just the limit of a segmented speedrun as the segment length approaches zero
I think the poll should be retained, as it's valuable, but its results shouldn't be dismissed so lightly. I think it should be taken more into consideration.
If the poll and the discussion in the thread agree with each other, then there's little problem in terms of judging. However, if there seems to be some kind of disagreement between the poll and the discussion, this could induce the judge to consider the tier decision more carefully, and perhaps even ask the community some questions in the thread if necessary, and ask for more opinions and clarification.
I have in the past suggested a slight change to the meaning of the poll answers, to be more like:
* I see no problem in this being published, and I liked it.
* I see no problem in this being published, but I found it unentertaining.
* I see no problem in this being published, and I have no strong feelings one way or the other with regards to entertainment.
* This shouldn't be published (explain in the thread why).
If the there are any votes on that last one, this ought to catch the attention of the judge.
If the poll and the discussion in the thread agree with each other, then there's little problem in terms of judging. However, if there seems to be some kind of disagreement between the poll and the discussion, this could induce the judge to consider the tier decision more carefully, and perhaps even ask the community some questions in the thread if necessary, and ask for more opinions and clarification.
This is pretty much how I use the polls.
[16:36:31] <Mothrayas> I have to say this argument about robot drug usage is a lot more fun than whatever else we have been doing in the past two+ hours
[16:08:10] <BenLubar> a TAS is just the limit of a segmented speedrun as the segment length approaches zero
1) Should this movie be published (does it adhere to the rules as you understand them)?
Yes
No
I'd like to note that this question would not contribute to the actual publication process. It's explicitly the judge's role to determine this, not the viewer.
I know that the judge is the ultimate decision maker regarding acceptance/publication and not the community.
I suggested this question for two reasons.
1) It will help us understand if the community as a whole generally understand the rules.
2) With either acceptance or rejection, having the results from this question provides the judge with information that may allow for formulation of more detailed judgement notes in regards to the rules. If the majority of voters are in agreement with the judge, the judgement notes may not need to be as detailed. However if the majority of voters are in disagreement with the judge, the judge knows that a more thorough explanation of why the movie does/doesn't fit the rules is warranted as part of the judgement note.
Perhaps a third answer option for this particular question is warranted:
1) Should this movie be published (does it adhere to the rules as you understand them)?
Yes
No
I Don't Know
Memory wrote:
Tbh I prefer when people vote honestly as to how they feel about entertainment of a movie. If you are not entertained by a movie, vote that you are not entertained by the movie. Don't vote yes instead because you think the movie should go to Moons. It makes our job a lot harder when people overthink things because that results in TASes getting moved between tiers.
I completely agree that the poll question (regardless of whether it's the current wording or using my suggestions) should be answered truthfully.
I think that the ambiguity I've mentioned also plays a part in this issue. I expect that some members are fearful that some runs will not be published at all if they vote 'No' on a run.
Others thoughts regarding entertainment polling.
If we don't want to ask which tier voters think a run should be in, we could ask about degree of entertainment in the second question:
How entertaining did you find this movie?
Not at all entertaining
Very entertaining
Somewhat entertaining
Or this question could simply be used instead of the current question and remain a single-question poll:
Instead of using
Vote: Did you find this movie entertaining? (Vote after watching!)
No
Yes
Meh
use
How entertaining did you find this movie?
Not at all entertaining
Very entertaining
Somewhat entertaining
This would at least remove the ambiguity of the current question. And it would be even less change to the current site code, it'd simply be a rewording. All the calculations would remain the same, yet the results would be more valid regarding the true entertainment value of a submission. (Assuming everyone is truthful in their votes.)
I have sporadically objected to the notion that Vault be considered some kind of garbage dump where undeserving "boring" runs are shoved into if they aren't "good enough". Some/many people seem to have the attitude that's it's pretty much effectively a sign of shame if the run is relegated to Vault and doesn't get enough recognition to get the privilege to get into Moons.
In other words, the Vault tier has a really bad reputation.
It just occurred to me that the poll question probably helps reinforcing this notion, which is a shame.
Unfortunately I don't know what the solution to this is (even assuming it could be solved).
Joined: 4/17/2010
Posts: 11475
Location: Lake Chargoggagoggmanchauggagoggchaubunagungamaugg
Lobsterzelda wrote:
There are a lot of times where I see a run on the workbench which is pretty boring for me to actually watch, but is extremely optimized and a good example of TASing. However, by convention I vote yes on the workbench poll for these, even though I'm not very entertained by them.
Even if I believe that the quality of a run lends itself to publication in moons or vault tier, that doesn't necessarily mean that I found the run very entertaining. Even for movies of popular games which get accepted into moons, I still sometimes run into the issue of not wanting to vote no because I think that the movie should be published (but I don't necessarily find it entertaining).
On the other hand, there are also occasionally runs submitted to the workbench which have some unoptimal or sloppy parts that should prevent them from being published. However, when I watch the movie itself, I am entertained by it anyway. However, I feel obliged to vote no on the workbench poll, because me voting yes would imply to some that I believe that the movie is optimized enough that it should be published, when in reality, all I want to say is that I was entertained by watching the movie.
Just like the current, single question can be completely misinterpreted in so many ways regardless of what we actually use it for when judging a movie, imagine how much misinterpretation 2 or 3 questions will introduce. When assessing ideas we shouldn't limit our take on them to just their pros. We need to ponder their contras and compare ideas based on these 2 scales. Only then we will be able to tell which idea is better, clearer, harder to misuse for a large crowd.
If people don't want to post and explain their impression behind just one vote, what will make expressing those details via several anonymous polls more reliable? It will just skew the overall feedback even heavier. And quite predictably, it will encourage people to completely disregard the thread, since every tiny detail can now be expressed via these 3 polls.
Memory wrote:
Warp wrote:
If the poll and the discussion in the thread agree with each other, then there's little problem in terms of judging. However, if there seems to be some kind of disagreement between the poll and the discussion, this could induce the judge to consider the tier decision more carefully, and perhaps even ask the community some questions in the thread if necessary, and ask for more opinions and clarification.
This is pretty much how I use the polls.
That's how we all do it.
DrD2k9 wrote:
However if the majority of voters are in disagreement with the judge, the judge knows that a more thorough explanation of why the movie does/doesn't fit the rules is warranted as part of the judgement note.
If all you have is votes and no posts, there's nothing to argue about, because no posts provide no insights or hints regarding viewers' opinions. So we ask for them, and then apply our knowledge about tiers, deciding if the silent votes stand against it. But this data is never totally ignored. If we have 0 No votes and 50 Yes votes, we will not blindly send a movie to Vault.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Considering how often people misinterpret the single question, it is a great idea to have two questions instead.
I get that the judges decide if something is to be published, not the audience. But if you ask a single question, part of the audience IS going to misinterpret that as "should it be published". Far better to ask that separately (and ignore the answer) than to muddle the answer of what we're actulally asking.
In other words, the Vault tier has a really bad reputation.
It just occurred to me that the poll question probably helps reinforcing this notion, which is a shame.
Unfortunately I don't know what the solution to this is (even assuming it could be solved).
The tier system is in fact supposed to award entertainment. Some may see it from the other way around, that is, punishing lack of entertainment. It sure would be nicer to have people seeing it from the positive point of view rather than from the negative one, but I'm afraid that attempting to fix misconceptions of people would go far beyond the purpose of this site. I apologize for my nihilism.
On the other hand, while it could be true that the presence of the poll question may be reinforcing this negative point of view, in my opinion it's still better than having no poll, even though there are already other aspects that the judges use for determining tier, beside that. Because the poll has the unique advantage to allow to share an opinion anonymously, even if it's a very simplified (and often misinterpreted) opinion. Sometimes some people may be having troubles sharing their opinion openly for whatever reason, so it's still better than nothing. It's part of the judge job to understand when and how much each poll may be more or less trustworthy, by putting in use as many informations and aspects as possible, for each case independently. And in case anyone is wondering: we judges can't see the voter's identity, and we don't need to know it either, as that wouldn't help us with understanding the reasons behind a vote, and it would also uselessly damage the privacy of the users.
In any case, as I already wrote recently in another thread, if enough people consider the requirement for voting to be too loose, we could propose to raise the required post count to a higher amount, or even figuring out additional requirements.
DrD2k9 wrote:
Memory wrote:
DrD2k9 wrote:
1) Should this movie be published (does it adhere to the rules as you understand them)?
Yes
No
I'd like to note that this question would not contribute to the actual publication process. It's explicitly the judge's role to determine this, not the viewer.
I know that the judge is the ultimate decision maker regarding acceptance/publication and not the community.
I suggested this question for two reasons.
1) It will help us understand if the community as a whole generally understand the rules.
2) With either acceptance or rejection, having the results from this question provides the judge with information that may allow for formulation of more detailed judgement notes in regards to the rules. If the majority of voters are in agreement with the judge, the judgement notes may not need to be as detailed. However if the majority of voters are in disagreement with the judge, the judge knows that a more thorough explanation of why the movie does/doesn't fit the rules is warranted as part of the judgement note.
I agree that's it's useful to know about how the community feels about a judgment, but I don't think that a poll could help much; actually, it would make things even more confusing. Only speaking out your mind with a post can help the judge to understand what does need to be clarified, or sometimes even considering a revision of the verdict (we're not infallible). It happened in the past and it always worked out (people never keep quiet when they are unhappy), so I really don't see any reason to implement a poll for this. Or do you have any example of cases where things couldn't work out?
I'm simply trying to clarify a confusing and potentially (likely) misinterpreted poll question.
If all we want the poll question to answer is how people perceive the entertainment value of a submission, we need to ask HOW entertained they were....not IF they were entertained. Simply because there are degrees of perceived entertainment. The fact that we can rate post publication on the degree of entertainment on a scale of 0-10 acknowledges that this spectrum of entertainment exists.
I still believe that if the poll question were changed to a question of degree instead of a yes/no question, we would get better results.
Joined: 4/17/2010
Posts: 11475
Location: Lake Chargoggagoggmanchauggagoggchaubunagungamaugg
Better compared to what?
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
I still believe that if the poll question were changed to a question of degree instead of a yes/no question, we would get better results.
I like this idea, and I want to take it one step further. I would like to see the Poll question changed to the 'Rate this Movie' option found on the submission home page. Ratings are so often used in difficult submission judgments and in discussions over movie tiers and in award season... even though there are entirely too few on most movies to be representative.
This system would also allow people (like Lobsterzelda) to give a more honest opinion of entertainment when they can give the submission its due credit for technical quality.
Better in regards to getting the information the poll is seeking to derive in the first place, namely 'How entertaining does the community, as a whole, find this submission?'
Whether or not someone finds something entertaining is binary. Either they do or they don't. HOW entertaining someone finds something is not binary but exists on a spectrum.
The present poll question is inherently flawed.
As the current poll is presented, it asks IF someone was entertained. The available responses are YES, NO, and MEH. This IF question is a binary question yet we present 3 possible responses, two of which are binary answers while the third is an answer of degree; this in itself is illogical. The option of MEH doesn't make logical sense to even present as a possible answer to a binary question.
When someone is not entertained at all, the only appropriate response to IF they were entertained is "NO." Conversely, when someone is entertained (even in the slightest bit), the only appropriate answer to IF they were entertained is "YES." It is not possible to be both entertained and not entertained at the same time.
If every viewer who watched a particular submission found it entertaining, but only very slightly so, they would be forced to answer the current poll truthfully with a response of YES. This result would yield a 100% value for entertainment. But from a standpoint of entertainment value along a spectrum, those same watchers may not consider the run entertaining ENOUGH for moons tier. Therefore, if those voters would prefer the run be in vault and wanted that preference known; they would have to comment in the forum with some variation of
I was slightly entertained, which is why I voted yes, but I also think this shouldn't be published in moons tier.
Frankly, this is inefficient (which is ironic considering so much of what we do with the TASes themselves is efficiency driven).
The other option is to answer the poll untruthfully and vote NO when they did indeed find at least some entertainment value in the video.
It is this discrepancy that makes the current question's wording a problem. If they answer truthfully, they must then also comment to yield the publication in the correct tier. This makes answering the poll question pointless in the first place; because they could outright ignore answering the question and simply post a comment to yield the same publication result.
If they are going to lie on the poll, there's no point in asking them their opinion in the first place.
In short; because the way that the current poll is presented is itself flawed, the results derived from it are also flawed.
The absolute simplest way to remove the inherent flaw with the current question is to remove MEH as a response option. But this solution would not remove the ambiguity in how the question is answered. There remains an equal likelihood that users may vote based on IF they think the submission should be published in moons or not, as opposed to truthfully answering the question regarding IF they were entertained or not; this then returns to the problem of voters 'lying' and voting NO just to make sure the run ends up in vault. It also would not solve the issue of truthful voters still having to comment (in addition to voting) to have their perceived entertainment level understood by the judge; meaning it would still be less efficient than it could be.
A question that isn't inherently flawed.
A poll question that is worded regarding the degree of entertainment value would be less likely to have these problems. Viewers of the submission could simply vote their level of entertainment. There would be no YES/NO options for them to choose regarding if a run should be published or not. It would yield results that would provide the judge a better perspective of how the community perceives the degree of entertainment, without the voters having to also comment in the forum; thus efficiency is improved. This could also potentially save the judge from having to read as much text before making the judgement determination; further improving efficiency of the site as whole.
Ultimately, asking about the degree of entertainment directly, yields the information we're seeking with the poll in the first place....'How entertaining the community, as a whole, finds the submission.'
We use a spectrum based value of entertainment to move already published runs from one tier to another. We should be using a spectrum based (not binary) evaluation to publish into those tiers in the first place.
TL:DR When attempting to determine the degree to which something is or isn't entertaining: Logic suggests that a question asked about the degree of entertainment will yield better results than a question asked about IF something held any entertainment value at all.
As the ultimate purpose of the workbench poll is to answer how entertaining the community, as a whole, finds the submission, there is nothing that the current poll question answers that the following wouldn't also answer.
How entertaining did you find this movie?
Not at all entertaining
Very entertaining
Somewhat entertaining
Changing this question would mitigate the inherent problems and ambiguity with the current poll question.
In starting this thread, I never intended to imply that the current poll wording hasn't worked up to this point. I'm suggesting that different wording could yield a more efficient site.
Better, to me:
An inherently sound question is better than an inherently flawed question.
More efficient is better than less efficient.
I have another idea. How about the poll question being like:
Rate this run:
[ ] 0, [ ] 1, [ ] 2, [ ] 3, [ ] 4, [ ] 5
(The question could also be eg. "how much did you like this run?")
I don't think the poll requires any more explanation than that. Also, that kind of poll would be less likely to be confused with "should this be published?" (which I feel the current one resembles a bit too much.)
Yes, this would conceptually overlap with movie ratings, and some people would suggest them to be merged (and IIRC this was tried at one point and it didn't really pan out), but I really think that a submission poll should both have more nuance than just 3 options, but doesn't need as much nuance as the current rating system. A rank from 0 to 5 ought to be more than enough for people to express a justifiable opinion with enough nuance.
Joined: 4/17/2010
Posts: 11475
Location: Lake Chargoggagoggmanchauggagoggchaubunagungamaugg
It's not the first time we're having this talk. See this post for some answers: Post #421554
<feos> Moth: do you have a link explaining why this idea failed? http://tasvideos.org/forum/viewtopic.php?p=487669#487669
<feos> this looks like a FAQ thing at this point
<Mothrayas> feos: it caused huge overinflation of movie ratings, as most people can't give a proper rating during the first view/submission process
<Mothrayas> inflation as in excessively high ratings
<feos> I know, just don't have a link
<adelikat> yelah, huge over inflation
<adelikat> when people want a movie published, they are going to do 10/10 very commonly during the submisison process
<adelikat> anything less feel like less of a yes vote
[...]
<WarHippy> Moth, Feos, Adelikat: I just read your replies here. I didn't realize people skewed their votes so much in the past. I didn't even know that used to be how it was.
<Mothrayas> it was only like that for a little while
<Mothrayas> like less than a year, from 2007-2008ish
<WarHippy> ok
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
How about a question that demands an objective amount of engagement to sincerely vote yes on, such as 'were you entertained enough to want to watch this again?' Or 'were you entertained enough to recommend this to your friends?' This demands a sufficient amount of entertainment to get a yes vote out of.
Joined: 4/17/2010
Posts: 11475
Location: Lake Chargoggagoggmanchauggagoggchaubunagungamaugg
You can't fight human nature. The best thing you can do is working with it to get the least terrible result.
Poll being imperfect encourages people to actually post their thoughts. If a poll is perfect, no "real human" feedback is encouraged. But it can't be perfect because of point 1. So we'll end up having 2 things that don't work instead of just one.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
You can't fight human nature. The best thing you can do is working with it to get the least terrible result.
Poll being imperfect encourages people to actually post their thoughts. If a poll is perfect, no "real human" feedback is encouraged. But it can't be perfect because of point 1. So we'll end up having 2 things that don't work instead of just one.
emphasis mine
Regarding #1...I'm trying to improve what the "least terrible result" could be and make it less terrible! Yes, by our nature, humans are imperfect and nothing we do can be guaranteed to be perfect. I'd argue that most other times that this has been discussed, those wanting the poll question changed had the same intent as I do now. Not to make it perfect, but to improve what information we do collect. I never claimed that any of my suggestions were perfect, but I feel that I've well explained why they are better than the current usage.
Regarding #2: This thought process is flawed. Poll votes ARE "real human" feedback even when they are a lie. When someone feels they have to lie on the poll question to yield the result they desire, it shows that the poll question doesn't collect the proper information that we need when making a judgement using that very information.
Unless what you mean by "real human" feedback is 'readable language in the form of a forum post'; which, by the way, could also be imperfect or contain lies.
If the only type of feedback we're going to place a high value on is obtaining people's perspectives explained through language-based posts, we should just eliminate the poll outright. Then there's no speculation as to what the votes meant to begin with. There's no chance of someone lying on the poll. If someone likes a movie and wants their perspective known, they comment. If someone doesn't like a movie, they comment. If someone is unwilling to comment, then their voice isn't heard regrading that particular submission; and they would have no grounds to complain about the result if it differed from their opinion. If you don't want human feedback in the form of a clicked button, eliminate the button.
The other problem with #2 is that the poll question's imperfect nature doesn't inherently encourage feedback. If (by it's imperfect nature) the poll question encouraged language-based feedback in the form of posts, our forums would always be filled with comments. No submission would have fewer people commenting than it did votes, and judges would never have to ask for more feedback on a submission before they could properly discern the poll results. This also solves the potential of having 2 problems from the poll. No poll means no question about its results; leaving only the human nature expressed in forum posts as a problem for a judge to discern when making a judgment.
I realize that (some of) the site staff may feel discussing this topic again and again is a merry-go-round that goes nowhere. But the fact that the topic keeps cropping up suggests that it's a problem for which solutions need to (at least) be considered and (at most) be attempted. You've been willing to try new things before, what's the harm in trying more new ideas? If it fails as the last attempt did, we can always come back to the current method (then we should try and improve it yet again).
We claim that TASes are never perfect because there could always be future discoveries that would make room for improvement in them....why don't we treat our site the same way? The whole point of TASing is doing something better than status quo, why are we settling for status quo on something we know isn't that great instead of trying to make it better?
So let's keep inviting suggestions, and perhaps someone will have a usable idea that might acceptably solve these problems.
If anyone in the community is willing acknowledge that the poll question could be worded better, yet wouldn't want to use any of the suggestions already in this thread; what suggestions do you have to make the poll question itself better? Let's at least try something to make the poll and our site better. If it doesn't work...try again....and again...and again, if necessary.
We should never stop trying to improve our site, nor should we discourage suggestions for improvements.
Joined: 4/17/2010
Posts: 11475
Location: Lake Chargoggagoggmanchauggagoggchaubunagungamaugg
Now you make it sound like the submission poll is something particularly bad. It would be bad if its results always contradicted the posts, making things extremely confusing for any judge and the audience. But there's very little contradiction. When feedback is confusing (and doesn't match the publication ratings), it's confusing in both votes and posts.
As it's been explained a ton of times already, we neither (literally) completely disregard the votes, not do we assign absolute meaning to them. They contribute to the final decision. They also give people an easy way to express themselves. Since its easy, its value is low. But it's never been zero.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Emulator Coder, Site Developer, Site Owner, Expert player
(3573)
Joined: 11/3/2004
Posts: 4754
Location: Tennessee
<adelikat> because everytime a border line case happens, judges over-emphasize how they don't look at polls as justification
<adelikat> then people get upset
<feos> I don't think it was even caused by a borderline submission. more likely by binary thinking
<adelikat> well, I think judges always give that impression in these cases more than they should, votes absolutely matter a lot
<adelikat> you don't get into moons without the votes
<feos> can you please post this?
After lengthy and well appreciated (at least on my part) conversation with feos off-forum, I think I can sum things up this way.
The poll question as it exists makes "meh" an illogical option; because even if the movie is generally super dull or annoying, having just a single entertaining scene means that the viewer has somehow been entertained. In fact, judges as well as most viewers treat the poll as this question: "Did you find this movie generally entertaining?" Treating it this way makes all answer options sane and everything works perfectly. The problem is, that's not how the question is currently worded. So let's simply reword it.
I'm making a final suggestion in attempt to reconcile the various perspectives presented in this thread. I'm suggesting that we simply add the word "generally" into the current poll question. Making it
Did you find this movie generally entertaining?
This makes the question explicitly subjective in nature instead of only having implied subjectivity. It also makes all three current response choices completely valid and reconciles various perspectives of the current poll question.
