Post subject: Are voters biased.
Editor
Joined: 11/3/2013
Posts: 506
A thought occurred to me while I was thinking about these two submissions: http://tasvideos.org/4378S.html http://tasvideos.org/4388S.html The first of these TASes is done by a newcomer to the site, the second by a very experienced and well-respected TASer. And they have had very different receptions. For my part, I immediately watched Jigwally's movie and wrote on the discussion page that I thought it looked sloppy. A few days later I saw TASeditor had produced a faster version, and I immediately thought to myself "Oh, it's TASeditor, that'll be optimised". Without even watching it. I've now watched them back to back and I honestly struggle to differentiate the runs in real time. I realised that I just perceived TASeditor's run as more accurate because of the name attached to it. Even more surprising are the results to the "did you enjoy this movie" question. For jigwally's movie the reception was a strong lack of enjoyment, whereas for TASeditor's movie the reaction was pretty much split down the middle, so more people found TASeditor's movie entertaining despite the fact that it looks very similar and some of the people watching it had presumably seen pretty much the same movie a few days before (and the fact that the game is still horrible for TASing entertainment value). There is clearly some bias going on here due to the reputation of the TASer. This is a problem for future movies as it may quash high-quality runs by newcomers and let mistakes from skilled TASers slip through the net. How should we counter this bias?
Editor, Skilled player (1536)
Joined: 7/9/2010
Posts: 1319
It is possible to get suboptimal movies published as a newbie. I did this with [1814] SNES Super Adventure Island by TASeditor in 16:56.87, but the days when this worked are over. The game choice affects the acception of a TAS. A suboptimal TAS of a simple game has a lower rate getting accepted, than a suboptimal TAS of a complicated game. Mostly because errors are more obvious in simple games than in complex ones. Well Super Adventuer Island is also simple, so it is the judges fault.
Favorite animal: STOCK Gt(ROSA)26Sortm1.1(rtTA,EGFP)Nagy Grm7Tg(SMN2)89Ahmb Smn1tm1Msd Tg(SMN2*delta7)4299Ahmb Tg(tetO-SMN2,-luc)#aAhmb/J YouTube Twitch
Samsara
She/They
Senior Judge, Site Admin, Expert player (2238)
Joined: 11/13/2006
Posts: 2822
Location: Northern California
Well... Yes, voters are biased. It's nearly impossible for them not to be. Bias is naturally wired into the human brain: We're more trusting of familiar things and we tend to stray away from unfamiliar things. If we enjoy a game, we're more likely to enjoy any content surrounding it. If we dislike a person, we're more likely to avoid any content they produce. We're hardwired to make comparisons, whether it be comparing a new TASer to an experienced TASer, an old run to a new run, games to other games, runs to other runs, et cetera. There's always a "best" and "worst" choice, and it's different from person to person, and unless the choice is something like "chocolate ice cream" or "nails being slowly driven into your flesh", people are going to be slightly biased towards one choice over the other based on what they like and dislike. It's not something we can avoid in the slightest unless all of humankind were turned completely emotionless and apathetic, having the exact same reaction to literally everything in the world. We can compare this new run from TASeditor to other runs that he's done and it makes sense, which makes us happy as human beings. We've seen TASeditor make good and optimized runs before, so we're immediately biased into thinking that every run he makes is going to be good and optimized. We haven't seen a previous Jigwally run, so the only way we can make a comparison is to compare it to other first-time TASers submitting their first runs. For the most part, first runs are highly suboptimal (and painful to watch in retrospect) and most of them aren't accepted, so our immediate bias to any first run is that it's going to be unoptimized. Even if we get past being biased towards or against certain people, there are still a lot more bias obstacles to overcome. Honestly, the system works well because multiple people communicating means that everyone can keep everyone else's biases in check, and the ultimate decision relies on the judge who makes a decision that isn't just their thoughts, but everyone's thoughts.
TASvideos Admin and acting Senior Judge 💙 Currently unable to dedicate a lot of time to the site, taking care of family. Now infrequently posting on Bluesky
warmCabin wrote:
You shouldn't need a degree in computer science to get into this hobby.
Player (146)
Joined: 7/16/2009
Posts: 686
I think it's in part due to the amount of people that still vote on publishableness instead of entertainment: TASeditors' version is more optimized and as such more publishable, resulting in more yes votes.
Patashu
He/Him
Joined: 10/2/2005
Posts: 4043
For the record, I voted 'meh' on both Elmo TAS submissions.
My Chiptune music, made in Famitracker: http://soundcloud.com/patashu My twitch. I stream mostly shmups & rhythm games http://twitch.tv/patashu My youtube, again shmups and rhythm games and misc stuff: http://youtube.com/user/patashu
Eszik
He/Him
Joined: 2/9/2014
Posts: 163
From TASEditor run thread:
Bamahut wrote:
I will give it a yes for the vault
So I agree with Scepheo‚ many people (including me) will give a yes to optimized movies even if the aren't really entertaining
I problably made mistakes, sorry for my bad English, I'm French :v
Editor, Expert player (2479)
Joined: 4/8/2005
Posts: 1573
Location: Gone for a year, just for varietyyyyyyyyy!!
I looked at the list of cognitive biases from Wikipedia to see if there is something that is relevant to the TASVideos voting system or to the kind of situation thatguy describes. These are what I found at the first glance: Anchoring or focalism The tendency to rely too heavily, or "anchor," on one trait or piece of information when making decisions (usually the first piece of information that we acquire on that subject) Yes. This can happen when regular voters "anchor" their decision to the first piece of information they get about the TAS, which by the way happens to be the name of the author. Availability cascade A self-reinforcing process in which a collective belief gains more and more plausibility through its increasing repetition in public discourse (or "repeat something long enough and it will become true"). I'm not sure how often "availability cascade" happens in the submission discussions, but this looks like something that might happen occasionally. Bandwagon effect The tendency to do (or believe) things because many other people do (or believe) the same. Related to groupthink and herd behavior. This is a strong effect in human behavior and probably happens often here too. In retrospect, I remember some occasions (not necessarily related to submissions) where I jumped to the wagon. Cheerleader effect The tendency for people to appear more attractive in a group than in isolation. This might actually happen with submissions that have more than one author. Confirmation bias The tendency to search for, interpret, focus on and remember information in a way that confirms one's preconceptions. Yes, this will definitely happen often. It is a common human thing to do. Like, if you believe that the submissions of newbies are generally unoptimized, you will pay more attention to the mistakes. I can imagine myself thinking "that trick looks unoptimized" vs. "that trick looks unoptimized, but since the TASer is experienced, he must know what he is doing". Conservatism or regressive bias A certain state of mind wherein high values and high likelihoods are overestimated while low values and low likelihoods are underestimated. Yes, this looks like something that could happen here sometimes. I might overestimate the probability that the TAS is close to optimal when it is made by an expert TASer and underestimate the probability when it was done by a newbie. Conservatism (Bayesian) The tendency to revise one's belief insufficiently when presented with new evidence. I can't think of an example, but this probably happens here sometimes. Contrast effect The enhancement or reduction of a certain perception's stimuli when compared with a recently observed, contrasting object. Yes, this might happen when somebody has just seen a very good TAS and then watches an average TAS and perceives the latter as worse than it actually is. Curse of knowledge When better-informed people find it extremely difficult to think about problems from the perspective of lesser-informed people. This probably happens here, but I can't think of an example. Distinction bias The tendency to view two options as more dissimilar when evaluating them simultaneously than when evaluating them separately. Yes, comparing two TASes to each other vs. watching them separately. Essentialism Categorizing people and things according to their essential nature, in spite of variations. Definitely happens: "A submission by a newbie. Must be unpotimized then." Experimenter's or expectation bias The tendency for experimenters to believe, certify, and publish data that agree with their expectations for the outcome of an experiment, and to disbelieve, discard, or downgrade the corresponding weightings for data that appear to conflict with those expectations. Seems like something that could happen when watching TASes with expectations of it being good/bad. Focusing effect The tendency to place too much importance on one aspect of an event. Yes. This can happen when we focus too much on who did the TAS. Framing effect Drawing different conclusions from the same information, depending on how or by whom that information is presented. Yes, I think any submission whether made by a newbie or not will receive better feedback, if the the run is submitted by somebody with good reputation. Also, good submission text and text formatting will have a positive impact on the audience. Hot-hand fallacy The "hot-hand fallacy" (also known as the "hot hand phenomenon" or "hot hand") is the fallacious belief that a person who has experienced success has a greater chance of further success in additional attempts. Not sure about this. Maybe you can fall to this fallacy, when you overestimate the probability that an expert taser will make a good TAS in the future? Illusion of control The tendency to overestimate one's degree of influence over external events. This happens every time someone votes. (Just kidding.) Selective perception The tendency for expectations to affect perception. Happens all the time? Stereotyping Expecting a member of a group to have certain characteristics without having actual information about that individual. "A specimen from the noob group? Definitely didn't use frame advance."
Chamale
He/Him
Player (182)
Joined: 10/20/2006
Posts: 1355
Location: Canada
I'd say the voters are biased, but probably not so biased that it prevents worthwhile movies from getting published. One of the biggest biases is probably the fact that movies by well-known TASers seem to get more views and more votes after being submitted. We could try hiding the names of authors for a day or two after a movie is submitted, but this would cause serious logistical issues, wouldn't work for prominent projects, and I'm not convinced that this problem is in serious need of solving.
Joined: 7/2/2007
Posts: 3960
You know what I look for when I'm pre-judging a submission? The submission text. If it's mostly empty then I'll subconsciously assume that the author doesn't know what they're doing. If, on the other hand, it contains details about how it's an improvement over whatever prior submissions/WIPs/unsubmitted TASes/etc., talks about techniques it uses, and so on, then I'll be far more favorably-inclined to it. We've had runs submitted by complete newcomers who've had that kind of detailed breakdown in the submission text, and those runs have also generally been of high quality. Meanwhile, the majority of runs that have basically-empty submission texts are also clearly not optimized when you watch them.
Pyrel - an open-source rewrite of the Angband roguelike game in Python.
ALAKTORN
He/Him
Former player
Joined: 10/19/2009
Posts: 2527
Location: Italy
I just think the answers to the question are wrong, they shouldn’t be “yes, no, meh”, they should be “I was entertained, I wasn’t entertained, I was indifferent” or something like that, you won’t have people spamming YES just because they think it should be published anymore. edit: although that’s kind of a different issue, but what the OP says can’t be helped.