Joined: 11/13/2006
Posts: 2792
Location: Northern California
It's been a while since the Vault was introduced, and the recent Hydlide submission has called the system into question. A few valid criticisms were brought up regarding tiering and there was a suggestion that a discussion thread be made. I've held off on making the thread since I figured it should be posted by site staff, but I got indirect permission to post it... So, here it is.
How does everyone feel about the Vault tier or tiering in general?
What's good about it?
What's bad about it?
Should we keep Vault or remove it?
I'll post my thoughts separately after I write them up.
----
Mod edit: Added poll. -Mothrayas
Note that by "removing the Vault", what is meant is just removing the tier itself and merging all the Vault and Moon runs into a singular default category that still accepts all runs that would normally qualify for Vault.
Runs will not be removed or unpublished.
TASvideos Admin and acting Senior Judge 💙 | CohostwarmCabin wrote:
You shouldn't need a degree in computer science to get into this hobby.
Joined: 3/31/2010
Posts: 1466
Location: Not playing Puyo Tetris
On the positive, the valut does allow "bad" games to be submitted and accepted.
On the negative, vault runs get less attention then their Moon and Star brothers.
The vault will allow me to submit Gun.Smoke (auto scroller) and have it be accepted.
The House of the Dead 2/3 Returns (light gun) can be accepted. Under the old rules, no way.
When TAS does Quake 1, SDA will declare war.
The Prince doth arrive he doth please.
Joined: 11/13/2006
Posts: 2792
Location: Northern California
hegyak wrote:
On the positive, the valut does allow "bad" games to be submitted and accepted.
On the negative, vault runs get less attention then their Moon and Star brothers.
I feel like the negative there outweighs the positive: We can still accept "bad" games using the Vault guidelines without needing to put them in a lower tier that hardly gets any attention.
I'm having trouble putting all my thoughts into words, so a full rambly post will have to wait until tomorrow, but my main problem with Vault is how much negativity it implies. I know it's not intentional, but it's still there in my opinion. "Vault" as a name implies that the runs are just going to be locked up and forgot about. "Vault" as the lowest tier implies that the runs are comparatively low quality compared to Moons or Stars and thus aren't worth watching or rating. "Vault" as a tier in general imprints a certain kind of bias on the viewer, and post-publication ratings plummet because of it: People will even rate Vault runs low on technical quality, when the entire point of the tier is to showcase runs that are high technical quality but may not be entertaining. Aesthetically, Vault is the only major tier that isn't part of the Mario theme. The icon is dark and grayscale, whereas the other icons are bright and cheery. It's jarring to say the least.
If we're going to keep the tier, I'd at least recommend changing the name and icon. Something like Mushroom or Flower would be consistent with the Mario theme and have a lot more positivity attached.
TASvideos Admin and acting Senior Judge 💙 | CohostwarmCabin wrote:
You shouldn't need a degree in computer science to get into this hobby.
People will even rate Vault runs low on technical quality, when the entire point of the tier is to showcase runs that are high technical quality but may not be entertaining. Aesthetically, Vault is the only major tier that isn't part of the Mario theme. The icon is dark and grayscale, whereas the other icons are bright and cheery. It's jarring to say the least.
If we're going to keep the tier, I'd at least recommend changing the name and icon. Something like Mushroom or Flower would be consistent with the Mario theme and have a lot more positivity attached.
1. If it helps, I rate technical based on how hard it would be for me to TAS it. :P
2. Isn't dark and grayscale pretty much Bowser's castle?
But yea, it does have a negative connotation. :|
Also one thing commented on that thread was ratings. I suggest rather than using a separate page, it displays the dropboxes for rating a movie automatically for users logged on. At the same time, some indicator that said user has already rated the movie would be nice.
The tiers were partly created as a result of a desired shift in ideology of tasvideos.
For a very long time the main ideology of tasvideos was that we only publish the highest-quality tool-assisted speedruns. This did not only include technical quality, but more relevantly to this issue, entertainment quality. In other words, not only did we have (and still have) a very high standard of technical quality, we also wanted all published runs to be entertaining. Thus we rejected "boring" runs as being unfit for the goal of the site.
While that sounds good, at some point there was a slight shift in ideology: Rather than being just a collection of "most entertaining TASes", we wanted tasvideos to be like an archive for TASes of all games (that can be reasonably TASed). In other words, a principle of "every game deserves a TAS" (no matter how boring it might be). The concept "most entertaining TASes" can still be retained thanks to the rating system and also thanks to the tier system.
In my personal opinion, I approve of this change, and thus this change was for the better. I do think that all games deserve a TAS, and no game should be shunned (and I was a promoter of this idea). After all, what's "boring" is subjective, and what's boring to some people might not be boring to another. I like the idea that tasvideos is a kind of "archive" of all TASable games. Many people who played a certain game when they were young often would love to see it TASed, even if the game is "boring".
If the problem of the "vault" tier is that the name has some negative connotations (I'm not sure it has, but I understand if it can get that kind of negative fame), perhaps it could be renamed to something like "archive" or the like. (However, I doubt the problem is so severe that this is justified...)
Joined: 8/14/2009
Posts: 4083
Location: The Netherlands
I've added a poll to the topic. This is a valid discussion topic, and I see reasonable points being made for removing the Vault, but I think it's important to see whether or how many people do actually agree with it.
http://www.youtube.com/Noxxa
<dwangoAC> This is a TAS (...). Not suitable for all audiences. May cause undesirable side-effects. May contain emulator abuse. Emulator may be abusive. This product contains glitches known to the state of California to cause egg defects.
<Masterjun> I'm just a guy arranging bits in a sequence which could potentially amuse other people looking at these bits
<adelikat> In Oregon Trail, I sacrificed my own family to save time. In Star trek, I killed helpless comrades in escape pods to save time. Here, I kill my allies to save time. I think I need help.
Tier system
Like others have mentioned, implementing a Vault tier was a good idea to allow runs of less popular and/or not that stimulating games onto the site. I remember that before Vault, I was much more worried about my game choices, and I think I even aborted some TASes at the planning stage because I realized the resulting run would be slow-winded and probably not accepted. Fortunately all that's in the past now, and now the player has more freedom over their game choices and more incentive to work on the TASes they want, even if they are not what you'd deem as anything very entertaining.
I must admit that the age old fixation of things being "entertaining" on this site has seemed somewhat arbitrary to me. Entertainment is such a wildly subjective thing after all... I guess as long as entertainment is used as a criteria for different categories, there will always be semi-small issues like there currently is with the unstable line between Moon and Vault tiers.
I also think that Vault runs are handled in a slightly demeaning way on the site, kept in the sidelines and only featuring a grey, boring logo to show for it.
So, if getting rid of the Vault meant getting rid of this unstable line between Moons and Vaults; getting rid of the prejudice against Vault runs and the dull gray logo, then I would be for the idea!
Here's another stray thought. I would personally like to see a single category or maybe a tag for fastest game completions (any% TASes). This way you could generate some interesting statistics like what is the average fastest completion time of games on different consoles, or maybe between genres... This kind of thing would suit the philosophy of being an objective records site of TASes, and it would also please the statistics loving nerd in me (maybe in others too, considering how elaborate the "Statistics" page is on this site).
Rating system
>>I suggest rather than using a separate page, it displays the dropboxes for rating a movie automatically for users logged on.
Yeah, that would streamline things a bit.
As an aside note, I'd still like to be able to write a review to go along with a rating. I mean, if TASes really are art... there should be reviews.
In any tiered system, some tier will be the bottom one. Fair or not, the bottom tier will have some negative connotations to it for some people. If this is the problem, then changing the name and giving it a happier logo won't change anything. I can see use in sorting movies by entertainment, however, and of course there will be borderline cases.
That said, if I'm looking for a tas to watch, I click on the "all" option and scroll until I see a game that interests me. I don't pay particular attention to what tier a movie is in. I don't know if this is typical viewer behavior, but it's probably not unique.
So I suppose the option presented I find most agreeable is just to leave the system as-is. Either that or remove the vault but keep the idea it represents-- publish a tas for any game regardless of entertainment value, so long as the quality of the run itself is high.
The poll is confusing because it doesn't make it clear what "No, remove the Vault" means.
Does it mean "remove the tier and all TASes belonging to it", or does it mean "merge the Vault and Moon tiers"? If it means the latter, I think the poll options should reflect that more clearly.
I like the tier system which the Vault is a core part of a lot. It avoids nearly all of the problems that the pre-tier publication system had, such as "bias against unpopular games/Pokémon and Final Fantasy being accepted while other RPGs are rejected", "rejecting technically sound runs due to lack of entertainment", "non-standard categories being handled on a game-by-game basis and category limits for games like Super Metroid, and similar problems for hacks, like with Super Mario World and Super Mario Bros hacks" (solved by Moons allowing as many categories and hacks as viewers find entertaining for the most part), and maybe more. It's been a while since the old system was in place, so I could be missing some.
I agree that the Vault icon is a bit dull, though, and the Moons icon was also recycled from the old "Notable Publication" icon, which was confusing at first, and the name had some confusion with the old Moons as well. I'd be fine with a snazzier name and icon for the Vault that's similarly themed to Stars and Moons, such as:
-Asteroids
-Meteors
-Rocks
-Stones
-Space dust
-Biomass
-Mushrooms
-Flowers
-Leaves
I would personally like to see a single category or maybe a tag for fastest game completions (any% TASes).
This has been discussed before, and it even went to an almost-agreement phase where it was almost going to be done, but in the end it turned out to be too ambiguous (and even controversial).
The idea would have worked perfectly something like 5 years ago, but today it's too fuzzy. The reason for this is that there are more and more "total control" TASes (and IMO also reset-corruption TASes) that just jump to the end screen of the game rather than completing it. Does that count as "fastest completion", or should it be a TAS that actually plays the game through?
(Personally I would oppose the idea of classifying a total control run, and even runs that use resetting to corrupt the game, as "legit" world record completions, and from that point of view I'm somewhat glad that the system wasn't implemented after all. Although it's also a bit of a pity, as it would have conveyed useful information in the form of "this game can be hypothetically completed this fast".)
I don't see why all the runs can't be merged and published in the same list, for ease of access if nothing else. You could simply add the vault icon, similar to moons or stars and state what it means on the top of the page. That way people know it wasn't as "entertaining" as other movies, but they can still find the games they want relatively easy.
I don't see why all the runs can't be merged and published in the same list, for ease of access if nothing else. You could simply add the vault icon, similar to moons or stars and state what it means on the top of the page. That way people know it wasn't as "entertaining" as other movies, but they can still find the games they want relatively easy.
I agree with being less user-friendly for searching if merged, but then TAS'es would get stigmatized occasionally (ie. Dump it to vault comments), so I'm not sure what to make of it. A search based on ratings could work, but some movies are unrated/rated by too few people to make that reliable at the moment.
Joined: 8/14/2009
Posts: 4083
Location: The Netherlands
Warp wrote:
The poll is confusing because it doesn't make it clear what "No, remove the Vault" means.
Does it mean "remove the tier and all TASes belonging to it", or does it mean "merge the Vault and Moon tiers"? If it means the latter, I think the poll options should reflect that more clearly.
It effectively means "merge the tiers". Nobody ever considered removing TASes from the site, and there is no intention to change site policy of "no unpublications".
http://www.youtube.com/Noxxa
<dwangoAC> This is a TAS (...). Not suitable for all audiences. May cause undesirable side-effects. May contain emulator abuse. Emulator may be abusive. This product contains glitches known to the state of California to cause egg defects.
<Masterjun> I'm just a guy arranging bits in a sequence which could potentially amuse other people looking at these bits
<adelikat> In Oregon Trail, I sacrificed my own family to save time. In Star trek, I killed helpless comrades in escape pods to save time. Here, I kill my allies to save time. I think I need help.
Joined: 10/12/2011
Posts: 6335
Location: The land down under.
Mothrayas wrote:
Warp wrote:
The poll is confusing because it doesn't make it clear what "No, remove the Vault" means.
Does it mean "remove the tier and all TASes belonging to it", or does it mean "merge the Vault and Moon tiers"? If it means the latter, I think the poll options should reflect that more clearly.
It effectively means "merge the tiers". Nobody ever considered removing TASes from the site, and there is no intention to change site policy of "no unpublications".
Agree. Unpublications will also screw over the actual speed TASes that were meant to be in Vault to begin with and with it's playaround counterpart in Moons or Stars (Mainly Fighters).
WebNations/Sabih wrote:
+fsvgm777 never censoring anything.
Disables Comments and Ratings for the YouTube account.Something better for yourself and also others.
The good:
It provides a way to filter TASes by quality (although filtering by movie rating range would work too*). The vault did enable more games to be accepted, though that should have been allowed all along. Vault does get credit for fixing that, and that is a big improvement to this site.
The bad:
The default movie lists hide vault entries, causing them to be overlooked (and less likely to be rated*), even if they actually are high quality. In some respect, if there is a TAS of a game that is considered "bad", then it will tend to be put in Vault regardless of how well the TAS was done because it would be perceived as "boring". I would prefer to have the full list by default, and then let people filter the list however they want.
The ugly:
The vault category name/icon does make those TASes feel second rate. Also, there is also a "penalty" applied to the score calculation to the player score for vaulted TASes. I do not agree with any of the multipliers for tiers; I don't think they are necessary. Because of how the rating system has always worked (which I like), there already is a penalty for runs with a lower rating; that is two 4.5 rated TASes do not equate to one 9.0 TAS. But with the current system, the 2 TASes would be in separate tiers, and so the separation is enhanced with multipliers, which is rather unfair to TASes that happen to be put in vault, especially if they are borderline cases.
*Provided that the movie rating system is working well. Right now, that system is broken along with the submission voting, and I think that is the primary factor why there are complaints. It may be that the tiers are generally working, but it needs the voting to be working to help the judges decide.
Submission voting:
The question is "Did you find this movie ENTERTAINING?" (emphasis added) But some people have been voting as if it was "Should this movie be published?", especially when the TAS is an improvement. This makes it difficult for a judge to determine the entertainment response. I would propose that there be 2 poll questions, the ones mentioned above. That way the "yes for improvement" votes can be moved to the published poll, and then the entertainment can be voted on properly. It might be worth having 5 choices for the entertainment poll, sort of like a 1-5 rating, but with wording like: yes, probably, maybe, not really, no. I think these changes will help the judges determine the entertainment better.
Movie voting:
I honestly have no clue how I am supposed to rate movies, so I refrain from doing so. I don't see a 100 point system as useful, even 10 point is somewhat difficult. But having a description for what that rating means would be very useful, and it would help people give useful ratings, especially help avoiding voting 7 because that is the default. Also, having it be less daunting would help encourage people to vote. As for the technical rating, I can't even imagine what the numbers mean, and the votes from people make no sense either. That definitely needs text to help understand it. Perhaps another category could be useful to either add to diversify, or replace another if it were to make more sense. I can't think of any right now, but it should be considered. I agree with AKheon that an optional review would be useful with the voting. It is sort of like the comments in the submission thread that go along with the voting.
Conclusion:
I am not so sure what to do about tiers right now because I think we should focus on fixing the voting system, which I think is the main cause of the symptom, tier complaints. If the vote results were more useful, it would help make better sense of the tiers.
But in the meantime, I think we should change the name and icon of vault so it is less negative. The original system sort of had tiers (nothing, moon, star) though they were really category marker tags. In fact, I think it is better to think of tiers as just category marker tags for sorting, like they were before the change.
We should drop the score multipliers for tiers as well since that penalizes lower rated movies(or perceived lower) more than is deserved.
Adding reviews(comments) for movies would be interesting. I suppose it could be done now by creating a thread for each movie and having people post there.
I like the vault and many runs in it, I don't want it to be removed or something (also, I have a TAS published in there ^^).
Personally, I would even be okay with allowing even more games to be accepted there, like pure party games or ports that just have minor differences, so the statistics nerds like me can be happy.
But the thing I would like the most for this site, would still be ILs. If a system for publishing those could be implemented, this would basically be the best thing ever.
Current project: Gex 3 any%
Paused: Gex 64 any%
There are no N64 emulators. Just SM64 emulators with hacky support for all the other games.
In some respect, if there is a TAS of a game that is considered "bad", then it will tend to be put in Vault regardless of how well the TAS was done
What does that even mean? If it was done poorly, then there would be clear improvements and it'd generally be rejected. If you mean perceived effort, there are so many problems with that and the only benefit seems to be self-esteem hugs.
We need an appeal to a higher court! :D
Actually, I think... First, we need to understand goals. Second, we need to locate issue, and find out where its stink comes from.
Goals: as far as I understand, main goal is entertainment. Second goal is history. Star/Moon means that it's high entertainment. Vault category supposed to be as history of top records for particular game.
Issue: I don't know exactly what issue you see, but I see this:
If run going in Vault, it marked as "not entertainment", literally: "don't watch it, it's only waste of time", and as result: less votes about ratings, and so on. More than that, vault runs are hidden in list by default.
With runs in Moons/Stars category, completely different situation, they marked "this run cool, watch it", and result: more votes about ratings, and so on. And they are visible in list by default.
Where is issue here? Good runs forces its ratings to be good, and not entertainment runs forces its ratings to be bad. In other words, it's not good that rating is affected by tier.
Why I didn't post that? It actually doesn't represent actual situation. There isn't enough votes in ratings to make statistical conclusions. But now, I'm posting that, to reveal my way of thinking.
We need more votes to be able to work with staticsics. It would be nice to force somehow people to vote. For example, me, I didn't ever vote for publicated movies, actually, until some moment, I didn't even know that it's possible. It's hidden somewhere in list of links, and it's not noticeable enough. Vote input must SCREAM! Vote me! Do it! Now! But instead of it, you have to find yourself where it is... then go on another page, with vote form. Also, why it's showing only entertainment rating by default? Why not show both?
In Workbench situation completely different: you have encode, and you have poll. All in one page. May be that's why I vote so much in Workbench.
For me, vault is like "black mark". All going bad after you got it. :D
I can't vote in this poll, because there is no such option.
Idea is simple: why I, myself, don't have tier: vault? Some people has rewards like "tas of 2013" and so on, why I don't have reward "He don't have reward, he looser."
For me, moons, and star - must be like rewards, like tags. And all runs that is not moon or star, must be clean, without tier. It does not mean that it is bad, it means that it's not cool enough to be Moons.
It's my opinion what needs to be "at least".
Now, what I would like to see in best case:
1) No vault: as I said, just clean, without tiers.
2) Auto moons/star: based on rating. It must be not tier in my view of things. It must be like tag. Moons meaning will be "it's more than X by rating, but less than Y by rating". Star: "It's more than Y by rating" where Y > than X. And searching for "stars" will be equal to looking for publications with rating above some value.
3) Tag "New": means that it was publicated recently. Period can be chosen as month or two, or week, I don't know, depends on voting speed. Then, after selected period, tag would be selected automaticaly (moon or star).
In this system judges duty would be to check technical part, accept for publication, select branch, in other words: manage submissions.
But it won't work unless you have enough votes.
Where is my "looser tier"? :D
In some respect, if there is a TAS of a game that is considered "bad", then it will tend to be put in Vault regardless of how well the TAS was done
What does that even mean? If it was done poorly, then there would be clear improvements and it'd generally be rejected. If you mean perceived effort, there are so many problems with that and the only benefit seems to be self-esteem hugs.
It's because voting on entertainment is subjective and biased, which can't be helped. The bias is see is people voting on the game rather than the TAS. This goes for vault as well as stars. Of course the tier placement can bias the voting as well.
r57shell wrote:
1) No vault: as I said, just clean, without tiers.
Now that I think about it, we can use the old system's methodology. Change the name 'vault' to something along the lines of 'normal' and change the icon to nothing. That way it can still be a tag for filtering, but will appear as if there is no tag. I would also say that moons could go back to to being more like a 'notable TASes' tag, with a somewhat higher rating requirement.
I've read only a few posts, but I have an impression we're mixing everything possible in one monstrous issue.
First of all, the main problems always was and still is with borderline cases. Those are exactly what got people complain about rejection through pages. Those are exactly what still causes rejection if the goal is unvaultable. Those are what makes the Vault look bad - runs that don't look deadly bad get placed there.
Then, we still are uncertain about the criteria. But what else do we have besides judge's notion, crowd's voting, posts and ratings? Nothing. So we must take it into account, just try to do it wise.
I suggest to add an undefined tier. It's like postponing the tier judgment until it's more obvious. And if it's not, not make it at all!
Things suddenly look resolved: definitely bad runs still go to where only few people care about them, definitely good ones go to where all see them. Undefined-tier runs go to the Moons area, but have some icon on them shouting NEW or RATE ME. RATE ME makes more sense, because it can still have little to no ratings after a year, when it's in no way NEW anymore. Also, putting them with Moons gives them more chances to be noticed and rated.
The only 2 problems that are possible here are:
- ambiguity of what is ambiguous (how do we know it needs to go to the undefined tier?) and
- should that tier be used to accept unvaultable runs of uncertain entertainment value (pretending they will be Moons, to justify acceptance)?
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Why should the vault tier need to feel like just a dump to throw the "garbage TASes" into? That's a rather unfair characterization of the tier. IMO vault is the default tier for every TAS. Getting to the moon tier is like a recognition for high entertainment value or other notable characteristic, but that doesn't need to mean that vaulted runs are just garbage.
(The star tier is special. It's basically moon-tiered runs hand-picked for a showcase of TASing. They aren't necessarily the best of the best, even though some of them might be. In fact, I would even say that a run that would otherwise be vaulted, could well be hand-picked for a star, if there's something special about it that makes it worth showcasing.)
I do understand, however, if some people might feel that their TAS, which they spent so much effort doing, kind of gets "shunned" by not getting so much recognition as to get the privileged moon tier label.
I suppose a relevant question would be: What does the two-tier system achieve that the ratings don't?
And the answer to that question is (according to my understanding): We want to give all games a fair chance to get a TAS, but we don't want arbitrary goals for all of them (because else the site would be inundated). The "all games deserve a TAS" principle is reserved for completion time only (with any% and 100% being the two possible goals). Only if an alternative goal is entertaining enough for a particular game, will it get a chance to be published as well (and this is what the moon tier represents).
Thus the even more relevant question is: If we merge the vault and moon tiers, would this mean that all games can be TASed with all possible goals, even non-speed-oriented ones?
Edit: Which gives me the idea: Change the tiers to "the any%/100% tier" and "the everything else tier". (I don't have good ideas for the actual names.)
I'm for keeping the idea of the Vault Tier on the grounds that any game I would Solo TAS falls into it by default (Nail 'n Scale just isn't entertaining enough on its own to those unfamiliar with it, and Rugrats Time Travelers has too many anti-speed features built into it for a TAS to look significantly different from any extant Realtime Speedrun). With that said, I like the replacement name "Coin" for the Vault Tier - the line of thinking is like this:
Coin: Common, but good to find.
Moon: Rare, in SMW one of these is worth 300 coins.
Star: Very rare - you can't recreate the effect with coins or even a Moon, but only by finding a Star.
I guess this kinda goes along with r57shell said, though I would point out that Gruefood is the loser tier when not deliberately sought for (such as with the April Fools' submissions).
