Joined: 3/9/2004
Posts: 4588
Location: In his lab studying psychology to find new ways to torture TASers and forumers
After the creation of ytu, the next logical step is to allow our encoders to sideload through the site to our official YouTube channel.
Documentation of this feature and more details is now available.
Note: Right now this feature is currently being tested, and some bugs may yet be discovered. Also, the part which ties files you uploaded to ytu, to sideload them to YouTube is not in place yet, but will be once we deem this working.
If you're an encoder, or better, please help with testing this feature.
Warning: Opinions expressed by Nach or others in this post do not necessarily reflect the views, opinions, or position of Nach himself on the matter(s) being discussed therein.
Explain the guidelines of usage the uploader with submissions/publications. For example: "Upload only the published movies encodes, for submissions you can use your own channel".
Also for status.html it would help to have a field "High Definition" with yes/no. Or it can use just "no" (or N/A) if HD is unavailable, but report the highest definition available if it is HD.
EDIT: And it would be insane to create "Current" and "Obsoleted" playlists and toss the encodes between them when necessary. Having playlists by platform is also awesome.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Joined: 3/9/2004
Posts: 4588
Location: In his lab studying psychology to find new ways to torture TASers and forumers
feos wrote:
Explain the guidelines of usage the uploader with submissions/publications. For example: "Upload only the published movies encodes, for submissions you can use your own channel".
Good idea, that should be added somewhere.
feos wrote:
Also for status.html it would help to have a field "High Definition" with yes/no. Or it can use just "no" (or N/A) if HD is unavailable, but report the highest definition available if it is HD.
This is beyond the scope of the project. We have other pages which give that information. Right now this feature is only for sideloading to our channel (or download queued encodes).
If there is interest down the line, perhaps we'll enlarge the scope.
feos wrote:
EDIT: And it would be insane to create "Current" and "Obsoleted" playlists and toss the encodes between them when necessary. Having playlists by platform is also awesome.
Also beyond the scope of this project. Although I think YouTube does have a playlist API that perhaps we'll look into.
Warning: Opinions expressed by Nach or others in this post do not necessarily reflect the views, opinions, or position of Nach himself on the matter(s) being discussed therein.
The resolution pulled from YT is reported by some natt's page, but it has ALL published encodes and all resolutions table. But for reencoders, only the availability of HD for current runs is necessary, but it's quite hard to get that idea from natt's page.
Also I'm interested what to do if the ENCODE gets obsoleted (if tasblend one is replaced with deblink or so). Can existing encodes be deleted from the channel, or marked as non-public?
The playlist thing is necessary for YT viewers. If they look for movies right on the site, they don't care what channel hosts them. But if we start having all encodes at one place, we must make it easy to navigate through these (currently) 600+ (!) videos.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Joined: 3/9/2004
Posts: 4588
Location: In his lab studying psychology to find new ways to torture TASers and forumers
feos wrote:
The resolution pulled from YT is reported by some natt's page, but it has ALL published encodes and all resolutions table. But for reencoders, only the availability of HD for current runs is necessary, but it's quite hard to get that idea from natt's page.
YouTube offers no public API to get that info, and we have not created one yet. Further, the current design of the encoder site is that once video is siteloaded to YouTube, it's never looked at again.
feos wrote:
Also I'm interested what to do if the ENCODE gets obsoleted (if tasblend one is replaced with deblink or so). Can existing encodes be deleted from the channel, or marked as non-public?
In theory yes.
We don't have a delete API up yet, but we do have access to changing its view mode.
Edit:
To elaborate on the last point. We don't want to blindly give people access to delete encodes. Otherwise a disgruntled staff member can wreak havoc.
Right now we're reserving such features to admin level types of staff members, who have direct access to the channel itself.
Warning: Opinions expressed by Nach or others in this post do not necessarily reflect the views, opinions, or position of Nach himself on the matter(s) being discussed therein.
The resolution pulled from YT is reported by some natt's page, but it has ALL published encodes and all resolutions table. But for reencoders, only the availability of HD for current runs is necessary, but it's quite hard to get that idea from natt's page.
YouTube offers no public API to get that info, and we have not created one yet. Further, the current design of the encoder site is that once video is siteloaded to YouTube, it's never looked at again.
Hmmm... then how did natt get it?
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Joined: 3/9/2004
Posts: 4588
Location: In his lab studying psychology to find new ways to torture TASers and forumers
feos wrote:
Nach wrote:
feos wrote:
The resolution pulled from YT is reported by some natt's page, but it has ALL published encodes and all resolutions table. But for reencoders, only the availability of HD for current runs is necessary, but it's quite hard to get that idea from natt's page.
YouTube offers no public API to get that info, and we have not created one yet. Further, the current design of the encoder site is that once video is siteloaded to YouTube, it's never looked at again.
Hmmm... then how did natt get it?
He created his own hack.
Warning: Opinions expressed by Nach or others in this post do not necessarily reflect the views, opinions, or position of Nach himself on the matter(s) being discussed therein.
There's no officially blessed api for that, so I just get (http request) video pages and then scrape them for info. Very unreliable, nonportable (might even screw up if I was served by a different node of youtube's CDN?), and can always break tomorrow.
Joined: 3/9/2004
Posts: 4588
Location: In his lab studying psychology to find new ways to torture TASers and forumers
The sideloading part is now done. We are live!
However, we probably need to tweak a few things, and create automation of movie entry updates and similar.
Warning: Opinions expressed by Nach or others in this post do not necessarily reflect the views, opinions, or position of Nach himself on the matter(s) being discussed therein.
Joined: 3/9/2004
Posts: 4588
Location: In his lab studying psychology to find new ways to torture TASers and forumers
A command line client, as well as upload API is now on the wiki page.
Warning: Opinions expressed by Nach or others in this post do not necessarily reflect the views, opinions, or position of Nach himself on the matter(s) being discussed therein.
Here are the steps I followed.
1. Using a modern browser
2. Install the site certificate
3. Goto site
4. Generate password
5. Goto Upload
6. Title = movie number (ex. 1234M)
7. Description = movie page (ex. http://tasvideos.org/1234M)
8. Tags = display tags (ex. yt:stretch=4:3)
9. Browse for file
10. UPLOAD
11. Wait for youtube location to show up at [url=https://encoders.tasvideos.org/status.html
]Recent activity status[/url]
P.S. files are aggregated on the TAS server and sideloaded at 45 minutes past each hour.
Joined: 3/9/2004
Posts: 4588
Location: In his lab studying psychology to find new ways to torture TASers and forumers
12. Check YouTube that the encode is good.
13. Copy link to movie page.
Warning: Opinions expressed by Nach or others in this post do not necessarily reflect the views, opinions, or position of Nach himself on the matter(s) being discussed therein.
Joined: 3/9/2004
Posts: 4588
Location: In his lab studying psychology to find new ways to torture TASers and forumers
For those of you following this topic:
The limit on uploads is 3.5GB as is stated in the documentation.
TVCMan has no limit on what it can support (unless you consider 2^63-1 a limit). The site backend, and the browser based uploader both have a 3.5GB limit hard coded in.
The browser based uploader had a bug in its MD5 calculations for files >=2GB in size, effectively making the form not work for anything in that range. Thankfully, thanks to help from Ilari, this has been fixed. I tested with a 3GB file, and it worked, so things should be fine now.
Is the 3.5GB limit high enough? Should I raise it higher? I'd like to hear the opinion of various encoders on this.
Warning: Opinions expressed by Nach or others in this post do not necessarily reflect the views, opinions, or position of Nach himself on the matter(s) being discussed therein.
For those of you following this topic:
The limit on uploads is 3.5GB as is stated in the documentation.
TVCMan has no limit on what it can support (unless you consider 2^63-1 a limit). The site backend, and the browser based uploader both have a 3.5GB limit hard coded in.
The browser based uploader had a bug in its MD5 calculations for files >=2GB in size, effectively making the form not work for anything in that range. Thankfully, thanks to help from Ilari, this has been fixed. I tested with a 3GB file, and it worked, so things should be fine now.
Is the 3.5GB limit high enough? Should I raise it higher? I'd like to hear the opinion of various encoders on this.
Personally, I think it's way too low. I remember uploading a DKC2 TAS that was slightly over 12GB in size.
What would it cost to raise such limit?
Emulator Coder, Site Developer, Site Owner, Expert player
(3584)
Joined: 11/3/2004
Posts: 4738
Location: Tennessee
The intent of the limit is allow most every encode through. I think the target is about 90%. The remaining 10% can be handled with direct TVC access (which I don't mind giving out to trusted and experienced publishers).
The lower the limit the better to avoid potential disk space problems, and allow us to scale up the umber of encodes using it.
Unfortunately disk space is not an issue that can be fixed with "cost".
So what % of your encodes would do you think are greater than 3.5 gb?
As technical background:
* The disk space reserved for storing the video files is 48GB total.
* Each file is deleted after sideloading completes.
* The maximum delay from completion of upload to start of sideload is 1 hour.
* The sideload (if everything goes properly) usually proceeds at multiple megabytes per second.
Scaling by 8 the filesize decreases extremely comparing to x6 and stuff. Someday I'll see the size for DKC of 1.5 hour length. It shall not overcome the 3 GB border.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Joined: 3/9/2004
Posts: 4588
Location: In his lab studying psychology to find new ways to torture TASers and forumers
What's new:
Fixed some minor bugs in the backend.
Fixed some minor bugs in the browser based uploader.
Significantly improved hashing speed in browser based uploader.
Fixed progress bar and progress percentage with huge files in browser based uploader.
Removed upload limit from browser based uploader.
Note: You may want to force refresh the page to ensure you get the new JavaScipt.
Note: New JavaScript was tested with files as large as 32GB, so issues in that area should now be gone.
The backend still limits files to 3.5GB, but that will be changing in the future.
In general due to the slow hashing speed in JavaScript, I recommend the command line uploader - TVCMan, especially for huge files. However with the new speed improvements, The JavaScript isn't as horrible as it used to be.
Warning: Opinions expressed by Nach or others in this post do not necessarily reflect the views, opinions, or position of Nach himself on the matter(s) being discussed therein.
New storage page explaining storage status and current working limits.
Warning: Opinions expressed by Nach or others in this post do not necessarily reflect the views, opinions, or position of Nach himself on the matter(s) being discussed therein.
Joined: 3/9/2004
Posts: 4588
Location: In his lab studying psychology to find new ways to torture TASers and forumers
I just uploaded a new version of TVCMan.
This fix reporting file size to site with files >4GB in size.
Note: This issue did not affect ytu.
Warning: Opinions expressed by Nach or others in this post do not necessarily reflect the views, opinions, or position of Nach himself on the matter(s) being discussed therein.
4826933103/4826933103 [*************************] 100% 264.533 KB/s 0:00:00:00
Runtime error: Upload failed, 400
----------
Authentication-Info: cnonce="MDMxMDMx", nc="00000003", qop="auth", rspauth="398a
55c5dc2b36ab23d7f7a6b109652c"
Connection: close
Content-Type: text/html; charset=UTF-8
Date: Sat, 02 Feb 2013 20:38:01 GMT
Server: Apache/2.2.16 (Debian)
Transfer-Encoding: chunked
Vary: Accept-Encoding
----------
Requests are expected to be made via PUT.
An X-File-Md5 header is required.
X-File-Md5 should consist of 32 hex characters of the entire file's MD5 digest.
A Content-Range header is required.
Content-Range is in the form "bytes <begin>-<end>/<total>".
Content-Length must also match the amount of bytes specified by the Content-Rang
e.
Example 1:
Content-Range: bytes 0-9/10
Content-Length: 10
Example 2:
Content-Range: bytes 150-199/2300
Content-Length: 50
For status check, the following Content-Range is also permitted:
Content-Range: bytes */*
Content-Length: 0
Before upload is allowed to begin, a status check MUST occur.
>4GB+ file. TVCman hits a site bug with those.
Edit: You also seem to use older version that can't even register such upload correctly.
Edit 2: Also there looks to be some sort of bug that the site didn't reject the transfer with size mismatch before the real transfer started.
Joined: 3/9/2004
Posts: 4588
Location: In his lab studying psychology to find new ways to torture TASers and forumers
I tracked down the bug an found it in PHP. It would be nice if they fixed the bug: https://bugs.php.net/bug.php?id=64187
You can go there and (without an account) vote that the bug is important to you, and that you want it fixed.
Warning: Opinions expressed by Nach or others in this post do not necessarily reflect the views, opinions, or position of Nach himself on the matter(s) being discussed therein.
Joined: 3/9/2004
Posts: 4588
Location: In his lab studying psychology to find new ways to torture TASers and forumers
Okay, I fixed our PHP on the server, uploading in chunks >=4GB should now be possible.
Warning: Opinions expressed by Nach or others in this post do not necessarily reflect the views, opinions, or position of Nach himself on the matter(s) being discussed therein.
