Post subject: Setting up an encoding workflow
Experienced player (764)
Joined: 6/17/2008
Posts: 146
All examples herein are asspulled for illustrative purposes. Any resemblance to actual workflows is purely accidental. TODO: make this suck less It's come to my attention that a lot of people aren't encoding as optimally as they could be. In this post, I seek to remedy this. Let's look at at an example scenario. Our heroine, Elly the Encoder, is encoding 3523S. She does the following: 0. Downloads and unzips movie file and locates the ROM and dumps with snes9x 1.43 and checks the dump for A/V desyncs and other issues. 1. Edits in the dump filenames to a template AviSynth script that resembles BrandonE's Hybrid Encoding Script. 2. Edits subtitles in the AviSynth script and previews the result in VirtualDub and possibly adjusts them afterward. 3. Clicks a batch for a first pass. 4. Edits the AviSynth script for a second pass and clicks another batch 5. Loads up VirtualDub and extracts the audio to a manually specified file 6. Clicks yet another batch to encode the audio 7. Clicks another batch to mux the primary 8. Clicks a batch to mux the 10bit444 9. Clicks a batch to mux the YT encode. 10. Checks the resulting files; if the quality or size is unsatisfactory, she goes back to step 4. 11. Names the files according to site guidelines 12. Uploads the files to archive.org and YT This is quite a bit of work. How much of this actually requires her input? How could we optimize the process? Unzipping the movie file and placing it to a more appropriate directory could be automated; for some consoles, the ROM can also be located automatically. Some emulators feature reliable automated dumping. The base subtitle text can often be automatically derived from the submisson. Subtitle() subtitles look awful and can take quite a bit of effort to time and place, especially if the game has complex action right at the beginning. Previewing them with VirtualDub is also rather slow. We can quickly see that batches are clicked multiple times. That's not too complex; we can certainly automate this. Editing filenames manually is quite time-consuming; perhaps we could dump to a specific directory and automatically locate the files after inputting only the directory. Firing up VirtualDub at this point is inconvenient and unnecessary. We can replace it with avs2pipemod, which can be called without user input. Naming the files is a bother. We can nearly always automatically derive the file name from the submission. FADE TO BLACK And after working for a while, Elly would now do the following when encoding 3523S: 0. Elly downloads the submission's movie archive; the file is automatically extracted and the relevant ROM is located and copied to an easily accessible location. 1. Elly dumps the run to a specific location and checks them for problems. 2. The base subtitles are automatically filled in; Elly simply times and styles them. 3. Elly clicks a single batch file - a few hours later, she gets back to the keyboard and finds that her computer has left her a present: encodes that she simply needs to check, rename and upload. If she finds the quality unsatisfactory, she adjusts the CRF, clicks the batch again and ends up with the another pair of relevant encodes a few hours later. tl;dr: optimize, optimize, optimize; teach a lady to encode and she can spend more time TASing and less time encoding
Editor, Emulator Coder, Site Developer
Joined: 5/11/2011
Posts: 1108
Location: Murka
For comparison, here's my workflow for now. It has strengths and weaknesses: 1. Download movie file, dump. 2. Create avisynth script (usually this means just copy+pasting the old one). 3. Manually enter subtitle times, check a/v sync, trim length. 4. Prepare a slightly different HD script which includes HD resize, and 30fps conversion. 5. Hit the batch button. Automatically does: a. dedup first pass b. all audio encodes directly from the avs, including sample undelay for aac c. resize of _512kb encode, and AR labelling of other encodes d. all video encodes at specified CRF e. all muxing 6. Examine resulting encodes. If I don't like one, change CRF and repeat step 5 (only the needed parts will be redone).
Post subject: My workflow
Joined: 3/31/2010
Posts: 84
Location: Middle of Nowhere
Here's the way I do things: 1.) Get Movie File (Unzip if Emu doesn't support play movie straight from zip.), Dump Movie 2.) Splice segments and trim with Vdub, resave spliced file as source.avi (repeat for other games if multigame) 3.) Edit script for subtitles + timings, set other options if needed, check results 4.) Run encoding batch, choosing what encodes I want to produce in-batch 5.) Check encodes for quality, upload
creaothceann
He/Him
Editor
Joined: 4/7/2005
Posts: 1874
Location: Germany
Some notes: - always use the same filenames (00a.avi+00b.avi & 00.avs; 01.avi & 01.avs; etc.), only change the directory name - use AvsPmod and a command-line encoder instead of VirtualDub(Mod) - extracting the audio can be done with avs2wav or mencoder
Player (66)
Joined: 4/21/2011
Posts: 232
You say optimize, but what you are describing is automation. You can automate the framerate reduction of youtube encodes by using TASblend on everything, but to optimize you would need to look at the frames and apply different methods to different parts. The encoding step is where you are likely to see the biggest gains. Ideally you'd be able to drop a .avs on a .bat and get out your presents. A fully automatic hybrid encode script, hum. FerretFaucet made one, not sure where to find it. There are 4 files that need to be made. 1) high bitdepth 2) primary 3) _512kb 4) YouTube A. resolution 1+2 native resolution, flag SAR 3 reduce height to 240, resize to square pixels (mod 2) 4 8x8 pointupsize B. framerate 1+2 ExactDedup 3 full framerate 4 half fraterate by TASblend (this part probably shouldn't be automated) C. bitrate 1+2 --crf 20 3 cap video at 512, sound at 64 4 lossless D. codec 1+2 h264 aac mp4 3 h264 aac mp4 (hinted for streaming) 4 h264 wav (youtube does have trouble with lossless rgb from x264) E. colorspace 1 10 bit YUV444 PC.601 (send x264 RGB and let it do the conversion?) 2+3 8 bit YUV420 PC.601 4 8 bit YUV420 Rec709; RGB
Player (66)
Joined: 4/21/2011
Posts: 232
Alright, so here is what I have so far. If the .avs plays like you want it to, then this should encode it. NGcode.bat NGcode.avs Here is where to find stuff if you don't already have it. Avisynth installed AviSynth_110525.exe avs2pipemod.exe avs2pipemod-0.4.1.7z ExactDedup.dll exactdedup-0.03.zip MP4Box.exe GPAC.Framework.Setup-0.4.6-DEV-rev4013.exe (If you move it, you may have to move some dlls too) mkvmerge.exe mkvtoolnix-unicode-5.5.0.7z mtee.exe mtee.zip neroAacEnc.exe NeroAACCodec-1.5.1.zip (The encoder is version 1.5.4) NHMLFixup2.exe NHMLFixup2-v2.zip x264_8.exe x264.exe (Save link as... and change the name) x264_10.exe x264.exe Adjust the set PATH in the batch to reflect where the files are. Adjust the set ng_lib in the batch to point to NGcode.avs. Adjust the LoadPlugin in NGcode.avs to point to ExactDedup.dll.
Player (66)
Joined: 4/21/2011
Posts: 232
This site freaks out when you mix less_than_equal and greater_than_equal.
Former player
Joined: 2/19/2007
Posts: 424
Location: UK
I guess I'll post mine too. My encodes are not publication quality as they don't include any subtitles or intro etc., but they are quite high on the automation scale. It only works on smv and fm2 for now, though. After I've downloaded an .smv file, the encoding process looks like this:
Language: bash

$ smvenc2 foo.smv foo.mkv
That produces a lossless yuv444 encode at normal resolution. If I want to use a lua script with that, and perhaps have a double-sized yuv420 encode, I would do
Language: bash

$ smvenc2 -lua script.lua -fmt yuv420 foo.smv foo.mkv
The encode is totally headless. That is, no snes9x window pops up during encoding, and no sound is played (but of course, the output video contains sound and image), so this can be run on a system with no running X server, which is nice for making an automated encoding service as I did before at one time. The script itself looks like this (link):
Language: bash

#!/usr/bin/env bash # Automatic smv encode. Given an smv file and and output filename, # produces a lossless encode in one pass player="smv" popts="-r 5 -mute -dumpstreams" prefix=".smvenc" afifo="audiostream0.dat" vfifo="videostream0.dat" afile="$prefix.audio.mp3" vfile="$prefix.video.mkv" fmt="yuv444" double="auto" proginterval=0.5 # My snes9x has problems with complicated file names due to the # dchshell wrapper, so copy things to simple names in the work dir. worksmv="$prefix.work.smv" workmkv="$prefix.work.mkv" worklua="$prefix.work.lua" qp=0 function show_help() { echo "smvenc2: Encode smv files to mkv Syntax: smvenc2 [options] in.smv out.mkv Options: -lua script.lua: Run with this lua script. -v: Verbose mode: Show output from subcommands. -frames: Output video will be this many frames long. -qp: Encode with this qp. Default is 0 (lossless). -fmt: Color format to use. Rgb, yuv444 or yuv420. -2: Double size before encoding. Useful with yuv420. -1: Do not double size. -0: Determine doubling automatically based on color format. -h: Display this help." } # Parse arguments nf=0 args=() argv=("$@") for (( i=0; i <argv>&2; exit ;; *) args[$nf]="${argv[$i]}" ; nf=$(($nf+1)) ;; esac done case "$fmt" in rgb) fmtcmd="--output-csp rgb" ;; yuv444) fmtcmd="--output-csp i444" ;; yuv420) fmtcmd="--output-csp i420" if [[ "$double" == "auto" ]]; then double="yes"; fi;; *) echo "Invalid format $fmt" >&2; exit ;; esac if [[ $double == yes ]]; then resize="--vf resize:512,448,method=point"; fi smvfile="${args[0]}" outfile="${args[1]}" frames="$(bread "$smvfile" 16 4)" frames=$((frames+1000)) if [[ $frames_override ]]; then frames=$frames_override; fi # Prepare work directory cp "$smvfile" "$worksmv" if [[ $lua ]]; then cp "$lua" "$worklua"; luacmd="-loadlua $worklua"; fi rm -f "$afifo"; mkfifo "$afifo" rm -f "$vfifo"; mkfifo "$vfifo" popts="$popts $luacmd" junkdev=/dev/null if [[ $verbose ]]; then junkdev=/dev/stderr; fi # Be ready to clean up trap 'kill $xvfb $ppid $apid $vpid $spid 2>/dev/null rm -f "$vfile" "$afile" "$worksmv" "$worklua" "$workmkv" "$afifi" "$vfifo" ' EXIT # Don't display anything display=":40.1" Xvfb "$display" -screen 1 800x600x24 >$junkdev 2>&1 & xvfb=$! export DISPLAY="$display" function progreport { while true; do s=$(stat -c %s "$afile" 2>/dev/null) rsec=$((s/96/1024*8)) h=$((rsec/3600)) m=$((rsec/60-h*60)) s=$((rsec-h*3600-m*60)) if [[ ! $verbose ]]; then printf "\b\b\b\b\b\b\b\b%2d:%02d:%02d" $h $m $s; fi sleep $proginterval done } # Start producing output "$player" "$worksmv" $popts -maxframes $frames >$junkdev 2>&1 & ppid=$! # Encode audio lame -r -s 32 -h "$afifo" "$afile" >$junkdev 2>&1 & apid=$! # Encode video x264 --input-csp bgra --input-res 256x224 --range pc --colormatrix bt470bg $fmtcmd $resize -o "$vfile" "$vfifo" --qp $qp --preset veryslow --fps 60.0375 >$junkdev 2>&1 & vpid=$! # Progress report progreport >&2 2>/dev/null& spid=$! wait $ppid $apid $vpid { kill $spid $xvfb; wait $spid $xvfb; } 2>/dev/null # And merge them together mkvmerge -o "$outfile" "$vfile" "$afile" >$junkdev 2>&1 rm -f "$vfile" "$afile" "$worksmv" "$worklua" "$workmkv" "$afifi" "$vfifo" printf "\ndone\n"
This depends on a couple of other utilities. The "smv" is a script I use for watching smv files. It handles the automatic lookup of the rom and then runs snes9x (link):
Language: bash

#!/usr/bin/env bash index="$HOME/tas/snes/index" snes=snes9x-1.43 smvfile="$1" shift echo n | "$snes" "$index/$(smvcrc "$smvfile")" -autodemo "$smvfile" "$@"
There are some other short utilities that are needed, but the forum mangles them, so I'll just link to them instead: smvcrc is used to extract a crc from an smv file, for the purpose of looking up the correct rom. romcrc is used to compute the crc of a snes rom, wihch is used to poulate a directory of crc -> rom symlinks. bread is used to read the frame count from the smv file. For fm2 files I use a similar setup fm2enc, fm2, fm2md5, nesmd5 The md5 stuff is mostly extracted from fceux.
Post subject: Here's my batch+script
Joined: 3/31/2010
Posts: 84
Location: Middle of Nowhere
I remember when I got laughed at for attempting automated encoding. There's a few things that bug me. You have to set the --sar and AR manually. What if you don't want to do all the encodes at once, or only need to do downloadables? Or what if you're encoding something NOT in 4:3? Or something that you don't know the --sar for? My Batch. ReplaceText.vbs. Since I lacked the tools to replace text from a batch, I use this VBS script to do it. The Script I use. Vdub jobs file for automated audio extraction. Things needed: Avisynth (2.58 if 64-bit, 2.6 if 32-bit), Blinkmask and Shakemask won't work on 2.58, but I never use them (and still working on a way to only use them on parts of the encode anyways) ExactDedup, mt_masktools, and mvtools2 (and 64-bit versions if needed) mp4box & neroaacenc mkvmerge, oggenc2, and flac x264 8-bit and x264 10-bit sox lua and NHMLFixup-v10 (I'm outdated a bit) Whatever dll files the above need to work A lot of things here in my works that don't revolve around TASVideos. It's not readable at a distance. It won't work on linux to the best of my knowledge. Parts are sloppy and unorganized. DIRs need to be configured because I lack programming knowledge for AVIsynth and Batch. And my script is also heavily based on the current Hybrid. The script and my batch are flexible and easily customizable however. I wanted to give people a choice in everything rather than needing to rewrite everything any time they wanted something (or PER ENCODE). If you want mp4s, my batch lets you make mp4s, if you want MKVs, same deal. Things are easily toggleable. Once subtitles are inputted, everything else is automated (my batch automatically produces all encodes if nothing is hit).
Post subject: Re: Here's my batch+script
Player (66)
Joined: 4/21/2011
Posts: 232
Velitha wrote:
There's a few things that bug me. You have to set the --sar and AR manually. ...what if you're encoding something NOT in 4:3? Or something that you don't know the --sar for?
Yes, I could read in the frame dimensions and calculate one from the other, maybe next time.
What if you don't want to do all the encodes at once, or only need to do downloadables?
I'd use a text editor and :: comment out the lines that aren't needed. What is the difference between the 512 and SDstreamable?
Player (66)
Joined: 4/21/2011
Posts: 232
Velitha wrote:
--vf resize:,,,,i420:8,lanczos
It looks like you end up with an undefined colormatrix in your encodetypes 1,2,3 ? Though I guess by the time nhml is done with them, any information I passed to x264 is lost as well. So, does it matter, or just expect everything SD will be treated as PC.601 and hope for the best???
Joined: 3/31/2010
Posts: 84
Location: Middle of Nowhere
nanogyth wrote:
Yes, I could read in the frame dimensions and calculate one from the other, maybe next time.
I find it's much simpler and elegant to calculate that stuff in-script. One less point of failure, as well as ensuring that it's done properly each time.
I'd use a text editor and :: comment out the lines that aren't needed.
Part of the point of my batch was that I didn't need to manually edit it every time per encode. it's tedious and inelegant. You should be able to configure what encodes need to be done in-batch rather than have to edit the batch/script every single time. Besides, some people like not having to edit files as well. Some pictures of my batch in action: Main Menu 1. Sub Menu. Main Menu 2.
What is the difference between the 512 and SDstreamable?
512 is for Archive.org. SD Streaming uses tasblend, doesn't upscale the encodes, and is capped at whatever the streaming FPS cap is set at (in most cases it's 30, but it's easily changeable). I threw this in my batch despite it not being needed because "why the heck not".
It looks like you end up with an undefined colormatrix in your encodetypes 1,2,3 ?
Ilari told me to use that (and let x264 handle the color matrices), so what I'm using is what I was told to use. I'm also sorry that I can't have the EncodeType as a string. Replacetext does NOT like things with quotes, so I had to use a number. Every other program I tried failed to work in a batch, required some GUI (and manual intervention, which eliminates the purpose of what I'm doing), refused to work at all, or had no documentation/was too complicated to figure out (Looking at you SED). I'm working on a few changes for the batch as well, but they're going to take a lot of time to implement. They should add more flexibility (although it might cost a few extra cpu cycles). I never change my links for pastebin, so the changes will be uploaded to those links when I'm done with them. Something I would like to add, but not sure the proper batch coding would be 'directory independence'. Everything right now is heavily tied down to the folders I use on my computer. Which makes testing outside of my computer difficult. Also, if you expect me to have a clue what I'm doing, you're out of luck :P (everything is 'just try it and see what happens', goggling stuff, splicing commands, and learning everything that goes wrong).
Player (66)
Joined: 4/21/2011
Posts: 232
Velitha wrote:
--vf resize:,,,,i420:8,lanczos
Ilari told me to use that (and let x264 handle the color matrices), so what I'm using is what I was told to use.
I'm sure x264 uses a predetermined color matrix, I just didn't know which. Is it 601? Is there a way to signal that after mp4box combines the nhml? Does nhmlfixup need the nhml from the raw h264? Would pulling the nhml from an mp4 stream keep the metadata? This issue also seems to kill the SAR. A video I flagged as 7:6 ended up as 5:4 according to mediainfo. Is there a program to check if the video is tv/pc range?
Emulator Coder, Skilled player (1129)
Joined: 5/1/2010
Posts: 1217
nanogyth wrote:
I'm sure x264 uses a predetermined color matrix, I just didn't know which. Is it 601?
Yes, it is 601 (TV or full depending on parameters).
nanogyth wrote:
Does nhmlfixup need the nhml from the raw h264? Would pulling the nhml from an mp4 stream keep the metadata?
The x264 MP4 muxer is worse than useless. The timecode data it writes is WRONG and NHMLFixup CAN NOT fix it.
nanogyth wrote:
This issue also seems to kill the SAR. A video I flagged as 7:6 ended up as 5:4 according to mediainfo.
NHMLFixup has aspect ratio correction mode (but only 4:3 and 16:9).
Player (66)
Joined: 4/21/2011
Posts: 232
The timecode problem is just something with mp4, right? So by storing my dedupped files as mkv I could avoid that hassle? But mp4 is still needed for the 512 because archive.org won't stream mkv?
Emulator Coder, Skilled player (1129)
Joined: 5/1/2010
Posts: 1217
nanogyth wrote:
The timecode problem is just something with mp4, right?
As far as it is known, yes.
nanogyth wrote:
But mp4 is still needed for the 512 because archive.org won't stream mkv?
Right.
creaothceann
He/Him
Editor
Joined: 4/7/2005
Posts: 1874
Location: Germany
What about using MP4Box to create an .mp4 file? (or some other tool than x264)
Player (66)
Joined: 4/21/2011
Posts: 232
creaothceann wrote:
What about using MP4Box to create an .mp4 file? (or some other tool than x264)
That is what NHML lets you do, but it loses some metadata. MP4Box has an option to change the SAR, but I didn't see anything to put the color matrix tags back. Probably not a big deal since most software ignores those tags anyway.
Experienced player (764)
Joined: 6/17/2008
Posts: 146
nanogyth wrote:
That is what NHML lets you do, but it loses some metadata. MP4Box has an option to change the SAR, but I didn't see anything to put the color matrix tags back. Probably not a big deal since most software ignores those tags anyway.
Nothing is lost because the VUI flags are in the h.264 stream itself, not the container. Some players read the SAR/DAR from the container, some read them from the h.264 stream. Therefore, you want them both to have the correct information. Also, get on #tasvideos while you're at it. You'll be set up much faster and more conveniently.
Site Admin, Skilled player (1256)
Joined: 4/17/2010
Posts: 11496
Location: Lake Char­gogg­a­gogg­man­chaugg­a­gogg­chau­bun­a­gung­a­maugg
nanogyth wrote:
This site freaks out when you mix less_than_equal and greater_than_equal.
if x < 0 then x = 0 elseif x > 193 then x = 193 end
Disable HTML in the post.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Player (66)
Joined: 4/21/2011
Posts: 232
Velitha wrote:
--vf resize:,,,,i420:8,lanczos
I see this warning when using that
resize [warning]: converting range from TV to PC
Do you see that too? --vf resize:,,,,i420:16,lanczos Might prevent excess color loss.
Experienced player (764)
Joined: 6/17/2008
Posts: 146
nanogyth wrote:
resize [warning]: converting range from TV to PC
Do you see that too? --vf resize:,,,,i420:16,lanczos Might prevent excess color loss.
If your input is RGB, there's no issue. The warning occurs because RGB input is labeled as "TV range" even though it is nothing of the sort due to an swscale bug. Using 16-bit precision is pointless in this case since the output will be dithered to 8-bit anyway. If you were using Hi10P, there would be negligible benefit.
Player (66)
Joined: 4/21/2011
Posts: 232
smb2.avi #dump from emu 4.avs
Language: avisynth

avisource("smb2.avi") assumefps(60) converttorgb32 num = AudioLengthF * FrameRateNumerator denom = FrameCount * FrameRateDenominator AssumeSampleRate(Round(num / denom)) ResampleAudio(48000) trim(0,38628) dissolve(last, last.trim(38429, 38628).amplify(0), 200)
4_yt.avs
Language: avisynth

import("C:\Users\vehi\Desktop\NGcode\lib\NGcode.avs") src=avisource("4.avs") ReplaceFramesSimple(src, src.ng_deblink(), \ mappings=" [2378 3071] #first boss [28627 29181] #wings") ReplaceFramesSimple(src.ng_deblink(1.0), \ mappings=" [2025 2053] #shyguy twitching [2564 2708] #toad twitching [4254 4341] #falling creeps [6137 6182] #falling logs [17391 17439] #invinc [18675 18679] #footsie [19699 19746] #invinc [22756 22803] #invinc [25922 26137] #egg+invinc [26596 26615] #toad twitch [27401 27455] #toad twitch")
smb2.avs
Language: avisynth

import("C:\Users\vehi\Desktop\NGcode\lib\NGcode.avs") multi = 1 src=avisource("M:\TAS\smb2\4.avs") import("subs.avs")
smb2_yt.avs
Language: avisynth

SetMemoryMax(2048) import("C:\Users\vehi\Desktop\NGcode\lib\NGcode.avs") multi = 8 src=avisource("M:\TAS\smb2\4_yt.avs") import("subs.avs") SelectEven()
subs.avs
Language: avisynth

subff = 425 system = "NES" title = "Super Mario Bros. 2" author = "DonamerDragon" time = "07:53.32" rerecord = "15215" sub_str1 = title + \ " by " + author + "\n" + \ " in " + time + \ " Rerecord Count: " + rerecord sub_str2 = "This is a tool-assisted recording.\n" + \ "For details, visit http://TASVideos.org/" src main=pointresize(multi*width, multi*height) AviSource("M:\TAS\logo.avi") ChangeFPS(main) audiodub(blankclip(main)) logo=spline36resize(main.width, main.height) logo++main ng_bighalo(sub_str1, y=17*multi, align=2, first_frame=subff, size=(13.0 * multi), halo_radius=(1*multi), memo = "1") ng_bighalo(sub_str2, y=19*multi, align=2, first_frame=(subff + 300), size=(15.0 * multi), halo_radius=(1*multi), memo = "2")
Using import("subs.avs") like this makes it act like a function that returns a clip with the subs applied.