As I said it's lossy compression - the recording will have compression artifacts, and they will be much more noticeable with low resolutions like the Genesis' 256x240 or 320x240 graphics modes. Also, each new encoding step will add more artifacts, just like saving a picture to JPG over and over again. This is why I'd record with a lossless codec, do my editing, and then finally spend a lot of CPU time to create a much smaller h.264 file that can be shared or uploaded to YT.
Also note that emulators will create recordings with uncompressed PCM audio. You can halve that audio's size with a codec like FLAC (you can think of it as ZIP for audio), and for the final encoding use something like MP3 (via LAME MP3 encoder) or OGG. You can export the PCM audio from VirtualDub to a WAV file, compress it, and then insert it into your final file. I use MKVToolNix GUI for that.
For TASes always use a lossless one, like ZMBV, Lagarith, Camstudio or ffdshow 'uncompressed'. They are / can be very fast and will record all pixels just as they are displayed on screen; the disadvantage is that they create relatively big files. ZMBV is the best afaik because it only records changes from one frame to the next (Camstudio always records a full frame, Lagarith at least supports null frames), but it only accepts 16-bit and 32-bit RGB, whereas some older emulators like SNES9x output 24-bit RGB. (If you use ZMBV then jumping around in the recorded file might be very slow because afaik it doesn't insert keyframes by itself; VirtualDub has an option to force a keyframe in regular intervals.)
Formats like h.264 are like JPEG compression: they convert the RGB data to YUV first (not sure about x264's "RGB" mode) and by default apply lossy compression, which is relatively slow. The advantage is much smaller files.
x264vfw has a text field in its configuration dialog for extra parameters.
Yes, it's the right formula... the only change is that YT doesn't call 720p "HD" any more.
MeGUI might use Avisynth internally? It's been years since I used it.
Regarding the other thread: you can use x264vfw for VirtualDub. Just know that by using VirtualDub you no longer have a completely automatic encoding process.