Given that GLideN64 is now available in the development versions of Bizhawk (and hopefully soon a release), I thought I'd quickly go over some relevant thoughts I've had on the matter. Feel free to discuss / disagree / add / whatever:
GLideN64:
Multisampling (anti-aliasing) should likely be enabled when encoding, except in special circumstance.
Max Anisotropy (anisotropic filtering) should likely be enabled when encoding, except in special circumstance
There's also the native resolution factor integer. This is an interesting one. At 0, it's disabled and the games run at whatever resolution you've specified. At 8, if the game's 320x240 (which it will be unless it's a rare case), it'll run at 2560x1920 and then be scaled down to whatever resolution you're using. For this purpose, I'd recommend dumping at the same resolution as the resolution factor you're using, so in this case dumping at 2560x1920.
The reason why this is relevant is that sometimes there are object misalignments. A good example is on the most recent Banjo-Kazooie run, the pictures that Mumbo is holding at the end (which show Banjo finding secret items) are made up of several smaller squares. When this value is set to 0, they're slightly misaligned, causing a grid to show up in the spaces between them. Using a high native resolution factor fixes this. It should also be noted that if you use too high a value (I haven't experimented at what value exactly, but below 20), severe graphical glitches start to show up, so you can't boost this value indefinitely. If necessary, when an official release of Bizhawk contains GLideN64 I can provide a savestate highlighting this issue.
A similar misalignment can be found with the arrows under the "number of players" selection in the Mario Kart 64 main menu, but it has a different fix. There's an option something along the lines of "native res texrects". Using that solves the issue, but it's only available at when native resolution factor is disabled / set to 0.
GLideN64 has many graphical accuracy improvements (and it affects the timing of runs, so that extends to timing, I believe) so provided it's a viable option it should be the default when doing N64 runs. Of course, plugin selection should probably be discussed on a per-game basis, as I'm sure there are cases where other plugins are more suitable candidates than GLideN64.
Additionally, there have been notes of users experiencing per-system graphical glitches. In some cases there is flickering (every alternate VI is black), in other cases fade transitions don't show in (for example, in the Mario Kart 64 menus). These are per-system glitches and often aren't actually present in the plugin itself. Drivers, opengl requirements, certain hardware (and certain software in rare cases) can all interfere with the plugin and it should be noted that a glitch plaguing one user may not be a result of the plugin as a whole (and therefore it should be tested on multiple systems before it's confirmed as a graphical anomaly).
Bizhawk in general:
Most N64 games have crackling audio using the regular sync. In some games it's much more severe than others but it's present in almost every ROM, at least to some degree (even if only for a frame here and there). It can present itself as a brief audio glitch, a click, or consistent crackling. Dumping using "alternate sync" fixes this and should probably be used in all cases, to be safe. A good example of this is the "King Jingaling Gets Zapped" cutscene in Banjo-Tooie. Once again, if requested, when an official release of Bizhawk contains GLideN64 I can provide a savestate highlighting this issue.
N64 emulators in general (both Bizhawk and Mupen):
It has been mentioned over
here that for the plugins that don't have anti-aliasing built in (and on the chance that Rice Video's anti-aliasing doesn't work the way it's supposed to) you can force anti-aliasing through your respective GPU control panel. The maximum settings available should be chosen.
The alternative to this is downsampling, in which the game's rendered at a much higher resolution and then resized down to the target resolution. This has the advantage of being more predictable and controllable, although depending on the resizing method the output can be less sharp. Additionally, some games experience glitches (especially if native resolution factor is required on GLideN64) at the resolutions required to produce the same level of anti-aliasing as the GPU control panel method would (for instance, on a 1920x1440 encode to achieve 16x supersampling (4x4) you would have to render at 7680x5760).
It is worth investigating which of these methods actually looks better (Aktan strongly recommends downsampling, but users such as feos and myself have noticed slightly sharper images with the GPU control panel method) and if there are any problems with either method. In any case, using BOTH would probably be the best outcome of this.
If anyone has anything else to add, feel free! I want this to be an open discussion to achieve the best results out of N64 encoding!
The problem with YV12 and YUY2 is that they're chroma subsampled. YV12 is set at 4:2:0 and YUY2 is set at 4:2:2 if I'm not mistaken. These exist for a variety of purposes, but do not really cater to pixel-oriented game videos. The issue is that in the case of 4:2:0, this means that there is only one chroma (or color) pixel for every 2x2 block of luma resolution, as shown in the picture above. This is why the black pixels in your examples remain unblurred, whilst color starts to spill into other pixels and is also why a resize with a factor of 2 solves this problem.
So assuming you have a video in RGB to begin with, you resize with a factor of 2 using point size or nearest neighbor. The point of this is to double (or quadrouple, etc) each pixel both horizontally and vertically, making each 2x2 block contain four identical pixels. This means that when the chroma is compressed down to represent the 2x2 block, no data is discarded and then no blurring/blending occurs.
Hopefully that helps you understand the situation more thoroughly. I darted around your question a bit, but the problem itself doesn't require you to resize the resolution and then the luma separately - they're linked. The luma is the brightness and the chroma is the color data and in a 4:4:4 colorspace (or in RGB) each pixel has it's own unique values.