You need to use glsl shaders. Cg shaders didn't work when I tried them.
You can find a huge collection of glsl shaders in Retroarch for Android. So, download the latest stable version and extract the contents of the apk file (apk is just a zip). The shaders are in 'assets\shaders_glsl'.
EDIT: Automatically updated GLSL conversions of the Libretro Cg shaders:
https://github.com/reedolph/common-shaders-glsl
You can also convert a cg shader to glsl yourself with this Python3 script:
https://github.com/libretro/RetroArch/blob/master/tools/cg2glsl.py
Once you have the shaders you want to use, you must create a preset file (cgp) for each one of them.
From all the shaders that Bizhawk is compatible with, I believe that 4xbr-v3.8d offers the best upscale. Let's see how we can use it.
The shader is here:
...\BizHawk\shaders\xbr\4xbr-v3.8d.glsl
Create a text file (change the extension to cgp) and put it in the shaders folder:
...\BizHawk\shaders\Preset (4x) 4xbr-v3.8d.cgp
Open "Preset (4x) 4xbr-v3.8d.cgp" with a text editor and put inside the following.
shaders = 1
shader0 = "xbr/4xbr-v3.8d.glsl"
filter_linear0 = false
float_framebuffer0 = false
scale_type0 = source
scale0 = 4
Select the cgp in the options and that's it. In the Final Filter it is best to select 'Bilinear'.
Note:
Many shaders come with presets that are ready to use. *Don't* use them exactly as they are. Specifically, notice the scale option. Normally it's like this "scale_0 = 4.000000" or "scale_x0 = 2.000000" & "scale_y0 = 2.000000". But in Bizhawk you have to use an integer.
Some shaders will require a specific scale value, or no value at all. But with shaders like 4xbr-v3.8d you can use whatever scale you like. In a very weak GPU or a low res monitor you may want to use scale0 = 2 or 3. On the other hand, you could also create a preset with scale0 = 6 or 8, to use with Atari Lynx or Gameboy for example.
Be careful with the N64 and the Saturn. If you have set the resolution in the N64 video plugin to something like 1280x960, it's pointless to have that upscaled to 4x. The GPU will work really hard for nothing.