I've ran a few lossless codecs on SM64 in bizhawk 2.6.1 at 4K, 10k frames of no input, GLideN64, all default settings. fps is from Mario's face.
UTVideo
(fastest setting + best compression)
-c:a pcm_s16le -c:v utvideo -pred median -pix_fmt gbrp -f avi
13-14fps
31.6GB
FFV1
-c:a pcm_s16le -c:v ffv1 -pix_fmt bgr0 -level 1 -g 1 -coder 1 -context 1 -f avi
3fps
15.4GB
AVC
-c:a pcm_s16le -c:v libx264rgb -qp 0 -preset ultrafast -g 1 -pix_fmt rgb24 -f avi
11-13fps
28.3GB
-c:a pcm_s16le -c:v libx264rgb -qp 0 -preset ultrafast -g 5 -pix_fmt rgb24 -f avi
13-14fps
9.91GB
-c:a pcm_s16le -c:v libx264rgb -qp 0 -preset ultrafast -g 10 -pix_fmt rgb24 -f avi
13-14fps
7.65GB
-c:a pcm_s16le -c:v libx264rgb -qp 0 -preset ultrafast -g 15 -pix_fmt rgb24 -f avi
13-14fps
6.84GB
Lagarith
(multithreading and null frames on, RGB)
14-17fps
9.72GB
Camstudio
(LZO)
6-7fps
59.7GB