Here is a document I wrote up describing my technique for making Genesis runs into DVD, and I've linked in this thread to a post I wrote over on doom9.org detailing my NES technique.
As for the 4GB limit, I would advise you to use the CamStudio Lossless Codec which is available
here. I has several compression settings, include a fast (but large) compression called LZO, and also a GZIP mode with settings from 1 to 9 (the higher the setting, the slower the encode, but the smaller the file - I recommend using 9 for the absolute smallest possible filesize). Using this codec, my "raw" for Gigafrost's Zelda 2 run ended up being 107841400 bytes for 8:40 of video, about 40% of which was audio. Since 107841400 goes into 4GB about 40 times, that means that you should easily be able to get some longer runs under the 4GB mark, at least an hour easily (although I wouldn't take that result too literally, because it depends on the game's graphics content and amount of on-screen motion). Audio is of course a constant with the NES (and would be exactly double with stereo consoles). And, I also use NTFS, and for the most part it seems the tools I use can support >4GB files on NTFS, so I haven't really taken that into consideration.
Also, some emulators do indeed automatically segment an AVI recording. Gens Movie Test, for example, splits to a new AVI about every 1GB, and I'm pretty sure that Famtasia splits, but as I haven't used Famtasia in ages, I don't know exactly where it splits to a new AVI.