I have a 1733 MHz Athlon. A typical 300kbit/s 256x240 video encoded with x264 uses about 12%-15% CPU to decode and display, using a recent mplayer. With emulator playback captures, there's no excuse to scale the video to a larger resolution.
On a ~645kbps 720x352 video (unresized cropped dvd widescreen movie) I get about 45% CPU usage.
On ~700kbps 720x480 video (unresized uncropped 4:3 dvd video) I see up to 50% CPU usage.
libavcodec-based players (mplayer, ffdshow, vlc) using an older libavcodec offer h.264 decoding that is both slower and buggier. If you use windows, you can get a more recent ffdshow here:
http://www.aziendeassociate.it/cd.asp?dir=/ffdshow
Likewise there are recent mplayer builds for windows:
http://www.aziendeassociate.it/cd.asp?dir=/mplayer
As for encoding: if you are getting better video quality with XviD, you're simply doing something wrong. XviD is of course excellent for an ASP codec but on "typical" source, you should be able to match its quality with x264 using 5%-30% less bitrate. There is a
guide for using mencoder to encode videos with x264; while it's neither complete nor entirely up-to-date, it will at least point you in the right direction.