This post attempts to explain how NesVideoAgent's automatic screenshot feature works.
It should answer all questions, though it is a little technically oriented.
Architecture
Outer layer: A php program
It is run manually by me, though it could be run automatically too.
This commandline program searches the database for submissions that
match these rules:
- Status is not published, rejected or canceled
- Has no screenshots yet
- Is for NES, SNES, GB, SGB, GBC, GBA or Genesis
For each matching submission, it does the following:
- Tries to determine which ROM should be used to watch the submission
- Scans the list of Good* ROM names and selects ~90 ones that most closely match the submitted ROM name or game's name
- If it finds one whose checksum (CRC32, MD5 or other) matches what the submitted movie file has, uses it
- Runs the emulator with specific parameters
- A requirement is that the emulator can run on parameters without need for keyboard or mouse input
- The emulator must also accept the fact that it can't produce sound (server has no soundcard)
- The emulator is run under Xvfb so it can be graphical yet non-interactive
- Updates the submission message with
- links to images produced by the emulator
- notes about the ROM selection
Inner layer: Modified emulator
The emulator is slightly patched version of the normal emulator
used to run the movies.
- It is completely uninteractive (movie file, sync options, play length specified on command line)
- It includes a beauty analysis module.
The beauty analysis module
The beauty analysis module is a component in the emulator that
works like this:
- At every frame when the movie plays, it observes the emulated screen (in a similar manner as AVI encoder does).
- At every frame, it can also save or load a savestate, or terminate the emulator.
Description of operation
First, it plays the movie normally.
- Every frame, it analyses the image for beauty, and records it as a numeric value for that particular frame.
- Every 100 frames, it makes a savestate.
When the movie ends (actually, 2 frames before but that's a detail):
- It divides the movie into N evenly long sections
- Where N is selected so that N <= 50 but that a section length must be at least 1 second and not more than 10 seconds
- And from that section, it determines the most beautiful shot, using the numeric values recorded earlier.
- Then it picks the 20 most beatiful sections, and remembers the frame numbers where the single most beatiful screen occured during that section.
- For each selected beauty:
- It loads the latest savestate that was recorded before that particular frame
- Runs the emulator until the framenumber matches the desired frame
- And saves the screen as a PNG file.
- Then it terminates the emulator.
Analysis of beauty
The algorithm used by the emulator for determining the beauty of a particular screenshot
is written hoping to meet the following guidelines. A good screenshot contains:
- distinctly different colours at different sections of the screen (saturated colors preferred over grayish ones)
- a lot of movement at each portion of the screen (less desirable to have it localized in a single corner)
The algorithm works by dividing the screen into 12 sections (4 horizontally, 3 vertically),
and for each section, determining:
- The average tint on that section of the screen
- The number of sprites that have changed position on that section of screen compared to the previous frame
The score is determined by the sum of sprite jitter in all sections, and matching each diagonal pair of sections and calculating how much the tint differs in them.
The tint is calculated by sampling the pixels in that section (actually,
about a third of them in a random gridlike pattern for speed but that's
again an implementation detail), converting
them to the YIQ colorspace and calculating average of the complex number
comprised by the I and Q components, weighted using the Y value, and the
final number weighted using a logarithmic transformation.