Here is how the RNG and encounters work:
At every step there is an RNG roll with about 1/16 chance of getting a fight. After 20 steps you get a fight no matter how lucky you are. The RNG only changes once per step outside of battle but goes crazy in battle and seems to change every frame. You can delay leaving or finishing a battle by a small number of frames to manipulate the maximum 20 steps between battles. Killing enemies might also make a difference depending on how they use the RNG.
Interesting memory locations:
C0A0 - RNG seed (32 bits)
C685 - Step counter (8 bits)
C69C - Encounter rate (16 bits)
Here is a disassembly of the subroutine at $FB5C that calculates the next random number:
MOVEM.l D1, -(A7) [Saves off D1]
MOVE.l $FFFFC0A0.w, D1 [Copy C0A0 - the RNG seed - to D1]
BNE.w *+$8 [Skip next unless that was 0]
MOVE.l #$2A6D365A, D1 [Initialization]
MOVE.l D1, D0 [Copy D1 to D0]
ASL.l #2, D1 [Shift D1 2 bits]
ADD.l D0, D1 [Add D0 to D1]
ASL.l #3, D1 [Shift D1 3 bits]
ADD.l D0, D1 [Add D0 to D1]
MOVE.w D1, D0 [Copy low 16 bits of D1 to D0]
SWAP D1 [Swap low and high 16 bits of D1]
ADD.w D1, D0 [Add low 16 bits of D1 to D0]
MOVE.w D0, D1 [Copy low 16 bits of D0 to D1]
SWAP D1 [Swap low and high words of D1]
MOVE.l D1, $FFFFC0A0.w [Save new RNG to memory]
MOVEM.l (A7)+, D1 [Restore old D1]
RTS
After this routine the value in register D0 is used as the random number. Here is how that gets used for encounters:
AND.w $FFFFC69C.w, DO [16 bit and with random number - that mem has $#000F]
BEQ.b *+$4 [Skip next 2 if we got 0 - leads to fight]
CLR.w D0
RTS [Return with 0 in D0]
MOVEQ #1, D0
RTS [Return with 1 in D0]
BNE.b -- If we had 1 in D0 then fight
-- If not then increment step count. If it hits 20, fight anyway.
C69C controls the encounter rate. It is $000F outside the starting village so that is how I figure 1/16 chance. The more bits set, the lower the encounter rate.
