After checking on a couple of things this evening, the game seems to have 4 different routines for random encounters and what enemy group you get. 0x7E1700 holds a value that goes from 00-03.
00: Earth worldmap
01: Underground worldmap
02: Moon worldmap
03: Dungeons/Towns
Earth World Map:
008951: BNE 00895c ($9)
00895C: LDA $1700
00895F: CMP #$00
008961: BNE 00899d ($3a)
008963: LDA $1707 // Load 7E1707 into the accumulator
008966: LSRA // Shift the byte right 2 times
008967: LSRA
008968: AND #$f8 // And it against 0xF8
00896A: STA $06 // Store the result in 0x7E0606
00896C: LDA $1706 // Load 0x7E1706 into the accumulator
00896F: LSRA // Shift the byte right 5 times
008970: LSRA
008971: LSRA
008972: LSRA
008973: LSRA
008974: CLC // Clear the carry flag
008975: ADC $06 // Add the value at 0x70606 to the accumulator
008977: TAX // Copy the value of the accumulator into register X
008978: LDA $0ec300,X // Load 0x0EC300 + Register X into the accumulator
00897C: STA $06 // Store the result into 0x7E0606
00897E: PHX // Push Index Register X
00897F: PLY // Pull Index Register Y
008980: LDA $c0 // This loads 7E06C0 into the accumulator, to zero out the accumulator
008982: BNE 008994 ($10)
008984: LDA $86 // Load 7E0686 into the accumulator
008986: TAX // Copy the accumulators value into X
008987: LDA $14ee00,X // Load 0x14EE00 + X into the accumulator
00898B: CLC // Clear the carry flag
00898C: ADC $17ef // Add the value at 0x7E17EF to the accumulator, this seems to only change on a reset
00898F: CMP $06 // Compare the accumulator's value against 0x7E0606, if 0x7E0606 is greater than the accumulator, don't set the carry flag
008991: BCC 008994 ($1) // Branch on carry, if this does not branch, an encounter occurs
Underground:
0089A5: LDA $86 ; Encounter formula begins here
0089A7: TAX
0089A8: LDA $14ee00,X
0089AC: CLC
0089AD: ADC $17ef
0089B0: CMP $0ec340
0089B4: BCC 0089b7 ($1) ; Triggers battle if branch not taken
The moon Random Battle formula seems to be the exact same formula as the underworld formula since 0x0EC340 and 0x0EC341 contain the same value of 0x08. I assume it uses a different routine because the way it loads the enemies different, since it's all in the same routine.
Moon world map:
0089DC: LDA $86 ; Encounter formula begins here
0089DE: TAX
0089DF: LDA $14ee00,X
0089E3: CLC
0089E4: ADC $17ef
0089E7: CMP $0ec341
0089EB: BCC 0089ee ($1) ; Triggers battle if branch not taken
Dungeons: (And anywhere else that isn't the world map that can encounter enemies randomly)
008A35: LDA $1702 ; Encounter begins
008A38: STA $3d
008A3A: LDA $1701
008A3D: BEQ 008a41 ($2)
008A41: STA $3e
008A43: LDX $3d
008A45: LDA $0ec342,X
008A49: BEQ 008a60 ($15)
008A4B: STA $06
008A4D: LDA $c0
008A4F: BNE 008a73 ($22)
008A51: LDA $86
008A53: TAX
008A54: LDA $14ee00,X
008A58: CLC
008A59: ADC $17ef
008A5C: CMP $06
008A5E: BCC 008a73 ($13) ; Triggers battle if branch not taken
Already successfully added them in lua and it seems to be correct. I still haven't fixed the value of 0x7E0686 rolling over in my script however v_v.