The core issue ended up being something found on the libTAS discord. It's simply this line of code being wrong:
https://github.com/godotengine/godot/blob/3.2.1-stable/drivers/alsa/audio_driver_alsa.cpp#L190 as uint8_t* was the wrong type, uint16_t* was the correct one to use (that exact fix was later done in Godot anyways
https://github.com/godotengine/godot/commit/4bf141a47e9d3ee2d38347d643aeca907e4e602b).
The game runs on Godot 3.2.1, which predates this fix. I figured it should be possible to insert effectively a * 2 somewhere in the function's machine code to fix the issue. The way I found the function was probably more complicated than it needed to be. First, I compiled ALSA myself with -g (so debug symbols were available) and installed it on my system. Then I was able to simply run the game under gdb and put a breakpoint at snd_pcm_writei, and use the return address to figure out where the problematic function was located (this was the only callsite for this function, so this worked without much issue). With that, I had gdb spit out a disassembly the entire function, and with that I was able to deduce the assembly for calculating the second arg, and was able to figure out where I could place a * 2 for it.
The patch specifically changes a movsxd rsi,esi (48 63 f6) to add rsi,rsi (48 01 f6). The original opcode is effectively a no-op in this context if esi is positive (if it is negative it would crash regardless), so this provided 3 free bytes to form a * 2 opcode. And it so happens changing it to add rsi,rsi (a+a is the same as a*2) is just a 1 byte change here, so I used that.
This probably means modding the swf isn't really needed? For what it's worth, the only nondeterministic part would appear to be the beginning, there aren't any nondeterministic areas afterwards it would appear. Would it be preferred to not do this determinism hack due to the "slowly frame advance at the beginning" seeming to be a substitute for a clean swf?