Just wanted to throw some updates in here as I continue to work on some new projects (that I'm not ready to go into detail on yet).
First off, I have gotten a proper frame counter working in Citra which as of now is merged into the latest nightly builds. This should hopefully be very helpful for anyone wanting to work on TAS stuff.
One issue I outlined in my Ocarina of Time 3D submission notes involves not being able to perform input on the first frame after loading a savestate. For whatever reason, this only seems to happen when using generic controllers for i.e. anything that isn't keyboard or a gamecube controller. I have still not figured out what difference between how these inputs are handled is causing this to happen, but I will continue to investigate. In the meantime, I have had a lot of success mapping keyboard inputs to a controller using Joy2Key. This won't work for analog circle pad or c-stick inputs, but it's a decent workaround for the time being.
Some users on discord expressed issues with desyncs previously, but I hadn't really experienced these issues until messing with a couple of other games recently. Thus far, these issues have only come about while running in New 3DS mode with games that take advantage of the c-stick and extra shoulder buttons. It's possible that there is another form of these desyncs that can occur, but this is the only type of issue I've run into. The good news, is that I have been able to easily and consistently correct every one of these desyncs that I have encountered with some minor hex editing. Basically, Citra movie files record different types of inputs at the same rate they are polled. This means that circle pad, button, and touch screen inputs are always recorded 234 times a second but the input group that handles the New 3DS inputs is usually polled at a different rate. It seems that sometimes the order at which these input types are recorded gets messed up. This means that the when the order should be BUTTONS -> TOUCH -> NEW 3DS, the order can occasionally be recorded as NEW 3DS -> BUTTONS -> TOUCH. I have not found the root of this issue, but seems to be caused by saving and loading states as I have never had it occur during a movie that was created without using savestates. Because of this and the fact that the polling rate for each input type should be consistent for any given movie regardless of the inputs used as long as it's the same game being played, the easiest way I have found to remedy this is to record a long movie with no savestates and then compare it to the movie file with the desync. Below is an example of what I look for:
Desync:
Some explanation on what you're seeing here - the first chunk of 7 bytes shown outlined in purple are the New 3DS inputs. You can tell this because this chunk starts with 05. The next chunk is the button inputs outlined in green which always starts with 00. Finally, touch inputs outlined in orange, starting with 01. This order these the inputs were recorded at at this specific point in the movie file can then be read as NEW 3DS -> BUTTONS -> TOUCH. However, in this particular situation, this order desyncs. When comparing this exact address with a movie file using the same game, this point in the movie file will look like this.
Sync:
The order of these three chunks of data have been rearranged. This will most likely result in the inputs during this frame being dropped any of the input types that were misplaced. In this example, the movie playback will be expecting button inputs but will instead read the New 3DS inputs. Then when it's expecting touch inputs on the next chunk, it will receive button inputs. So on and so forth. Fixing this desync is as easy as finding the area where the order of your movie file conflicts with the order of a base movie file which did not load savestates.
On the topic of hex editing, I have been working on a template to use with 010 Hex Editor that should make reading CTM files a little easier. It won't be nearly as useable as TAStudio, but it should be helpful for TASing. I'm still messing with making the template universal, but so far it only works for certain games. I'll post updates as I improve it.
The file on the left is TheRandomMaster's movie file. The one on the right is my test file. The blue and the red color coding is used to separate data into the frames they are relevant to. The area I have highlighted is frame 46 of both of our movie files. The short and simple of it is that each red chunk and each blue chunk should be identical in size and position between the two files. However, frame 46 in my movie file is 7 bytes longer than frame 46 in TheRandomMaster's. The reason for this is because my movie started recording data for accelerometer inputs slightly before TheRandomMaster's movie. I have no idea why this is happening. By inserting 02 00 00 00 FE 00 00 at 0x0AD8 (the accelerometer data Citra is polling for at this time in the movie file), I fix the desync and the TAS plays just fine. I have confirmed sync with this fix.
I have uploaded the fixed movie to my user files: User movie #638023496031275421
