I think this might be an excellent idea!
We already have movie files that usually store the held buttons for every single frame. You could read such a file and as a first step translate the button data into lua-code like this
frame 55: hold B for 800 frames
hold RIGHT for 800 frames
frame 88: hold A for 8 frames
frame 150: hold A for 12 frames
frame 855: hold LEFT for 1 frames
frame 856: hold RIGHT for 1 frames
frame 857: hold LEFT for 1 frames
frame 858: hold RIGHT for 1 frames
->
hold(55,800,"b")
hold(55,800,"right")
...
Then in a second step, you could apply some game-specific translations. This would maybe require an AI to identify what's going on on the screen atm or some user input to work well (for context specific glitches, etc).
frame 55: Run to the right for 800 frames
frame 88: Jump with strength 8
frame 150: Jump with strength 12
frame 855: Wobble for 4 frames
->
--game specific part
function run(length, direction)
hold(frame,length,"b")
hold(frame,length,direction)
end
.....
--movie specific part
frame = 55
run(800,"right")
frame += 33
jump(8)
.....
I think that once a movie has been successfully converted in this way, editting it should become infinitely easier already.
Then you could maybe even make the lua script display the commands on screen, maybe in an abbreviated form, instead of raw button presses:
run ->
run ->, jump 8
run ->
run ->, jump 12
run ->
wobble
This third step would make it easier for somebody who watches a TAS to identify what's going on.
If you'd be able to look at and edit the results of the first step next to the emulator window, it'd probably make TASing a bit easier already. If you manage to do the same for the second step, (maybe with some nice visualisation/gui), it'd essentially be macros on a piano roll. Now, that'd have the potential to considerably improve TASing efficiency already imo.
Smart macros would obviously be the next step, but I think that even without them this'd be a pretty awesome idea!