I don't participate in the things going on on this site, but I do have some ideas to share.
I think that there needs to be a universal movie format for the classic emulators. I have no idea what to call it ("Letting programmers name flagship applications makes as much sense as letting marketers write them").
Rationale:
- Different emulators for the same system have different movie formats.
- Similarly, the movie formats for different consoles are unrelated.
- The movie formats are frequently undocumented, and the emulator source code is frequently not published.
- Desynchronization with the movie stream is common and I believe much of it is avoidable.
- Current movie formats don't offer useful features such as subtitles and embedded cheat codes.
- It's basically impossible to play back current movies on actual hardware.
- Movies can be quite large unnecessarily.
- Security and authenticity.
I hope that these problems can be mostly solved by introducing a new movie format. Here are my high-level proposals:
- The format shall be formally documented in the hopes that various emulators would add support for it.
- The format shall be compatible across multiple consoles. By this I mean that the format's specifications will apply equally to NES and Genesis emulators equally, minus of course the system-specific stuff.
- Rather than encode the state of the controllers at each frame, I propose that instead the format record the state of the controllers at each read probe. The meaning of "frame" can be ambiguous, especially due to emulators' inexact timing, and with a frame encoding it's difficult to do playback on real hardware. Timing using the read probe is much more deterministic. For Nintendo's 2D consoles, the concept of "read probe" is well-defined; I don't know about the others.
- Controller data would be encoded as delta-time. The controller will remain the same state until otherwise changed. Each "line" of the recorded controller input shall count the number of read probes since the last update.
- Cheat codes can be embedded into the data stream. Cheat codes can consist of ROM patches, RAM patches, disk patches, and any other useful thing. Typically they will be enabled for the whole movie but this is not mandatory. This is designed to allow things like 150cc in Super Mario Kart. With all cheats explicitly noted in the file, auditing tools could be written to enforce submission requirements of bodies such as tasvideos.org.
- Saved states can be embedded into the stream at any point, including the beginning. This serves two purposes: special movies that don't start at the beginning, and checkpoints in the middle of a movie. Checkpoints allow rewinding. Periodic checkpoints embedded into the file automatically by a "rerecording" emulator will help the author during development. A "release" version of a movie could be made by stripping these.
- The delta-time stream shall support subchannels that are synchronized to the read probe time. Such data can include notes and on-screen subtitles.
- Security: The ability to digitally sign a movie is helpful. The format shall allow multiple signatures of the same file. An author can sign her movies to prove she made them. A site like tasvideos.org can sign a movie as being officially approved. Another use is as a timestamping authority: a neutral party can notarize that the movie was submitted by someone at a certain time, which might resolve disputes.
- Extensibility: The format shall be extensible. New tags can be defined that go into a data stream. The header shall specify which extensions are required for playback and which extensions are optional. A new controller type is a required feature; a subtitle extension is an optional feature.
What do you think?
Melissa
My thoughts:
- Different emulators for the same system have different movie formats.
Different systems have different input requirements. I see this as natural, and not really a problem.
- The movie formats are frequently undocumented, and the emulator source code is frequently not published.
That doesn't seem to be true, at least here. Source is usually given out at least when the release is first out (would be nice to have a page here hosting these emulators / source to keep it current though...), and there are FCM, SMV, GMV, VBM pages here as well which are quite helpful.
- Desynchronization with the movie stream is common and I believe much of it is avoidable.
Because of differing formats between consoles? I don't follow.
- It's basically impossible to play back current movies on actual hardware.
How would a universal format reach toward this goal? And... so what?
- Movies can be quite large unnecessarily.
Adding in things like cheat codes, subtitles, and especially savestate checkpoints will make this a much larger problem. Personally I think movie files are a very nice size, especially compared to avi.
- Controller data would be encoded as delta-time. The controller will remain the same state until otherwise changed. Each "line" of the recorded controller input shall count the number of read probes since the last update.
Is this the same thing FCM does? If so then I think it's a very bad idea, unless some great hex-editing tools were designed as well (and I don't just mean some converter either).
So to sum up, I personally don't see the need for it, at least at this point. It also doesn't help that the emulation updates are supported by the site in any way, as it only further encourages individual attention to them, which will drive them further apart.
<Swordless> Go hug a tree, you vegetarian (I bet you really are one)
Sounds interesting, though perhaps excessive for the needs of the community. I'm more than likely the wrong person to comment here, but it just doesn't seem likely that there are going to be "new" emulators being supported in the near future that could benefit from such a standard (i'm guessing nitsuja/bisqwit would know best here).
If, however, this is something that blows peoples skirts up (so to speak), i'd be happy to help :)
Different systems have different input requirements. I see this as natural, and not really a problem.
My format would make the input data consistent for a given controller type on a given console. Ideally, ZSNES and SNES9x would be able to read and record the same movie format.
That doesn't seem to be true, at least here. Source is usually given out at least when the release is first out (would be nice to have a page here hosting these emulators / source to keep it current though...), and there are FCM, SMV, GMV, VBM pages here as well which are quite helpful.
I've never done any TAS stuff before, so I assume you're correct.
Because of differing formats between consoles? I don't follow.
I mean the sorts of problems people have where someone's movie won't play back correctly because it's the wrong emulator version or whatever. I'm just guessing really, but I think timing changes to the emulator could be partly at fault for this. A situation where a game barely avoids missing a frame could turn into missing a frame if the emulator timing changes, and this is catastrophic for movie playback.
A format that times the recorded controller to the controller port probe signal is easier to get correct.
How would a universal format reach toward this goal? And... so what?
It's not the universal format that would allow hardware playback, it's the redesign to use the controller port probe mechanism. On NES and SNES at least, the controllers receive a signal to read their button states. By timing itself to this mechanism, a hardware device simulating a controller would easily know what data to send to the console. (Presumably, you'd put a filtered version of the movie file into the device's memory.)
Adding in things like cheat codes, subtitles, and especially savestate checkpoints will make this a much larger problem. Personally I think movie files are a very nice size, especially compared to avi.
Cheat codes and subtitles are very small compared to the size of the controller data. You are definitely right about the save states; however, the reason for the embedded saved states is for the author's benefit in the movie creation process. Some tool would be used to strip out the saved states in the data stream to make a "release" version of the movie for publication on this site.
Is this the same thing FCM does? If so then I think it's a very bad idea, unless some great hex-editing tools were designed as well (and I don't just mean some converter either).
I don't know what FCM does. My format is clearly not designed for direct editing; some tools would definitely be necessary. Perhaps a parallel XML-based format could be made that could be manually edited as required.
It also doesn't help that the emulation updates are supported by the site in any way, as it only further encourages individual attention to them, which will drive them further apart.
I don't know what the "emulation updates" you're referring to are.
Melissa
I recommend reading this discussion about a new format for Genesis movies:
http://tasvideos.org/forum/viewtopic.php?t=3995
... and the specifications we have worked out during it:
http://tasvideos.org/GM2.html
It includes some of your ideas: subtitles, savestates for rewinding, extensibility.
Some other comments:
>The format shall be formally documented in the hopes that various emulators would add support for it.
See http://tasvideos.org/FAQ.html#emulators_ -- all the movie formats are reasonably well documented.
>Controller data would be encoded as delta-time.
FCM does this and everyone hates it, because it's practically impossible to hex edit the movie files. The movies are rather small anyway, and compress very well. Emulators can read ROMs in zip files, so there's no reason why they shouldn't be able to read movies in zip files.
>Rather than encode the state of the controllers at each frame, I propose that instead the format record the state of the controllers at each read probe.
This sounds like a good idea in theory at least. M64 (Mupen movie file) does this but has still had a fair share of desynch problems, but perhaps they are unrelated.
Hmm, I fail to see the point of a global movie format. Is your plan to have a video that is recorded on SNES9x play back properly in zsnes and on the console? Heck, the emulators have a hard enough time keeping sync running a video back using it's own cores, let alone a trying to play back movies on other emulators. It's just not feasible. It's not the movie format which is the causes these problems...
Also each system has a different input method, so you will have to store the input data differently for each system. That leaves the header being the only thing which remains more or less constant. However each emulator has to store different information in the headers for it to know which settings to play the movie with. As a result there is differences in the headers too. To make a global header you are going to need some sort of bloated catch all set of fields which each emulator has pick it's way through to see which fields apply to it.
While it's an interesting idea I don't really see it working. As it stands the current movie formats we use are well documented and for the most part are very similar. Their differences tend to be where it is needed to add something for playback on the specific emulator.
My format would make the input data consistent for a given controller type on a given console. Ideally, ZSNES and SNES9x would be able to read and record the same movie format.
The problem I see with this is that, while they might be recording and reading the same format, the generated movie files won't be interchangeable. In other words, a movie created with zsnes will have a high probability of desyncing in snes9x (even though it will work in zsnes) and the other way around.
This is because, as far as I know, different emulators emulate the console differently. They will have lag frames here and there, and it will depend on the emulator exactly where. These don't make any difference when playing normally, but with a recording, where perfect frame-by-frame replication is necessary, it makes a big difference.
Thus we would have one format which is still emulator-specific. This can only add to the confusion because there's no external way of telling which emulator was used to create the recording. With the differently-named formats it's at least possible to differentiate. (Of course nothing would stop the emulator from naming the files differently, but then, what's the point? What would be the advantage?)
I mean the sorts of problems people have where someone's movie won't play back correctly because it's the wrong emulator version or whatever. I'm just guessing really, but I think timing changes to the emulator could be partly at fault for this. A situation where a game barely avoids missing a frame could turn into missing a frame if the emulator timing changes, and this is catastrophic for movie playback.
A movie format won't change this. It would require a change in all the emulators to make them emulate in an identical way.
A format that times the recorded controller to the controller port probe signal is easier to get correct.
I don't see how that would solve the problems. Usually desyncs happen because of changing lag frames: Keypresses have been recorded on a per-frame basis, and if a lag frame gets wrongly emulated the second time, the recording will obviously desync because it was created with a different (or lacking) lag frame position. I don't see how timing the keypresses in some other way could solve this problem: If a lag frame which was/wasn't there previously is introduced/removed during playback, the movie will desync. It doesn't matter how the keypresses have been timed.
I don't know what FCM does. My format is clearly not designed for direct editing; some tools would definitely be necessary. Perhaps a parallel XML-based format could be made that could be manually edited as required.
I thought your idea was to save space, not to bloat it. :P
>I don't see how that would solve the problems. Usually desyncs happen because of changing lag frames: Keypresses have been recorded on a per-frame basis, and if a lag frame gets wrongly emulated the second time, the recording will obviously desync because it was created with a different (or lacking) lag frame position.
It possibly solves the problem if the console does not read controller states during the lag frame, like Myria explained. There could be 10 lag frames on one emulator and it would still only read the next input after the screen updated next time, like an emulator which didn't lag at all.
This is assuming that no games probe the controller during lag, and that the state after a frame of lag is the same as if the system hadn't lagged at all. Which to me sounds doubtful, but I have no knowledge of consoles' inner workings. Perhaps someone experienced with emulator cores can help?
- The movie formats are frequently undocumented, and the emulator source code is frequently not published.
this isn't true, the formats used on this site are documented, and the related emulators are open source.
famtasia being the exception, but famtasia wouldn't support the new format anyway.
- Desynchronization with the movie stream is common and I believe much of it is avoidable.
which is true, but that's a problem in the emulator core, not in the movie format. The universal format wouldn't help.
- Rather than encode the state of the controllers at each frame, I propose that instead the format record the state of the controllers at each read probe.
that's ok if there's only one read probe per frame, or per every few frames. If a game is lazy and probes the input a couple of times per frame, that'd make the movie file grow too fast.
a better suggestion might be: "only record the controller state in frames, where the state has been read at least once". But then again, why? Filesize is not an issue (see below), and it'd make hex-editing more difficult. Thinking that a poll-based input recording would allow playback on real consoles is wrong as well - desyncs don't happen because the input doesn't match, but because the machine behaves differently.
Adding a display, whether the controller state has been read, would be useful to detect lag-frames, but that's not a problem of the movie format.
I'm also sceptical - do we really want to be able to play back TAS-input on a real console? That'd pretty much destroy unassisted speedrunning.
Controller data would be encoded as delta-time.
why? it makes hex-editing more difficult. Most of the overhead disappears when you zip the movie file prior to distribution anyway.
- Saved states can be embedded into the stream at any point, including the beginning. [..] and checkpoints in the middle of a movie.
a feature that adds checkpoints to a movie to allow seeking would be nice, indeed. In terms of filesize, these checkpoints should be kept outside the movie file though. The small movie file can be distributed, and the emulator will automatically create new savestates for seeking when playing the movie for the first time.
- The delta-time stream shall support subchannels that are synchronized to the read probe time. Such data can include notes and on-screen subtitles.
to keep the file structure simple, those should probably go into a seperate file. subtitle formats are well established. I don't see any advantage of using read probe synchronisation over simple timestamps though.
there could also be meta-information about "chapters", e.g. "frame 12345: beginning of Kraid Fight". The emulator would create additional savestates on those frames, and allowed skipping right to them.
Those ideas aren't new, but require an amount of work that nobody has done yet.
- Security: The ability to digitally sign a movie is helpful. The format shall allow multiple signatures of the same file. An author can sign her movies to prove she made them. A site like tasvideos.org can sign a movie as being officially approved. Another use is as a timestamping authority: a neutral party can notarize that the movie was submitted by someone at a certain time, which might resolve disputes.
you can always take a movie from someone else, and sign it as your own. Over here, the "neutral party" for submission time is simply the submission script.
A movie is officially approved by tasvideos.org when it's available for download here.
Adding a signing function is possible, but I fail to see the uses.
My format would make the input data consistent for a given controller type on a given console. Ideally, ZSNES and SNES9x would be able to read and record the same movie format.
as said above, that would be useless. You still can't play back a zsnes movie on snes9x, because of subtle emulation differences. Conversion tools for different emulator movie files exist, but they're mostly useless.
same goes for different releases of the same emulator: if the emulator core's timings changed, no movie format can guarantee that it'll stay desync-free.
It possibly solves the problem if the console does not read controller states during the lag frame, like Myria explained. There could be 10 lag frames on one emulator and it would still only read the next input after the screen updated next time, like an emulator which didn't lag at all.
The lag is introduced precisely because the CPU is doing something. What it is doing during the lag may well affect subsequent frames. In other words, subsequent frames may be different depending on the lag. That is, lag frames are not necessarily (nor probably) no-ops, but they may well affect what happens next. Thus if what happens next is different, the input will be invalidated (ie. will desync) regardless.
Joined: 2/28/2006
Posts: 2275
Location: Milky Way -> Earth -> Brazil
Also, you wouldn't know what is that movie file about, since all will have the same extension...
"Genuine self-esteem, however, consists not of causeless feelings, but of certain knowledge about yourself.
It rests on the conviction that you — by your choices, effort and actions — have made yourself into the
kind of person able to deal with reality. It is the conviction — based on the evidence of your own volitional
functioning — that you are fundamentally able to succeed in life and, therefore, are deserving of that success."
- Onkar Ghate
Bisqwit wrote:
Joined: 3/9/2004
Posts: 4588
Location: In his lab studying psychology to find new ways to torture TASers and forumers
ZMVs support most of what was requested here.
Warning: Opinions expressed by Nach or others in this post do not necessarily reflect the views, opinions, or position of Nach himself on the matter(s) being discussed therein.
Joined: 2/28/2006
Posts: 2275
Location: Milky Way -> Earth -> Brazil
When will you finish ZSNES...? You know how everyone hates SNES9x.
"Genuine self-esteem, however, consists not of causeless feelings, but of certain knowledge about yourself.
It rests on the conviction that you — by your choices, effort and actions — have made yourself into the
kind of person able to deal with reality. It is the conviction — based on the evidence of your own volitional
functioning — that you are fundamentally able to succeed in life and, therefore, are deserving of that success."
- Onkar Ghate
Bisqwit wrote:
Joined: 3/9/2004
Posts: 4588
Location: In his lab studying psychology to find new ways to torture TASers and forumers
JXQ wrote:
Adding in things like cheat codes, subtitles, and especially savestate checkpoints will make this a much larger problem. Personally I think movie files are a very nice size, especially compared to avi.
ZMV supports all that and is smaller than SMVs for the same movie in the tests I ran.
Warning: Opinions expressed by Nach or others in this post do not necessarily reflect the views, opinions, or position of Nach himself on the matter(s) being discussed therein.
Joined: 2/28/2006
Posts: 2275
Location: Milky Way -> Earth -> Brazil
So, can people start using ZSNES now?
"Genuine self-esteem, however, consists not of causeless feelings, but of certain knowledge about yourself.
It rests on the conviction that you — by your choices, effort and actions — have made yourself into the
kind of person able to deal with reality. It is the conviction — based on the evidence of your own volitional
functioning — that you are fundamentally able to succeed in life and, therefore, are deserving of that success."
- Onkar Ghate
Bisqwit wrote:
Joined: 3/9/2004
Posts: 4588
Location: In his lab studying psychology to find new ways to torture TASers and forumers
Wait for the 1.5 release. See news here: http://zsnes.com/index.php?page=news
Although it'll probably need some touching up before the masses will want to use it.
Warning: Opinions expressed by Nach or others in this post do not necessarily reflect the views, opinions, or position of Nach himself on the matter(s) being discussed therein.
Adding in things like cheat codes, subtitles, and especially savestate checkpoints will make this a much larger problem. Personally I think movie files are a very nice size, especially compared to avi.
ZMV supports all that and is smaller than SMVs for the same movie in the tests I ran.
You're saying a ZMV with savestate anchors is small than what - SMV with savestate anchors as well, or SMV in general?
<Swordless> Go hug a tree, you vegetarian (I bet you really are one)
Joined: 3/9/2004
Posts: 4588
Location: In his lab studying psychology to find new ways to torture TASers and forumers
SMV in general. Even if the ZMV has ZSTs in it, and the SMV doesn't have any save states in it.
Of course there's cases where this doesn't hold true, but it's true for your average movie of reasonable length.
Warning: Opinions expressed by Nach or others in this post do not necessarily reflect the views, opinions, or position of Nach himself on the matter(s) being discussed therein.
Joined: 3/9/2004
Posts: 4588
Location: In his lab studying psychology to find new ways to torture TASers and forumers
JXQ wrote:
Just looked at ZMV.html. It seems that this is true because it's like FCM - it updates only on frames that controller states change. Groan.
No groan or I smack you.
Enhanced ZMV Parser 1.16 Copyright (c) 2005 ZSNES Team
Usage: ezp <params>[<value>] <zmv filename>
<Params>
-co: - chapter offsets sorted by ascending frame #
-cd: - interactive deletion of chapters - not implemented yet
-fc: - pad input changelog
-ff: - pad input full log
-fi value: - pad input for frame #value
-as letter: - auto pad #letter input subtitle generator
Note: the -fc/ff params output should be piped into less or cat
Default: no params - only displays basic header/footer info
RLE can easily be decompressed and recompressed.
Warning: Opinions expressed by Nach or others in this post do not necessarily reflect the views, opinions, or position of Nach himself on the matter(s) being discussed therein.
So can non-RLE, like SMV. The difference is that every time I hex-edit, I don't have to do two stupid conversions that I hope keep all the metadata intact.
GROAN. *dodges*
<Swordless> Go hug a tree, you vegetarian (I bet you really are one)
Joined: 2/28/2006
Posts: 2275
Location: Milky Way -> Earth -> Brazil
Yeah... that would force us to blackmail Zefiris, to make him improve his TASEdit.
"Genuine self-esteem, however, consists not of causeless feelings, but of certain knowledge about yourself.
It rests on the conviction that you — by your choices, effort and actions — have made yourself into the
kind of person able to deal with reality. It is the conviction — based on the evidence of your own volitional
functioning — that you are fundamentally able to succeed in life and, therefore, are deserving of that success."
- Onkar Ghate
Bisqwit wrote:
