Related page: Bizhawk/SavestateFormat
Excerpt from a Conversation on Sep 29, 2013
<natt> i haven't identified the source of the problem in the code yet, but what i'm seeing going on is
<natt> i made a movie, starts from savestate, then saved a backup of it
<natt> then i recorded more onto it per that user's directions, saved that, and am comparing the two
<natt> the second one
<natt> has two copies of the original savestate embedded in the movie file
<natt> and both copies are slightly mangled
<adelikat> that makes sense
<adelikat> the problem is the loading of the movie
<adelikat> it looks for headers it recognizes, and keywords, and input log
<adelikat> anything it doesn't recognize it just stuffs into "comments"
<natt> yeah, you've got it
<adelikat> when saving it dumps comments in there, and I"m guessing also does some logic to save a savestate in there too
<natt> yup
<natt> the mangling is because one particular line in the savestate for an NES starts with the letters "PAL"
<natt> so that gets eaten away as the header "PAL" line
<adelikat> yeah, that loading code isn't very smart
<adelikat> once upon a time, that's all it needed to do
<adelikat> but there was no pal, and whatever logic to save the state didn't exist
<adelikat> I literally cheated and just stuffed everything in as comments ^_^
<natt> other cores could conceivably have problems with other header values if they have a line in the savestate that coincidentally matches
<adelikat> yeah, ideally there is some notion of "hey we are in the savestate section, don't load this stuff as header info"
<natt> but since the savestate section adheres to no global rules about formatting or marking
<natt> well, i kind of like the ideas that i put together for the binary movie savestates
<adelikat> oh, you mean lsnes style?
<natt> i think it is lsnes style, isn't it
<adelikat> yeah, I like that too
<natt> right, so the current bizhawk, when you are recording a movie and the core is using binary savestates
<adelikat> it does simplify problems when a movie file is an archive of files
<natt> the savestates it creates are actually zip files
<natt> looking at one now, there are 4 lumps
<natt> "BizState 1.0", empty file, marks the format
<natt> "Core" is whatever the core wants to put in as a binary state
<natt> "Framebuffer" is a binary dump of the framebuffer, if so selected in user config to dump framebuffer to savestates
<natt> "Input Log" is the input log
<natt> as a text file, taken from the movie stuff
<adelikat> nice
<adelikat> movie files could be done simarly
<adelikat> "BizMovieState 1.0"
<adelikat> Savestate
<adelikat> InputLog
<adelikat> header
<adelikat> subtitles
<natt> honestly, looking at this, i doubt movies starting from savestates have ever been anywhere close to bulletproof
<adelikat> no, but they worked, enough
<adelikat> when there was only 3 cores to worry about ^^
<natt> i'll believe that, but i think it was a delicate arrangement doomed to break when someone sneezed
<adelikat> yup.
<adelikat> 1.5.0 was the tipping point for a lot of things
<adelikat> old solutions just didn't scale to this size
<adelikat> I think the real rolution is a lsnes style movie format, and to break all backwards compatibility
<adelikat> and after that, come up with some old format importer in the importer section
<natt> well, i'm willing to move forward with it, but i do think we should proceed slowly and carefully and thoughtfully
<natt> one thought i'm having is about repeated code
<natt> these new movies will be very similar to savestates in many ways
<natt> do we want to try to combine it all into one fatso code path?
<natt> to expand on the old "a savestate is a movie is a savestate" concept
<adelikat> yeah, that sounds reasonable
<adelikat> and I like the concept that it is both
<natt> the 1.0 reality is "a movie is almost a savestate is almost a movie"
<natt> so then it's one format with just possibly a different marker and different lumps
<natt> inputlog lump doesn't exist in non movie savestates
<natt> header lump doesn't exist in non movie savestates
<natt> framebuffer lump and core lump might not exist in movie
<natt> this would also be a chance to add "starts from SRAM" movies. i know you don't like them, and i'm not advocating for them really, just saying; if we do them, now is the time
<adelikat> well, my statement is actually "a movie savestate and a movie are the same file"
<adelikat> or at least work as both files
<adelikat> nah, I don't know what I"m saying
<adelikat> maybe we can throw in sram with an off switch?
<adelikat> so it is a matter of politics to not support them
<natt> although i do kinda like sram starts... it gives you some anchoring possibilities without the can of worms that is savestates
<adelikat> starting from sram is just as "dirty" in terms of possible hackery
<adelikat> and sram is basically a subset of savestate is it not?
<adelikat> anything contained in that sram is also contained in a savestate
<adelikat> but if this is an archive of files, it is trivial to do both
<adelikat> look for a savestate file in there, else look for a sram file
<natt> agree with just as dirty in terms of hackery
<natt> yes, sram is a subset of savestate (on all of the stuff we work with, anyway; could see that as being not true when you have mountable and removable memcards)
<natt> my thought was about ease of support across versions
<natt> the sram file format is usually rather simple
<adelikat> so my point is that it is no cleaner, and you can do everything with savestates that you can with sram
<adelikat> oh, that's a good point!
<natt> i made a movie, starts from savestate, then saved a backup of it
<natt> then i recorded more onto it per that user's directions, saved that, and am comparing the two
<natt> the second one
<natt> has two copies of the original savestate embedded in the movie file
<natt> and both copies are slightly mangled
<adelikat> that makes sense
<adelikat> the problem is the loading of the movie
<adelikat> it looks for headers it recognizes, and keywords, and input log
<adelikat> anything it doesn't recognize it just stuffs into "comments"
<natt> yeah, you've got it
<adelikat> when saving it dumps comments in there, and I"m guessing also does some logic to save a savestate in there too
<natt> yup
<natt> the mangling is because one particular line in the savestate for an NES starts with the letters "PAL"
<natt> so that gets eaten away as the header "PAL" line
<adelikat> yeah, that loading code isn't very smart
<adelikat> once upon a time, that's all it needed to do
<adelikat> but there was no pal, and whatever logic to save the state didn't exist
<adelikat> I literally cheated and just stuffed everything in as comments ^_^
<natt> other cores could conceivably have problems with other header values if they have a line in the savestate that coincidentally matches
<adelikat> yeah, ideally there is some notion of "hey we are in the savestate section, don't load this stuff as header info"
<natt> but since the savestate section adheres to no global rules about formatting or marking
<natt> well, i kind of like the ideas that i put together for the binary movie savestates
<adelikat> oh, you mean lsnes style?
<natt> i think it is lsnes style, isn't it
<adelikat> yeah, I like that too
<natt> right, so the current bizhawk, when you are recording a movie and the core is using binary savestates
<adelikat> it does simplify problems when a movie file is an archive of files
<natt> the savestates it creates are actually zip files
<natt> looking at one now, there are 4 lumps
<natt> "BizState 1.0", empty file, marks the format
<natt> "Core" is whatever the core wants to put in as a binary state
<natt> "Framebuffer" is a binary dump of the framebuffer, if so selected in user config to dump framebuffer to savestates
<natt> "Input Log" is the input log
<natt> as a text file, taken from the movie stuff
<adelikat> nice
<adelikat> movie files could be done simarly
<adelikat> "BizMovieState 1.0"
<adelikat> Savestate
<adelikat> InputLog
<adelikat> header
<adelikat> subtitles
<natt> honestly, looking at this, i doubt movies starting from savestates have ever been anywhere close to bulletproof
<adelikat> no, but they worked, enough
<adelikat> when there was only 3 cores to worry about ^^
<natt> i'll believe that, but i think it was a delicate arrangement doomed to break when someone sneezed
<adelikat> yup.
<adelikat> 1.5.0 was the tipping point for a lot of things
<adelikat> old solutions just didn't scale to this size
<adelikat> I think the real rolution is a lsnes style movie format, and to break all backwards compatibility
<adelikat> and after that, come up with some old format importer in the importer section
<natt> well, i'm willing to move forward with it, but i do think we should proceed slowly and carefully and thoughtfully
<natt> one thought i'm having is about repeated code
<natt> these new movies will be very similar to savestates in many ways
<natt> do we want to try to combine it all into one fatso code path?
<natt> to expand on the old "a savestate is a movie is a savestate" concept
<adelikat> yeah, that sounds reasonable
<adelikat> and I like the concept that it is both
<natt> the 1.0 reality is "a movie is almost a savestate is almost a movie"
<natt> so then it's one format with just possibly a different marker and different lumps
<natt> inputlog lump doesn't exist in non movie savestates
<natt> header lump doesn't exist in non movie savestates
<natt> framebuffer lump and core lump might not exist in movie
<natt> this would also be a chance to add "starts from SRAM" movies. i know you don't like them, and i'm not advocating for them really, just saying; if we do them, now is the time
<adelikat> well, my statement is actually "a movie savestate and a movie are the same file"
<adelikat> or at least work as both files
<adelikat> nah, I don't know what I"m saying
<adelikat> maybe we can throw in sram with an off switch?
<adelikat> so it is a matter of politics to not support them
<natt> although i do kinda like sram starts... it gives you some anchoring possibilities without the can of worms that is savestates
<adelikat> starting from sram is just as "dirty" in terms of possible hackery
<adelikat> and sram is basically a subset of savestate is it not?
<adelikat> anything contained in that sram is also contained in a savestate
<adelikat> but if this is an archive of files, it is trivial to do both
<adelikat> look for a savestate file in there, else look for a sram file
<natt> agree with just as dirty in terms of hackery
<natt> yes, sram is a subset of savestate (on all of the stuff we work with, anyway; could see that as being not true when you have mountable and removable memcards)
<natt> my thought was about ease of support across versions
<natt> the sram file format is usually rather simple
<adelikat> so my point is that it is no cleaner, and you can do everything with savestates that you can with sram
<adelikat> oh, that's a good point!