Joined: 3/2/2010
Posts: 2178
Location: A little to the left of nowhere (Sweden)
This was definitely not the way I saw the project going when I decided I wanted to jump in and improve on nitsujas work. Most who are reading this are probably now asking "How did we get here?". I will try to cover that.
Final chapter serves kinda as a tl;dr, for you lazy people.
Basically, there was some information that I had overlooked. I received a tip on IRC regarding a book called Windows Internals, I believe it was Henke37 who suggested it, and this book is full of nightmares.
Besides showing me Hourglass isn't the way to go for re-recording Windows programs, it also turned me off on Windows entirely.
How bad could it really be?
There are some points discussed in this book that just sank my heart. Could windows be lying? No way! Yes way!
They have multiple shim-engines in their kernel that patches old programs at runtime to make them run better. They are also lying to debuggers about when a program has started, when DLLs have been loaded, etc.
This becomes problematic for a debugger such as Hourglass, since it needs to know these things when they happen, not when the kernel thinks it needs to know these things.
The straw that broke the camels back
To mitigate the troubles we'd need a kernel driver. This driver would poke at structures that the book Windows Internals doesn't describe (though, they are mentioned), and the Microsoft kernel debugger won't print full details of. This feels like a horrible idea, and also needs an additional computer as Windows can only be debugged over Ethernet for these things. I am not interested in buying another computer just for this.
This just to get an idea of when a program really begins to start, memory allocations, thread execution orders etc.
There would be drivers Microsoft aren't going to approve of. Which becomes annoying really fast since Windows 10 only accepts signed drivers, outside of "hidden" debug settings.
But, WinDbg has re-recording?
Yes, it turns out that WinDbg has added re-recording, and great for them. They avoid several issues Hourglass is facing by recording program state rather than input.
I did not research this a lot, so I am not sure how they solved some of the problems that we'd have in common with WinDbg, such as memory allocation handling.
"We're doomed! Doooooooooooooooomed!"
Not quite, there is at least one kernel that provides a lot of this control, and contains a fairly well documented API, the Linux kernel. There's also a program that enables the launching of Windows applications on Linux, Wine.
There's a project that I got to see the insides of that looks really promising in terms of making this happen.
Finally, I want to say thank you to everyone who contributed. I hope some of you will come back when the next project is up and running.
Joined: 4/17/2010
Posts: 11495
Location: Lake Chargoggagoggmanchauggagoggchaubunagungamaugg
There's also a way to grab a PC emulator with rerecording, install linux on it, and run windows games via wine. Or install windows right away. The problem is, there's no such emulator yet. But this is an option, we'll see which approach happens sooner, and which approach turns out to be more stable.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Joined: 3/9/2004
Posts: 4588
Location: In his lab studying psychology to find new ways to torture TASers and forumers
feos wrote:
The problem is, there's no such emulator yet.
MAME cannot do it?
Also, maybe we should consider VirtualBox-RR.
Warning: Opinions expressed by Nach or others in this post do not necessarily reflect the views, opinions, or position of Nach himself on the matter(s) being discussed therein.
Sounds like improving keylie’s Linux TAS tools ( https://github.com/clementgallet/libTAS ) then in the future TASing windows games by emulating them through TAS’d Linux is the way to go, lol
Thanks for the post mortem Warepire, I'm sure it comes at the end of many frustrating hours of work. Too bad Windows is such a mess, I was really hoping something revolutionary would come out of this project, thanks for trying though.
Joined: 4/17/2010
Posts: 11495
Location: Lake Chargoggagoggmanchauggagoggchaubunagungamaugg
Nach wrote:
feos wrote:
The problem is, there's no such emulator yet.
MAME cannot do it?
Also, maybe we should consider VirtualBox-RR.
My current mame-rr wasn't compiled with support of anything but arcades, and it's from 2010, so I dunno how well it works there even if I make such a build. And upstream mame doesn't have rerecording yet, I was planning to add it.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Joined: 3/2/2010
Posts: 2178
Location: A little to the left of nowhere (Sweden)
Nach wrote:
Also, maybe we should consider VirtualBox-RR.
VirtualBox-RR? I am not touching anything Oracle has had their hands in. Do you recall my whine about their MySQL drivers? Until someone proves otherwise this is how I see the Oracle development process: https://xkcd.com/1926/
(I've also used VirtualBox for a lot of things, the number of times it cost me my foot rather than won me anything is pretty high)
Patashu wrote:
Sounds like improving keylie’s Linux TAS tools ( https://github.com/clementgallet/libTAS ) then in the future TASing windows games by emulating them through TAS’d Linux is the way to go, lol
There are way better methods on Linux actually, the kernel is quite a bit more powerful in regards to these things than Windows is.
Joined: 4/17/2010
Posts: 11495
Location: Lake Chargoggagoggmanchauggagoggchaubunagungamaugg
Regarding VirtualBox-RR, recently we had an Hourglass submission created in a VB WinXP. It only synced for the author, but for none of the people who had an identical VB environment. It worked for me partially on native XP. I would very much try to avoid anything that heavily depends on the host hardware, even though the possibilities that running a modern system emulator would open are quite tempting.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Well if we get a much improved emulator then I would be able to TAS New 'n' Tasty and Abe's Exoddus with the advanced re-location glitch (ARG).
Enjoys speedrunning, playing and TASing Oddworld games!
Has TASed:
Oddworld: Abe's Oddysee in 12.06.13 (with Dooty)
Oddworld: Adventures II in 20.03.78 (with Dooty)
Oddworld: Abe's Exoddus 100% in 2:08:28.4 (with Dooty)
Oddworld: Abe's Oddysee 100% in 1:05:01.65
Oddworld: Abe's Exoddus in 37:18
Oddworld: Abe's Exoddus in 37:15
Oddworld: Abe's Exoddus 100% in 2:!5.44.12
Oddworld: Abe's Oddysee any% in 13:01.3
Oddworld: Abe's Oddysee any% in 12:59.95
Oddworld: Abe's Oddysee 100% in 1:04:16.27
Oddworld: Abe's Oddysee 100% in 1:04:01.07
Currently working on:
Waiting for Windows TAS Tools to work so I can TAS PC version of Exoddus.
Joined: 3/9/2004
Posts: 4588
Location: In his lab studying psychology to find new ways to torture TASers and forumers
Warepire wrote:
Nach wrote:
Also, maybe we should consider VirtualBox-RR.
VirtualBox-RR? I am not touching anything Oracle has had their hands in. Do you recall my whine about their MySQL drivers?
They're not actually from the same teams. MySQL was created by MySQL AB, originally a stand alone company. Then every major server vendor and huge social sites contributes all kinds of crazy patches.
VirtualBox was created by people at Sun Microsystems.
True, Oracle runs them both now, but it doesn't mean you should expect any similarities in their code.
feos wrote:
Regarding VirtualBox-RR, recently we had an Hourglass submission created in a VB WinXP. It only synced for the author, but for none of the people who had an identical VB environment. It worked for me partially on native XP. I would very much try to avoid anything that heavily depends on the host hardware, even though the possibilities that running a modern system emulator would open are quite tempting.
VirtualBox has settings as to whether you want to use the host hardware or not. Obviously anything tightly coupled to the host shouldn't be used. That's where Hourglass has also had issues, where games acted slightly differently depending on the video card used.
Warning: Opinions expressed by Nach or others in this post do not necessarily reflect the views, opinions, or position of Nach himself on the matter(s) being discussed therein.
Joined: 4/17/2010
Posts: 11495
Location: Lake Chargoggagoggmanchauggagoggchaubunagungamaugg
Nach wrote:
VirtualBox has settings as to whether you want to use the host hardware or not. Obviously anything tightly coupled to the host shouldn't be used. That's where Hourglass has also had issues, where games acted slightly differently depending on the video card used.
Well I can't check this particular case anymore, but the main question is, was VB even designed with determinism in mind? I think it takes a dedicated guru to dig into the code and do the checks, as well as through usage. But even though this environment would allow games more modern that PCem probably, I really doubt games that require modern hardware are going to work at tolerable speeds if at all.
By the time there appears someone to start this work, PCem might be already way ahead by what it supports. Also linux TASing is already a reality, and if Wine is possible to get to work, quite some Windows games also become available for TASing without speed decrease (unless you're on a VM of course).
Bottomline: there are several directions to explore, it just takes code maniacs.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
VirtualBox-RR? I am not touching anything Oracle has had their hands in. Do you recall my whine about their MySQL drivers? Until someone proves otherwise this is how I see the Oracle development process: https://xkcd.com/1926/
(I've also used VirtualBox for a lot of things, the number of times it cost me my foot rather than won me anything is pretty high)
Patashu wrote:
Sounds like improving keylie’s Linux TAS tools ( https://github.com/clementgallet/libTAS ) then in the future TASing windows games by emulating them through TAS’d Linux is the way to go, lol
There are way better methods on Linux actually, the kernel is quite a bit more powerful in regards to these things than Windows is.
Well, why not QEMU-RR then? It's Open Source and people claim that it's a lot better than VB or VMWare. There's a Windows port of QEMU that works kind of well, but it's not easy to use for a beginner. The Windows port have extremely limited documentation too. But still :)
Joined: 9/12/2014
Posts: 543
Location: Waterford, MI
Looking at this again, and I think its safe to say libtas is superior to hourglass at this point. Although running wine so far has desync problems, you could still slow down the game and use frame advance. Which is basically a tas, and could be useful for segmented runs.
Unfortunately, linux has its share of annoying issues as well similar to hourglass. Like running arch linux behaves differently at times and certain distributions cause some functionality problems. Having to swap operating to test 1 thing can be pretty annoying. And when programs update, they may not start anymore until fixed again.
I read the above message and understand why windows xp has the best functionality.
Would it be easier to just focus on windows xp only games? As in games that work with windows xp?
It really is a shame the project was abandoned. I can understand why with the whole kernel thing, but I don't have Linux, and I don't know a thing about virtual machines or anything like that. I wanted to make a TAS of a few Windows games without crashes or anything like that. Hourglass R81 is giving me a lot of crashes for IWbtG, and I really wanted to make a TAS of A Blind Legend. This is disappointing to say the least. I understand this is from a while back, but I was looking into downloading something better than R81, for Windows, but I just don't know what to do anymore.
DJ Incendration
Believe in Michael Girard and every speedrunner and TASer!
Shame project died. There was a leak of Windows source code which allowed to compile Windows server 2003 and contained nt5src and some thoughts on KernelSourceCode. Can`t it possibly revive the project?
Shame project died. There was a leak of Windows source code which allowed to compile Windows server 2003 and contained nt5src and some thoughts on KernelSourceCode. Can`t it possibly revive the project?
Not likely. We had a discussion about the leaked source code of Xbox OS which could be useful to improve emulation and eventually make TASing possible.
TASing is like making a film: only the best takes are shown in the final movie.
Shame project died. There was a leak of Windows source code which allowed to compile Windows server 2003 and contained nt5src and some thoughts on KernelSourceCode. Can`t it possibly revive the project?
Not likely. We had a discussion about the leaked source code of Xbox OS which could be useful to improve emulation and eventually make TASing possible.
I wanted to ask and wasn't sure if this was a good place.
I was interested in hourglass mostly for the save state functionality, basically being able to save the state of a game, and then load it, similar to how emulators do it.
I was wondering how possible it is to do this with windows or if anyone happens to know any projects like these around?
