Experienced Forum User, Published Author, Player
(125)
Joined: 6/26/2018
Posts: 157
Alyosha wrote:
Do you think working with original hardware from the time period would give better results? Would it be harder or easier to work with?
That's a thought. An old graphics card will likely be needed at some point, to get an actual 70 Hz signal for games of the era. An older, slower CPU may better line up intra-frame timings. A rotating hard disk, I would guess, would be more variable in timing than an SSD, but I've been trying to avoid games with loads from disk anyway.
A next step is probably to start eliminating variables. I'm thinking it would be good to program a custom "game" that is well-behaved: no randomness, no video mode changes, no filesystem I/O, just a very simple loop locked to vsync, with visual feedback. Like a single pixel that you control with the keyboard. The screen could draw a predetermined path, and the input sequence would try to make the pixel follow the path. That would make it easy to see, with 1-frame resolution, whether current setup is able to sync at all under ideal conditions, and if not, whether it fails to sync in a consistent way.
Here, you can see some failed attempts at syncing Dangerous Dave. Seek to 8:10 for the attempt that gets the farthest, into level 3. The farthest I think I've ever seen it get is the end of level 3, stopping just short of the door.
Link to video
There are two things I want to talk about:
It's not immediately apparent in the video, but the game is running slower than it should. This is because the game loop is locked to the screen refresh, and this computer's video card outputs a 60 Hz signal despite the game using mode 13h, which is supposed to be 320x200@70Hz.
The refresh rate discrepancy should be okay, because the Arduino playback device reads the VGA VSYNC signal and adjusts its own timing to the vertical refresh, rather than using absolute timestamps. But it still doesn't work.
60 Hz video output
My preliminary attempts were very far from syncing, more than can be explained by the gain or loss of a few milliseconds. The cause was unexpected to me: the video signal output by the PC's VGA port had a refresh rate of 60 Hz, not 70 Hz as it should have had. I checked this by enabling an option on my monitor to show the current video mode. You can also verify it by counting cycles of the trophy animation in level 1 of Dangerous Dave. On hardware the animation runs slower than in emulation with DOSBox.
Every computer and graphics card I had at hand had the same problem, outputting a 60 Hz signal for graphics modes that are nominally 70 Hz:
Many LCD monitors from the recommended time period already have digital DVI inputs that can provide better image quality than the analog VGA input. Still, using DVI is not recommended for DOS games because refresh rate issues may appear where VGA modes run at 60 Hz refresh rate instead of 70 Hz standard. A typical indication are speed and/or sound issues in games and demos that use the refresh rate for time synchronization.
Support for Mode 13h, and especially for undocumented VGA tweaks like those used by Doom, has dropped to near non-existence in modern video hardware and operating systems.
The problem arises under modern machines that do not actually support mode 13h - this support is becoming exceedingly rare, especially on newer ATi video cards, some of which will not even set into ANY screen mode lower than 640x480.
TLDR: I patched the VGA BIOS ROM of my 9600 Pro to output 1280x1024@70.08 Hz instead of the VESA default of 75 Hz.
I would have assumed that playing games on hardware under FreeDOS was a more authentic experience than playing in emulation, but that is not necessarily so. Whatever Crystal Caves does with the video mode fails in an especially noticeable way: it's like the top row of pixels gets stretched over the whole screen. I suppose the only remedy is to track down a vintage video card that implements the right timings. I'm curious what all-in-one games-oriented hardware packages like the weeCee (LGR video) do with video timing.
VGA VSYNC to the Arduino
I altered the Arduino circuit and program to be able to read VGA VSYNC signals. I connect a Y splitter to the PC's VGA port. One end of the splitter goes to the monitor and pin 14 (the VSYNC pin) of the other end goes to one of the Arduino's digital inputs.
The JRSR format does not store video refreshes, only nanosecond timestamps. But you can often infer where the video refreshes are, because of how authors tend to make TASes: hit frame advance, enter inputs for the frame, then hit frame advance again. Most input events occur at times that are multiples of the video refresh interval, plus a constant. The constant depends, usually, on non-frame-aligned activities at the start of the movie, the boot process and DOS prompt.
For example, here are the event timestamps from [1718] DOS Dangerous Dave by Ilari in 06:32.80, modulo the VGA refresh interval of 14.268 ms (= 800 × 449 / 25.175 MHz). Notice how most timestamps are aligned at an offset of about 9.507 ms. There is a secondary line of timestamps about 1.3 ms after the primary one—this happend when there are two events in the same frame.
Similarly, there are strong primary, secondary, and tertiary timing alignments in the events of [3706] DOS Crystal Caves: Volume 1 - Troubles with Twibbles by DungeonFacts in 20:33.15. But there is also a substantial minority of events that are not so well aligned. I looked into this case—the unaligned events are where 200 ms timed advance was used, rather than frame advance.
In contrast, the timestamps of [2119] DOS Avoid the Noid by turska & Ilari in 03:33.52 are mostly not aligned to the VGA refresh. This suggests that this TAS was not made using frame advance. (Another reason for a lack of alignment could be an incorrect refresh rate, but I'm pretty sure this game is 70 Hz like the rest.)
I wrote a program, guess-vsync, that tries to infer VSYNC parameters, given the event timestamps in a JRSR file. With these VSYNC parameters, the program gen-table produce a table of input events; not only RELEASE, PRESS, and WAIT events, but also VSYNC which means to wait until the next VSYNC pulse from the VGA port. Having a reference for the PC's timing, the Arduino should (in principle) be able to cope with timing variations, like the 60 Hz for 70 Hz I talked about above. The source code of these programs is available in 20220124-dave-pc-tas-verification-attempts.zip, or from git clone https://www.bamsoftware.com/git/pc-tas-arduino.git.
Here's what I did to program the Arduino for the video above. First, run gen-table, just to get a timestamp for when the boot-related events are over and we're ready to start the program. We will pass this timestamp to later commands as the argument to the --skip option.
From this, we see that the timestamp of pressing Enter to start the program is 493728396ns. Now we run guess-vsync, telling it to ignore the events before that timestamp:
$ ./guess-vsync --skip 493728396ns ilari-ddave.jrsr
14268123.1380338ns+9507455.38628574ns
average delay between VSYNC and event: 0.386ms
The interval+offset 14268123.1380338ns+9507455.38628574ns is the guess for the VSYNC parameters. We'll tweak this guess, appending -10%, to shift the VSYNC phase earlier by 10% of a frame—this is because we don't want our inputs to occur at the exact same time as a vertical refresh. Now run gen-table again, this time passing the VSYNC parameters and the skip timestamp, and save the output to playback/table.h.orig:
We are not quite done yet. The timing of events is most uncertain at the beginning of the movie. I manually tuned the timing of the first few events and saved it as playback/table.h. Then I uploaded the program to the Arduino.
But it still doesn't work
As the video proves, all these considerations are still not enough for successful verification. Here are some of the things I've tried:
Scheduling inputs early in the frame (--vsync interval+offset-10%)
Scheduling inputs late in the frame (--vsync interval+offset+10%)
Removing WAIT events and using only VSYNC for timing
Different debounce parameters in the VSYNC interrupt handler
I've noticed the first run after a reboot is likely to desync earlier. My guess as to the cause is that the program is slower to start up when the disk cache is cold. Timings are affected by disk delays more than I would have thought. The disk image of #2961: Ilari's DOS Dangerous Dave in 06:32.80 contains only a single file, DAVE.EXE. Once, I mistakenly ran the input file in the emulator with a disk image containing three files: DAVE.EXE, DSCORES.DAV, and EGADAVE.DAV. The presence of those two extra files was enough to desync emulator playback in level 2, despite the fact that the main program does not even open those other files by that point (I think).
I started the movie with pressing Enter at the DOS prompt. This is problematic because it means there is a video mode change in the middle of the event stream. I had initially tried to start playback at the title screen, or at the beginning of the first level, but that was worse. There's a sort of "frame rule" at the beginning of a level, while Dave is blinking. He must finish a blinking cycle before he can begin moving, which means that if you start the playback at a random point in the blinking cycle, the inputs that follow will be off by a random amount.
I am pretty naive about electrical engineering. To give you an idea: I didn't know what a pull-down resistor was until a few months ago when I had to wire up a pushbutton for the first iteration of the playback device. It's possible that the circuit I'm using for reading VSYNC signals doesn't make sense, or is susceptible to noise that causes timing errors. I'm not sure it's right that the VSYNC GND pin is not used, nor that the VGA input and the rest of the circuit do not have the same ground reference potential. Here is the circuit:
arduino-verification-circuit-20220124.cddx from https://www.circuit-diagram.org/
I'm using the attachInterrupt interface to react to changes on the VSYNC input pin; i.e., attachInterrupt(digitalPinToInterrupt(VSYNC_PIN), vsync_isr, FALLING). I've read that attachinterrupt goes through a lot of abstractions that add overhead, and that for high performance you need to program interrupts at a lower level. But the overhead estimates I've seen are in the realm of single-digit microseconds, which seems like it shouldn't matter much for VGA vertical refresh, which is in the realm of milliseconds. I think that the USB HID key events can be no more precise than 1 ms, anyway.
Experienced Forum User, Published Author, Player
(125)
Joined: 6/26/2018
Posts: 157
adelikat wrote:
media links have mostly been moved to files. All wiki pages should be updated accordingly, and forum posts (that had non-broken images before go-live
Thanks, that's great. One small bug: revision 71 of Text Formatting Rules fixes the example markup to refer to files.tasvideos.org, but revision 70 had already removed the live markup for the image.
Experienced Forum User, Published Author, Player
(125)
Joined: 6/26/2018
Posts: 157
feos wrote:
We just don't link to Archive items, only directly to files. So maybe this part can be improved.
I like this idea. I have often wished that the archive.org links went to the item's /details/ page, not a direct /download/ URL. (I usually "Copy Link", paste, and manually edit the URL.) The details page not only has easy download and torrent links, but also favorites and review posts, and automatically derived lower-bandwidth video files you can play in the inline player.
There's an opportunity for archive.org to serve even more of an archival purpose, by hosting emulator input files alongside the video, since an item can contain more than one file.
Like MUGG, I used torrents a long time ago (I have a memory of watching a Mega Man X run while the video was half downloaded and enjoying the resulting video glitches), but I have not used them in years.
Experienced Forum User, Published Author, Player
(125)
Joined: 6/26/2018
Posts: 157
There are a number of wiki pages and forum posts that load images from media.tasvideos.org, many of which are now broken. For example, at Wiki: TextFormattingRules#ReferencesLinks:
Experienced Forum User, Published Author, Player
(125)
Joined: 6/26/2018
Posts: 157
TiKevin83 wrote:
I'm looking into next whether the player can stream from resources on archive.org. It should actually be possible to use this for cross domain resources if they send the correct CORS response, so I might have options to play back streaming video if from sites that supply the right CORS info.
I have found that archive.org sends CORS responses for video files if you request from cors.archive.org and use a /cors/ path prefix.
Start from the item's details page:
I don't remember how I figured this out. I cannot now find where it is documented. Strangely, in my experience, for audio files, you get access-control-allow-origin: * even with a normal archive.org/download URL; it's only video files that need cors.archive.org/cors.
Experienced Forum User, Published Author, Player
(125)
Joined: 6/26/2018
Posts: 157
It seems to me that good candidate games would not be too long, not do much loading from disk, have a low frame rate, not use subframe inputs, and not use the real-time clock (this last criterion eliminates the Commander Keen games).
From a skim of the DOS movies list, some good first runs to try could be
I don't know whether fast-paced adventure games like [4391] DOS Hero's Quest by c-square, mrprmiller & davidtki in 00:40.90 would be good or not. Because something is happening on nearly every frame, even a small inaccuracy could desync the run. On the other hand, they are short in real time, and may buffer keyboard inputs, providing some resilience against timing inaccuracies.
The Arduino is wired to a simple circuit with a button to begin playback. I copied the circuit from the Arduino KeyboardMessage tutorial. I paid about USD 40 all together for the Arduino, breadboard, pushbutton, jumper wires, etc.
I made some changes to the original JRSR file, for convenience or to get the run to sync. You can see the changes by diffing tas/comic1.jrsr (original) and tas/arduino-verification/comic1.jrsr (modified) in the Git repo. Apart from the first two changes in the list, the modified file still syncs in JPC-RR.
I was not interested in playing back the boot sequence, so I trimmed the inputs up to the title screen. You have to boot the computer and run COMIC.EXE yourself, then press the button to begin playback.
I shortened the delay between the title screen and the first game inputs by about 165 ms (equivalent to 1.5 game ticks). This is for sync; without it, Comic always made the first jump too late. I attribute this to the real laptop running the game's initialization code faster than the emulated CPU does.
After the first death in CASTLE0, I moved the release of the Right key to happen about 1 s earlier. I don't know why this was necessary, but without it, the game would not register this particular key release and would desync.
The original JRSR file continues holding the Left and Space keys at the end of input. On a real PC, this led to automatic keyboard repeat and annoying beeping at the high score screen. I added release events for these keys at the end of the input.
Even with the above changes to the JRSR file, the playback does not sync every time. At the archive page, you can see all 11 attempts I made, of which 3 synced to the end. The embedded video above is attempt #7.
I attribute the sync failures to a lack of synchronization between the externally timed keyboard inputs and the game loop. In the emulator, inputs occur at certain times relative to the game's tick cycle. Because the USB keyboard inputs are not locked to the game loop, they may occur earlier or later in each tick, perhaps sometimes even splitting inputs that are meant to be in the same tick across adjacent ticks. Captain Comic is a forgiving game in that it has ticks of long duration (110 ms). The same technique may not work directly with other games that require more precise timing—though an idea for the future is to connect the VSYNC pin of the laptop's VGA port to the Arduino, as a timing source for games whose game loop is locked to the screen refresh.
Experienced Forum User, Published Author, Player
(125)
Joined: 6/26/2018
Posts: 157
I initially felt apprehension when I heard about the rewrite project. My experience with redesigns and migrations is that they can have the discouraging side effect of breaking years-old stable links and losing history. The gain—in new technologies, improved URL design, or what have you—was diminished by a break in continuity.
I was therefore gladdened when I tried an old-style forum link:
https://tasvideos.mistflux.net/forum/viewtopic.php?p=508259#508259
and saw that it redirected to the same post in the new forum, down to the HTML anchor!
https://tasvideos.mistflux.net/Forum/Topics/22808?CurrentPage=1&Highlight=508259#508259
This is, honestly, tremendous. It gives me confidence in the developers and their attention to detail. It was this experience that made me feel the rewrite project was worth some of my time and attention.
This site is not only a place for ongoing contemporary discussion and the development of new speedruns. Its history is one of its greatest assets. At a time when much of speedrunning history is falling into the black hole of Discord or random Google docs, where it is bound to disappear in a few years, TASVideos serves as an example of how to do things right. Think of those Summoning Salt videos that cite forum posts from a decade or more ago, and how those vital historical links might have been lost if not for careful stewardship. I see the preservation of past history as one of the main responsibilities of the site as it proceeds into the future. After all, cool URIs don't change 😎.
I appreciate the effort that has gone into the rewrite. It is good to feel that what I contribute today will not be lost tomorrow. You have my support!
Experienced Forum User, Published Author, Player
(125)
Joined: 6/26/2018
Posts: 157
OnehundredthCoin wrote:
I'm pretty sure this new glitch can save time in the 100% TAS. As a quick summary, you can store the most recent block a sprite detects, and if it's a bumped block when Boom Boom spawns in, they are instantly defeated.
This is a great find. I added information about the trick to the Game Resources page. Let me know if I got any facts wrong and I'll fix it.
Experienced Forum User, Published Author, Player
(125)
Joined: 6/26/2018
Posts: 157
I watched the explanation video at https://www.youtube.com/watch?v=wzrKrAvyDZQ. Very interesting, particularly the detail about horizontal position mod 20. The technique of encoding TAS inputs into the code of the game is similar to what Typhon is doing with Nintendo Nightmare, another hard-to-emulate game.
$ python3 -mzeep http://tracker.tasvideos.org/api.wsdl
...
ValueError: No visitor defined for '{http://www.w3.org/2001/XMLSchema}br'
Here is the stack trace as a table:
[td colspan="5"]( ! ) Notice: Array to string conversion in /home/tasvideos/domains/tracker.tasvideos.org/public_html/api/wsdldesc_compiler.php on line 257[/td]
Experienced Forum User, Published Author, Player
(125)
Joined: 6/26/2018
Posts: 157
Great work.
Speaking for myself, I prefer the "final input" ending criterion.
At Post #506089 you mentioned using a solver; can you say more about how it works?
Experienced Forum User, Published Author, Player
(125)
Joined: 6/26/2018
Posts: 157
I enjoyed the movie, being familiar with the game.
Thanks for the detail about the HOLL0350 naming convention. From it, I found The Adventure Family Tree (and a PDF chart). The version I know is KNUT0350.
Dacicus wrote:
The game does not work in Gnash or Lightspark. It seems to be a problem with the textbox used for input.
Through the family tree link, I found the HOLL0350 adventure.swf. I thought about uploading it to the Internet Archive to use their new in-browser Flash emulation. However, I tried the SWF at https://ruffle.rs/demo/ and also encountered a problem with text input: I can type, but cannot complete a command by pressing Enter.
Is is possible to post a text transcript of commands entered? I stepped through the video slowly and the route looks reasonable. Besides the pour and fill shortcuts, I appreciated the use of destination names (e.g. pit) instead of step-by-step directional commands when possible.
After toss eggs, does it save time to cross the chasm using ne instead of cross? (Similarly with sw after free bear.)
Does the black rod count as a treasure in this version? I'm wondering if it's possible to skip dropping the rod in the house at the end.
Experienced Forum User, Published Author, Player
(125)
Joined: 6/26/2018
Posts: 157
I will add my support to SmashManiac's statement and say that HTTPS is important to me as well. Whenever I do something at TASVideos that requires logging in, I go to https://direct.tasvideos.org/ and click through the MD5 warning, because even bad TLS is vastly better than no TLS.
Experienced Forum User, Published Author, Player
(125)
Joined: 6/26/2018
Posts: 157
Typhon wrote:
Should we merge the threads somehow?
I'm not sure how to request a thread merge—I see that XTREMAL93 has requested some merges in the "Feature Requests" thread in Post #499548, but I clicked on a few and they haven't been merged.
Experienced Forum User, Published Author, Player
(125)
Joined: 6/26/2018
Posts: 157
This is quite an interesting approach. If I understand you correctly, the source code of the game is available (nintendo-nightmare.gmk at the home page), and rather than provide inputs externally, you edit the source code to act as if the player had taken certain actions at certain times. I don't think there's anything wrong with calling it a bot. Did you use the same approach in your 24:04 run?
The primary technique used in this run is the CAD (control-alt-delete) exploit. To elaborate, on Windows pressing the CAD keyboard combination brings up a secondary desktop security window. Through errors in the underlying game engine that are not fully understood, large swaths of game code are not executed while in this menu and similar UAC menus.
That's a strange glitch. Do you know if it affects other GameMaker games, or is it specific to this one?
I'm curious about the calls to random_set_seed. Does the game's RNG desync otherwise?
The JRSR format does not store video refreshes, only nanosecond timestamps. But you can often infer where the video refreshes are, because of how authors tend to make TASes: hit frame advance, enter inputs for the frame, then hit frame advance again. Most input events occur at times that are multiples of the video refresh interval, plus a constant. The constant depends, usually, on non-frame-aligned activities at the start of the movie, the boot process and DOS prompt.
For example, here are the event timestamps from [1718] DOS Dangerous Dave by Ilari in 06:32.80, modulo the VGA refresh interval of 14.268 ms (= 800 × 449 / 25.175 MHz). Notice how most timestamps are aligned at an offset of about 9.507 ms. There is a secondary line of timestamps about 1.3 ms after the primary one—this happend when there are two events in the same frame.
Similarly, there are strong primary, secondary, and tertiary timing alignments in the events of [3706] DOS Crystal Caves: Volume 1 - Troubles with Twibbles by DungeonFacts in 20:33.15. But there is also a substantial minority of events that are not so well aligned. I looked into this case—the unaligned events are where 200 ms timed advance was used, rather than frame advance.
In contrast, the timestamps of [2119] DOS Avoid the Noid by turska & Ilari in 03:33.52 are mostly not aligned to the VGA refresh. This suggests that this TAS was not made using frame advance. (Another reason for a lack of alignment could be an incorrect refresh rate, but I'm pretty sure this game is 70 Hz like the rest.)
I wrote a program, guess-vsync, that tries to infer VSYNC parameters, given the event timestamps in a JRSR file. With these VSYNC parameters, the program gen-table produce a table of input events; not only RELEASE, PRESS, and WAIT events, but also VSYNC which means to wait until the next VSYNC pulse from the VGA port. Having a reference for the PC's timing, the Arduino should (in principle) be able to cope with timing variations, like the 60 Hz for 70 Hz I talked about above. The source code of these programs is available in 20220124-dave-pc-tas-verification-attempts.zip, or from git clone https://www.bamsoftware.com/git/pc-tas-arduino.git.
Here's what I did to program the Arduino for the video above. First, run gen-table, just to get a timestamp for when the boot-related events are over and we're ready to start the program. We will pass this timestamp to later commands as the argument to the --skip option.
arduino-verification-circuit-20220124.cddx from https://www.circuit-diagram.org/
I'm using the attachInterrupt interface to react to changes on the VSYNC input pin; i.e., attachInterrupt(digitalPinToInterrupt(VSYNC_PIN), vsync_isr, FALLING). I've read that attachinterrupt goes through a lot of abstractions that add overhead, and that for high performance you need to program interrupts at a lower level. But the overhead estimates I've seen are in the realm of single-digit microseconds, which seems like it shouldn't matter much for VGA vertical refresh, which is in the realm of milliseconds. I think that the USB HID key events can be no more precise than 1 ms, anyway.
I made some changes to the original JRSR file, for convenience or to get the run to sync. You can see the changes by diffing tas/comic1.jrsr (original) and tas/arduino-verification/comic1.jrsr (modified) in the Git repo. Apart from the first two changes in the list, the modified file still syncs in JPC-RR.