Posts for Sand

1 2
7 8
Sand
He/Him
Experienced Forum User, Published Author, Player (146)
Joined: 6/26/2018
Posts: 198
I'm impressed by the routing and inventory management in this one.
Sand
He/Him
Experienced Forum User, Published Author, Player (146)
Joined: 6/26/2018
Posts: 198
AmaizumiUni wrote:
I checked wram in mainmemory.read_u8 but the numbers are different, so it looks like I'm looking at ram.
I hope I understand your question. To read WRAM, you should use memory.read_u8 (not mainmemory.read_u8), and specify "WRAM" for the memory domain. Like this:
memory.read_u8(addr, "WRAM")
You can see the available memory domains with memory.getmemorydomainlist:
memory.getmemorydomainlist() --> {[0]="RAM", "System Bus", "PPU Bus", "CIRAM (nametables)", "PALRAM", "OAM", "Battery RAM", "PRG ROM", "VRAM", "WRAM"}
mainmemory.read_u8(addr) is like memory.read_u8(addr, mainmemory.getname()). The return value of mainmemory.getname() depends on the core. In NesHawk, it is "RAM". (This comes from the common implementation of IMemoryDomains and the ordering of memory domains in NesHawk.)
Sand
He/Him
Experienced Forum User, Published Author, Player (146)
Joined: 6/26/2018
Posts: 198
MUGG wrote:
Why didn't the first code work?
It's because you used the name val both as the name of the outer table and as one of the parameters in the anonymous callback function.
local val = {
function(addr, val, flags)
In the environment of the anonymous callback function, val refers to the function parameter, not the outer table you intend. BizHawk is calling the callback with an integer value in the val parameter, which is why the val.ball operation results in "attempt to index a number value". You could use a different name in one place or the other. Or you could omit the function parameters from the callback (since you aren't using them), so that values that are passed into the function are not bound to variable names.
Post subject: Lua/Fennel polyglot
Sand
He/Him
Experienced Forum User, Published Author, Player (146)
Joined: 6/26/2018
Posts: 198
As you may know from Post #535321, I've been experimenting with writing scripts for BizHawk in Fennel, a lisp programming language based on Lua. Fennel code has to be compiled to Lua before BizHawk can run it. To do that, I have been dividing programs into two files: a main ".fnl", written in Fennel, that has the actual code; and a stub ".lua" file that compiles and runs the .fnl file. You load the .lua file in BizHawk, then the .lua file runs the .fnl file. For example, I might have a "script.fnl" that contains the code of the script:
Language: fennel

(console.log "Hello Fennel") (console.log (string.format "ROM name: %q" (gameinfo.getromname))) (local fennel (require :fennel)) ;; for fennel.view (console.log "joypad" (fennel.view (joypad.get) {:one-line? true}))
And then a "script.lua" stub that executes the main Fennel file:
Language: lua

require("fennel").install().dofile("script.fnl")
The "fennel" module comes from fennel.lua, which you just have to install in the directory alongside your scripts. The two-file approach works fine. But I found a way to get the same effect with just one file. The file is simultaneously a Lua program and a Fennel program; i.e., a polyglot. It looks like this:
Language: lua

;; return require("fennel").install().eval([==[ (console.log "Hello Fennel") (console.log (string.format "ROM name: %q" (gameinfo.getromname))) (local fennel (require :fennel)) ;; for fennel.view (console.log "joypad" (fennel.view (joypad.get) {:one-line? true})) ;; ]==])
The ;; marks a comment in Fennel. So from the point of view of a Fennel interpreter, the first and last lines of the file disappear, leaving only the lisp code in between. In Lua, ; is an empty statement, a no-op separator. So the Lua interpreter sees and executes the require("fennel").install().eval(...). The [==[ marks the beginning of a long literal string that lasts until the ]==] on the last line—encompassing the entire text of the Fennel program. To the Lua interpreter, the Fennel part of the program looks like one long string, which gets passed to the fennel.eval function. It would be nice if only required adding something to the top of the file, not both top and bottom, but I couldn't think of a way to do that. I've found just one problem with the polyglot approach, which is that BizHawk won't let you load a file with a ".fnl" filename extension with the --lua option on the command line. I want to name the program file "script.fnl", so that when I edit it, I get syntax highlighting and automatic indentation appropriate for Fennel. But BizHawk's --lua option only allows ".lua" and ".txt" as filename extensions, and silently ignores files with any other extension. (Except for ".luases", which I don't know what that is.) You can load .fnl files from the "Open Script" UI in the Lua Console; it's just the --lua option that doesn't work. So I have to name the file "script.lua", which adds inconvenience when editing it as Fennel code. So I might stick with the two-file method anyway. If you want to see how the Fennel code compiles to Lua, you can paste it in here.
Post subject: Roguelike Celebration talk
Sand
He/Him
Experienced Forum User, Published Author, Player (146)
Joined: 6/26/2018
Posts: 198
Has the NetHack TAS team considered doing a talk at Roguelike Celebration about the design for a minimum-turns run? The 2025 call for presentations is up:
We can’t wait to welcome you all to this year’s Roguelike Celebration on Oct 25th and 26th - but first it is time to open the door to this year’s batch of speakers! Our conference is only as good as our speakers and presenters, and luckily every year we welcome a breadth of creative, passionate, and talented individuals to step onto the virtual conference stage. The call for presenters is now open, with a submission form here. Submissions will be open until June 30th. Please be sure to pass the link along to anyone you think would have something compelling to share. We welcome all types of presentations and performances, and encourage anyone with the hint of an idea to look at our past years to see just how wide an array of topics we accept and how many different backgrounds and life experiences can make up a Roguelike Celebration speaker!
I've watched this conference for years and I'm sure the topic would be well received. I don't want to create an obligation for anyone, but if you're looking to share in a venue that is more roguelike-oriented than speedrun-oriented, this would be a good one. Just walking through the post–turn 2000 actions à la t2000timings.txt (Post #475904) would be enough for a complete talk, I think. People will be interested to know about zero-time actions, how turns are structured, and the obscure features that come up in a TAS scenario. (It makes me think of the "NetHack: Tech Tourist Mode" talk from 2018.)
Sand
He/Him
Experienced Forum User, Published Author, Player (146)
Joined: 6/26/2018
Posts: 198
There's no memory.readword, but there are memory.read_s16_be, memory.read_s16_le, memory.read_u16_be, memory.read_u16_le: Wiki: BizHawk/LuaFunctions. You have to specify whether you want to read as signed (s) or unsigned (u), and big-endian (be) or little-endian (le).
Sand
He/Him
Experienced Forum User, Published Author, Player (146)
Joined: 6/26/2018
Posts: 198
MUGG wrote:
After I discovered my visualisation luascript doesn't run correctly due to syntax errors that have somehow slipped into that post… Here is the script with syntax errors "seemingly fixed".
Language: diff

-for n=1,table.getn(addressList) do +for n=1,#addressList do
I don't think it was a syntax error, exactly. It's just that the old script used the table.getn function, which was deprecated in Lua 5.1 and removed in Lua 5.2. BizHawk currently uses Lua 5.4. You fixed it correctly by changing it to use the # operator.
MUGG wrote:
I tried to fix the script but now it will output information that's different from before (when compared to the screenshot I had posted). When running the "seemingly fixed" script, I noticed the information output is different between v1.0 and v1.1 of the game. V1.0 (Every few frames of running right) V1.1 (Every few frames of running right)
If I understand you right, there are two problems:
  1. After you made changes to the script, it produces different output than it used to, for the same version of the game.
  2. The updated script produces different output for v1.0 and v1.1.
On point (1), I'm not sure what the cause could be. I looked at the differences between the old and new script in Post #462326 and https://pastebin.com/qjMjp6ZP. The only significant change seems to be that, in addition to an onmemoryexecute handler, the script installs onmemoryread and onmemorywrite handlers for each address in addressList. On point (2), the script is looking for the execution of specific addresses, and it would not be surprising for code addresses to be different in different versions of the game. You'll need a separate table of addresses for each different version. From a quick check, judging by the text labels in addressList, it looks like the current table is tuned for the 1.1 version. For example, the "READ FFEA" and "READ FFE9" entries:
Language: lua

[2]={0x2258,0xFFFFE000,"READ FFEA"}, [3]={0x225D,0xFFFFE000,"READ FFE9"},
match up with the listed addresses in the 1.1 ROM:
0x2258:	ld	a, [0xffea]
0x225a:	cp	0x01
0x225c:	ret	nZ
0x225d:	ld	a, [0xffe9]
But in the 1.0 ROM, those instructions appear a few bytes earlier, at addresses 0x224f and 0x2254:
0x224f:	ld	a, [0xffea]
0x2251:	cp	0x01
0x2253:	ret	nZ
0x2254:	ld	a, [0xffe9]
Post subject: Twitch archiving
Sand
He/Him
Experienced Forum User, Published Author, Player (146)
Joined: 6/26/2018
Posts: 198
KusogeMan wrote:
i just finished playing this game and i wanted to watch some speedruns and TASes, and the speedrun.com page says they might get removed by twitch!!! this is scary, also some of these links in the conversations don't work anymore, i hope we don't lose material for future TASing because of this inactivity!
There is a lot of Twitch video saved at the Internet Archive (archive.org). I'm sure it's a tiny fraction of all Twitch video ever broadcast, but it's hundreds of thousands of items. Here are two collections for example (not all Twitch stuff is in these collections): Archive Team has notes on archiving Twitch data (e.g. VODs, clips, chat), with suggestions of some software to use. The easiest automated way might actually be tubeup.py, which works with Twitch (and other video sites) in addition to YouTube: https://github.com/bibanon/tubeup. I've been disappointed in the quality of the metadata of tubeup.py items I've seen, but it's better than no archive at all. You can find existing tubeup.py items by searching for scanner:tubeup (currently 2,254,431), or just Twitch items with something like scanner:tubeup twitch (currently 284,589). The Archive Team IRC channel #burnthetwitch may have people who can give advice on archiving.
Post subject: Username highlighted on /Profile/UserFiles only(?)
Sand
He/Him
Experienced Forum User, Published Author, Player (146)
Joined: 6/26/2018
Posts: 198
When logged in, the username in the navbar is highlighted (has the CSS active class) at /Profile/UserFiles:
<a class="nav-link active" title="Manage" href="/Profile"><i class="fa fa-user d-inline"></i>&nbsp;Sand</a>
The username is not highlighted at other pages I looked at, for example /Forum, /GameResources, /Subs-List, and /UserFiles/Upload:
<a class="nav-link" title="Manage" href="/Profile"><i class="fa fa-user d-inline"></i>&nbsp;Sand</a>
I don't know if it's intended. I initially thought the highlighted username was a notification indicator, or something like that.
Post subject: Programmatic recreation
Sand
He/Him
Experienced Forum User, Published Author, Player (146)
Joined: 6/26/2018
Posts: 198
I used this movie as a target for a programmatic reimplementation (whole run as a Lua/Fennel script). The movie file is User movie #638789633114541718, and source code and notes are in Post #535321. I had the idea of using this implementation to help with looking for a faster set of names, but after reading pirohiko's glitch notes in Post #505719, I see that the current exploit is already subtle and sophisticated. It's not obvious if there might be a way to improve on it.
Post subject: Recreation of 4468M, implementation in Fennel
Sand
He/Him
Experienced Forum User, Published Author, Player (146)
Joined: 6/26/2018
Posts: 198
I've continued to experiment with scripting TAS in BizHawk in a data-oriented style. I've managed to write a script that functionally recreates [4468] NES Final Fantasy "game end glitch" by AmaizumiUni, Spikestuff & DJ Incendration in 01:36.47 (the inputs are not exactly the same, but it's equivalent in length). The movie file is User movie #638789633114541718. But more interesting than the movie itself is how it was made. Here is the source code in a Git repository and a zip snapshot: git clone https://www.bamsoftware.com/git/ff1.git (currently at commit c8b0cbad2c13b3a9c210b0daf9ae7777b4c00d06) https://www.bamsoftware.com/computers/tasvideos/ff1-20250330.zip The script to run is script/gameend.lua. It's using an evolution of the support code from Post #532299, now factored into modules. It's built on the same concept of immutable savestate objects and computation over values rather than dynamic control of the emulator state. It's also now written in Fennel, a Lisp language that compiles to Lua. The file gameend.lua is actually just a stub that compiles and run gameend.fnl, which is Fennel. I apologize slightly for that—I realize that presenting the savestate manipulation ideas in a less familiar language doesn't make them any more accessible. But I'm doing this for fun, and this was my project to learn Fennel. Writing Fennel for BizHawk turns out to be nice anyway. Here's the top-level gameend.fnl program and the main function:
Language: fennel

;; Unload our own modules, to avoid old versions being cached in package.loaded ;; from a previous execution. (each [_ modname (ipairs [:ff1 :ff1jp :game :savestate :ucs :util])] (tset package.loaded modname nil)) ;; Here we load a version-specific ff1 module, then additionally store it as ;; package.loaded.ff1. Other modules will require it simply as :ff1. (local ff1 (require :ff1jp)) (tset package.loaded :ff1 ff1) (local game (require :game)) (local {: run : restore-savestate!} (require :savestate)) (local {: Up : Down} game) ;; Call f iteratively n times, starting with the value init. (lambda times [init n f] (faccumulate [acc init _ 1 n] (f acc))) (lambda main [savestate] (-> savestate ;; Do the title menu and party selection. (game.title-menu 0) (game.party-select {:class ff1.CONST.CLS.FT :name [0x8b 0x90 0x91 0x96]} ; いきくす {:class ff1.CONST.CLS.TH :name [0xaf 0xb2 0x8d 0x90]} ; よるえき {:class ff1.CONST.CLS.BB :name [0x4c 0x51 0x50 0x55]} ; ごぞぜで {:class ff1.CONST.CLS.RM :name [0x9a 0x4c 0x9f 0xaa]}) ; ちごにむ ;; Enter the overworld. (game.await-screen-wipe) ;; Enter Coneria Castle. (times 6 #(game.move-on-map $ Up)) (game.await-screen-wipe-2) ;; Approach the stairs. (times 16 #(game.move-on-map $ Up)) ;; Start the repeated stair traversals. (times 18 #(-> $ (game.move-on-map Up) ; Step onto stairs. (game.await-screen-wipe-2) ; Screen transition. (game.move-on-map Down))) ; Step off of stairs. ;; Open and close the menu. (game.open-menu) (game.await-palette-cycle) (game.close-menu) (game.await-palette-cycle) ;; More stair traversals. (times 13 #(-> $ (game.move-on-map Up) ; Step onto stairs. (game.await-screen-wipe-2) ; Screen transition. (game.move-on-map Down))) ; Step off of stairs. ;; One last stair traversal. (game.move-on-map Up) (game.await-screen-wipe-2) ;; Open the menu. (game.open-menu) ;; Sync the emulator state to the final computed savestate, in order to ;; save a movie. (restore-savestate!))) (assert (run main))
gameend.fnl gives the overall plan of the TAS, fairly removed from the nitty-gritty of savestate manipulation. The code loads an ff1jp module for some memory address constants. The game module has subroutines specific to Final Fantasy, things like "open a menu" and "move on the map". The main function should be easy to follow. We start by advancing through the title menu and selecting a party with the given classes and names. Walk up 6 times to enter the castle (using an Up constant imported from the game module), then walk up another 16 times to reach the stairs. Walk up/down the stairs 18 times, open and close the menu, then walk up/down the stairs 13 more times. Finally, walk down the stairs a final time and open the menu to jump to the game ending sequence. How the glitch works is explained by pirohiko in Post #505719. The (restore-savestate!) at the end of main is only there for saving a movie file. As I've tried to explain, the whole point of programming in this way is not to have to think about the dynamic state of the emulator, but rather to think in terms of pure functions operating on immutable data. The emulator is just a tool that computes functions over savestates. But saving a movie from the BizHawk menu is one of the rare times we do care about the exact emulator state. The restore-savestate! function syncs the emulator with the final computed savestate and its input log, so you can "Stop Movie" in BizHawk and have a movie that ends exactly there. The function's name ends in an exclamation point to emphasize that it is not a pure function, but depends on or alters global state (which is a naming convention in Fennel). The next level of abstraction down is the game module, in game.fnl. The functions in this module describe how to do actions that are specific to the game of Final Fantasy, implemented on top of the lower-level code in the savestate module. The functions vary in complexity: title-menu, for example, accounts for input differences in the Japanese and USA releases of the game, and party-select computes an optimal sequence of inputs to enter the given names. Let's look at an easy function, await-screen-wipe, which waits for the horizontal screen wipe transition effect to finish:
Language: fennel

(lambda await-screen-wipe [savestate] (assert (await-exec savestate 128 ff1.CODE.ScreenWipe_Finalize)))
The implementation is simple: using the await-exec function from the savestate module, let the emulator run for up to 128 frames until the code address ScreenWipe_Finalize is executed, then return the resulting savestate. If the address is not executed within 128 frames, something is not as expected, so raise an error. await-screen-wipe is a good example of how Fennel compiles to Lua:
Language: lua

local function await_screen_wipe(savestate) return assert(await_exec(savestate, 128, ff1.CODE.ScreenWipe_Finalize)) end
There's also await-screen-wipe-2, which just waits for two screen wipes in sequence, such as happen when climbing or descending stairs:
Language: fennel

(lambda await-screen-wipe-2 [savestate] (-> savestate (await-screen-wipe) (await-screen-wipe)))
The -> macro is useful for sequences of operations like this, where the output of one function becomes the input to the next function. We pass the input savestate to await-screen-wipe to get a new savestate, then pass that new savestate into another invocation of await-screen-wipe. You'll see -> used also in the main function in gameend.fnl. This is how await-screen-wipe-2 compiles to Lua:
Language: lua

local function await_screen_wipe_2(savestate) return await_screen_wipe(await_screen_wipe(savestate)) end
The savestate module is the lowest level of abstraction, the code that interfaces directly with the running state of the emulator. It does all the event registration / coroutine yield / callback operations as were described in Post #532299. await-screen-wipe was defined in terms of await-exec from the savestate module, so let's look at await-exec:
Language: fennel

;; Return a new savestate representing the state of the emulator at the frame ;; boundary after the address addr is executed, or nil if ?frame-limit is ;; reached. (lambda await-exec [savestate ?frame-limit addr] (await-event savestate ?frame-limit #(bizhawk.event.on_bus_exec $ addr)))
As you can see, await-exec just passes the given savestate and ?frame-limit (name prefixed by a question mark to indicate it may be nil) to the await-event function, along with a function that describes how to register an event listener for the specific BizHawk event we're looking for (using the # and $ syntax for anonymous functions). await-event is the analog of what was called run_until_event in Post #532299, a core function whose purpose is to let the emulator run until an event happens, then return. The savestate module is what makes the data-oriented programming style possible. It's where the most complicated code is, but if you're interested in looking at it yourself, it's really not that bad: just 400 lines, plenty of which are comments.
Sand wrote:
The current version of the support code wouldn't work well for press-and-hold inputs (Kwirk only needs one-frame presses). In order to hold buttons in a convenient way, you may have to enrich the savestate object to support some kind of "sticky" inputs that don't become unpressed when the frame is advanced.
I implemented slightly enhanced joypad support in this version of the code. A savestate object stores, alongside a BizHawk-native savestate and the set of currently pressed joypad inputs, a schedule of inputs to happen in future frames. With functions like joypad-set and joypad-press-release, you can schedule button presses or releases to happen in the current frame or in a future frame. This is nice for press-and-hold inputs, as well as overlaying several actions that use different buttons.
Sand
He/Him
Experienced Forum User, Published Author, Player (146)
Joined: 6/26/2018
Posts: 198
YoshiRulz wrote:
CPP pushed a fix for pcall, so please try again with a new dev build.
That's great. A dev build after 0572266 works for me. Thanks for your help. It would have taken me much longer to find it on my own. Interesting that the cause was similar to JPC-RR builtin functions returning on the wrong thread. In that case, it affected every return of a builtin function, not just exceptions. So in one sense it was easier to notice. But also, JPC-RR only runs one Lua script at a time, rather than running each script in its own Lua thread, so it would only happen if you actually created new coroutines yourself, which was not the case here.
Post subject: Video of debugging and fix
Sand
He/Him
Experienced Forum User, Published Author, Player (146)
Joined: 6/26/2018
Posts: 198
I guess I never linked the video of fixing this bug. It's from 29:35 to 47:35 in the video below, which is part of my series of live TAS videos from [4345] DOS The Adventures of Captain Comic: Episode 1 - Planet of Death by Sand & Kabuto in 06:30.98. Link to video
Post subject: Name entry cursor movement
Sand
He/Him
Experienced Forum User, Published Author, Player (146)
Joined: 6/26/2018
Posts: 198
I haven't seen documentation of optimized cursor movement on the name entry menu, even though some published runs take advantage of it. I've posted new notes at Wiki: GameResources/NES/FinalFantasy1?revision=3#NameEntry. In short, the name entry of [2079] NES Final Fantasy by TheAxeMan in 1:09:57.70 could be done up to 5 frames faster; the name entry in [4468] NES Final Fantasy "game end glitch" by AmaizumiUni, Spikestuff & DJ Incendration in 01:36.47 is already optimized. Naively, to move the cursor from one point to another on the name entry screen, you would alternate between 1 frame of pressing a directional button and 1 frame of no input. When you need to do a diagonal movement, you save time by alternating the two directional buttons you need to press. For example, moving 2 spaces down and 3 spaces right (2D3R), a total distance of 5 spaces, can be done in 5 frames by alternating Right and Down inputs: ...R .D.. ...R .D.. ...R. But you can do even better than that. The name entry menu reacts to simultaneous directional inputs in unintuitive ways. For example, if you were pressing ...R on the previous frame, and press U..R on the current frame, the cursor moves 1 space down, despite the Down button never being pressed. Only U... and ..L. can move the cursor up or left, but there are many ways to move the cursor down or right, which means that you can often move at the maximum speed of 1 space per frame, even when moving straight horizontally or vertically.
A B C D E F G H I J
K L M N O P Q R S T
U V W X Y Z ’ , .  
0 1 2 3 4 5 6 7 8 9
a b c d e f g h i j
k l m n o p q r s t
u v w x y z - ‥ ! ?
[2079] NES Final Fantasy by TheAxeMan in 1:09:57.70 (and in turn namegenerator/inputgeneration.py from lightwarriorcode-v1.0.zip) applies the diagonal-movement optimization but not the simultaneous-buttons optimization. For example, the diagonal 2D2R movement from 'A' to 'W' in the name "Wedg" is already as fast as it can be:
|..|...R....| 'B'
|..|.D......| 'L'
|..|...R....| 'M'
|..|.D......| 'W'
|..|.......A|
But the 3R horizontal movement from 'd' to 'g' takes 5 frames when it could take only 3 frames:
|..|...R....| 'e'
|..|........|
|..|...R....| 'f'
|..|........|
|..|...R....| 'g'
|..|.......A|
You can move from 'd' to 'g' in only 3 frames by alternating ...R and ..LR:
|..|...R....| 'e'
|..|..LR....| 'f'
|..|...R....| 'g'
|..|.......A|
User movie #638766622812817201 is a demonstration of entering the "Wedg", "Bigs", "Axe ", "Viks" names from [2079] NES Final Fantasy by TheAxeMan in 1:09:57.70 5 frames faster. The inputs were computed using a shortest-path Lua script (a further development of the savestate manipulation technique from Post #532299) using the movement table from Wiki: GameResources/NES/FinalFantasy1#NameEntry. User movie #638766593655640386 is a partial resync of the original .fm2 to BizHawk 2.10, for easier comparison.
あいうえおがぎぐげご
かきくけこざじずぜぞ
さしすせそだぢづでど
たちつてとばびぶべぼ
なにぬねのぱぴぷぺぽ
はひふへほゃゅょっ。
まみむめも01234
らりるれろ56789
やゆよわん-!?‥ 
[4468] NES Final Fantasy "game end glitch" by AmaizumiUni, Spikestuff & DJ Incendration in 01:36.47 already applies both the diagonal-movement optimization and the simultaneous-buttons optimization. The only place the latter optimization is needed is in the name of the 4th party member, "ちごにむ". I believe the optimization was applied by Spikestuff in Post #505707 and User movie #71345684705421713. (The original submission in #7120: AmaizumiUni, Spikestuff & DJ Incendration's NES Final Fantasy "game end glitch" in 01:36.47 was replaced so I can't be sure.) To move 3D1R from 'あ' to 'ち', the movie uses a frame of U.L. to move the cursor down immediately after another down movement:
|..|.D......|........| 'か'
|..|...R....|........| 'き'
|..|.D......|........| 'し'
|..|U.L.....|........| 'ち'
|..|.......A|........|
The same trick is used to move 4D2R from 'ご' to 'に'. The 2D1R movement from 'に' to 'む' uses .D.. ...R U.L., but in this case the simultaneous buttons are unnecessary; .D.. ...R .D.. would also have worked.
Post subject: Re: Inconsistencies with savestate.save and pcall between Lua console and script
Sand
He/Him
Experienced Forum User, Published Author, Player (146)
Joined: 6/26/2018
Posts: 198
YoshiRulz wrote:
Grab a dev build.
I mean, sure—f409cc4 fixes the immediate problem of checking for error from savestate.save. But I think you have overlooked the more important part of my post. Namely:
Sand wrote:
  1. The pcall function actually runs and returns. The first return value is false, as expected.
  2. However, the second return of pcall is not the expected LuaScriptException error message, but rather the argument to savestate.save, "/root/foobar".
  3. Even though pcall catches the error, the script crashes with a LuaScriptException stack trace anyway, before the print("after") line.
The fact that pcall returns one of its arguments, rather than the expected error messages, and that the script doesn't crash until the next call to print, makes me suspect something is getting mixed up with the Lua virtual stack.
The problem is more general than savestate.save. For example, try any of these in the Lua Console input box and in a script file:
  • print("before") print(pcall(client.screenshot, "/root/foobar")) print("after")
  • print("before") print(pcall(memory.hash_region, memory.getcurrentmemorydomainsize(), 1)) print("after")
In the Lua Console, they work as expected: pcall catches the error and the program finishes. In a script, the second return value of pcall is wrong, and the program crashes after pcall returns. Surely that's not intended? Having to use pcall sometimes when calling Wiki: BizHawk/LuaFunctions doesn't bother me. That's normal Lua programming. The problem is the inconsistency between how it works in the Lua Console input box and in a script file. I spent some time trying to understand what's different when Lua code is run from the console versus in a script file. When code is run from the input box in the console, it calls LuaSandbox.Sandbox with a null thread and a null exceptionCallback. When a Lua file is enabled, thread is null and exceptionCallback is non-null. When a Lua file is restarted or resumed, both thread and exceptionCallback are non-null. But I could not see how these differences would lead to the described inconsistency. LuaMethodWrapper.Call looks to be where the Lua virtual stack manipulations happen. On seeing a bug like this one, where function parameters are wrongly interpreted as return values, and values from a previous function call affect the next function call, I would tend to first suspect code like this that does low-level stack manipulations. But again it was not clear to my why anything in this function should be different when run from the Lua Console input box and from a script. Note that there is no problem when the error occurs before the function is actually called, such as when an attempt is made to call a function with the wrong number of parameters. print("before") print(pcall(joypad.get, "wrong type")) print("after") works as expected both in the Lua Console input box and in a script file.
Post subject: Inconsistencies with savestate.save and pcall between Lua console and script
Sand
He/Him
Experienced Forum User, Published Author, Player (146)
Joined: 6/26/2018
Posts: 198
I'm trying to make a wrapper around savestate.save with a return value that indicates whether the save happened or not. The obvious way of doing it, calling savestate.save inside pcall, works the way I expect when I type it into the input box of the Lua Console, but it doesn't work the same way inside a script. Inside a script, pcall returns unexpected values, and the script then crashes with a LuaScriptException anyway. This is with BizHawk 2.10 on Linux. Saving a savestate to a file could fail for various reasons, such as insufficient permissions. savestate.save calls SaveStateLuaLibrary.Save, which doesn't have a return value; it relies on throwing an exception to signal error. SaveStateLuaLibrary.Save in turn defers to APIs.SaveState.Save and _mainForm.SaveState. In the Lua Console, if I try to save a savestate to a path to which I do not have permission, I get a LuaScriptException stack trace (which is expected):
savestate.save("/root/foobar")
NLua.Exceptions.LuaScriptException: [string "input"]:1: A .NET exception occured in user-code
System.UnauthorizedAccessException: Access to the path "/root/foobar" is denied.
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.Boolean anonymous, System.IO.FileOptions options) [0x001ef] in <12b418a7818c4ca0893feeaaf67f1e7f>:0
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.Boolean isAsync, System.Boolean anonymous) [0x00000] in <12b418a7818c4ca0893feeaaf67f1e7f>:0
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access) [0x00000] in <12b418a7818c4ca0893feeaaf67f1e7f>:0
  at (wrapper remoting-invoke-with-check) System.IO.FileStream..ctor(string,System.IO.FileMode,System.IO.FileAccess)
  at BizHawk.Client.Common.FrameworkZipWriter..ctor (System.String path, System.Int32 compressionLevel) [0x00006] in <2abb4b338b4a4b62a6b02f91d843b06c>:0
  at BizHawk.Client.Common.ZipStateSaver..ctor (System.String path, System.Int32 compressionLevel) [0x00006] in <2abb4b338b4a4b62a6b02f91d843b06c>:0
  at BizHawk.Client.Common.SavestateFile.Create (System.String filename, BizHawk.Client.Common.SaveStateConfig config) [0x00007] in <2abb4b338b4a4b62a6b02f91d843b06c>:0
  at BizHawk.Client.EmuHawk.MainForm.SaveState (System.String path, System.String userFriendlyStateName, System.Boolean fromLua, System.Boolean suppressOSD) [0x00091] in <f3ae97cbaae74109853bdae22c2312c8>:0
  at BizHawk.Client.Common.SaveStateApi.Save (System.String path, System.Boolean suppressOSD) [0x00000] in <2abb4b338b4a4b62a6b02f91d843b06c>:0
  at BizHawk.Client.Common.SaveStateLuaLibrary.Save (System.String path, System.Boolean suppressOSD) [0x00023] in <2abb4b338b4a4b62a6b02f91d843b06c>:0
  at (wrapper managed-to-native) System.Reflection.RuntimeMethodInfo.InternalInvoke(System.Reflection.RuntimeMethodInfo,object,object[],System.Exception&)
  at System.Reflection.RuntimeMethodInfo.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0007c] in <12b418a7818c4ca0893feeaaf67f1e7f>:0
In the Lua Console, wrapping the call in pcall works as I expect. Saving the savestate fails, and pcall returns false and a string representation of the error:
pcall(savestate.save, "/root/foobar")
False	NLua.Exceptions.LuaScriptException: A .NET exception occured in user-code
The above technique with pcall would be adequate for my needs. The problem is, it doesn't work the same way inside a script. I create a file test.lua:
print("before")
print(pcall(savestate.save, "/root/foobar"))
print("after")
When I run the script with ./EmuHawkMono.sh --lua=test.lua ROMFILE, I get one line of output for the "before", one line of output for the pcall, and then a LuaScriptException stack trace:
before
False	/root/foobar
NLua.Exceptions.LuaScriptException: A .NET exception occured in user-code
System.UnauthorizedAccessException: Access to the path "/root/foobar" is denied.
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.Boolean anonymous, System.IO.FileOptions options) [0x001ef] in <12b418a7818c4ca0893feeaaf67f1e7f>:0
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.Boolean isAsync, System.Boolean anonymous) [0x00000] in <12b418a7818c4ca0893feeaaf67f1e7f>:0
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access) [0x00000] in <12b418a7818c4ca0893feeaaf67f1e7f>:0
  at (wrapper remoting-invoke-with-check) System.IO.FileStream..ctor(string,System.IO.FileMode,System.IO.FileAccess)
  at BizHawk.Client.Common.FrameworkZipWriter..ctor (System.String path, System.Int32 compressionLevel) [0x00006] in <2abb4b338b4a4b62a6b02f91d843b06c>:0
  at BizHawk.Client.Common.ZipStateSaver..ctor (System.String path, System.Int32 compressionLevel) [0x00006] in <2abb4b338b4a4b62a6b02f91d843b06c>:0
  at BizHawk.Client.Common.SavestateFile.Create (System.String filename, BizHawk.Client.Common.SaveStateConfig config) [0x00007] in <2abb4b338b4a4b62a6b02f91d843b06c>:0
  at BizHawk.Client.EmuHawk.MainForm.SaveState (System.String path, System.String userFriendlyStateName, System.Boolean fromLua, System.Boolean suppressOSD) [0x00091] in <f3ae97cbaae74109853bdae22c2312c8>:0
  at BizHawk.Client.Common.SaveStateApi.Save (System.String path, System.Boolean suppressOSD) [0x00000] in <2abb4b338b4a4b62a6b02f91d843b06c>:0
  at BizHawk.Client.Common.SaveStateLuaLibrary.Save (System.String path, System.Boolean suppressOSD) [0x00023] in <2abb4b338b4a4b62a6b02f91d843b06c>:0
  at (wrapper managed-to-native) System.Reflection.RuntimeMethodInfo.InternalInvoke(System.Reflection.RuntimeMethodInfo,object,object[],System.Exception&)
  at System.Reflection.RuntimeMethodInfo.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0007c] in <12b418a7818c4ca0893feeaaf67f1e7f>:0
Important points to notice:
  1. The pcall function actually runs and returns. The first return value is false, as expected.
  2. However, the second return of pcall is not the expected LuaScriptException error message, but rather the argument to savestate.save, "/root/foobar".
  3. Even though pcall catches the error, the script crashes with a LuaScriptException stack trace anyway, before the print("after") line.
But get this: if I don't immediately print the return values of pcall, but instead store them in variables, then the stack trace does not occur until after the print("after") line:
print("before")
x, y = pcall(savestate.save, "/root/foobar")
print("after")
before
after
NLua.Exceptions.LuaScriptException: A .NET exception occured in user-code
System.UnauthorizedAccessException: Access to the path "/root/foobar" is denied.
...etc...
The x and y variables that were assigned to in the script remain set, and their values can still be seen in the Lua Console:
print(x, y)
False	/root/foobar
I'm not sure if this is a bug in BizHawk, or if I am doing something wrong. The fact that pcall returns one of its arguments, rather than the expected error messages, and that the script doesn't crash until the next call to print, makes me suspect something is getting mixed up with the Lua virtual stack. Below is the function I would like to be able to write. In fact, I can define the function in a script, and call it from the Lua Console, and it works as expected. But if I try to call it from the same script, it exhibits the inconsistencies described above.
function save_savestate_to_file(path)
	local ok, err = pcall(savestate.save, path)
	if ok then
		return true
	else    
		return nil, err
	end
end
Post subject: Corrected JP character table
Sand
He/Him
Experienced Forum User, Published Author, Player (146)
Joined: 6/26/2018
Posts: 198
pirohiko wrote:
Character List
I discovered some errors in the character table. Here is a corrected table. It is found at 0xa013 in the JP ROM. The corresponding table is at 0xa011 in the NA ROM, and is called lut_NameInput in the disassembly.
8A 8B 8C 8D 8E 48 49 4A 4B 4C
あ い う え お が ぎ ぐ げ ご
8F 90 91 92 93 4D 4E 4F 50 51
か き く け こ ざ じ ず ぜ ぞ
94 95 96 97 98 52 53 54 55 56
さ し す せ そ だ ぢ づ で ど
99 9A 9B 9C 9D 57 58 59 5A 5B
た ち つ て と ば び ぶ べ ぼ
9E 9F A0 A1 A2 70 71 72 73 74
な に ぬ ね の ぱ ぴ ぷ ぺ ぽ
A3 A4 A5 A6 A7 7D 7E 7F 7C B9
は ひ ふ へ ほ ゃ ゅ ょ っ 。
A8 A9 AA AB AC 80 81 82 83 84
ま み む め も 0 1 2 3 4
B0 B1 B2 B3 B4 85 86 87 88 89
ら り る れ ろ 5 6 7 8 9
AD AE AF B5 B6 C2 C4 C5 C3 FF
や ゆ よ わ ん - ! ?  ‥  
The corrections are:
  • ゃゅょっ is 7D 7E 7F 7C, not 7C 7D 7E 7F.
  • らりるれろやゆよ is B0 B1 B2 B3 B4 AD AE AF, not AD AE AF B0 B1 B2 B3 B4.
Here it is as a Python table, for convenience:
{
    0x8a: "あ", 0x8b: "い", 0x8c: "う", 0x8d: "え", 0x8e: "お", 0x48: "が", 0x49: "ぎ", 0x4a: "ぐ", 0x4b: "げ", 0x4c: "ご",
    0x8f: "か", 0x90: "き", 0x91: "く", 0x92: "け", 0x93: "こ", 0x4d: "ざ", 0x4e: "じ", 0x4f: "ず", 0x50: "ぜ", 0x51: "ぞ",
    0x94: "さ", 0x95: "し", 0x96: "す", 0x97: "せ", 0x98: "そ", 0x52: "だ", 0x53: "ぢ", 0x54: "づ", 0x55: "で", 0x56: "ど",
    0x99: "た", 0x9a: "ち", 0x9b: "つ", 0x9c: "て", 0x9d: "と", 0x57: "ば", 0x58: "び", 0x59: "ぶ", 0x5a: "べ", 0x5b: "ぼ",
    0x9e: "な", 0x9f: "に", 0xa0: "ぬ", 0xa1: "ね", 0xa2: "の", 0x70: "ぱ", 0x71: "ぴ", 0x72: "ぷ", 0x73: "ぺ", 0x74: "ぽ",
    0xa3: "は", 0xa4: "ひ", 0xa5: "ふ", 0xa6: "へ", 0xa7: "ほ", 0x7d: "ゃ", 0x7e: "ゅ", 0x7f: "ょ", 0x7c: "っ", 0xb9: "。",
    0xa8: "ま", 0xa9: "み", 0xaa: "む", 0xab: "め", 0xac: "も", 0x80: "0", 0x81: "1", 0x82: "2", 0x83: "3", 0x84: "4",
    0xb0: "ら", 0xb1: "り", 0xb2: "る", 0xb3: "れ", 0xb4: "ろ", 0x85: "5", 0x86: "6", 0x87: "7", 0x88: "8", 0x89: "9",
    0xad: "や", 0xae: "ゆ", 0xaf: "よ", 0xb5: "わ", 0xb6: "ん", 0xc2: "-", 0xc4: "!", 0xc5: "?", 0xc3:  "‥", 0xff: " ",
}
Sand
He/Him
Experienced Forum User, Published Author, Player (146)
Joined: 6/26/2018
Posts: 198
Dimon12321 wrote:
I took the opportunity to comment on this TAS on Game Done Quick 2024. https://www.twitch.tv/videos/2273560382?t=22h58m35s
Thanks for this link. I hadn't been aware of this showcase. I enjoyed the run and your commentary.
Sand
He/Him
Experienced Forum User, Published Author, Player (146)
Joined: 6/26/2018
Posts: 198
Sand
He/Him
Experienced Forum User, Published Author, Player (146)
Joined: 6/26/2018
Posts: 198
CasualPokePlayer MUGG
Sand
He/Him
Experienced Forum User, Published Author, Player (146)
Joined: 6/26/2018
Posts: 198
Alyosha for console verification work and a cooperative attitude, in addition to TASes completed this year.
Post subject: Level 23 score tally
Sand
He/Him
Experienced Forum User, Published Author, Player (146)
Joined: 6/26/2018
Posts: 198
Great job. It's a yes vote for me. I was especially interested in the end-of-stage score tally. I recreated the graph from Post #522668 with this submission included. The graph, which shows the number of blocks remaining and the time required to tally the score for every frame in level 23, is attached below. To my surprise, this submission ties [5327] NES Arkanoid "warpless" by eien86 in 10:56.12 in level 23, taking 1588 frames. Both movies are 26 frames slower than [3943] NES Arkanoid "warpless" by Chef Stef in 11:11.18. If this submission takes the score tally into account, why is it that the optimizer did not find the faster solution? If I understand the submission notes correctly, it should be possible to teleport the paddle instantly to any required coordinate, and it should be possible to re-align the RNG sequence when earlier inputs change. Shouldn't it then be possible to import a known faster solution, even if it was not found automatically by the optimizer? Perhaps there is some important factor I am misunderstanding. I looked in the source code of QuickerArkBot to try and understand what was going on, but I may not have found the right places. I see UpdateScore, which does the tens-and-hundreds score tally logic on pendingScore. But I did not understand how it fits into determining when a level is finished. In the upstream arkbot, I see a place that checks for state.currentBlocks == 0 && state.pendingScore <= 0, but I did not find an analogous check in QuickerArkBot. I did not look at the code too hard, so it's possible I overlooked something. This is the graph that shows this submission and [5327] NES Arkanoid "warpless" by eien86 in 10:56.12 ultimately taking longer than [3943] NES Arkanoid "warpless" by Chef Stef in 11:11.18, even though they clear the final block faster, because they accumulate more pendingScore that takes time to burn down.
Sand
He/Him
Experienced Forum User, Published Author, Player (146)
Joined: 6/26/2018
Posts: 198
#9396: staphen, AJenbo, ephphatha & dwangoAC's Windows Diablo in 04:10.31 has this interesting note:
Diablo uses a type of pseudo-random number generator called a Linear Congruential Generator (LCG). … Each dungeon seed is picked by advancing the RNG state then treating the 32 bit state as a signed integer value and transforming it into a positive integer value between 0 and 231 using the C standard library function abs() (yielding a 31 bit seed[6]). [6]: Plus an extra value; because the absolute value of -231 cannot be represented as a positive signed 32 bit integer, Diablo ends up using this value as-is.
I found more information at The Cutting Room Floor:
Very, very rarely, the random number generator can generate a negative number when the code expects a number to fall within a random range of positive numbers. The issue lies with a broken implementation of Borland's random function (as listed in the Hellfire source code):
//***************************************************************************
//***************************************************************************
long GetRndSeed() {
	SeedCount++;
	static const DWORD INCREMENT = 1;
	static const DWORD MULTIPLIER = 0x015a4e35L;
	sglGameSeed = MULTIPLIER * sglGameSeed + INCREMENT;
	return abs(sglGameSeed);
}


//***************************************************************************
//***************************************************************************
long random(byte idx,long v) {
	if (v <= 0) return 0;

	// high order bits are more "random" than low order bits
	if (v < 0x0ffff) return (GetRndSeed() >> 16) % v;

	return GetRndSeed() % v;
}
The problem is with the call abs(sglGameSeed). The evident intention is that GetRndSeed should never return a negative number; which is why the function takes an absolute value before returning. But there is one case where the output of abs may be negative. When sglGameSeed has the value 1457187811, the calculation MULTIPLIER * sglGameSeed + INCREMENT results in −2147483648; i.e., −231; i.e., 0x80000000, the most negative signed 32-bit integer. Because this value has no positive counterpart in two's complement arithmetic, the call abs(sglGameSeed) returns the same negative value, −2147483648. This is the only case where GetRndSeed returns a negative value, and this is the only negative value it may return. When this happens, a call to random will return a value that is ''non-positive''. When v evenly divides −2147483648 (i.e., when v is a power of 2), the return value will be 0; otherwise it will be negative. To put it more plainly: if the game needs to generate a random number within a range (and the upper limit of the range is not a power of 2), there is a very small chance that the number generated will actually be a negative number, instead of a number within that range. This can wreak havoc in the game, causing anything from memory corruption to random sound effects being played. Since the chances of this happening are incredibly small and the effects are so random, it's unlikely that this has been observed more than a handful of times over the years.
The idea of an out-of-bounds write is intriguing, because that is the kind of thing that arbitrary code execution exploits and credits warps are made of. If the code is trying to write into a random index of an array of size 10, for example, with the correct RNG seed the game will write to index −8 instead. You would need to find a place in the code where random is called and its return value used in the address of a subsequent write, with either the value written or the modulus v being usefully controllable. You'd likely get only one chance, unless there is way to re-seed the RNG, because it's only one point in the RNG's period where this happens.
Post subject: Re: races
Sand
He/Him
Experienced Forum User, Published Author, Player (146)
Joined: 6/26/2018
Posts: 198
Do you have the start, end coordinates for each race in a convenient machine readable format? 150 vertices is few enough to at least approximate a solution to traveling salesman. I was looking at traveling salesman solvers recently. (Even though it turned out not to be necessary for the problem I was looking at.) For this problem, you set up a directed graph over 150 vertices. The weight of the edge between vertex i and vertex j is the distance from the end of race i to the start of race j. wij = distance(Ri.end, Rj.start) To take resets in to account, you'll need to figure out some conversion factor from distance on the map to time (i.e., seconds), and measure the reset_time in seconds. Then the weight between vertex i and vertex j is the smaller of the no-reset option (the one above) and the reset option. wij = min( distance_to_time × distance(Ri.end, Rj.start), distance_to_time × distance(Ri.start, Rj.start) + reset_time ) Here's an example of approximating a solution using NetworkX in Python. I used the pixel coordinates of the first few races from the map you posted. You'll have to fill in the full table of coordinates. There's documentation on traveling_salesman_problem and simulated_annealing_tsp.
Language: python

import collections import math import random import networkx import networkx.algorithms.approximation.traveling_salesman as TSP Point = collections.namedtuple("Point", ("x", "y")) Race = collections.namedtuple("Race", ("label", "start", "end")) RACES = ( Race(1, Point(1263, 1724), Point(6888, 556)), Race(2, Point(1400, 1775), Point(7555, 979)), Race(3, Point(1477, 1798), Point(1728, 1715)), # ... ) def distance(p_1, p_2): return math.sqrt((p_2.x - p_1.x)**2 + (p_2.y - p_1.y)**2) G = networkx.DiGraph() for j in range(len(RACES)): r_j = RACES[j] for i in range(len(RACES)): if i == j: continue r_i = RACES[i] G.add_edge(r_i.label, r_j.label, weight = distance(r_i.end, r_j.start)) print(G) path = TSP.traveling_salesman_problem( G, cycle = False, method = lambda g, weight: TSP.simulated_annealing_tsp(g, "greedy"), ) print("path", path) print("weight", networkx.path_weight(G, path, "weight"))
For me this outputs
DiGraph with 3 nodes and 6 edges
path [3, 1, 2]
weight 6086.839929148261
1 2
7 8