keylie
He/Him
Editor, Emulator Coder, Expert player (2822)
Joined: 3/17/2013
Posts: 391
libTAS is a software for Linux that provides TAS tools to games, such as frame advance, inputs recording, savestates. It is not a Linux emulator as games are running natively on the user system, but it creates an intermediate layer between the game and the operating system, feeding the game with altered data (such as inputs, system time). We can call such tool a translayer - a code translation layer. It tries to make the game running deterministically, although it is still an issue when dealing with multithreaded games. This layer connects to an external program to provide a graphical interface to the user, with tools such as input editor or RAM watch/search. Wiki pages: http://tasvideos.org/EmulatorResources/LibTAS.html Code repository: https://github.com/clementgallet/libTAS Stable releases: https://github.com/clementgallet/libTAS/releases
</hr> Original post: last edit: 07/2018 Hey everyone, I would like to present a project that I've been working on in the past months that brings TAS tools to GNU/Linux games. In short, this program works similarly to Hourglass. It consists on a program and a library. The library is loaded with the game, to override some API functions that the game calls. The program and the library communicate with a Unix socket. The program is in charge of gathering inputs and send them to the game, record/playback movie files, savestates, and offer a user interface. This project is still highly experimental and there are many things still to implement, but it already runs a bunch of commercial games correctly. The main lacking functionality right now is savestate support. Many ideas come from the Hourglass project, thanks to all the people that took part in its development. The source code can be found here. If think there is a benefit in developing a tool for GNU/Linux even if there is already one for Windows and the catalog of games is much larger on Windows: - OS are different, and games may be coded differently on different platform, so the compatibility list can be different even with the same approach - Part of the design are much easier on GNU/Linux that on Windows. For example, hooking functions in Windows requires multiple operations (replacing the first instruction of the function with a jump, saving that instruction to execute the original function). On GNU/Linux, it simply consists on declaring a function with the same name as the function to hook and use the LD_PRELOAD trick - If necessary, it is easier to mess with the core system by modifying the Linux kernel Here is a description of the different features currently implemented in the program: Game compatibility This project aims primary at supporting games based on SDL/OpenGL. This concerns many indie games (Super Meat Boy, Volgarr, VVVVVV, Limbo, Braid, Dustforce, etc.), including games written originally on XNA framework and ported to GNU/Linux using FNA (TowerFall Ascension, FEZ). There is also a support for low-level functions, so games using other librairies (Unity, GM:S) may have basic support. Frame Advance Frame Advance is performed by overriding the different draw functions. Frame boundaries are located just after the screen draw, and it is where the communication between the program and the game is done. Fast-forward All sleep calls by the game are intercepted and are delayed until the next frame boundary. A sleep call is done at each frame boundary to get a normal game speed, or is bypassed to obtain a fast-forward. All OpenGL render commands are skipped during fast-forward for a big speed boost. Joystick support SDL1 Joystick and SDL2 GameController APIs are captured, as well as Linux jsdev and evdev interfaces. The program support mapping keys to joystick buttons, and there is a specific controller input window to send analog inputs (sticks, triggers). Mouse support SDL Mouse API and Xlib API are captured. The pointer is made visible to be able to perform inputs frame by frame. For games using the absolute values of the coordinates (FTL, Dustforce), it works fine. For games using an ugly combination of pointer warping and using relative movement (Braid), there is a constant offset between the OS pointer and the game pointer that needs to be addressed. Input recording/playback Keyboard, mouse and controllers inputs can be saved in a file and played back. There is a basic input editor to modify inputs during recording/playback (including analog inputs), and to insert or delete frames. Determinism There is a deterministic timer implemented to advance time by 1/fps at each frame boundary. It gives the time to every function accessing time. There are several problems on multithreaded games: - Threads can have a loop that waits for time to advance. The current solution is to advance time by a tiny amount after enough time queries have been made by this thread, which is a potential source of desyncs. - Extra threads are basically uncontrolled because they are not related to frame increments. One example of such thread causing sync issues are loading threads, i.e. threads that perform a job while the main thread is still running, and notify the main thread to continue when it has finished. Video dumping Video coming from either an OpenGL rendering or a SDL software rendering is captured and encoded using ffmpeg's avcodec/avformat library. Audio playback and dumping Audio sources are captured and the mixing is done by our program. This allows us to control the playback of the audio (pause it, skip it during fast-forward, etc.). Audio can also be dumped that way, and encoded using ffmpeg. Currently SDL Audio and OpenAL librairies are supported. Also, ALSA and PulseAudio Simple are hooked so that unsupported librairies (e.g. FMOD) that output sound using one of those drivers can still be captured. HUD Some information can be displayed on top of the game screen. Currently frame count, inputs, log messages and ram watches are displayed. This works for OpenGL and SDL software renderers. Savestate Savestates are implemented by suspending all threads (using signals) and dumping the whole game process memory. Savestates can be either stored on disk or on RAM. Also, there is an incremental savestate feature to lower the size of savestates: after the first savestate, every memory page that is written to by the game is flagged (using the recent soft-dirty bit), so only modified memory is saved on a subsequent savestate. This requires extra work to save/load a savestate, but drastically reduces savestate size. Savefile management For TAS, we want to prevent the game from saving, and also store savefiles in savestates. I hooked file IO functions to detect the opening of files that are possibly savefiles (currently regular files with write access). When such a file is detected, I load the file in memory and return a file descriptor to it (using memfd_create or open_memstream). That way, modifications are made on memory and the file content is intact. User interface The user interface was implemented usng Qt5. RAM Search/Watch A basic RAM Search/RAM Watch feature is implemented. You can search on every kind of memory segments of the game (code/static data/heap/anonymous memory/etc.). You can search for integers (signed/unsigned 1-byte -> 8-byte) and float/double. You can filter results by comparing against the previous value or a constant value. For games with high memory usage, the memory search can break if stores too many values (> 1 GB for me). RAM watches can also store pointers (and pointers to pointers, etc.). To be able to get watches that are still valid upon game restart, there is a pointer scanning feature (similar to Cheat Engine). You can search for chains of pointers->offset->pointer->offset->... from a static address to the address you are looking at. Finally, you can poke values into game memory.
Dimon12321
He/Him
Active player (478)
Joined: 4/5/2014
Posts: 1120
Location: Ukraine
Who would bring to mind our current Hourglass...? Anyway, good luck in development! I don't know how useful it will be. As for now, major part uses Windows.
TASing is like making a film: only the best takes are shown in the final movie.
Skilled player (1703)
Joined: 9/17/2009
Posts: 4952
Location: ̶C̶a̶n̶a̶d̶a̶ "Kanatah"
Would there be any plans for RAM watch? I wish you luck on this. :)
Fog
Experienced player (625)
Joined: 4/5/2014
Posts: 459
jlun2 wrote:
Would there be any plans for RAM watch? I wish you luck on this. :)
RAM Watch would be just using something like Cheat Engine? Anyways, I'm definitely interested in trying this out with some games that I've wanted to do for a while. Is support for WINE a possibility?
Site Admin, Skilled player (1234)
Joined: 4/17/2010
Posts: 11251
Location: RU
Hot damn, you're a true hero! Question: You will be able to run sdl-driven emulators that way too, right? You know... TASing emulators with no TAS tools in them.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Editor, Player (44)
Joined: 7/11/2010
Posts: 1022
I was working on something like this a while ago, but at a different level of abstraction (hooking at the system call level, rather than the library call level). My version would likely be more accurate, but yours is likely easier to write. There's some functionality that you mark as missing that I may have solutions to already. For example, one of the things I was working on was placing the game within an isolated environment so that things like save files were easily detected, and things like process IDs (which some games use as input into their random number generators) would be the same on every run. (I don't think it would translate exactly, though.) The way you're trying to go about savestating could possibly work in theory, but is necessarily fragile (many games use simpler allocators) and is also very complex. There's a simpler approach using the program's memory map tables (savestating at the level of mmap/sbrk, rather than malloc). Some savestating programs, such as CRIU, already exist on Linux. You might want to get into contact with dwangoAC, who was looking at CRIU and related programs in the context of TASing. (He's likely busy with GDQ stuff at the moment, but may be able to give you some pointers as to how to get started with that.) The main problems I was running into were to do with threading (I wanted to handle it deterministically in all circumstances). I'd just about reached the point of deciding that a kernel patch would be helpful for that when I got distracted by other things and stopped. There are some fundamental differences between our models for running the program, e.g. I assume you're allowing the program to connect to the screen directly, whereas I was using xvfb (and hadn't reached the point of simulating clicks yet). However, I doubt there's anything insurmountable there. I'll be idling in #tasemu on Freenode over the next few days if you want to talk to me about anything.
keylie
He/Him
Editor, Emulator Coder, Expert player (2822)
Joined: 3/17/2013
Posts: 391
jlun2 wrote:
Would there be any plans for RAM watch?
This is not a priority, but yes. It is not a complex feature to implement, but requires time for the user interface mainly.
Fog wrote:
Is support for WINE a possibility?
I guess it may be possible, but for now you would still require to launch SDL games, which are quite uncommon on Windows. There would be several issues to address first I guess. I don't know if the hook method would work natively. Also, I don't know if the calling convention is the same for Windows DLLs and Linux shared libraries. If it is not, you have a problem here.
feos wrote:
Question: You will be able to run sdl-driven emulators that way too, right? You know... TASing emulators with no TAS tools in them.
Yes, I guess. This is conceptually making things much harder that it should be, because the emulator has much more knowledge of the game. Thanks ais523 for your feedback. I agree that my solution is less accurate in design and may completely fail on some games. This is partly because of my weak programming background and skills that I decided to go along an easy path. It is also better for my motivation to be able to get results rather quickly. I tried to look at CRIU code, but it is a very large project to understand. Also, I was thinking that as opposed to CRIU who must save the state of the process as it is, maybe we could make things easier to save the game state by shutting down some stuff. As an example, we could close all opened files before doing the savestate.
Senior Moderator
Joined: 8/4/2005
Posts: 5769
Location: Away
Would this potentially enable us to TAS the Linux versions/ports of PC games that Hourglass can't presently handle?
Warp wrote:
Edit: I think I understand now: It's my avatar, isn't it? It makes me look angry.
Joined: 9/6/2009
Posts: 24
Location: Renton, WA
Another potential source of inspiration or implementation is Mozilla's RR project (rr-project.org). This records a single execution of a Linux program and later plays it back deterministically. (It's intended for debugging. For instance, if you've got an intermittent crashing bug, you can always run your program under rr, and when it crashes play back that execution with a debugger to see what caused the crash.) It works at the system call level. It supports multiple threads (and even multiple processes), by simulating a single-core machine and deterministically scheduling threads onto that single core. (I've read about RR, but never actually used it. It looks very cool, though.)
keylie
He/Him
Editor, Emulator Coder, Expert player (2822)
Joined: 3/17/2013
Posts: 391
Here is an update on this project. In the past few months, I worked on a GUI so that it is easier for people to use it. Also, I made savestating mostly work. In short, I pause all threads, dump the memory sections of the game process and thread registers into a file. Same goes when loading, except I run the code in an alternate stack to restore the original stack. It is not currently possible to load a state when the set of active threads has changed, but it is not a big problem in practice. Extra care was done on the connection to the X server, I keep the struct holding the connection data intact when loading. Also, the game must be running using Mesa's opengl software rendering if savestates are to be used, because I cannot copy the content of the video card state. There are currently still some crashes when loading (about 1/100 for the few games I tested). I still managed to produce a TAS of TowerFall Ascension: Post #457565
Joined: 9/12/2014
Posts: 535
Location: Waterford, MI
Would freej2me run at this point?
Site Admin, Skilled player (1234)
Joined: 4/17/2010
Posts: 11251
Location: RU
I have another question about Wine. Dunno if the old answer still fully applies, and I don't mean this particular implementation this time, but with all the Linux features, should it be possible to TAS Wine application itself regardless of what the games are? I mean, once Wine is able to run some app, can Linux theoretically provide TAS tools to all such apps? Or maybe turn Wine itself into something capable, using Linux features? I ask because Hourglass-Resurrection is now effectively dead as well (sorry if you only know it from me), and Linux TAS tools are gonna be the only chance for PC games. Well, unless we decide to install Windows on computer emulators and TAS that instead.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Alyosha
He/Him
Editor, Expert player (3514)
Joined: 11/30/2014
Posts: 2713
Location: US
feos wrote:
I ask because Hourglass-Resurrection is now effectively dead as well (sorry if you only know it from me), and Linux TAS tools are gonna be the only chance for PC games. Well, unless we decide to install Windows on computer emulators and TAS that instead.
Is it really dead? D: I was really hopeful for that one. I hope there will be some kind of post mortem if this is the case.
Site Admin, Skilled player (1234)
Joined: 4/17/2010
Posts: 11251
Location: RU
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
keylie
He/Him
Editor, Emulator Coder, Expert player (2822)
Joined: 3/17/2013
Posts: 391
I managed to launch a flash file and print the first frame, but then it freezes. It is probably because the tool does not support the specific screen draw function that was used and could not call all the frame boundary routines (advance time, read inputs, etc.). The tool is game-oriented, so it is less likely to support general purpose graphical programs.
Joined: 10/18/2011
Posts: 64
On the Steam topic, this exists: https://forum.csmania.ru/viewtopic.php?f=21&t=42071 I've never used it but I'd expect all the heavy lifting is already done for you. ....That's assuming that the source is available anywhere, and open... which are huge assumptions. I'm curious why you wouldn't think it would be legal though. Distributing the Steam DLL would be illegal, but using a fake Steam framework would be a violation of the EULA at worst, which might get a Steam account de-activated (if they somehow caught you?) but wouldn't be illegal. Edit: Updated link - [MOD EDIT: Removed link to piracy content website. -Mothrayas]
keylie
He/Him
Editor, Emulator Coder, Expert player (2822)
Joined: 3/17/2013
Posts: 391
Here is an update on the libtas project: GUI The GUI was completely rewritten to use Qt5. This results in a much better look, user experience and also a much cleaner code. This also fixed a few bugs from the old toolkit Controller inputs Together with the GUI rewrite, the controller inputs window was finished. Now controller inputs can be sent to the game, including analog inputs from sticks and triggers. I'm simulating inputs from an xbox360 controller, it is a rather standard controller that most games recognize and have a mapping for. RAM Search/Watch A basic RAM Search/RAM Watch feature is implemented. You can search on every kind of memory segments of the game (code/static data/heap/anonymous memory/etc.). You can search for integers (signed/unsigned 1-byte -> 8-byte) and float/double. You can filter results by comparing against the previous value or a constant value. For games with high memory usage, the memory search can break if stores too many values (> 1 GB for me) Releases I start using version numbering and releasing compiled versions (both deb file and plain binaries) so that people don't have to compile it if they don't want to. I also started to write usage instructions on the github wiki page. ToDo - Increase game compatibility - Implement incremental savestates for much lower savestate size and faster reading/writing - Try to handle multithreading undeterministic behavior (run some thread code in the main thread, play with the thread scheduler...) - Improve savestate code so that it can recover destroyed threads. For now, the set of running threads must not have changed to be able to load a savestate
juef
He/Him
Player (123)
Joined: 1/29/2007
Posts: 204
Location: Québec, Canada
Press Start and hope.
:D Seriously though, this is amazing. Thank you for your hard work!
Joined: 9/12/2014
Posts: 535
Location: Waterford, MI
So this project supports OpenGL games? A lot of emulators out there rely on OpenGL. Such as rpcs3 and ppsspp. I don’t have Linux installed on my machine and have really bad internet back here, so could someone test those emulators?
keylie
He/Him
Editor, Emulator Coder, Expert player (2822)
Joined: 3/17/2013
Posts: 391
Some major updates, and I also edited the first post of this topic: - an input editor, to modify inputs during recording/playback. It is quite minimal right now, but it also supports inserting/deleting frames - incremental savestates. Except for the first one, savestates now take much less space by only storing memory that has been modified since last savestate. Savestates can also be stored either in disk or RAM. Both features are trying to address the very long savestate time for games that use a lot of memory (and users without an SSD) - pointer scanning. Inspired by Cheat Engine, we can scan for a chain of pointers->offset->pointers->offset->... to a specific address, and store this in a ram watch, so that watches can be valid upon game restart.
Post subject: This works with love 11.1, almost! I'm making a mari0 TAS.
Experienced player, Moderator, Senior Ambassador (897)
Joined: 9/14/2008
Posts: 1007
First, you're a true hero for starting on this. This is an amazing project and I'll be highlighting it substantially. I've started making a TAS in the game mari0 which uses the love framework (with the current version at https://github.com/Stabyourself/mari0 working with the latest 11.1 release of love). The love framework combines OpenGL and SDL2 in a Lua game engine and there are a number of interesting games for the project but I'm most amused at the idea of Mario holding a Portal gun so I'm starting with that. I've run into an issue which I've submitted at https://github.com/clementgallet/libTAS/issues/52 - in brief, if we allow music to play it calls love.audio.stop() which causes the game to hang (no frames advance). I've been working with the devs on their Discord server at https://discord.gg/znze7SQ and we've hacked together a version that removes all those calls which means it's possible to make a TAS but without music. I plan to play around with making a -1 run sans the music on my stream tonight at http://twitch.tv/dwangoAC and I'll let you know how it goes but I'm very excited!
I was laid off in May 2023 and could use support via Patreon or onetime donations as I work on TASBot Re: and TASBot HD. I'm dwangoAC, part of the senior staff of TASVideos as the Senior Ambassador and BDFL of the TASBot community; I post TAS content on YouTube.com/dwangoAC based on livestreams from Twitch.tv/dwangoAC.
Joined: 9/12/2014
Posts: 535
Location: Waterford, MI
How do I compile latest dev build? And how would I test wine with it? I would like to send bug reports and add to capability list. Thanks in advance!
Alyosha
He/Him
Editor, Expert player (3514)
Joined: 11/30/2014
Posts: 2713
Location: US
This is some cool stuff, great work getting this to the level where it is accepted by the site . I’ll be really looking forward to what crazy tases come out of it.
Dimon12321
He/Him
Active player (478)
Joined: 4/5/2014
Posts: 1120
Location: Ukraine
Not a wide list to play. Is there any 3D games compatible with libTAS? Does Steam Play work?
TASing is like making a film: only the best takes are shown in the final movie.
Patashu
He/Him
Joined: 10/2/2005
Posts: 4000
Dimon12321 wrote:
Not a wide list to play. Is there any 3D games compatible with libTAS? Does Steam Play work?
Jazzpunk is 3D! Everything else confirmed to work seems to be 2D (well, Fez is 2.5D...) Steam probably won't work, because it says 'Also, Steam games won't work because Steam messes up with our code injection, so you must check if a drm-free version of the game exists.'.
My Chiptune music, made in Famitracker: http://soundcloud.com/patashu My twitch. I stream mostly shmups & rhythm games http://twitch.tv/patashu My youtube, again shmups and rhythm games and misc stuff: http://youtube.com/user/patashu