Post subject: Networking with lua: chat client
Joined: 1/26/2009
Posts: 558
Location: Canada - Québec
I believe there aren't many projects which uses lua with sockets. So far, there's this project by XKeeper that implements a netplay feature for Tetris, but looking at what the source may look like, things seem to get messy quick. There's also my attempt at using remote calls over sockets, but this I guess it may not be easy to use in practice. That being said, there are certainly other projects with networking features in "real time" in lua related to emulators, but I didn't find source code about it. Anyway, while it's definitely not easy to debug what's going on a network, I believe there are a couple of ways to reduce the overhead, such as:
  • Avoid blocking calls which depends on emu.frameadvance
  • Make a minimal script version that can run directly into the native lua console, for debug/testing purpose
  • If you use a particular resource, make sure you control access in a mutual exclusive way
  • Make modular code, favor good idiom, use message queue, etc
So, where's to begin? Well, I just made a simple chat client. So maybe this can be used as a start point for something more complex. The chat client is made in MoonScript. Basically, this can be "recompiled" to Lua 5.1 and run in BizHawk without issue. MoonScript works the same way as C++ can wrap C... or just as "CoffeeScript/TypeScript/ES6/..." can wrap JS. I figured this worked well and syntax reminds of python. Also, it's kinda easier to write modules and you can use fancy features such as "continue"(for the "for" loop), "switch" feature, etc. Also, there's a plugin available with Atom to provide a nice lint. Here's the code. Couple of technical note:
  • The Luasocket library is used for udp sockets with another client. At some point, the script could be extended to connect with other multiple clients in a P2P fashion. Other improvements that I'm still thinking is adding a TCP mode, since it can often be easier to connect with anyone on the web using TCP without packet loss, etc. Furthermore, sometimes it can also be a bit easier to tunnel an app with TCP across the web.
  • The iup library is used as a "timer", which is basically the only way to interrupt the code flow of the application, since lua is single threaded. As long, as there's no native timer functionality in the lua API, a library such as iup is being required. Without the timer, the application will have to depend on emu.frameadvance and can cause much annoyance while receiving packets,etc.
  • Currently the UI makes use of the forms library in BizHawk, so you can configure your address/ports to connect to. This means, you'll probably need to change what's going in the formConnect module, to get this working with other emulator.
  • As alternative functionality, you can also use the lua interpreter in the LuaConsole window(example: say "ohoh!")
  • For testing purpose, you can run client_udp.lua from the command line as well if you don't want to start another instance of BizHawk.
Other thoughts:
  • Despite using Lua/MoonScript to directly handle sockets in BizHawk, I guess one easier way to handle complex case would be to directly make a lua library dll in C and works from there.
  • If someone really doesn't want to have any relation to lua, etc. Well, I guess it's always possible to hack directly in the emulator and add a web/HTTP API inside BizHawk and have fun from there.
Couple of screenshots: Usage:
  • Drag the startServer.lua into BizHawk.
  • Type the udp port and ip and press connect
Troubleshoots:
  • If you have dll issues, you may have to put the dll the socket folder next to EmuHawk.exe
  • If you can't connect, make sure you can ping the remote machine and verify your firewall accept the port used.
Possible improvements:
  • Support more than 2 clients
  • Find and support a free service to tunnel message
  • Improve the serialization/deserialization
  • Support Joystick input
That's about it, let me know if you have any questions or interested in any project related.
Site Admin, Skilled player (1236)
Joined: 4/17/2010
Posts: 11268
Location: RU
We need people to actually test it over web! (I have a dynamic IP via router so I can't.) Also, mention Bizhawk in to the thread title?
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Joined: 1/26/2009
Posts: 558
Location: Canada - Québec
feos wrote:
We need people to actually test it over web! (I have a dynamic IP via router so I can't.)
Yes, it should work over the web if both clients have static IP. Maybe a free alternative that can replaces DynDNS such as noip could work. I guess it would be worth a try.
feos wrote:
Also, mention Bizhawk in to the thread title?
Right now, the UI is handled with the BizHawk forms, since I preferred to use as few libraries as possible for the project needs. Thought, the script could be adjusted to support the UI with the IUP library alone.
Site Admin, Skilled player (1236)
Joined: 4/17/2010
Posts: 11268
Location: RU
I think in the end we should try supporting dynamic IPs too. Or maybe the problem was just my router? I remember configuring it to use some ports that I can also force programs like QIP to use for file transferring.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Joined: 1/26/2009
Posts: 558
Location: Canada - Québec
Sorry for the late reply. Dynamic ip aren't required to communicate across the world if you can setup a VPN. If you need a free VPN you can probably get a free one such as Hamachi or anything else.
Site Admin, Skilled player (1236)
Joined: 4/17/2010
Posts: 11268
Location: RU
VPN sounds like an overhead. It's known that Kaillera, or example, doesn't seem to care about this. There's also an open source Kaillera replication http://p2p.kaillera.ru/ though I'm not sure if it works with dynamic ips too. Probably the mothod used there could give some hints in our case too. Anyways, I think if I enable the port in my router, I'll be able to connect using it, like it helps me with QIP file sharing.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Joined: 1/26/2009
Posts: 558
Location: Canada - Québec
Well, looking at the Kaillera case, I believe the first action you get while using the client is to contact a "main server" which return the list of other servers. I guess this main server can also help to do a handshake with other servers/clients to ease support for dynamic ip. Now, a new interesting project could be to implement a custom lobby for a chat client. Ultimately, it could be possible to join a chat room from the Kaillera server with a bit of reverse engineering, but since the project is so old, I don't think it's worth it. Maybe there are other netplay projects such as MAMEHub which could be used as a baseline for this kind of project.
Site Admin, Skilled player (1236)
Joined: 4/17/2010
Posts: 11268
Location: RU
There's also netplay in Mesen, an NES emulator whose UI is done with .net. So probably it's easier to adapt its solutions than to reinvent the wheel. https://github.com/SourMesen/Mesen/tree/master/GUI.NET/Forms/NetPlay
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Joined: 1/26/2009
Posts: 558
Location: Canada - Québec
Yes, Mesen NetPlay feature doesn't require a third party server, so it's good a baseline. First, I guess a client-spectator feature could be done in Lua as well. Once the client is connected, the server shall send a savestate to his client, then the server should send his input for a specific amount of time to the client.
Site Admin, Skilled player (1236)
Joined: 4/17/2010
Posts: 11268
Location: RU
Sour wrote:
feos wrote:
Sour, hello there! I know that your GUI is in C# and it has netplay. Could you give us some hints on how to port this feature over to bizhawk, which is also C#/.net? Also, does it connect when the user is using a router and needs a specified port, or it just connects no matter what like kaillera?
Hey :) The netplay itself is all coded in C++, but there's no reason you couldn't implement the same concept in C#. The concept is pretty simple: the server receives the current state of each client's controller, and applies that to the input ports ($4016/4017). The server then sends the result in the form of a "movie" stream (i.e what inputs were pressed at what moment) and the clients play that movie. The server/client never go out of sync, but it's not that great when ping is high (i.e 150+ms) - other than that it works pretty nicely (and it's certainly better than the experience I had with ZSNES 10+ years ago :p). It has a lot of other features (e.g: selecting which controller you're using, server loading games/states, server-controlled cheat codes, etc), but nothing that is all that hard to implement. I'm not too certain on the specifics of Kaillera, but Mesen doesn't use a central server/hub. It does however uses UPnP to contact the router & add a port forwarding rule when the server is started, which means it should work out of the box with most routers and the typical LAN setup most of us have at home. Technically, you can also get around this by using UDP hole punching, but Mesen currently uses TCP, so I never looked into it much.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Site Admin, Skilled player (1236)
Joined: 4/17/2010
Posts: 11268
Location: RU
Mesen has been tested and it has an increasing lag.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Alyosha
He/Him
Editor, Expert player (3522)
Joined: 11/30/2014
Posts: 2728
Location: US
feos wrote:
Mesen has been tested and it has an increasing lag.
How much lag feos? Anything quantitative? I wonder if it would be better to just send the client audio/video instead of having it run the movie. They can be streamed as they are generated I suppose which might be faster but i dont know anything about networking.
Joined: 6/29/2016
Posts: 53
feos wrote:
Mesen has been tested and it has an increasing lag.
If it does, it's probably only because of this: https://github.com/SourMesen/Mesen/blob/master/Core/GameClientConnection.cpp#L152 The netplay gradually increases the buffer's size whenever it becomes empty (typically because of network lag), but never tries decreasing it again (it probably should). When/if it reaches 10, that's typically at least 10 frames of lag, which is pretty noticeable. I haven't really looked at the netplay code much in the last couple of years, so it could definitely be improved.
Joined: 5/16/2017
Posts: 4
feos wrote: Mesen has been tested and it has an increasing lag. If it does, it's probably only because of this: https://github.com/SourMesen/Mesen/blob/master/Core/GameClientConnection.cpp#L152 The netplay gradually increases the buffer's size whenever it becomes empty (typically because of network lag), but never tries decreasing it again (it probably should). When/if it reaches 10, that's typically at least 10 frames of lag, which is pretty noticeable. I haven't really looked at the netplay code much in the last couple of years, so it could definitely be improved.
Even 3 frames is too much (tested with 100 mbps and 47ms ping). When I set delay frames to 3 from 0, then no lag for client, but random buttons on lag. Netplay comparing(Subjective test): 1. Mednafen type: client-server-client system: nes, smd, psx etc +: savestates online up to 8 players(or more?) take/swap control between players you can host your own server low latency reconnect -: drop fps even if you have good connect server does saves a pressed buttons of gamepad on frameskip (looks like a random buttons on lag) if one player have good internet and other have poor internet, it's not playable for second player(massive lag) savestates doesnt present on all systems(example snes) Summary: probably best emulator for online playing with 2+ players, but if you have good internet 2. Kaillera(nestopia, mame, gens etc) type: p2p,client-server-client system: nes, arcade etc +: up to 4 players(or more?) in server mode lobby (server mode) you can host your own server(dont really needed, optional) individual frame latency -: no reconnection no savestates freeze on lag even if good internet desync(extremely rare, but...) Summary: if you have poor internet, it's probably best solution for you. 3. KegaFusion type: p2p system: MD, SCD, SMS, GG etc +: up to 4 players swap control until start no lags at all (but look first negative point) -: frame latency for every player based on maximum ping of player(but still playable with 150 ms ping) no reconnection no savestates desync(extremely rare, but...) Summary: best Sega emulator for netplay, but without savestates... 4. Mesen type: p2p system: nes +: savestates online reconnect(really not tested, but based on previous point) no lags (but look first negative point) -: frame latency very high for client even if both players have good internet(not playable) Summary: ~0,1s delay with 40ms ping until 1s(!!!) with 100+ms ping? 5. RetroArch type: p2p system: libretro =) +: savestates online reconnect(really not tested, but based on previous point) -: desync without check frames with check frames some platforms unplayable(or laggy) stateless mode is terrible even with 100mbps and 10 ms ping(?!) Summary: unstable netplay with reconnection and savestates. Playable only with perfect internet. PS Any P2P netplay worse than server netplay, PSS What about Open Kaillera? Will anyone add savestates to this project?
Joined: 6/29/2016
Posts: 53
doomday45 wrote:
-: frame latency very high for client even if both players have good internet(not playable) Summary: ~0,1s delay with 40ms ping until 1s(!!!) with 100+ms ping?
This sounds like a bug in the netplay logic more than anything else. I've played several hours' worth of time with my brother via netplay in the past, and it was definitely very playable for both of us. I've also gotten feedback from other users via email that they used the netplay feature to play with friends, too. So it may not work perfectly in all scenarios, but definitely some. Like I said though, it could most likely be improved - it just hasn't been my focus. The problem is that developing network code that is highly sensible to latency on my own (e.g on a single computer) is relatively complex to test/debug.
Joined: 5/16/2017
Posts: 4
Sour wrote:
doomday45 wrote:
-: frame latency very high for client even if both players have good internet(not playable) Summary: ~0,1s delay with 40ms ping until 1s(!!!) with 100+ms ping?
This sounds like a bug in the netplay logic more than anything else. I've played several hours' worth of time with my brother via netplay in the past, and it was definitely very playable for both of us. I've also gotten feedback from other users via email that they used the netplay feature to play with friends, too. So it may not work perfectly in all scenarios, but definitely some. Like I said though, it could most likely be improved - it just hasn't been my focus. The problem is that developing network code that is highly sensible to latency on my own (e.g on a single computer) is relatively complex to test/debug.
I did corrected test. Now 0,1 delay with 20ms ping and 0,5-1s with 50ms ping. Looks like a 3 and 10 not frames really. PS KegaFusion have similar logic and there is no problem.
Joined: 6/29/2016
Posts: 53
I did a few tests using clumsy: https://github.com/jagt/clumsy With my ping set to 20ms, I get around 30-40ms of delay. A ping of 100ms, gives about 160ms of delay. A ping of 500ms, gives ~650ms of delay. These are pretty much what I would expect the numbers to be: 500ms of ping + 10 frames of buffer means ~660ms in theory. The 3/10 buffer size values are not exactly a number of frames, but rather (in the case of standard controllers) the number of times the controller data is refreshed. In the majority of games this happens once per frame, though. What game(s) are you testing with? What OS are you testing with? If it's Linux, the network socket code for Linux hasn't been tested much, so it might have more issues than the Windows version.
Joined: 5/16/2017
Posts: 4
Sour wrote:
I did a few tests using clumsy: https://github.com/jagt/clumsy With my ping set to 20ms, I get around 30-40ms of delay. A ping of 100ms, gives about 160ms of delay. A ping of 500ms, gives ~650ms of delay. These are pretty much what I would expect the numbers to be: 500ms of ping + 10 frames of buffer means ~660ms in theory. The 3/10 buffer size values are not exactly a number of frames, but rather (in the case of standard controllers) the number of times the controller data is refreshed. In the majority of games this happens once per frame, though. What game(s) are you testing with? What OS are you testing with? If it's Linux, the network socket code for Linux hasn't been tested much, so it might have more issues than the Windows version.
Windows 7 x64 both PC. (both have router with NAT and firewall) Battletoads and Double Dragon(I think best game for testing netplay, because many random values) I did tested with this player Kaillera(server) with settings - LAN (1st player - 21 ms to server, 2nd player - 29 ms to server) and there is no lag at all(sometimes frameskip). When I tested Mesen with other player - 20 ms ping, but anyway there was low delay. You can think that doesn't matter. But I did tested with this player at Retroarch(P2P) after. And It was ideally netplay without any lag or frameskip.
Joined: 5/16/2017
Posts: 4
Most likely, Mesen did not receive the corrected netplay. One of the options that would make netplay honest: 1. The host should also play with a delay Ideally, the emulator should adjust the frame delay according to the ping between the players. 2. The frame delay for both players should be the same or differ in 1-2 frames. This option is the most honest. The client will not receive an increased frame delay if the ping between players is 50 ms. This method is used in Kega Fusion and supports up to 4 players. And in fact is the best and most honest version of the netplay.