1 2
5 6 7
Post subject: Lua code for mouse control in Mario Paint
Active player, Editor (296)
Joined: 3/8/2004
Posts: 7468
Location: Arzareth
Here is my Lua function for producing a mouse click at a given coordinate in Mario Paint. It usually achieves the movement in a few frames, but sometimes it gets stuck in an infinite loop, circling the intended coordinates in an everlasting attempt to compensate for the perceived error. Does anyone have will to try to improve it? This is at the core of my Mariopaint TAS.
Language: lua

local sendx,sendy = 128,127 local click = false local curx,cury = "?","?" local function getcur() curx,cury = memory.readbyte(0x7E0226), memory.readbyte(0x7E0227) end local function next() joypad.set(1, { x = sendx, y = sendy, left=click } ) emu.frameadvance() getcur() local valid_location = AND(memory.readbyte(0x7E0426),1) gui.text(10,160, "cur="..curx..","..cury..";send="..sendx..","..sendy..",valid="..valid_location) end local function ismousedown() return memory.readbyte(0x7E00DE) >= 0x40 end local function mousegoto(x,y) local s = savestate.create(1) local prompt = "curx="..curx..","..cury..";goal="..x..","..y..";send="..sendx..","..sendy maxadjust = 97 while(true) do -- Go forward a few frames to get a stable reading getcur() local origx,origy = curx,cury gui.text(40,140, prompt) savestate.save(s) local n for n = 1,3 do next() gui.text(40,140, prompt) end if cury == 244 then break end local valid_location = AND(memory.readbyte(0x7E0426),1) == 0 savestate.load(s) --if((origx ~= curx) or (origy ~= cury)) then prompt = "cur="..curx..","..cury..";orig="..origx..","..origy..";goal="..x..","..y..";send="..sendx..","..sendy -- Find out how much the current address must be corrected -- to reach at the desired endpoint local errx = x - curx local erry = y - cury if (errx == 0) and (erry == 0) then if(valid_location) then break end errx = 128 end --if(errx > 128) then errx = errx - 256 end --if(errx < -128) then errx = errx + 256 end --if(erry > 128) then erry = erry - 256 end --if(erry < -128) then erry = erry + 256 end if(errx > maxadjust) then errx = maxadjust end if(errx < -maxadjust) then errx = -maxadjust end if(erry > maxadjust) then erry = maxadjust end if(erry < -maxadjust) then erry = -maxadjust end next() sendx = AND(sendx + errx, 255) sendy = AND(sendy + erry, 255) --end if((origx ~= curx) or (origy ~= cury)) then maxadjust = maxadjust*math.random(6,9)/10 + math.random(1,2) end end end function clickat(x,y) click = false; while(ismousedown())do next()end while(true)do mousegoto(x,y) local s = savestate.create(1) savestate.save(s) click = true ; while(not ismousedown())do next()end if((cury == 244)or((curx == x)and(cury == y)))then break end savestate.load(s) click = false; next() end click = false; while(ismousedown())do next()end end
Active player, Editor (296)
Joined: 3/8/2004
Posts: 7468
Location: Arzareth
For the record. For the original picture (resized and sharpened): Conversions with Nitsuja's version of Scolorq (filter=3, presumably gamma=1.0). From left to right: Dithering level = 1.25, 1.0, 0.75, 0.6 and 0.5 respectively. Conversions using Yliluoma-2 dithering @ 32-color mix (no precombines), with gamma=1.0 (NOTE: gamma 1 should not be used, it is wrong): From left to right: original; RGB; CIE 76; CIE 94; CIEDE 2000. The same, with gamma=2.0: The same, with 4 colors (pointless, except to illustrate the different ΔE formulae): The same, with 2 colors: A selection of error diffusion filters, gamma 1.0: From left to right: Original, Floyd-Steinberg, Jarvis-Judice-Ninke, Sierra-3 and Sierra-2-4A The same, with gamma 2.0: Finally, Yliluoma-2 dithering with gamma=2, with 16 colors, from 16 precombined maximum-16-color unique color mixes, at 16x16 matrix, with different delta-E formulae: From left to right, RGB; CIE 76; CIE 94; CIEDE 2000; CMC; BFD. Each of these small images took something from 10―30 minutes to render (on a 4-core), but the result is definitely worth it. Lifting the unique-color restriction would possibly yield even better results. (Note that as of this posting they're still rendering, but will come up.) (Do you now see why I prefer ordered-dithering over floyd-steinberg?) (Disclaimer: It's possible though, that my implementation of error-diffusion dithers is buggy.) Appendix: 16 colors; RGB and CIE76 side-by-side. Gamma=2.0 Floyd-Steinberg: Scolorq (gamma=1.0, RGB, default options; original shown at side to show how the dithered version is lighter due to wrong gamma): 4 colors (gamma=2.0, RGB and CIE76): 2 colors 4 colors, with less choices and more duplicates 2 colors, with less choices and more duplicates Mega Man at 64 colors (minimal precombines to make it fast) at RGB,CIE76,CIEDE2000,CMC,BFD, gamma 2.2: Same, but 4 colors: Same, but 2 colors: Fewer colors are preferred if you want to increase the chances of Mario Paint's predefined dithering brushes being used. Note: When I say "64 colors" in this context, it means that for each pixel, a selection of 64 colors is formed from the 15-color palette. From that selection, the color is selected according to the dithering matrix. It does not mean that the image has 64 colors. It would also be possible to create custom dithering brushes according to whichever dithering patterns are the most common ones in the source picture (assuming you still have enough one-pixel brushes to complete the image). I considered this option, but could not figure out how to achieve it optimally. This possibility, however, exists only when using ordered dithering. EDIT: Oh, one more thing. I forgot. It is possible to mix ordered dithering and error diffusioning! It works by diffusing the error that remains after ordered-dithering, as opposed to diffusing the error that remains after nearest-color quantization. From left to right: Yliluoma-2 dithering, Yliluoma-2 + Floyd-Steinberg, Floyd-Steinberg. Gamma = 2.2, colors=15, precombine=minimal. The fourth image is of Nitsuja's version of Scolorq (where gamma=presumably 1.0) and the fifth is the original (resized+sharpened). Same, with 4 colors: Same, with 2 colors & minimum premixes: ----------- Conclusions: ― Any proper dithering algorithm should operate on gamma-corrected RGB values rather than on linear RGB values. Failure to do this will produce an image that is obviously lighter in tone than the original. This error is most pronounced when the dithered colors differ significantly from each others. ― With static images, for certain images, ordered dithering can produce at least as good pictures as error-diffusion dithering, but for many images, it is vice versa. ― All error diffusion dithers, except for scolorq, seem to suffer from a bias towards gray values. Gray is apparently a good approximation for most colors. Therefore, where error diffusion is deemed appropriate over ordered dithering, I recommend using scolorq except where its lack of gamma correction is too obvious. ― Especially at low color candidate counts, ordered dithering produces distinguished repetive patterns, which may be very beneficial in optimizing the production of the picture when patterned brushes are available (such as in Mario Paint). ― From the six color difference formulae (ΔE), an euclidean distance in the RGB space appeared to be sufficient for most purposes. In a few cases, an euclidean distance in the LAB C*i*e* space (aka. CIE 76) produced better results. The more expensive formulae (CIE 94, CIEDE 2000, CMC, BFD) produced no advantage to justify their significantly more expensive calculation. (And indeed, BFD often seemed even to produce inferior results). ― For customized needs, it often is warranted to do extensive testing with different values for gamma, and color collection/mixing, in order to produce the image that is best for the particular need. Remember, that the eye is more sensitive to local differences in color/tone than to global differences of color/tone. Therefore, it is forgivable to tweak the colorscape of the entire image at once (such as by darkening, brightening, or colorizing it), if it helps bringing out a more accurate local contrast. Such testing was not done here. ― For animation, ordering dithering should always be used rather than error diffusion dithering. To prove the point, study these three examples (from DemonStrate's Portal Done Pro speedrun... rendered in Mario Paint palette.): ―― http://bisqwit.iki.fi/kala/snap/mp/pdp_fs.gif (1.3 MB, Floyd-Steinberg @gamma=2) ―― http://bisqwit.iki.fi/kala/snap/mp/pdp_sc.gif (1.5 MB, Scolorq @gamma=1) ―― http://bisqwit.iki.fi/kala/snap/mp/pdp_y2.gif (913 kB, Yliluoma-2 @gamma=2).
Lex
Joined: 6/25/2007
Posts: 732
Location: Vancouver, British Columbia, Canada
It's clear to me that nitsuja's scolorq is the best choice!
Post subject: Re: Lua code for mouse control in Mario Paint
Emulator Coder, Skilled player (1299)
Joined: 12/21/2004
Posts: 2687
Bisqwit wrote:
Here is my Lua function for producing a mouse click at a given coordinate in Mario Paint. It usually achieves the movement in a few frames, but sometimes it gets stuck in an infinite loop, circling the intended coordinates in an everlasting attempt to compensate for the perceived error. Does anyone have will to try to improve it? This is at the core of my Mariopaint TAS.
This seems to work better for me:
Language: lua

local sendx,sendy = 128,127 local click = false local curx,cury = "?","?" local function getcur() curx,cury = memory.readbyte(0x7E0226), memory.readbyte(0x7E0227) local absx = memory.readwordsigned(0x7E04DC) if absx < 64 and curx > 192 then curx = curx - 256 end if absx > 192 and curx < 64 then curx = curx + 256 end end local function isbusy() return memory.readbyte(0x7E016A) == 0 end local function ismousedown() return memory.readbyte(0x7E00DE) >= 0x40 end local function next() joypad.set(1, { x = sendx, y = sendy, left=click } ) emu.frameadvance() if isbusy() then local s3 = savestate.create(3) repeat savestate.save(s3) joypad.set(1, { x = sendx, y = sendy, left=click } ) emu.frameadvance() until not isbusy() savestate.load(s3) joypad.set(1, { x = sendx, y = sendy, left=click } ) end getcur() local valid_location = AND(memory.readbyte(0x7E0426),1) gui.text(10,160, "cur="..curx..","..cury..";send="..sendx..","..sendy..",valid="..valid_location) end function mousegoto(x,y) local s = savestate.create(1) local s2 = savestate.create(2) savestate.save(s) local joy = joypad.get(1) sendx,sendy = joy.x, joy.y getcur() for attempt=1,8 do xoff,yoff = x-curx, y-cury sendx,sendy = sendx + xoff, sendy + yoff for frame=1,6 do if curx == x and cury == y then if frame ~= 1 and not isbusy() then -- TODO: sometimes this is safe to do even when isbusy() savestate.load(s2) -- save 1 frame end return true -- success end savestate.save(s2) next() end -- we landed in the wrong place. -- this will happen if the hotspot changed on us, -- or if some unknown force is preventing our motion. -- we'll adjust our target and try again a few more times. savestate.load(s) if xoff == x-curx then if x-curx < 0 then curx = curx + 127 elseif x-curx > 0 then curx = curx - 127 end end if yoff == y-cury then if y-cury < 0 then cury = cury + 127 elseif y-cury > 0 then cury = cury - 127 end end end print('mousegoto('..x..','..y..') failed. it could only get to '..curx..','..cury..'. rolled back.') -- failure end function clickat(x,y) click = false if mousegoto(x,y) then while ismousedown() do next() end click = true while not ismousedown() do next() end click = false while ismousedown() do next() end end end
Post subject: Re: Lua code for mouse control in Mario Paint
Active player, Editor (296)
Joined: 3/8/2004
Posts: 7468
Location: Arzareth
On startup screen:
mousegoto(214,194) failed. it could only get to 214,186. rolled back. mousegoto(222,194) failed. it could only get to 222,186. rolled back. mousegoto(230,194) failed. it could only get to 230,186. rolled back.
Aww.
Emulator Coder, Skilled player (1299)
Joined: 12/21/2004
Posts: 2687
The only way I can reproduce those results is by first clicking on the T to switch to the pen tool on the title screen, but isn't 194 an invalid Y coordinate in that case? I can't get it to go below 190.
Active player, Editor (296)
Joined: 3/8/2004
Posts: 7468
Location: Arzareth
nitsuja wrote:
The only way I can reproduce those results is by first clicking on the T to switch to the pen tool on the title screen, but isn't 194 an invalid Y coordinate in that case? I can't get it to go below 190.
I see. It was a consequence of the clicks being faster now. Sorry about that. However, when attempting to access the bottom-screen tools in the drawing screen, I get: mousegoto(233,207) failed. it could only get to 223,145. rolled back. mousegoto(60,207) failed. it could only get to 60,145. rolled back.
Emulator Coder, Skilled player (1299)
Joined: 12/21/2004
Posts: 2687
This is because the target position is off the bottom of the screen for the cursor that was active before moving (due to the changing hotspot affecting the position). One way to fix this would be to switch (everywhere) to using the hotspot-independent coordinates that you would get if getcur() uses readwordsigned(0x7E04DC),readwordsigned(0x7E04DE) instead of readbyte(0x7E0226), readbyte(0x7E0227). That would probably also allow the script to run faster overall (due to fewer retries when the cursor changes). Another way that appears to work fine is to put in a hack like this:
Language: lua

local sendx,sendy = 128,127 local click = false local curx,cury = "?","?" local function getcur() curx,cury = memory.readbyte(0x7E0226), memory.readbyte(0x7E0227) local absx = memory.readwordsigned(0x7E04DC) if absx < 64 and curx > 192 then curx = curx - 256 end if absx > 192 and curx < 64 then curx = curx + 256 end end local function isbusy() return memory.readbyte(0x7E016A) == 0 end local function ismousedown() return memory.readbyte(0x7E00DE) >= 0x40 end local function next() joypad.set(1, { x = sendx, y = sendy, left=click } ) emu.frameadvance() if isbusy() then local s3 = savestate.create(3) repeat savestate.save(s3) joypad.set(1, { x = sendx, y = sendy, left=click } ) emu.frameadvance() until not isbusy() savestate.load(s3) joypad.set(1, { x = sendx, y = sendy, left=click } ) end getcur() local valid_location = AND(memory.readbyte(0x7E0426),1) gui.text(10,160, "cur="..curx..","..cury..";send="..sendx..","..sendy..",valid="..valid_location) end function mousegoto(x,y) local s = savestate.create(1) local s2 = savestate.create(2) savestate.save(s) local joy = joypad.get(1) sendx,sendy = joy.x, joy.y getcur() if y > 195 and AND(memory.readbyte(0x7E0426),0x12) == 0x12 then -- screen bottom hotspot hack curx,cury = memory.readwordsigned(0x7E04DC), memory.readwordsigned(0x7E04DE) end for attempt=1,8 do xoff,yoff = x-curx, y-cury sendx,sendy = sendx + xoff, sendy + yoff for frame=1,6 do if curx == x and cury == y then if frame ~= 1 and not isbusy() then -- TODO: sometimes this is safe even when isbusy() is true. savestate.load(s2) -- save 1 frame end return true -- success end savestate.save(s2) next() if y > 195 and AND(memory.readbyte(0x7E0426),0x12) == 0x12 then -- screen bottom hotspot hack curx,cury = memory.readwordsigned(0x7E04DC), memory.readwordsigned(0x7E04DE) end end -- we landed in the wrong place. -- this will happen if the hotspot changed on us, -- or if some unknown force is preventing our motion. -- we'll adjust our target and try again a few more times. savestate.load(s) if xoff == x-curx then if x-curx < 0 then curx = curx + 127 elseif x-curx > 0 then curx = curx - 127 end end if yoff == y-cury then if y-cury < 0 then cury = cury + 127 elseif y-cury > 0 then cury = cury - 127 end end end print('mousegoto('..x..','..y..') failed. it could only get to '..curx..','..cury..'. rolled back.') -- failure end function clickat(x,y) click = false if mousegoto(x,y) then while ismousedown() do next() end click = true while not ismousedown() do next() end click = false while ismousedown() do next() end end end
Active player, Editor (296)
Joined: 3/8/2004
Posts: 7468
Location: Arzareth
nitsuja wrote:
Another way that appears to work fine is to put in a hack like this:
Sadly, it did not seem to change the situation in any manner. It gives me the exact same message... My main code:
Language: lua

while(movie.framecount() < 152) do next() end getcur() if(movie.framecount() < 530)then if(cury ~= 244) then clickat(208,84) end for y = 90,146,8 do -- for x=214,230,8 do for x=198,230,8 do if(movie.framecount() > 286)or(cury == 244) then break end if(y == 146)and(x >= 230) then break end print('trying click at('..x..','..y..')') clickat(x,y) end end while(cury ~= 108) do next() end for n=1,50 do click=not click;next() end repeat click = not click; next() until(cury < 244) end if(movie.framecount() < 1009)then clickat(233,207) --; next() clickat(60,207) --; next() -- continues here with stuff. end
Emulator Coder, Skilled player (1299)
Joined: 12/21/2004
Posts: 2687
I guess this case is from telling it to click when it can't because the cursor is frozen. Making clickat() look like this fixes the first instance for some reason:
Language: lua

function clickat(x,y) click = false while ismousedown() do next() end if mousegoto(x,y) then click = true while not ismousedown() do next() end click = false while ismousedown() do next() end end end
But the clickat(60,207) still fails after that because isbusy() doesn't detect the cursor being frozen due to switching screens. Another check could probably be added to isbusy(), or clickat() could detect that the cursor isn't moving at all and wait a few frames before trying again... or of course a workaround would be to manually add a sufficient number of calls to next(). Maybe I can look at it again tomorrow.
Skilled player (1882)
Joined: 4/20/2005
Posts: 2160
Location: Norrköping, Sweden
I present to you, Mario Paint Vision (now with dithering!) in the form of an animated GIF: http://s3.postimage.org/64jd4xcpm/res6.gif
Active player, Editor (296)
Joined: 3/8/2004
Posts: 7468
Location: Arzareth
Floyd-steinberg in animation... Errrgh. With positional dithering: http://bisqwit.iki.fi/kala/snap/mp/mar.gif (Huge disclaimer subtitles, errrgh.)
Banned User
Joined: 3/10/2004
Posts: 7698
Location: Finland
Does that really need dithering?
Joined: 11/4/2007
Posts: 1772
Location: Australia, Victoria
Bisqwit wrote:
Floyd-steinberg in animation... Errrgh. With positional dithering: http://bisqwit.iki.fi/kala/snap/mp/mar.gif (Huge disclaimer subtitles, errrgh.)
I'll use fade in next time, if you want.
Active player, Editor (296)
Joined: 3/8/2004
Posts: 7468
Location: Arzareth
Warp wrote:
Does that really need dithering?
Not really, if you are ready to accept wrong colours. As seen here: http://bisqwit.iki.fi/kala/snap/mp/mar0.gif
Banned User
Joined: 3/10/2004
Posts: 7698
Location: Finland
Bisqwit wrote:
Warp wrote:
Does that really need dithering?
Not really, if you are ready to accept wrong colours. As seen here: http://bisqwit.iki.fi/kala/snap/mp/mar0.gif
I can see cyan right there in the Mario Paint palette. I think the algorithm that decides the gray is closer to cyan than cyan itself is not very well done.
Active player, Editor (296)
Joined: 3/8/2004
Posts: 7468
Location: Arzareth
Warp wrote:
I can see cyan right there in the Mario Paint palette. I think the algorithm that decides the gray is closer to cyan than cyan itself is not very well done.
The algorithm that chooses nearest palette colors used an euclidean RGB distance to determine the nearest color, and came up with what you saw. The theory of measuring color differences is currently an open area of science with a number of different formulas devised during decades. You can read about it at http://en.wikipedia.org/wiki/Color_difference . For the same price, I rendered that animation with a number of different color difference formulae. These are: RGB, CIE-76, CIE-94, CIEDE-2000, BFD, CMC The CIE-based formula (all but RGB) were based on the CIE L*a*b* colorspace (or LCh), which are created from RGB using a conversion matrix called "white point". IIRC my program uses the Adobe D65 whitepoint matrix. The cyan of Mario Paint is considerably darker and of different hue & saturation than the cyan background in that SMB3 level. Different enough to cause all of these algorithms to choose another color as the nearest one. As can be deduced from the dithering patterns in http://bisqwit.iki.fi/kala/snap/mp/mar.gif, it only becomes a close representation of the video's color by the addition of great amount of white, a token amount of yellow and a token amount of gray.
Banned User
Joined: 3/10/2004
Posts: 7698
Location: Finland
Bisqwit wrote:
The cyan of Mario Paint is considerably darker and of different hue & saturation than the cyan background in that SMB3 level.
Maybe it's just my monitor, but it looks pretty darn close to me: If the color picking algorithms are that bad, then perhaps the colors should be picked manually. After all, there aren't that many to choose from.
Active player, Editor (296)
Joined: 3/8/2004
Posts: 7468
Location: Arzareth
Warp wrote:
Maybe it's just my monitor, but it looks pretty darn close to me:[..] If the color picking algorithms are that bad, then perhaps the colors should be picked manually. After all, there aren't that many to choose from.
Side by side comparison of the two colors (MP: #00F8F8, SMB: #9CFBF0): Again, what's obvious to the human eye is not obvious to the mathematic formula. Also, a global change of color temperature / brightness is often completely ignored by a human, whereas a local change of colors is noticed more easier. A good algorithm could tweak the global color temperature or the global brightness to invoke better color matches, but I know of no such algorithm.* All of these color difference algorithms are extremely local: They work on two individual colors at a time. As for the "manual" option, I don't see how picking the colors manually is reasonably doable when the input image contains 10303 colors (due to filtered rescaling). *) Hmm, maybe I should work on that a bit.
Banned User
Joined: 3/10/2004
Posts: 7698
Location: Finland
Bisqwit wrote:
As for the "manual" option, I don't see how picking the colors manually is reasonably doable when the input image contains 10303 colors (due to filtered rescaling).
How about not scaling it, but use the original image data?
Active player, Editor (296)
Joined: 3/8/2004
Posts: 7468
Location: Arzareth
Warp wrote:
How about not scaling it, but use the original image data?
Like this, you mean?
Banned User
Joined: 3/10/2004
Posts: 7698
Location: Finland
Bisqwit wrote:
Like this, you mean?
I don't know. If that image is not scaled, then I suppose. I don't know why you were talking about there being tens of thousands of colors due to filtered scaling.
Active player, Editor (296)
Joined: 3/8/2004
Posts: 7468
Location: Arzareth
Warp wrote:
Bisqwit wrote:
Like this, you mean?
I don't know. If that image is not scaled, then I suppose. I don't know why you were talking about there being tens of thousands of colors due to filtered scaling.
Because filtered scaling (resizing) produces a myriad of tones. Not resizing it means that it would have to be cropped (portions from some edges are removed). If you unfiltered-scale it (i.e. just choose the nearest neighboring pixel), you will indeed keep the original colors, but it will still look bad, because now part of the original content is simply discarded, instead of being blended into remaining pixels. [img_left]http://bisqwit.iki.fi/kala/snap/mp/snaporig.png[/img_left] Left: Original picture. Bottom: Left to right: Lanczos-scaled; Cropped; Nearest-neighbor-scaled.
Banned User
Joined: 3/10/2004
Posts: 7698
Location: Finland
Bisqwit wrote:
Not resizing it means that it would have to be cropped (portions from some edges are removed).
Why would that be such a bad thing in this context? Just draw a portion of the image.
Active player, Editor (296)
Joined: 3/8/2004
Posts: 7468
Location: Arzareth
Warp wrote:
Bisqwit wrote:
Not resizing it means that it would have to be cropped (portions from some edges are removed).
Why would that be such a bad thing in this context? Just draw a portion of the image.
That would certainly be an option. But if the subject is an animation, then you need an algorithm (or artistic decision) that decides which section of the screen to crop to. A static crop might not be the best choice.
1 2
5 6 7