1 2
Posted: 5/1/2011 5:18 PM
Post subject: Mosaics of publication screenshots
Quote
Randil
Skilled player (1886)
Joined: 4/20/2005
Posts: 2160
Location: Norrköping, Sweden
I have made a few mosaic images of publication screenshots, using all images of all publication screenshots. I thought you might be interested. Here are a few: (Arcade) The Simpsons Donkey Kong Country 2 Super Mario World Gunstar Heroes These were made by a matlab program I wrote. Any thoughts or ideas?
My own page on this site.
Posted: 5/1/2011 5:22 PM
Quote
Derakon
Joined: 7/2/2007
Posts: 3960
Seems to work pretty well, but using all the screenshots to make a larger screenshot means that the larger image has large areas that are all the same color, which result in the same smaller screenshot being used over and over again. Maybe if you used the program to recreate one of the N64 or PSX screenshots?
Pyrel - an open-source rewrite of the Angband roguelike game in Python.
Posted: 5/1/2011 5:24 PM
Quote
GabCM
He/Him
Joined: 5/5/2009
Posts: 901
Location: QC, Canada
This is AWESOME! Nice work Randil, and like Derakon said, I'm looking forward to see some N64/PSX recreated that way.
Posted: 5/1/2011 5:26 PM
Quote
Patryk1023
He/Him
Joined: 3/1/2011
Posts: 288
Location: Inside out house.
I want to see Super Mario Bros (NES) image, if you can do it. btw. Images are very good, I'm not saying, they're bad!
<Nach> scrimpy is fretty with her sunglasses on I'm here. never visible.
Posted: 5/1/2011 5:39 PM
Quote
Randil
Skilled player (1886)
Joined: 4/20/2005
Posts: 2160
Location: Norrköping, Sweden
:) Here are some more, including N64 and PSX (I tried to choose colorful screenshots): Mischief Makers Super Mario Bros Mega Man 8 Regarding what you said Derakon, I think you are right. It shouldn't be too hard to modify the code to randomly pick, say, 1 of the 3 best matches. I might give that a try.
My own page on this site.
Posted: 5/1/2011 6:00 PM
Quote
DK64_MASTER
Joined: 10/24/2005
Posts: 1080
Location: San Jose
Can you post the source and/or algorithm used to generate these images? I'm assuming you grab a bunch of screenshots, assign them an "average pixel value", and then try to correlate each individual pixel in the publication image? It seems like if you post a histogram of the images uses, there are quite a few used extensively, and some outliers? Very awesome concept, I am very impressed :).
<agill> banana banana banana terracotta pie! <Shinryuu> ho-la terracotta barba-ra anal-o~
Posted: 5/1/2011 6:20 PM
Quote
Randil
Skilled player (1886)
Joined: 4/20/2005
Posts: 2160
Location: Norrköping, Sweden
Yeah, you're pretty much spot on. The script starts by calculating the average color value of all the images that will make up the mosaic. Then it loops through all the "mosaic windows" in the image you want to make a mosaic of, and calculate that average color. It then finds the picture in your database with the shortest "distance" to the average color of this window. The distance is measured by: d=abs(c(1)-d(1)) + abs(c(2)-d(2)) + abs(c(3)-d(3)) where c and d are the vectors with the RGB values. You can of course play around with other distance functions, but I found this one to work well. That's pretty much it. Here is the code (I tried to comment it as good as I could) http://pastebin.com/8MCGmzbY
My own page on this site.
Posted: 5/1/2011 8:21 PM
Quote
Derakon
Joined: 7/2/2007
Posts: 3960
Are you weighting R, G, and B equally? If I recall correctly human vision gives more weight to G than the other two so your distance algorithm should have custom weights for each channel. This page has some more information (scroll down to the "RGB -> XYZ conversion" section). I don't think it makes much difference, but it's interesting to know about.
Pyrel - an open-source rewrite of the Angband roguelike game in Python.
Posted: 5/1/2011 8:29 PM
Quote
Randil
Skilled player (1886)
Joined: 4/20/2005
Posts: 2160
Location: Norrköping, Sweden
Derakon wrote:
Are you weighting R, G, and B equally? If I recall correctly human vision gives more weight to G than the other two so your distance algorithm should have custom weights for each channel. This page has some more information (scroll down to the "RGB -> XYZ conversion" section). I don't think it makes much difference, but it's interesting to know about.
Very interesting, I didn't know about this. I'll give it a try and see how it works out.
My own page on this site.
Posted: 5/1/2011 8:32 PM
Quote
Brandon
He/Him
Editor, Player (190)
Joined: 11/21/2010
Posts: 913
Location: Tennessee
Awesome! Keep them coming! Maybe you could do [1755] Genesis Contra: Hard Corps "best ending" by Soig in 17:36.97? That movie was the first run I published, and I still love that screenshot.
All the best, Brandon Evans
Posted: 5/1/2011 10:06 PM
Quote
Tub
Joined: 6/25/2005
Posts: 1377
You will get much better results if you don't just compare the average color of "mosaic window" & tile screenshot, but consider image structure. A mosaic window that looks like 
######
#####.
####..
###...
##....
#.....
is better approximated by a screenshot with diagonal features than something that averages to gray. You can simply scale the screenshots to match the resolution of the mosaic window, then compare differences for each pixel. Sum up the errors (or squared errors) of each pixel and pick the best tile. Since this is computationally expensive, it may be wise to use your averaging method as a first step to filter out any screenshots that are way off, then use pixel-perfect matching on the best fits to determine the winner. Further improvements can be made by disallowing repeating screenshots. That's easy to code, but obviously needs a larger library of screenshots to work. (of course there are plenty of pre-made programs that can already do this, but where's the fun in that? ;))
m00
Posted: 5/2/2011 4:48 AM
Quote
Randil
Skilled player (1886)
Joined: 4/20/2005
Posts: 2160
Location: Norrköping, Sweden
Tub wrote:
You will get much better results if you don't just compare the average color of "mosaic window" & tile screenshot, but consider image structure. A mosaic window that looks like 
######
#####.
####..
###...
##....
#.....
is better approximated by a screenshot with diagonal features than something that averages to gray. You can simply scale the screenshots to match the resolution of the mosaic window, then compare differences for each pixel. Sum up the errors (or squared errors) of each pixel and pick the best tile. Since this is computationally expensive, it may be wise to use your averaging method as a first step to filter out any screenshots that are way off, then use pixel-perfect matching on the best fits to determine the winner. Further improvements can be made by disallowing repeating screenshots. That's easy to code, but obviously needs a larger library of screenshots to work. (of course there are plenty of pre-made programs that can already do this, but where's the fun in that? ;))
I have a version of my script that (kinda) works like this, that compares pixel by pixel, but it's not perfect. I will have to try this too. I just realized I now have a whole bunch of ideas to test. Thanks for all the help. :)
My own page on this site.
Posted: 5/2/2011 2:53 PM
Quote
DK64_MASTER
Joined: 10/24/2005
Posts: 1080
Location: San Jose
Nice commented source :). I wonder if spectral analysis can be done to improve some of the edge transitions. Heck, I wonder if comparing frequency content in general will make correlation better. With MATLAB it's pretty simple to experiment with, but I think about it more when I have time.
<agill> banana banana banana terracotta pie! <Shinryuu> ho-la terracotta barba-ra anal-o~
Posted: 5/2/2011 7:40 PM
Quote
Randil
Skilled player (1886)
Joined: 4/20/2005
Posts: 2160
Location: Norrköping, Sweden
Here's Mega Man 8 again with Tub's idea implemented: http://img40.imageshack.us/img40/4118/mosaic7346258239.png Comparing this to the old version, I think it looks better, but I'm open for debate. And here you go, Brandon. (I think this one turned out really nice - you can even clearly read the digits)
My own page on this site.
Posted: 5/2/2011 8:57 PM
Quote
Derakon
Joined: 7/2/2007
Posts: 3960
That does look better, yes. Nice work!
Pyrel - an open-source rewrite of the Angband roguelike game in Python.
Posted: 5/2/2011 9:25 PM
Quote
Tub
Joined: 6/25/2005
Posts: 1377
Ahh, nice. The immediate improvement is that the new algorithm will prefer low-contrast screenshots to fill in unicolor-areas, removing a lot of noise from the mosaic. Didn't think of that :) There seem to be a few places where the algorithm picked structured screenshots to emulate image structure, but that's barely visible in this example.
m00
Posted: 5/2/2011 9:29 PM
Quote
scrimpeh
Editor, Skilled player (1404)
Joined: 3/31/2010
Posts: 2086
TAS of 2017 First edition TAS of 2017 PSX TAS of 2017
This of course leads to another fun game: How many publication screenshots can you recognize?
Posted: 5/2/2011 9:29 PM
Quote
GabCM
He/Him
Joined: 5/5/2009
Posts: 901
Location: QC, Canada
What would be the best is that you can make full-resolution mosaics, like without any resize, so we can see every detail about the screenshots you've used. But, on the other hand, the current image files are already big, in terms of file size, and that would make them bigger.
Posted: 5/2/2011 9:33 PM
Quote
Nahoc
Editor, Experienced player (852)
Joined: 8/12/2008
Posts: 845
Location: Québec, Canada
TASer of 2012 3D Platforms TASer of 2012 TAS of 2012 Speedy TAS of 2012 3D Platforms TAS of 2012
Very impressive!
Posted: 5/2/2011 9:52 PM
Quote
Derakon
Joined: 7/2/2007
Posts: 3960
Mister Epic wrote:
What would be the best is that you can make full-resolution mosaics, like without any resize, so we can see every detail about the screenshots you've used. But, on the other hand, the current image files are already big, in terms of file size, and that would make them bigger.
I think a better way to try this might be to have each screenshot account for more pixels in the macro image. How good of a mosaic can you make when each screenshot represents a 3x3 grid of pixels? Then image structure becomes much more important.
Pyrel - an open-source rewrite of the Angband roguelike game in Python.
Posted: 5/3/2011 4:51 AM
Quote
Randil
Skilled player (1886)
Joined: 4/20/2005
Posts: 2160
Location: Norrköping, Sweden
The reason why the mosaics aren't bigger are frankly because matlab can't handle any bigger matrices. Any bigger and matlab simply tells you that it's "out of memory". On the other hand, matlab could be used to find what screenshots that make up the mosaic, and then another program could be used to put them together, but I don't know any good programs for this (I have tried ImageMagick, but it also has some problems with very large mosaics.)
My own page on this site.
Posted: 5/3/2011 5:01 AM
Quote
DarkKobold
Skilled player (1637)
Joined: 11/15/2004
Posts: 2202
Location: Killjoy
Arcade TASer of 2010 PCE TASer of 2010 Arcade TAS of 2010 Lucky TAS of 2009 Lucky TAS of 2007
Randil wrote:
The reason why the mosaics aren't bigger are frankly because matlab can't handle any bigger matrices. Any bigger and matlab simply tells you that it's "out of memory". On the other hand, matlab could be used to find what screenshots that make up the mosaic, and then another program could be used to put them together, but I don't know any good programs for this (I have tried ImageMagick, but it also has some problems with very large mosaics.)
matlab64? also, it would be interesting to export an entire run to PNG, and then have all of the screenshots possible. Note, that in a 'commercial' version of these, they never use the same scene twice.
Sage advice from a friend of Jim: So put your tinfoil hat back in the closet, open your eyes to the truth, and realize that the government is in fact causing austismal cancer with it's 9/11 fluoride vaccinations of your water supply.
Posted: 5/3/2011 5:02 AM
Quote
Derakon
Joined: 7/2/2007
Posts: 3960
Hm, I've used ImageMagick to compose an 18000x18000 image. It wasn't exactly what you'd call fast, but it worked just fine. In my case I started with an 18000x18000 base and overlaid smaller images on top of it (the final output being a 10x bigger version of this image), but I don't see why you couldn't achieve a mosaic through judicious use of the -append and +append commands and a few intermediary images.
Pyrel - an open-source rewrite of the Angband roguelike game in Python.
Posted: 5/3/2011 5:12 AM
Quote
Randil
Skilled player (1886)
Joined: 4/20/2005
Posts: 2160
Location: Norrköping, Sweden
I'm using matlab 32-bit. For Imagemagick, the command I used was "montage -adjoin -mode Concatenate -tile 3072x2304 (list all the images that will make up the mosaic) montage". But the list got too long for the command window, so I couldn't paste it in one chunk. Thinking about it now, maybe I could write a .bat-file for the script? I think I will give ImageMagick another try. :)
My own page on this site.
Posted: 5/3/2011 5:16 AM
Quote
Derakon
Joined: 7/2/2007
Posts: 3960
Yeah, I didn't try to do all the images at once. Much easier on my brain to do them one at a time and for me it didn't really matter since there were only ~20 images to append anyway. You have far more images to deal with so one at a time would probably involve too much memory loading/unloading to be performant, but one column at a time (or some similar limited-but-large number of macro pixels) would probably work just fine.
Pyrel - an open-source rewrite of the Angband roguelike game in Python.
1 2