Post by fsvgm777

Posted: 12/30/2014 5:33 AM

Post subject: Encoders: Compressing JPEG

Quote

Nach

Emulator Coder

Joined: 3/9/2004
Posts: 4588
Location: In his lab studying psychology to find new ways to torture TASers and forumers

I've noticed that we haven't been doing much to compress our JPEGs. Therefore I created a new application, Crushed JPEG, which can convert various images to Crushed JPEGs. Usage is: crushedjpeg input.image output.jpg Or Usage is: crushedjpeg input.image output.jpg 85 The last parameter here is quality to preserve (0 - 100). On default settings, I cut several of our JPEGs in half, and didn't notice any significant quality loss. Please play with this and post comments. Edit: While this can compress from JPEG, it's always best to compress directly from a lossless source (yes, from PNGs is supported). Edit 2: I'm not compatible with 3D, so I don't understand what I'm looking at, so I'm not the best judge on quality here, but with default settings: 2443M.jpeg.jpg: 43827 2443M.jpeg2.jpg: 9816 Difference: -34011

I can't tell the difference, and that's a huge size savings. (Which would work better if done from a lossless original source) (If you can tell the difference, you're free to tweak the quality setting.) Edit 3: Another test with lossy source using default settings (would be nice if someone can test with lossless source). 2376M.png.jpg: 43702 2376M.png2.jpg: 17759 Difference: -25943

Edit 4: Same deal. 2587M.jpg: 43548 2587M2.jpg: 8336 Difference: -35212

2562M.jpg: 39181 2562M2.jpg: 22351 Difference: -16830

2501M.jpg: 44861 2501M2.jpg: 12611 Difference: -32250

2089M.png.jpg: 43029 2089M.png2.jpg: 15128 Difference: -27901

2009M.jpg: 37315 2009M2.jpg: 14375 Difference: -22940

Warning: Opinions expressed by Nach or others in this post do not necessarily reflect the views, opinions, or position of Nach himself on the matter(s) being discussed therein.

Posted: 12/30/2014 8:49 AM

Quote

Warp

Banned User, Former player

Joined: 3/10/2004
Posts: 7698
Location: Finland

At least in the Luigi screenshot there's a quite remarkable visual difference (the more compressed one is clearly more blurred). Not that anybody would notice in practice...

Posted: 12/30/2014 9:11 AM
(Edited: 12/30/2014 11:57 AM)

Quote

AntyMew

It/Its

Encoder, Player (35)

Joined: 10/22/2014
Posts: 425

Comparisons between lossless, 85, 95, and normal jpeg: EDIT: Fuck dropbox

normal jpeg, lossless, 95, 85

lossless, 95, normal jpeg, 85

85, 95, lossless, normal jpeg

Just a Mew! 〜 It/She ΘΔ 〜 ███████

Posted: 12/30/2014 9:13 AM

Quote

Warp

Banned User, Former player

Joined: 3/10/2004
Posts: 7698
Location: Finland

Those images are not showing up (at least here).

Posted: 12/30/2014 10:31 AM

Quote

Nach

Emulator Coder

Joined: 3/9/2004
Posts: 4588
Location: In his lab studying psychology to find new ways to torture TASers and forumers

Looking good I can't tell the difference. We may even be able to take the number lower.

Warning: Opinions expressed by Nach or others in this post do not necessarily reflect the views, opinions, or position of Nach himself on the matter(s) being discussed therein.

Posted: 12/30/2014 10:40 AM

Quote

Nach

Emulator Coder

Joined: 3/9/2004
Posts: 4588
Location: In his lab studying psychology to find new ways to torture TASers and forumers

Default quality is 75, and here's notes from the manual:

       The -quality switch lets you trade off  compressed  file  size  against
       quality of the reconstructed image: the higher the quality setting, the
       larger the JPEG file, and the closer the output image will  be  to  the
       original  input.   Normally  you want to use the lowest quality setting
       (smallest file) that decompresses  into  something  visually  indistin-
       guishable  from  the original image.  For this purpose the quality set-
       ting should be between 50 and 95; the default  of  75  is  often  about
       right.  If you see defects at -quality 75, then go up 5 or 10 counts at
       a time until you are happy with the output image.  (The optimal setting
       will vary from one image to another.)

       -quality  100 will generate a quantization table of all 1’s, minimizing
       loss in the quantization step (but there is still information  loss  in
       subsampling,  as  well  as  roundoff error).  This setting is mainly of
       interest for experimental purposes.  Quality values above about 95  are
       not  recommended  for normal use; the compressed file size goes up dra-
       matically for hardly any gain in output image quality.

Warning: Opinions expressed by Nach or others in this post do not necessarily reflect the views, opinions, or position of Nach himself on the matter(s) being discussed therein.

Posted: 12/30/2014 11:30 AM

Quote

feos

Site Admin, Skilled player (1236)

Joined: 4/17/2010
Posts: 11274
Location: RU

Woah! Gotta spend some time on testing. But I already have a usage suggestion: can you make it work by drag'n'dropping the file on a program icon? For quality input, it could just open a command prompt and ask for user numbers. This way using it will take seconds, and tons of images could be processed in a row with various settings (which would be needed to determine the best quality to size ratio for a particular screenshot). Anty-Lemon: please put spaces between images, it unbreaks the page for narrow browser windows.

Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.

Posted: 12/30/2014 11:34 AM

Quote

Nach

Emulator Coder

Joined: 3/9/2004
Posts: 4588
Location: In his lab studying psychology to find new ways to torture TASers and forumers

Or you could do it yourself. Download crushed85.bat

Language: batch
crushedjpeg "%1" output85.jpeg 85

Download crushed80.bat

Language: batch
crushedjpeg "%1" output80.jpeg 80

Download crushed75.bat

Language: batch
crushedjpeg "%1" output75.jpeg

Download crushed70.bat

Language: batch
crushedjpeg "%1" output70.jpeg 70

Edit, or a single combined script: Download crushedmany.bat

Language: batch
crushedjpeg "%1" output85.jpeg 85
crushedjpeg "%1" output80.jpeg 80
crushedjpeg "%1" output75.jpeg
crushedjpeg "%1" output70.jpeg 70

Warning: Opinions expressed by Nach or others in this post do not necessarily reflect the views, opinions, or position of Nach himself on the matter(s) being discussed therein.

Posted: 12/30/2014 11:49 AM
(Edited: 12/30/2014 11:51 AM)

Quote

feos

Site Admin, Skilled player (1236)

Joined: 4/17/2010
Posts: 11274
Location: RU

That says to me "crushedjpeg is not a command, program, or a batch file"... Yes, I'm running it just near crashed.exe.

Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.

Posted: 12/30/2014 11:51 AM

Quote

Nach

Emulator Coder

Joined: 3/9/2004
Posts: 4588
Location: In his lab studying psychology to find new ways to torture TASers and forumers

feos wrote:

That says to me "crushedjpeg is not a command, program, or a batch file"...

You have to put it in the same directory as crushedjpeg.exe and run it from there. I'll look into adding a massive batch program too.

Warning: Opinions expressed by Nach or others in this post do not necessarily reflect the views, opinions, or position of Nach himself on the matter(s) being discussed therein.

Posted: 12/30/2014 12:12 PM

Quote

Nach

Emulator Coder

Joined: 3/9/2004
Posts: 4588
Location: In his lab studying psychology to find new ways to torture TASers and forumers

Okay feos, you can redownload http://nach.tasvideos.org/crushedjpeg.7z It includes batch.exe which you can drag and drop multiple images onto, and it will create for each image compressions in intervals of 5 from 60 to 95.

Warning: Opinions expressed by Nach or others in this post do not necessarily reflect the views, opinions, or position of Nach himself on the matter(s) being discussed therein.

Posted: 12/30/2014 2:17 PM

Quote

feos

Site Admin, Skilled player (1236)

Joined: 4/17/2010
Posts: 11274
Location: RU

Does cjpeg have an option to set subsampling to 4:4:4? Without it no matter how I try, the loss is visible against GIMP, even if the file size/quality is greater than GIMP's.

Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.

Posted: 12/30/2014 3:06 PM

Quote

OmnipotentEntity

He/Him

Player (36)

Joined: 9/11/2004
Posts: 2623

Have we tested this against MozJpeg? https://blog.mozilla.org/research/2014/07/15/mozilla-advances-jpeg-encoding-with-mozjpeg-2-0/

Build a man a fire, warm him for a day, Set a man on fire, warm him for the rest of his life.

Posted: 12/30/2014 3:41 PM

Quote

dekutony

Experienced player (633)

Joined: 11/23/2013
Posts: 2208
Location: Guatemala

Noticed many differences, just because it's me. What I do to compress JPEGs is to use Gimp and save as a JPEG and then other saving settings and stuff.

Here, my YouTube channel: http://www.youtube.com/user/dekutony

Posted: 12/30/2014 4:14 PM

Quote

Nach

Emulator Coder

Joined: 3/9/2004
Posts: 4588
Location: In his lab studying psychology to find new ways to torture TASers and forumers

feos wrote:

Does cjpeg have an option to set subsampling to 4:4:4? Without it no matter how I try, the loss is visible against GIMP, even if the file size/quality is greater than GIMP's.

That should be an option for convert, part of ImageMagick, not cjpeg. If you can find the correct parameter for it, I'll add it. Edit: Actually now that I think about it, cjpeg should be the one to set it, not sure if there's an option, we'll have to look for it.

OmnipotentEntity wrote:

Have we tested this against MozJpeg? https://blog.mozilla.org/research/2014/07/15/mozilla-advances-jpeg-encoding-with-mozjpeg-2-0/

If you'd look at what this does and what MozJpeg is, you'll see your question makes no sense.

Warning: Opinions expressed by Nach or others in this post do not necessarily reflect the views, opinions, or position of Nach himself on the matter(s) being discussed therein.

Posted: 12/30/2014 6:27 PM

Quote

Nach

Emulator Coder

Joined: 3/9/2004
Posts: 4588
Location: In his lab studying psychology to find new ways to torture TASers and forumers

New version uploaded. This now uses DCT float and 4:4:4. I'm noticing significantly higher visual quality at lower quality settings. I also fixed a bug in cjpeg misparsing the dct setting. Idiots left out {} in one place. Listen carefully programmers, even though {} is optional in most languages, NEVER LEAVE THEM OUT. Because of this, batch.exe now also outputs from 50 and up. fsvgm777: Want to look at Metroid Prime Pinball again? May get even higher quality at smaller size.

Warning: Opinions expressed by Nach or others in this post do not necessarily reflect the views, opinions, or position of Nach himself on the matter(s) being discussed therein.

Posted: 12/30/2014 7:12 PM

Quote

Nach

Emulator Coder

Joined: 3/9/2004
Posts: 4588
Location: In his lab studying psychology to find new ways to torture TASers and forumers

Using Anty-Lemon's PNG's and "regular JPEG" from above, and new version of Crushed JPEG: Order is PNG, Crushed JPEG, "Regular JPEG":

26,904 -> 18,105

31,118 -> 20,537

Warning: Opinions expressed by Nach or others in this post do not necessarily reflect the views, opinions, or position of Nach himself on the matter(s) being discussed therein.

Posted: 12/30/2014 8:02 PM

Quote

feos

Site Admin, Skilled player (1236)

Joined: 4/17/2010
Posts: 11274
Location: RU

Download test.bat

Language: batch
:: Supposed to be ran by itself, no drag'n'dropping
:: Hardcoded source name is 0.png, change it if you wish
@echo off
: start
echo Type 0-99 for the desired quality
echo Type r to roll by quality step
echo Type q to quit
set /p quality=
if "%quality%"=="q" goto end
if "%quality%"=="r" goto bystep
goto normal
: bystep
echo Type the step size
set /p step=
echo Type 0-99 for where to stop
set /p stop=
set /a quality=99
: loop
crushedjpeg 0.png %quality%.jpeg "%quality%"
set /a quality-=%step%
if %quality% GTR %stop% goto loop
: normal
crushedjpeg 0.png %quality%.jpeg "%quality%"
goto start
: end

Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.

Posted: 12/30/2014 9:40 PM
(Edited: 4/15/2018 9:01 PM)

Quote

feos

Site Admin, Skilled player (1236)

Joined: 4/17/2010
Posts: 11274
Location: RU

Did some real testings now. For games that make screenshots go over 45kb, it's hardly possible to get a jpeg with no visible loss under the site's limit. If it's not some exceptional case (like png being smaller than the best sub 45kb jpeg, or best jpeg being 40kb), it'd be best to just pick the first one that's below the limit, since in most cases there will already be compression artifacts there. However, to know which one fits best, you'd need to create jpegs with quality step of 1 (down to some 70). Here are the images that I would use for publication. Format: Game: quality / jpeg size in bytes. Comment PNG JPEG images Abe: 94 / 44,685. "0" at the monitor starts fading at lower qualities.

F-Zero: 92 / 48,624. Artifacts at the road stripes are already present on the first sub 50kb jpeg.

Pinball: 76 / 50,578 (woops). No comments.

Luigi 1: 98 / 40,649. Artifacts appear around the ligntning if you go under 98, but 98 is already small enough.

Luigi 2: 97 / 42,458. BG on the right gets blurred and blocked under 97.

Crash: 94 / 47,341. Hard to see, but artifacts around Crash and on the sky.

So here are the steps that I'd be using when making a publication screenshot for such a game:

If sub 45kb, publish it. If it's bigger:

Use the above batch for that.

If there are artifacts already, publish. Otherwise:

EDIT: Well, the limit is actually 45kb. Whatever.

Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.

Posted: 12/30/2014 9:51 PM

Quote

Nach

Emulator Coder

Joined: 3/9/2004
Posts: 4588
Location: In his lab studying psychology to find new ways to torture TASers and forumers

feos, in your opinion, are these images better quality than the ones we are using for publication? (Is the quality per size ratio higher?)

Warning: Opinions expressed by Nach or others in this post do not necessarily reflect the views, opinions, or position of Nach himself on the matter(s) being discussed therein.

Posted: 12/30/2014 10:05 PM

Quote

feos

Site Admin, Skilled player (1236)

Joined: 4/17/2010
Posts: 11274
Location: RU

Since we don't have the lossless sources of them anymore, I dunno how to make an educated judgment about it. But looking at how gimp compared to crashjpeg with proper parameters, we might be able to save a few kilobytes over the published jpegs (while not losing quality), because it seems they were compressed with gimp already (praise Ilari).

Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.

Posted: 12/30/2014 10:11 PM

Quote

Nach

Emulator Coder

Joined: 3/9/2004
Posts: 4588
Location: In his lab studying psychology to find new ways to torture TASers and forumers

Okay, good to hear. Also, the various tools in Crushed JPEG are slated to come out with new releases in the near future with significant improvements (seeing chatter of 5-10% improvements). So if this is a minor improvement now, it'll be a significant improvement then.

Warning: Opinions expressed by Nach or others in this post do not necessarily reflect the views, opinions, or position of Nach himself on the matter(s) being discussed therein.

Posted: 12/30/2014 11:23 PM

Quote

Nach

Emulator Coder

Joined: 3/9/2004
Posts: 4588
Location: In his lab studying psychology to find new ways to torture TASers and forumers

New version out. Uses significantly improved cjpeg, some improvements to jpegoptim, and some tweaks to crushedjpeg wrapper. This should provide more quality at lower file sizes.

Warning: Opinions expressed by Nach or others in this post do not necessarily reflect the views, opinions, or position of Nach himself on the matter(s) being discussed therein.

Posted: 1/1/2015 11:21 AM

Quote

Warp

Banned User, Former player

Joined: 3/10/2004
Posts: 7698
Location: Finland

feos wrote:

Does cjpeg have an option to set subsampling to 4:4:4?

Using that subsampling mode indeed increases quality (because it retains more color information). Note, however, that it will probably be useful mostly if the source image is lossless. Re-compressing a 4:2:2 jpeg to a 4:4:4 one is probably not going to increase the quality (because the original color information has already been lost.)