Thread: [GM-help] Best performance for resizing?

I am planning to use GraphicsMagick to batch process our back stock of
images at Etsy (ahem, a few million images).  The images are being shrunk
and cropped to fit 170x135.  I have been testing the following set of
options:

/usr/bin/gm convert input.jpg -filter triangle -resize "170x135^" -gravity
Center \
  -crop "170x135+0+0" -quality 90 -type TrueColor -interlace None output.jpg

Are there any options for "convert" that I am leaving out but should
consider using to improve the resize speed?  I already ran a test of the
available filters on a sample set of images and found "triangle" to have the
best balance of image quality, speed, and smaller file sizes.

I'm using version 1.3.12.

Thanks,
Mike

P.S. John Allspaw recommended I check in here to get the best advice on
GraphicsMagick!

-- 
Mike Brittain
Senior Software Engineer
Etsy.com

I am planning to use GraphicsMagick to batch process our back stock of
images at Etsy (ahem, a few million images).  The images are being shrunk
and cropped to fit 170x135.  I have been testing the following set of
options:

/usr/bin/gm convert input.jpg -filter triangle -resize "170x135^" -gravity
Center \
  -crop "170x135+0+0" -quality 90 -type TrueColor -interlace None output.jpg

Are there any options for "convert" that I am leaving out but should
consider using to improve the resize speed?  I already ran a test of the
available filters on a sample set of images and found "triangle" to have the
best balance of image quality, speed, and smaller file sizes.

I'm using version 1.3.12.

Thanks,
Mike

P.S. John Allspaw recommended I check in here to get the best advice on
GraphicsMagick!

-- 
Mike Brittain
Senior Software Engineer
Etsy.com

On Thu, 11 Mar 2010, Mike Brittain wrote:

> I am planning to use GraphicsMagick to batch process our back stock of images at Etsy (ahem, a few
> million images).  The images are being shrunk and cropped to fit 170x135.  I have been testing the
> following set of options:
> 
> /usr/bin/gm convert input.jpg -filter triangle -resize "170x135^" -gravity Center \
>   -crop "170x135+0+0" -quality 90 -type TrueColor -interlace None output.jpg
> 
> Are there any options for "convert" that I am leaving out but should consider using to improve the
> resize speed?  I already ran a test of the available filters on a sample set of images and found
> "triangle" to have the best balance of image quality, speed, and smaller file sizes.

Yes.  There are a few things.  Something which is vital to know for 
JPEG is that the JPEG decoder can return a smaller size 
(subresolution) of the image to GraphicsMagick.  That saves a lot of 
time.  This support is triggered by using the -size option prior to 
the input file name with a geometry argument which is at least the 
size that you need.  You will want to specify a size at least double 
the final size so that antialiasing works well.

Also, a -thumbnail option was added which works like -resize but takes 
steps to make resizing to much smaller sizes a lot faster.  It does 
that by using the 'sample' algorithm followed by the 'resize' 
algorithm using the box filter.  But since it knows the image size and 
the ratios, it does this intelligently.

So use something like

   gm convert -size 340x270 input.jpg -thumbnail "170x135^" \
      -gravity Center -crop "170x135+0+0" -quality 90 -type TrueColor \
      -interlace None output.jpg

Lastly, if you are batch processing a whole lot of files in a 
directory tree, you may want to use 'mogrify' instead since it is able 
to execute the same commands on a large set of files without starting 
a new process for each one.  The -output-directory and 
-create-directories options can be quite useful in order to process a 
sub-directory tree of images into another subdirectory tree.  See 
"http://www.graphicsmagick.org/FAQ.html#how-can-i-process-many-files-at-once".

> I'm using version 1.3.12.

Good.

> P.S. John Allspaw recommended I check in here to get the best advice on GraphicsMagick!

Many thanks to John Allspaw. :-)

Bob
--
Bob Friesenhahn
bfr...@si..., http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

Thanks for the info, Bob.  This is super helpful.

On Thu, Mar 11, 2010 at 10:34 PM, Bob Friesenhahn <
bfr...@si...> wrote:

> Something which is vital to know for JPEG is that the JPEG decoder can
> return a smaller size (subresolution) of the image to GraphicsMagick.  That
> saves a lot of time.  This support is triggered by using the -size option
> prior to the input file name with a geometry argument which is at least the
> size that you need.  You will want to specify a size at least double the
> final size so that antialiasing works well.
>

I had tried the -size option earlier with no improvements, but now that
you've mentioned this I found that I had it placed *after* the input file.

For my testing, I have 100 files I have been resizing.  Without -size, this
was taking 31 seconds.  After adding the -size option, the speed improved to
10 seconds.  Faster, indeed!


Also, a -thumbnail option was added which works like -resize but takes steps
> to make resizing to much smaller sizes a lot faster.  It does that by using
> the 'sample' algorithm followed by the 'resize' algorithm using the box
> filter.  But since it knows the image size and the ratios, it does this
> intelligently.
>

> So use something like
>
>  gm convert -size 340x270 input.jpg -thumbnail "170x135^" \
>     -gravity Center -crop "170x135+0+0" -quality 90 -type TrueColor \
>     -interlace None output.jpg
>

I replaced -resize with -thumbnail as you suggest in the options above, but
in this case the speed dropped.  The same set of 100 images took 14 seconds
to process when using -thumbnail.



> Lastly, if you are batch processing a whole lot of files in a directory
> tree, you may want to use 'mogrify' instead since it is able to execute the
> same commands on a large set of files without starting a new process for
> each one.
>

Looking at the directory options, I don't think this is going to be helpful
for my case.  The file layout I have is one full-size image and a number of
derivative sizes per directory.  For example, in a single directory we have
files something like: 12345_full.jpg, 12345_small.jpg, 12345_medium.jpg, and
12345_large.jpg.  The new sizes I am batch processing will be derivatives of
the *_full.jpg size but must live in the same directory.  Also, these are
new files, and not resizing a file in place of the original.  Correct me if
I'm wrong here.

I am glad that you brought this up, however, because we have an upcoming
project that *will* touch files in place.  Mogrify will probably come in use
there.

Mike

-- 
Mike Brittain
Senior Software Engineer
Etsy.com

On Fri, 12 Mar 2010, Mike Brittain wrote:
> 
> I replaced -resize with -thumbnail as you suggest in the options above, but in this case the speed
> dropped.  The same set of 100 images took 14 seconds to process when using -thumbnail.

Depending on the number of CPU cores you have active, -resize may be 
faster than -thumbnail in terms of wall-clock time, but it will 
usually use less total CPU.  Since the JPEG library is already 
returning a fairly small image, there is much less benefit from 
-thumbnail.

> Looking at the directory options, I don't think this is going to be helpful for my case.  The file
> layout I have is one full-size image and a number of derivative sizes per directory.  For example,
> in a single directory we have files something like: 12345_full.jpg, 12345_small.jpg,
> 12345_medium.jpg, and 12345_large.jpg.  The new sizes I am batch processing will be derivatives of
> the *_full.jpg size but must live in the same directory.  Also, these are new files, and not
> resizing a file in place of the original.  Correct me if I'm wrong here.

In this case, be sure to investigate the '-write filename' option 
which allows you to write the current image to the specified filename 
and then continue processing.  In this case you might not take 
advantage of the JPEG -size option, but you can use -thumbnail (or 
-resize, or -scale) in a sort of pyramid fashion to produce the 
various smaller sizes while reading the original image just once.

For example:

% time gm convert -verbose 100_4994.jpg -quality 80 +profile '*' \
    -write original.jpg -thumbnail 60% -write large.jpg -thumbnail 60% \
    -write medium.jpg -thumbnail 60% -write small.jpg -thumbnail 120x80 \
     thumb.jpg
100_4994.jpg JPEG 1728x2304+0+0 DirectClass 8-bit 1.3M 0.210u 0:01 (17.3M pixels/s)
100_4994.jpg=>original.jpg JPG 1728x2304+0+0 DirectClass 8-bit 798.5K 0.290u 0:01 (13.6M pixels/s)
100_4994.jpg=>large.jpg JPG 1728x2304=>1037x1382+0+0 DirectClass 8-bit 292.5K 0.100u 0:01 (38.0M pixels/s)
100_4994.jpg=>medium.jpg JPG 1728x2304=>622x829+0+0 DirectClass 8-bit 102.8K 0.040u 0:01
100_4994.jpg=>small.jpg JPG 1728x2304=>373x497+0+0 DirectClass 8-bit 31.6K 0.020u 0:01
100_4994.jpg JPEG 1728x2304=>60x80+0+0 DirectClass 8-bit 1.780u 0:01 (4.6M pixels/s)
100_4994.jpg=>thumb.jpg JPG 1728x2304=>60x80+0+0 DirectClass 8-bit 0.000u 0:01
gm convert -verbose 100_4994.jpg -quality 80 +profile '*' -write original.jpg  1.90s user 0.09s system 184% cpu 1.078 total

Just keep in mind that there could be a small bit of additional 
quality loss due to resizing an already resized image.

Regardless, when using JPEG format, the bottleneck is likely to be the 
JPEG decode/encode rather than GraphicsMagick.

Bob
--
Bob Friesenhahn
bfr...@si..., http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

On Fri, Mar 12, 2010 at 10:56 PM, Bob Friesenhahn <
bfr...@si...> wrote:

> Regardless, when using JPEG format, the bottleneck is likely to be the JPEG
> decode/encode rather than GraphicsMagick.
>

Does that imply I might be able to compile GraphicsMagick against an
optimized libjpeg to get better performance, or maybe even just a newer
libjpeg than whatever is on the system I'm using?

Mike

On Mon, 15 Mar 2010, Mike Brittain wrote:

> On Fri, Mar 12, 2010 at 10:56 PM, Bob Friesenhahn <bfr...@si...> wrote:
>       Regardless, when using JPEG format, the bottleneck is likely to be the JPEG decode/encode
>       rather than GraphicsMagick.
> 
> 
> Does that imply I might be able to compile GraphicsMagick against an optimized libjpeg to get better
> performance, or maybe even just a newer libjpeg than whatever is on the system I'm using?

There are modified libjpeg's available which use MMX instructions.  I 
have heard that these can cut the decode/encode time by as much as 
70%.  I understand that Flickr is using the Japanese libjpeg with MMX 
enhancements.

JPEG 7 and JPEG 8 are actually a bit slower than 6b.

Bob
--
Bob Friesenhahn
bfr...@si..., http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/