Learn how easy it is to sync an existing GitHub or Google Code repo to a SourceForge project! See Demo

Close

parallel job with graphicsmagick

Help
Roland
2012-10-04
2013-03-27
  • Roland
    Roland
    2012-10-04

    Hi Bob and others,
    I am still working on my \\\"massice image processing\\\" project, in which I have to manipulate large cmyk images in batch jobs.
    I manged to get help with all my questions, but now I have a new problem:

    I organised a large server with 32 cores an 256 GB of Ram with the intention to run multiple gm jobs parallel.

    In one example setup I have 4 files a1.tif, a2.tif, a3.tif and a4.tif
    and 4 other files b1.tif, b2.tif, b3.tif and b4.tif, all of them in cmyk and 200+ MB large.
    (this is the reason why i cannot share these files with you…)

    I do something like
    \\\"convert file a1.tif to a1_rgb.mpc and b1.tif to b1_rgb.mpc\\\"
    \\\"do my processing like adding labels\\\"
    combine these two files with \\\"gm convert a1_rgb.mpc b1_rgb_mpc -append c1_rgb.mpc\\\"
    and then convert back to c1_cmyk.tif

    this works fine.
    When I write a shellscript with all these command and run this script, it works great.
    When I start this script in background (with&) it works in background and does it very well, but:

    When I start the same script four times in background (of course with the four sets a1b1, a2b2, a3b3, a4b4)
    the whole thing stops satisfying me, because the result-images get randomly corruptet.
    Everytime a diffrent one, sometimes more, but alwas at least one of them.
    The interim mpc files seem to be less effected, but the tifs are almost always with lzw errors.

    to make it clear: the different jobs do NOT work on the same images, bu the try do do the same
    things at a time to different images.

    Is it possible, that gm gets confused when multiple gm processes try to do the same thing to different images?

    Hope you can help me!

    Greetings
    Roland

     
  • Roland
    Roland
    2012-10-04

    My test system (with only 4 core 32 bit) says:

    GraphicsMagick 1.3.15 2012-04-28 Q16 http://www.GraphicsMagick.org/

    Feature Support:
      Thread Safe              yes
      Large Files (> 32 bit)   yes
      Large Memory (> 32 bit)  no
      BZIP                     yes
      DPS                      no
      FlashPix                 no
      FreeType                 yes
      Ghostscript (Library)    no
      JBIG                     no
      JPEG-2000                no
      JPEG                     yes
      Little CMS               no
      Loadable Modules         no
      OpenMP                   yes (200805)
      PNG                      yes
      TIFF                     yes
      TRIO                     no
      UMEM                     no
      WMF                      no
      X11                      yes
      XML                      yes
      ZLIB                     yes

    Host type: i686-pc-linux-gnu

    Configured using the command:
      ./configure  \\\\\\\'-with-quantum-depth=16\\\\\\\' \\\\\\\'-with-tiff=yes\\\\\\\'

    Final Build Parameters:
      CC       = gcc -std=gnu99
      CFLAGS   = -fopenmp -g -O2 -Wall -pthread
      CPPFLAGS = -I/usr/include/freetype2 -I/usr/include/libxml2
      CXX      = g++
      CXXFLAGS = -pthread
      LDFLAGS  = -L/usr/lib -L/usr/lib
      LIBS     = -ltiff -lfreetype -ljpeg -lpng12 -lXext -lSM -lICE -lX11 -llzma -lbz2 -lxml2 -lz -lm -lgomp -lpthread

    my productivity system with 32 cores an 64 bit says:

    GraphicsMagick 1.3.15 2012-04-28 Q16 http://www.GraphicsMagick.org/
    Copyright (C) 2002-2012 GraphicsMagick Group.
    Additional copyrights and licenses apply to this software.
    See http://www.GraphicsMagick.org/www/Copyright.html for details.

    Feature Support:
      Thread Safe              yes
      Large Files (> 32 bit)   yes
      Large Memory (> 32 bit)  yes
      BZIP                     no
      DPS                      no
      FlashPix                 no
      FreeType                 yes
      Ghostscript (Library)    no
      JBIG                     no
      JPEG-2000                no
      JPEG                     yes
      Little CMS               no
      Loadable Modules         no
      OpenMP                   yes (200805)
      PNG                      yes
      TIFF                     yes
      TRIO                     no
      UMEM                     no
      WMF                      no
      X11                      yes
      XML                      no
      ZLIB                     yes

    Host type: x86_64-unknown-linux-gnu

    Configured using the command:
      ./configure  \\\\\\\'-with-quantum-depth=16\\\\\\\' \\\\\\\'-with-tiff=yes\\\\\\\' -enable-ltdl-convenience

    Final Build Parameters:
      CC       = gcc -std=gnu99
      CFLAGS   = -fopenmp -g -O2 -Wall -pthread
      CPPFLAGS = -I/usr/include/freetype2
      CXX      = g++
      CXXFLAGS = -pthread
      LDFLAGS  = -L/usr/lib
      LIBS     = -ltiff -lfreetype -ljpeg -lpng12 -lX11 -llzma -lz -lm -lgomp -lpthread

    both show the same behavior while parallel processing.

     
  • From your description it sounds as though some temporary file might
    be used by multiple processes instead of using a unique one
    for each process.  Can you tell whether the error happens during
    the first "convert" or during the "convert back"?

     
  • Roland
    Roland
    2012-10-04

    stupid me !!!
    The biggest problem sits in front of the screen !

    glennrp, you are right, I had a tempfile which I named without the jobnumber and all four jobs used the same tempfile.

    Sometimes talking about it helps…

    Thank you !

    Roland