Work at SourceForge, help us to make it a better place! We have an immediate need for a Support Technician in our San Francisco or Denver office.

Close

Unresponsive server

Help
syamashi
2012-12-11
2013-02-26
1 2 > >> (Page 1 of 2)
  • syamashi
    syamashi
    2012-12-11

    Hi,

    I'm experiencing an unresponsive state of the IIPImage Server. Here are the logs:

    [Mon Dec 10 16:51:34 2012] [warn] FastCGI: server "/usr/lib/cgi-bin/iipsrv.fcgi" (pid 15891) terminated due to uncaught signal '6' (Aborted)
    [Mon Dec 10 16:51:34 2012] [warn] FastCGI: server "/usr/lib/cgi-bin/iipsrv.fcgi" restarted (pid 16312)
    terminate called after throwing an instance of 'int'
    [Mon Dec 10 16:51:34 2012] [warn] FastCGI: server "/usr/lib/cgi-bin/iipsrv.fcgi" (pid 16312) terminated due to uncaught signal '6' (Aborted)
    [Mon Dec 10 16:51:39 2012] [warn] FastCGI: server "/usr/lib/cgi-bin/iipsrv.fcgi" restarted (pid 16313)
    terminate called after throwing an instance of 'int'
    [Mon Dec 10 16:51:39 2012] [warn] FastCGI: server "/usr/lib/cgi-bin/iipsrv.fcgi" (pid 16313) terminated due to uncaught signal '6' (Aborted)
    [Mon Dec 10 16:51:44 2012] [warn] FastCGI: server "/usr/lib/cgi-bin/iipsrv.fcgi" restarted (pid 16314)
    terminate called after throwing an instance of 'int'
    [Mon Dec 10 16:51:44 2012] [warn] FastCGI: server "/usr/lib/cgi-bin/iipsrv.fcgi" (pid 16314) terminated due to uncaught signal '6' (Aborted)
    [Mon Dec 10 16:51:44 2012] [warn] FastCGI: server "/usr/lib/cgi-bin/iipsrv.fcgi" has failed to remain running for 30 seconds given 3 attempts, its restart interval has been backed off to 600 seconds

    Is there a way to debug what's causing a problem? My guess is that there might be a failure in file conversion.

    Thanks,

     
  • Ruven
    Ruven
    2012-12-11

    What's in the iipsrv log file? If you are using one of our deb packages, this is configured to be in /var/log/iipsrv.log

     
  • syamashi
    syamashi
    2012-12-11

    Hi,

    This is all I have and nothing more gets printed out in the logs. Also the Apache requires restart every time the server freezes.

    Thanks,

    IIPImage Server. Version 0.9.9
    Ruven Pillay ruven@users.sourceforge.net

    Verbosity level set to 10
    Running in FCGI mode

    Setting maximum image cache size to 10MB
    Setting filesystem prefix to ''
    Setting default JPEG quality to 90
    Setting maximum CVT size to 5000
    Setting 3D file sequence name pattern to 'pyr'
    Setting up JPEG2000 support via Kakadu SDK
    Memcached support enabled. Connected to servers: 'localhost' with timeout 86400

    Initialisation Complete.
    <----------------------------------->

    CVT maximum viewport size set to 5000
    Full Request is FIF=/path/abc.jp2&HEI=80&QLT=100&CVT=JPEG
    [1/4]: Command / Argument is FIF : /path/abc.jp2
    FIF handler reached
    FIF :: URL decoding/filtering: /path/abc.jp2 => /path/abc.jp2
    FIF :: Image cache initialisation
    FIF :: JPEG2000 image requested
    FIF :: Created image

     
  • Ruven
    Ruven
    2012-12-11

    Hard to tell what's going on from that log. Is this on a specific image or with all images?

     
  • syamashi
    syamashi
    2012-12-11

    Not all but there are several images which result in frozen state of the server. I'm assuming the file conversions from TIF to JPEG2000 failed for whatever the reason but I don't have a way to tell what's wrong.

    thanks,

     
    Last edit: syamashi 2012-12-12
  • Ruven
    Ruven
    2012-12-12

    If you have an image which always crashes the server, please send it to me for me to try it out. Also tell me what encoding parameters you used for it.

     
  • syamashi
    syamashi
    2012-12-12

    A sample file is attached.

    convert input.jpg -define numrlvls=7 -define jp2:tilewidth=256 -define jp2:tileheight=256 -define jp2:rate=0.02348 -define jp2:prg=rpcl -define jp2:mode=int -define jp2:prcwidth=256 -define jp2:prcheight=256 -define jp2:cblkwidth=64 -define jp2:cblkheight=64 -define jp2:sop output.jp2

    Processed in CentOS 6.3, ImageMagick 6.7.8-2_64bit

     
    Last edit: syamashi 2012-12-12
    Attachments
  • syamashi
    syamashi
    2013-01-15

    Ok, so based on what's discovered so far is that those failed images are generated falsely on different machines. ImageMagick seems to have an option to support JP2 format and the ones which failed weren't installed with JP2 option.

    The problem is though, I still don't have a way for the server end to figure out what's going on. The worst part is that server hangs even though the proper images are requested. Hence I need to restart Apache manually every time when the viewer hits the faulty images. This is quite a pain. See the Apache errors when hangs:

    [Mon Jan 14 17:14:51 2013] [warn] FastCGI: server "/usr/lib/cgi-bin/iipsrv.fcgi" (pid 21519) terminated due to uncaught signal '6' (Aborted)
    [Mon Jan 14 17:14:56 2013] [warn] FastCGI: server "/usr/lib/cgi-bin/iipsrv.fcgi" restarted (pid 21520)
    terminate called after throwing an instance of 'int'
    [Mon Jan 14 17:14:57 2013] [warn] FastCGI: server "/usr/lib/cgi-bin/iipsrv.fcgi" (pid 21520) terminated due to uncaught signal '6' (Aborted)
    [Mon Jan 14 17:14:57 2013] [warn] FastCGI: server "/usr/lib/cgi-bin/iipsrv.fcgi" has failed to remain running for 30 seconds given 3 attempts, its restart interval has been backed off to 600 seconds
    [Mon Jan 14 17:15:50 2013] [notice] caught SIGTERM, shutting down
    [Mon Jan 14 17:15:50 2013] [warn] FastCGI: server "/usr/lib/cgi-bin/iipsrv.fcgi" has failed to remain running for 30 seconds given 3 attempts, its restart interval has been backed off to 600 seconds
    [Mon Jan 14 17:15:51 2013] [notice] FastCGI: process manager initialized (pid 21559)
    [Mon Jan 14 17:15:51 2013] [warn] FastCGI: server "/usr/lib/cgi-bin/iipsrv.fcgi" started (pid 21560)
    [Mon Jan 14 17:15:51 2013] [notice] Apache/2.2.22 (Ubuntu) mod_fastcgi/mod_fastcgi-SNAP-0910052141 configured -- resuming normal operations

    Are there any possible ways to avoid the server to hang?

    thanks,

     
  • Ruven
    Ruven
    2013-01-15

    I'm still investigating this issue and haven't quite been able to track down the problem yet. I'll get back to you as soon as I've fixed it.

     
  • syamashi
    syamashi
    2013-01-18

    Hi Ruven,

    Ok, please let me know if you have fixed it. Thanks.

    In the meantime, do you mind showing me how to disable Memcache? I want to disable it for debug purposes. Setting MEMCACHED_TIMEOUT=0 in iipsrv.conf didn't seem do.

    Thanks,

     
  • Ruven
    Ruven
    2013-01-20

    Try setting MEMCACHED_SERVERS to none and/or shutting down your Memcached server.

     
  • syamashi
    syamashi
    2013-01-22

    Neither commenting out the line or setting none won't start Apache. Hmm, I thought Memcache server is a part of Apache. How do I shutdown or disable the module without shutting down Apache? Sorry, I'm still new to this. Am I missing anything here?

    Syntax error on line 23 of /etc/apache2/mods-enabled/iipsrv.conf:
    FastCgiServer /usr/lib/cgi-bin/iipsrv.fcgi: invalid option: #-initial-env
    Action 'configtest' failed.
    The Apache error log may have more information.
    ...fail!

    [iipsrv.conf]

    FastCgiServer /usr/lib/cgi-bin/iipsrv.fcgi \

    -initial-env LOGFILE=/var/log/iipsrv.log \

    -initial-env VERBOSITY=10 \

    -initial-env JPEG_QUALITY=90 \

    -initial-env MAX_IMAGE_CACHE_SIZE=10 \

    -initial-env MAX_CVT=5000 \

    -initial-env MEMCACHED_SERVERS=none \

    -port 9000

     
    Last edit: syamashi 2013-01-22
  • Ruven
    Ruven
    2013-01-22

    No, memcached is not a part of Apache. It's an entirely separate application which you can shut down independently.

     
  • Mic
    Mic
    2013-01-30

    Hi,

    Using IIPImage in our project (a streeview like web app), we encounter the same server crashes as syamashi. The responsible files are corrupted, since other jpeg2000 viewer failed to open them. However, like syamashi, we have no way to figure out if the file is corrupted at runtime.

    Is there a way to avoid crashing the server, while a corrupted file is given to it ?

    Tks in advance.

    A sample corrupted file is also attached for testing.

     
  • Hi,
    One option would be to use a file format validation tool like jhove [1] to identify your corrupted images. I just tried it on your sample file and it reports that the file is "Not well-formed" with the error message "No JPEG 2000 header". Jhove has an audit feature you can run against whole directories of files.

    It would be nice if IIP handled corrupted jp2 files more gracefully, but a good first step is building validation into your jp2 generation step.

    Cheers,
    Eoghan

    [1] http://jhove.sourceforge.net/

     
  • I just tried jhove on syamashi's sample.jp2 file & it returns the same status/error: "Not well-formed"/"No JPEG 2000 header".

    Cheers
    Eoghan

     
  • Ruven
    Ruven
    2013-01-30

    OK, I've just committed a fix to this problem to github. Corrupted JPEG2000 files should no longer crash the server.

     
  • Mic
    Mic
    2013-01-30

    Thansk Eoghan for your suggestion and Ruven for the fix !

     
  • syamashi
    syamashi
    2013-01-31

    Do you have JPEG2000 builds for this fix? I don't have Kakadu.

     
  • Adam Cooper
    Adam Cooper
    2013-02-06

    I'd like to echo syamashi's request for built binaries for this issues fix. Linux and OSX ones would be nice :)

     
  • syamashi
    syamashi
    2013-02-22

    12.10 seems to have a different config format. Would you be able to point us to the doc?

    Syntax error on line 16 of /etc/apache2/mods-enabled/iipsrv.conf:
    Invalid command 'FcgidInitialEnv', perhaps misspelled or defined by a module not included in the server configuration
    Action 'configtest' failed.
    The Apache error log may have more information.
    ...fail!


    ScriptAlias /iipsrv/ "/usr/lib/iipimage-server/"

    <Directory "="" iipsrv="" "="">
    AllowOverride None
    Options None
    Order allow,deny
    Allow from all

    AddHandler fcgid-script .fcgi
    </Directory>

    FcgidInitialEnv VERBOSITY "1"
    FcgidInitialEnv LOGFILE "/var/log/iipsrv.log"
    FcgidInitialEnv MAX_IMAGE_CACHE_SIZE "10"
    FcgidInitialEnv JPEG_QUALITY "90"
    FcgidInitialEnv MAX_CVT "5000"
    FcgidInitialEnv MEMCACHED_SERVERS "localhost"

    FcgidIdleTimeout 0
    FcgidMaxProcessesPerClass 1

     
    Last edit: syamashi 2013-02-22
1 2 > >> (Page 1 of 2)


Anonymous


Cancel   Add attachments