#1475 STAF Crash

Unix::Linux
closed
STAFProc (180)
5
2013-07-15
2012-11-04
bfurey
No

Hi

STAF encountered a crash on one of our LA Servers

STAF Started : 20121102-08:20:55
STAF Crash : 20121104-12:02:16

# Jobs Ran : 2442

STAF Version : 3.4.8
STAX Version : 3.5.2
OS : Fedora 16 (64bit)
JVM : IBM Java 1.6.0 SR11

I have a 7GB tar fle made up the following
- core.29186
- trace.out
- JVM Logs
- STAFProc.out
- STAF.cfg
- top stats

Please advise how best to get that tar file to you. Or ping me to get access to the machine where the tar file is stored. Tar file too large to upload.

Regards
Brendan

Discussion

  • Sharon Lucas

    Sharon Lucas - 2012-11-05

    The STAFProc core file showed 1089 threads running in STAFProc (since a debug version of STAF was not being run, no other useful information could be obtained from the core file). Since this system was using the default STAF_THREAD_STACK_SIZE of 4096 (4M), then STAFProc would have needed 1089 * 4M of memory (4,356M) and STAFProc may have crashed due to being out of memory. The STAFProc.out file did not contain any error information.

    I recommended decreasing the STAF_THREAD_STACK_SIZE on this system to 2048 (2M). Section "4.13 Tuning" in the STAF User's Guide says:

    "STAF provides a way to tune its thread stack size. This is done via setting a "STAF_THREAD_STACK_SIZE" environment variable before STAFProc gets started. User can use this environment variable to set STAF's thread stack size in kilobytes. "

    To set the thread stack size to 2048KB (aka 2M) on Unix:

    export STAF_THREAD_STACK_SIZE=2048

    If this doesn't resolve the issue, then I may have to provide a debug version of STAF (where STAF was compiled with debugging information in it using the -g option for the gcc compiler) that you can run so that the core file will contain useful information like variables, functions, and line numbers, other than just the total number of threads.

     
  • Sharon Lucas

    Sharon Lucas - 2012-11-05
    • assigned_to: nobody --> slucas
     
  • Sharon Lucas

    Sharon Lucas - 2012-12-13

    Did setting STAF_THREAD_STACK_SIZE=2048 on this Linux system resolve the problem?

     
  • Sharon Lucas

    Sharon Lucas - 2013-01-28

    Provided a debug version of STAF V3.4.12.1 for Linux AMD64 to Brendan and James to install on domsvt009 so that more debug information will be provided when STAFProc crashes next time.

     
  • Sharon Lucas

    Sharon Lucas - 2013-07-15
    • status: open --> closed
     
  • Sharon Lucas

    Sharon Lucas - 2013-07-15

    Closing as assuming no longer occurring as no additional comments have been added since my last comment on Nov 5, 2012.

     

Log in to post a comment.