Nagios

If I'm running Nagios 3.0a1 it crashes always after a few seconds. 

In the log I only found:

Mar  8 15:35:18 nagios: Nagios 3.0a1 starting... (PID=24492)
Mar  8 15:35:18 nagios: LOG VERSION: 2.0
Mar  8 15:35:18 nagios: Finished daemonizing... (New PID=24493)
Mar  8 15:35:28 nagios: HOST ALERT: localhost;UP;SOFT;2;PING OK - Packet 
loss = 0%, RTA = 0.05 ms

Here the last signs of life captured by strace:

rename("/usr/local/nagios/var/nagios.tmpNdmpn0", 
"/usr/local/nagios/var/status.dat") = 0
gettimeofday({1173359616, 173629}, NULL) = 0
time([1173359616])                      = 1173359616
gettimeofday({1173359616, 173998}, NULL) = 0
time([1173359616])                      = 1173359616
gettimeofday({1173359616, 174363}, NULL) = 0
gettimeofday({1173359616, 174552}, NULL) = 0
time([1173359616])                      = 1173359616
gettimeofday({1173359616, 174968}, NULL) = 0
time([1173359616])                      = 1173359616
time([1173359616])                      = 1173359616
gettimeofday({1173359616, 175746}, NULL) = 0
umask(077)                              = 02
gettimeofday({1173359616, 176092}, NULL) = 0
getpid()                                = 3459
open("/tmp/nagiosYBuC2d", O_RDWR|O_CREAT|O_EXCL, 0600) = 7
fcntl64(7, F_GETFL)                     = 0x2 (flags O_RDWR)
fstat64(7, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) 
= 0xb7fcf000
_llseek(7, 0, [0], SEEK_CUR)            = 0
umask(02)                               = 077
gettimeofday({1173359616, 177807}, NULL) = 0
time([1173359616])                      = 1173359616
clone(child_stack=0, 
flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, 
child_tidptr=0xb7e610c8) = 3467
--- SIGCHLD (Child exited) @ 0 (0) ---
close(7)                                = 0
munmap(0xb7fcf000, 4096)                = 0
waitpid(3467, NULL, 0)                  = 3467
time([1173359616])                      = 1173359616
time(NULL)                              = 1173359616
gettimeofday({1173359616, 183489}, NULL) = 0
gettimeofday({1173359616, 183686}, NULL) = 0
nanosleep({0, 250000000}, 0)            = ? ERESTART_RESTARTBLOCK (To be 
restarted)
+++ killed by SIGSEGV +++

This error captured bei gdb

nagios1:/usr/local/nagios/bin# gdb nagios
GNU gdb 6.3-debian
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you 
are
welcome to change it and/or distribute copies of it under certain 
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for 
details.
This GDB was configured as "i386-linux"...Using host libthread_db library 
"/lib/tls/libthread_db.so.1".

(gdb) r /usr/local/nagios/etc/nagios.cfg
Starting program: /usr/local/nagios/bin/nagios 
/usr/local/nagios/etc/nagios.cfg
[Thread debugging using libthread_db enabled]
[New Thread -1209560960 (LWP 931)]

Nagios 3.0a1
Copyright (c) 1999-2007 Ethan Galstad (http://www.nagios.org)
Last Modified: 03-06-2007
License: GPL

Nagios 3.0a1 starting... (PID=931)
[New Thread -1209775184 (LWP 934)]
[New Thread -1218163792 (LWP 935)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1218163792 (LWP 935)]
0xb7eed215 in mallopt () from /lib/tls/libc.so.6
(gdb) bt
#0  0xb7eed215 in mallopt () from /lib/tls/libc.so.6
#1  0xb7eebf02 in realloc () from /lib/tls/libc.so.6
#2  0x0807785e in dbuf_strcat (db=0x80c6680,
    buf=0xb76448c0 
"1=1\n2=localhost\n3=\n4=0\n5=1\n6=1\n7=/tmp/nagiosUJ95U0\n8=nan\n9=1173431091.188930\n10=1173431095.208425\n11=0\n12=1\n13=0\n\n")
    at utils.c:4924
#3  0x08077d70 in check_result_worker_thread (arg=0x0) at utils.c:5373
#4  0xb7fb6b63 in start_thread () from /lib/tls/libpthread.so.0
#5  0xb7f5218a in clone () from /lib/tls/libc.so.6



This errors all happened on a Debian Sarge 32Bit Version with the Kernel 
2.6.20.1


Thanks in advance,
FliTTi

If I'm running Nagios 3.0a1 it crashes always after a few seconds. 

In the log I only found:

Mar  8 15:35:18 nagios: Nagios 3.0a1 starting... (PID=24492)
Mar  8 15:35:18 nagios: LOG VERSION: 2.0
Mar  8 15:35:18 nagios: Finished daemonizing... (New PID=24493)
Mar  8 15:35:28 nagios: HOST ALERT: localhost;UP;SOFT;2;PING OK - Packet 
loss = 0%, RTA = 0.05 ms

Here the last signs of life captured by strace:

rename("/usr/local/nagios/var/nagios.tmpNdmpn0", 
"/usr/local/nagios/var/status.dat") = 0
gettimeofday({1173359616, 173629}, NULL) = 0
time([1173359616])                      = 1173359616
gettimeofday({1173359616, 173998}, NULL) = 0
time([1173359616])                      = 1173359616
gettimeofday({1173359616, 174363}, NULL) = 0
gettimeofday({1173359616, 174552}, NULL) = 0
time([1173359616])                      = 1173359616
gettimeofday({1173359616, 174968}, NULL) = 0
time([1173359616])                      = 1173359616
time([1173359616])                      = 1173359616
gettimeofday({1173359616, 175746}, NULL) = 0
umask(077)                              = 02
gettimeofday({1173359616, 176092}, NULL) = 0
getpid()                                = 3459
open("/tmp/nagiosYBuC2d", O_RDWR|O_CREAT|O_EXCL, 0600) = 7
fcntl64(7, F_GETFL)                     = 0x2 (flags O_RDWR)
fstat64(7, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) 
= 0xb7fcf000
_llseek(7, 0, [0], SEEK_CUR)            = 0
umask(02)                               = 077
gettimeofday({1173359616, 177807}, NULL) = 0
time([1173359616])                      = 1173359616
clone(child_stack=0, 
flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, 
child_tidptr=0xb7e610c8) = 3467
--- SIGCHLD (Child exited) @ 0 (0) ---
close(7)                                = 0
munmap(0xb7fcf000, 4096)                = 0
waitpid(3467, NULL, 0)                  = 3467
time([1173359616])                      = 1173359616
time(NULL)                              = 1173359616
gettimeofday({1173359616, 183489}, NULL) = 0
gettimeofday({1173359616, 183686}, NULL) = 0
nanosleep({0, 250000000}, 0)            = ? ERESTART_RESTARTBLOCK (To be 
restarted)
+++ killed by SIGSEGV +++

This error captured bei gdb

nagios1:/usr/local/nagios/bin# gdb nagios
GNU gdb 6.3-debian
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you 
are
welcome to change it and/or distribute copies of it under certain 
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for 
details.
This GDB was configured as "i386-linux"...Using host libthread_db library 
"/lib/tls/libthread_db.so.1".

(gdb) r /usr/local/nagios/etc/nagios.cfg
Starting program: /usr/local/nagios/bin/nagios 
/usr/local/nagios/etc/nagios.cfg
[Thread debugging using libthread_db enabled]
[New Thread -1209560960 (LWP 931)]

Nagios 3.0a1
Copyright (c) 1999-2007 Ethan Galstad (http://www.nagios.org)
Last Modified: 03-06-2007
License: GPL

Nagios 3.0a1 starting... (PID=931)
[New Thread -1209775184 (LWP 934)]
[New Thread -1218163792 (LWP 935)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1218163792 (LWP 935)]
0xb7eed215 in mallopt () from /lib/tls/libc.so.6
(gdb) bt
#0  0xb7eed215 in mallopt () from /lib/tls/libc.so.6
#1  0xb7eebf02 in realloc () from /lib/tls/libc.so.6
#2  0x0807785e in dbuf_strcat (db=0x80c6680,
    buf=0xb76448c0 
"1=1\n2=localhost\n3=\n4=0\n5=1\n6=1\n7=/tmp/nagiosUJ95U0\n8=nan\n9=1173431091.188930\n10=1173431095.208425\n11=0\n12=1\n13=0\n\n")
    at utils.c:4924
#3  0x08077d70 in check_result_worker_thread (arg=0x0) at utils.c:5373
#4  0xb7fb6b63 in start_thread () from /lib/tls/libpthread.so.0
#5  0xb7f5218a in clone () from /lib/tls/libc.so.6



This errors all happened on a Debian Sarge 32Bit Version with the Kernel 
2.6.20.1


Thanks in advance,
FliTTi

matthias.flittner@... wrote:
> 
> If I'm running Nagios 3.0a1 it crashes always after a few seconds.
> 
> In the log I only found:
> Mar  8 15:35:18 nagios: Nagios 3.0a1 starting... (PID=24492)
> Mar  8 15:35:18 nagios: LOG VERSION: 2.0
> Mar  8 15:35:18 nagios: Finished daemonizing... (New PID=24493)
> Mar  8 15:35:28 nagios: HOST ALERT: localhost;UP;SOFT;2;PING OK - Packet 
> loss = 0%, RTA = 0.05 ms 
> <http://dict.leo.org/ende?lp=ende&p=/gQPU.&search=always>
> 
> 
> 
> Here the last signs of life captured by strace:
> rename("/usr/local/nagios/var/nagios.tmpNdmpn0", 
> "/usr/local/nagios/var/status.dat") = 0
> gettimeofday({1173359616, 173629}, NULL) = 0
> time([1173359616])                      = 1173359616
> gettimeofday({1173359616, 173998}, NULL) = 0
> time([1173359616])                      = 1173359616
> gettimeofday({1173359616, 174363}, NULL) = 0
> gettimeofday({1173359616, 174552}, NULL) = 0
> time([1173359616])                      = 1173359616
> gettimeofday({1173359616, 174968}, NULL) = 0
> time([1173359616])                      = 1173359616
> time([1173359616])                      = 1173359616
> gettimeofday({1173359616, 175746}, NULL) = 0
> umask(077)                              = 02
> gettimeofday({1173359616, 176092}, NULL) = 0
> getpid()                                = 3459
> open("/tmp/nagiosYBuC2d", O_RDWR|O_CREAT|O_EXCL, 0600) = 7
> fcntl64(7, F_GETFL)                     = 0x2 (flags O_RDWR)
> fstat64(7, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
> mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 
> 0) = 0xb7fcf000
> _llseek(7, 0, [0], SEEK_CUR)            = 0
> umask(02)                               = 077
> gettimeofday({1173359616, 177807}, NULL) = 0
> time([1173359616])                      = 1173359616
> clone(child_stack=0, 
> flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, 
> child_tidptr=0xb7e610c8) = 3467
> --- SIGCHLD (Child exited) @ 0 (0) ---
> close(7)                                = 0
> munmap(0xb7fcf000, 4096)                = 0
> waitpid(3467, NULL, 0)                  = 3467
> time([1173359616])                      = 1173359616
> time(NULL)                              = 1173359616
> gettimeofday({1173359616, 183489}, NULL) = 0
> gettimeofday({1173359616, 183686}, NULL) = 0
> nanosleep({0, 250000000}, 0)            = ? ERESTART_RESTARTBLOCK (To be 
> restarted)
> +++ killed by SIGSEGV +++
> 
> 
> 
> This error captured bei gdb
> nagios1:/usr/local/nagios/bin# gdb nagios
> GNU gdb 6.3-debian
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain 
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for details.
> This GDB was configured as "i386-linux"...Using host libthread_db 
> library "/lib/tls/libthread_db.so.1".
> 
> (gdb) r /usr/local/nagios/etc/nagios.cfg
> Starting program: /usr/local/nagios/bin/nagios 
> /usr/local/nagios/etc/nagios.cfg
> [Thread debugging using libthread_db enabled]
> [New Thread -1209560960 (LWP 931)]
> 
> Nagios 3.0a1
> Copyright (c) 1999-2007 Ethan Galstad (http://www.nagios.org)
> Last Modified: 03-06-2007
> License: GPL
> 
> Nagios 3.0a1 starting... (PID=931)
> [New Thread -1209775184 (LWP 934)]
> [New Thread -1218163792 (LWP 935)]
> 
> Program received signal SIGSEGV, Segmentation fault.
> [Switching to Thread -1218163792 (LWP 935)]
> 0xb7eed215 in mallopt () from /lib/tls/libc.so.6
> (gdb) bt
> #0  0xb7eed215 in mallopt () from /lib/tls/libc.so.6
> #1  0xb7eebf02 in realloc () from /lib/tls/libc.so.6
> #2  0x0807785e in dbuf_strcat (db=0x80c6680,
>    buf=0xb76448c0 
> "1=1\n2=localhost\n3=\n4=0\n5=1\n6=1\n7=/tmp/nagiosUJ95U0\n8=nan\n9=1173431091.188930\n10=1173431095.208425\n11=0\n12=1\n13=0\n\n")
>    at utils.c:4924
> #3  0x08077d70 in check_result_worker_thread (arg=0x0) at utils.c:5373
> #4  0xb7fb6b63 in start_thread () from /lib/tls/libpthread.so.0
> #5  0xb7f5218a in clone () from /lib/tls/libc.so.6
> 
> 
> 
> 
> 
> This errors all happened on a Debian Sarge 32Bit Version with the Kernel 
>  2.6.20.1
> 
> 
> Thanks in advance,
> FliTTi
> 

Looks like this is the NDOMOD module that's causing the segfault.  Some 
things changed with regards to object structures for 3.0a1, so I need to 
release a new version of NDOUtils that will work properly with it.


Ethan Galstad,
Nagios Developer
---
Email: nagios@...
Website: http://www.nagios.org

<snip_all>

Hi,

I've started a discussion thread on the german nagios-portal.de to
animate different users to do some testing against latest cvs code.

Following information where asked:

1. Operatingsystem + Version
2. Kernelversion
3. gcc -v
4. ldd against nagios binary
5. state of nagios

Following results came back up to now but I am not able to see a single
hint - "Crash Users" at the end of the list:

If there are more information we should supply, please feel free to tell
us what we should provice more.

##### User 1 (mess)- Nagios running fine without SIGSEGV #####

   1. OS: Ubuntu 6.10
   2. Linux galadriela 2.6.17-11-386 #2 Thu Feb 1 19:50:13 UTC 2007 i686
      GNU/Linux
   3. gcc -v
      Using built-in specs.
      Target: i486-linux-gnu
      Configured with: ../src/configure -v
      --enable-languages=c,c++,fortran,objc,obj-c++,treelang
      --prefix=/usr --enable-shared --with-system-zlib
      --libexecdir=/usr/lib --without-included-gettext
      --enable-threads=posix --enable-nls --program-suffix=-4.1
      --enable-__cxa_atexit --enable-clocale=gnu
      --enable-libstdcxx-debug --enable-mpfr --enable-checking=release
      i486-linux-gnu
      Thread model: posix
      gcc version 4.1.2 20060928 (prerelease) (Ubuntu 4.1.1-13ubuntu5)
   4. ldd /usr/local/nagios/bin/nagios
      linux-gate.so.1 => (0xffffe000)
      libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0xb7f05000)
      libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 (0xb7ef2000)
      libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0xb7eed000)
      libglib-2.0.so.0 => /usr/lib/libglib-2.0.so.0 (0xb7e5b000)
      libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7d27000)
      /lib/ld-linux.so.2 (0xb7f3f000)
      librt.so.1 => /lib/tls/i686/cmov/librt.so.1 (0xb7d1e000)
   5. Nagios running fine!!!

##### User 2 (myself)- Nagios running fine without SIGSEGV#####

   1. OS: Ubuntu 6.10 (Edgy Eft) (i686)
   2. uname -a: Linux abby 2.6.20-custom #1 SMP Tue Feb 27 18:45:20 CET
      2007 i686 GNU/Linux (kein Distrokernel)
   3. gcc -v:Using built-in specs.
      Target: i486-linux-gnu
      Configured with: ../src/configure -v
      --enable-languages=c,c++,fortran,objc,obj-c++,treelang
      --prefix=/usr --enable-shared --with-system-zlib
      --libexecdir=/usr/lib --without-included-gettext
      --enable-threads=posix --enable-nls --program-suffix=-4.1
      --enable-__cxa_atexit --enable-clocale=gnu
      --enable-libstdcxx-debug --enable-mpfr --enable-checking=release
      i486-linux-gnu
      Thread model: posix
      gcc version 4.1.2 20060928 (prerelease) (Ubuntu 4.1.1-13ubuntu5)
   4. ldd:ldd /usr/local/nagios-clean/bin/nagios
      linux-gate.so.1 => (0xffffe000)
      libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0xb7f46000)
      libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 (0xb7f33000)
      libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0xb7f2e000)
      libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7dfa000)
      /lib/ld-linux.so.2 (0xb7f7d000)
   5. Nagios running fine!!!


##### User 3 (cybit) - Nagios running fine without SIGSEGV#####

   1. OS: Ubuntu 6.10
   2. Linux galadriela 2.6.17-11-386 #2 Thu Feb 1 19:50:13 UTC 2007 i686
      GNU/Linux
   3. gcc -v
      Using built-in specs.
      Target: i486-linux-gnu
      Configured with: ../src/configure -v
      --enable-languages=c,c++,fortran,objc,obj-c++,treelang
      --prefix=/usr --enable-shared --with-system-zlib
      --libexecdir=/usr/lib --without-included-gettext
      --enable-threads=posix --enable-nls --program-suffix=-4.1
      --enable-__cxa_atexit --enable-clocale=gnu
      --enable-libstdcxx-debug --enable-mpfr --enable-checking=release
      i486-linux-gnu
      Thread model: posix
      gcc version 4.1.2 20060928 (prerelease) (Ubuntu 4.1.1-13ubuntu5)
   4. ldd /usr/local/nagios/bin/nagios
      linux-gate.so.1 => (0xffffe000)
      libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0xb7f05000)
      libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 (0xb7ef2000)
      libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0xb7eed000)
      libglib-2.0.so.0 => /usr/lib/libglib-2.0.so.0 (0xb7e5b000)
      libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7d27000)
      /lib/ld-linux.so.2 (0xb7f3f000)
      librt.so.1 => /lib/tls/i686/cmov/librt.so.1 (0xb7d1e000)
   5. Nagios running fine!!!


##### User 4 (pitchfork) - Nagios running fine without SIGSEGV#####

   1. OS OpenSuSE 10.2 PPC
   2. uname -a Linux lpa117 2.6.18.2-34-ppc64 #1 SMP Mon Nov 27 11:46:27
      UTC 2006 ppc64 ppc64 ppc64 GNU/Linux
   3. gcc -v Using built-in specs.
      Target: powerpc64-suse-linux
      Configured with: ../configure --enable-threads=posix --prefix=/usr
      --with-local-prefix=/usr/local --infodir=/usr/share/info
      --mandir=/usr/share/man --libdir=/usr/lib --libexecdir=/usr/lib
      --enable-languages=c,c++,objc,fortran,obj-c++,java,ada
      --enable-checking=release
      --with-gxx-include-dir=/usr/include/c++/4.1.2 --enable-ssp
      --disable-libssp --disable-libgcj --with-slibdir=/lib
      --with-system-zlib --enable-shared --enable-__cxa_atexit
      --enable-libstdcxx-allocator=new --program-suffix=-4.1
      --enable-version-specific-runtime-libs --without-system-libunwind
      --with-cpu=default32 --enable-secureplt --with-long-double-128
      --host=powerpc64-suse-linux
      Thread model: posix
      gcc version 4.1.2 20061115 (prerelease) (SUSE Linux)
   4. ldd /usr/local/nagios/bin/nagios
      linux-vdso32.so.1 => (0x00100000)
      libm.so.6 => /lib/power5/libm.so.6 (0x0ff35000)
      libpthread.so.0 => /lib/power5/libpthread.so.0 (0x0fefc000)
      libdl.so.2 => /lib/libdl.so.2 (0x0fed8000)
      libc.so.6 => /lib/power5/libc.so.6 (0x0fd5d000)
      /lib/ld.so.1 (0xf7fe1000)
   5. Nagios running fine!!!


##### User 5 (asa) - Nagios running fine without SIGSEGV#####

   1. OS SuSe Linux Enterprise Server 9 (SLES9) x86
   2. uname -a Linux <hostname> 2.6.5-7.244-default #1 Mon Dec 12
      18:32:25 UTC 2005 i686 i686 i386 GNU/Linux
   3. gcc -v
      Reading specs from /usr/lib/gcc-lib/i586-suse-linux/3.3.3/specs
      Configured with: ../configure --enable-threads=posix --prefix=/usr
      --with-local-prefix=/usr/local --infodir=/usr/share/info
      --mandir=/usr/share/man --enable-languages=c,c++,f77,objc,java,ada
      --disable-checking --libdir=/usr/lib --enable-libgcj
      --with-gxx-include-dir=/usr/include/g++ --with-slibdir=/lib
      --with-system-zlib --enable-shared --enable-__cxa_atexit
      i586-suse-linux
      Thread model: posix
      gcc version 3.3.3 (SuSE Linux)
   4. ldd /usr/local/nagios3/bin/nagios
      linux-gate.so.1 => (0xffffe000)
      libm.so.6 => /lib/tls/libm.so.6 (0x40030000)
      libpthread.so.0 => /lib/tls/libpthread.so.0 (0x40052000)
      libltdl.so.3 => /usr/lib/libltdl.so.3 (0x40062000)
      libglib-2.0.so.0 => /opt/gnome/lib/libglib-2.0.so.0 (0x40069000)
      libc.so.6 => /lib/tls/libc.so.6 (0x400d6000)
      /lib/ld-linux.so.2 (0x40000000)
      libdl.so.2 => /lib/libdl.so.2 (0x401f0000)
   5. Nagios running fine!!!


##### User 6 (mess - system nr 2) - Nagios running fine without SIGSEGV#####

   1. OS:Red Hat Enterprise Linux AS release 4 (Nahant Update 3)
   2. uname -a: Linux xxxxxxxxxx 2.6.9-34.EL #1 Fri Feb 24 16:44:51 EST
      2006 i686 i686 i386 GNU/Linux
   3. gcc -v:
      Reading specs from /usr/lib/gcc/i386-redhat-linux/3.4.5/specs
      Configured with: ../configure --prefix=/usr
      --mandir=/usr/share/man --infodir=/usr/share/info --enable-shared
      --enable-threads=posix --disable-checking --with-system-zlib
      --enable-__cxa_atexit --disable-libunwind-exceptions
      --enable-java-awt=gtk --host=i386-redhat-linux
      Thread model: posix
      gcc version 3.4.5 20051201 (Red Hat 3.4.5-2)
   4. ldd /usr/local/nagios/bin/nagios
      linux-gate.so.1 => (0xffffe000)
      libm.so.6 => /lib/libm.so.6 (0xb7f64000)
      libpthread.so.0 => /lib/libpthread.so.0 (0xb7f4c000)
      libltdl.so.3 => /usr/lib/libltdl.so.3 (0xb7f45000)
      libglib-2.0.so.0 => /opt/gnome/lib/libglib-2.0.so.0 (0xb7eb0000)
      libc.so.6 => /lib/libc.so.6 (0xb7d81000)
      /lib/ld-linux.so.2 (0xb7fa9000)
      libdl.so.2 => /lib/libdl.so.2 (0xb7d7d000)
      librt.so.1 => /lib/librt.so.1 (0xb7d74000)
   5. Nagios running fine!!!


##### User 7 (flitti) - SIGSEGV detected #####

   1. OS: Debian Sarge Netinstall (inkl. aptitude upgrade) [vergleichbar
      mit 3.1rev5]
   2. uname -a Linux nagios-test 2.6.20.1 #1 SMP PREEMPT Tue Feb 27
      13:04:57 CET 2007 i686 GNU/Linux
   3. gcc -v Reading specs from /usr/lib/gcc-lib/i486-linux/3.3.5/specs
      Configured with: ../src/configure -v
      --enable-languages=c,c++,java,f77,pascal,objc,ada,treelang
      --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info
      --with-gxx-include-dir=/usr/include/c++/3.3 --enable-shared
      --enable-__cxa_atexit --with-system-zlib --enable-nls
      --without-included-gettext --enable-clocale=gnu --enable-debug
      --enable-java-gc=boehm --enable-java-awt=xlib --enable-objc-gc
      i486-linux
      Thread model: posix
      gcc version 3.3.5 (Debian 1:3.3.5-13)
   4. ldd /usr/local/nagios/bin/nagio libm.so.6 => /lib/tls/libm.so.6
      (0xb7f62000)
      libpthread.so.0 => /lib/tls/libpthread.so.0 (0xb7f52000)
      libdl.so.2 => /lib/tls/libdl.so.2 (0xb7f4f000)
      libc.so.6 => /lib/tls/libc.so.6 (0xb7e1a000)
      /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0xb7f8a000)
   5. Nagios crashes even when a check was started!!!


##### User 8 (fwe) - SIGSEGV detected #####

   1. OS: Debian 3.1
   2. uname -a: Linux grp-nagios 2.6.8-3-386 #1 Wed Dec 6 00:38:53 UTC
      2006 i686 GNU/Linux
   3. gcc -v
      Lese Spezifikationen von /usr/lib/gcc-lib/i486-linux/3.3.5/specs
      Konfiguriert mit: ../src/configure -v
      --enable-languages=c,c++,java,f77,pascal,objc,ada,treelang
      --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info
      --with-gxx-include-dir=/usr/include/c++/3.3 --enable-shared
      --enable-__cxa_atexit --with-system-zlib --enable-nls
      --without-included-gettext --enable-clocale=gnu --enable-debug
      --enable-java-gc=boehm --enable-java-awt=xlib --enable-objc-gc
      i486-linux
      Thread-Modell: posix
      gcc-Version 3.3.5 (Debian 1:3.3.5-13)
   4. ldd /usr/local/nagios/bin/nagios
      libm.so.6 => /lib/tls/libm.so.6 (0x4001c000)
      libpthread.so.0 => /lib/tls/libpthread.so.0 (0x4003e000)
      libdl.so.2 => /lib/tls/libdl.so.2 (0x4004e000)
      libc.so.6 => /lib/tls/libc.so.6 (0x40051000)
      /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000)
   5. Nagios crashes even when a check was started!!!

End of List

Hi List,

a few members showed me up some nice points about the "crashed users".

Both don't have:

enable threads in gcc -v output...

might be a hint?!?

Hello developers,
Hendrik supposed me to build the nagios source without the option '-O2'. I 
have done this, but nothing changed. The SIGSEGV also happens in Version 
3.0a2 without the '-O2' option.
Output from gdb:
#0  0xb7e43e84 in mallopt () from /lib/tls/libc.so.6
#1  0xb7e42dcb in free () from /lib/tls/libc.so.6
#2  0x080774ac in my_free ()
#3  0x08073d4f in clear_volatile_macros ()
#4  0x0806c92c in log_host_event ()
#5  0x080706a8 in handle_host_state ()
#6  0x0805bfa8 in process_host_check_result_3x ()
#7  0x0805bc42 in handle_async_host_check_result_3x ()
#8  0x08057533 in reap_check_results ()
#9  0x0806ac7d in handle_timed_event ()
#10 0x0806aaea in event_execution_loop ()
#11 0x080554f1 in main ()

Output from strace:
[pid 21706] gettimeofday({1175066605, 112492}, NULL) = 0
[pid 21706] nanosleep({0, 250000000},  <unfinished ...>
[pid 21707] <... select resumed> )      = 0 (Timeout)
[pid 21707] read(4, "", 4096)           = 0
[pid 21707] select(0, NULL, NULL, NULL, {0, 500000} <unfinished ...>
[pid 21706] <... nanosleep resumed> NULL) = 0
[pid 21706] time([1175066605])          = 1175066605
[pid 21706] time(NULL)                  = 1175066605
[pid 21706] gettimeofday({1175066605, 368499}, NULL) = 0
[pid 21706] nanosleep({0, 250000000}, NULL) = 0
[pid 21706] time([1175066605])          = 1175066605
[pid 21706] time(NULL)                  = 1175066605
[pid 21706] gettimeofday({1175066605, 624476}, NULL) = 0
[pid 21706] nanosleep({0, 250000000},  <unfinished ...>
[pid 21707] <... select resumed> )      = 0 (Timeout)
[pid 21707] read(4, "", 4096)           = 0
[pid 21707] select(0, NULL, NULL, NULL, {0, 500000} <unfinished ...>
[pid 21706] <... nanosleep resumed> NULL) = 0
[pid 21706] time([1175066605])          = 1175066605
[pid 21706] time(NULL)                  = 1175066605
[pid 21706] gettimeofday({1175066605, 880492}, NULL) = 0
[pid 21706] nanosleep({0, 250000000}, NULL) = 0
[pid 21706] time([1175066606])          = 1175066606
[pid 21706] gettimeofday({1175066606, 136283}, NULL) = 0
[pid 21706] time([1175066606])          = 1175066606
[pid 21706] open("/tmp/nagios2r2UEC", O_RDONLY) = 7
[pid 21706] fstat64(7, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
[pid 21706] close(7)                    = 0
[pid 21706] unlink("/tmp/nagios2r2UEC") = 0
[pid 21706] time([1175066606])          = 1175066606
[pid 21706] rt_sigaction(SIGPIPE, {0xb7f3fa70, [], 0}, {SIG_IGN}, 8) = 0
[pid 21706] send(3, "<14>Mar 28 09:23:26 nagios: Warn"..., 163, 0) = 163
[pid 21706] rt_sigaction(SIGPIPE, {SIG_IGN}, NULL, 8) = 0
[pid 21706] open("/usr/local/nagios/var/nagios.log", 
O_RDWR|O_APPEND|O_CREAT, 0666) = 7
[pid 21706] time([1175066606])          = 1175066606
[pid 21706] fstat64(7, {st_mode=S_IFREG|0644, st_size=538, ...}) = 0
[pid 21706] mmap2(NULL, 4096, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7fd8000
[pid 21706] write(7, "[1175066606] Warning: Return cod"..., 149) = 149
[pid 21706] close(7)                    = 0
[pid 21706] munmap(0xb7fd8000, 4096)    = 0
[pid 21706] gettimeofday({1175066606, 140312}, NULL) = 0
[pid 21706] write(1, "Warning: Return code of 127 for "..., 135Warning: 
Return code of 127 for check of host 'localhost' was out of bounds. Make 
sure the plugin you're trying to run actually exists.) = 135
[pid 21706] write(1, "\n", 1
)           = 1
[pid 21706] time([1175066606])          = 1175066606
[pid 21706] time([1175066606])          = 1175066606
[pid 21706] munmap(0xd5db370d, 842745699) = -1 EINVAL (Invalid argument)
[pid 21706] munmap(0xd7024313, 1664839277) = -1 EINVAL (Invalid argument)
[pid 21706] --- SIGSEGV (Segmentation fault) @ 0 (0) ---
[pid 21706] mmap2(NULL, 2097152, PROT_NONE, 
MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0xb6c36000
[pid 21706] munmap(0xb6c36000, 827392)  = 0
[pid 21706] munmap(0xb6e00000, 221184)  = 0
[pid 21706] mprotect(0xb6d00000, 135168, PROT_READ|PROT_WRITE) = 0
[pid 21706] --- SIGSEGV (Segmentation fault) @ 0 (0) ---
[pid 21707] <... select resumed> )      = ? ERESTARTNOHAND (To be 
restarted)
[pid 21707] +++ killed by SIGSEGV +++
[pid 21708] <... read resumed> 0xb76358c0, 511) = ? ERESTARTSYS (To be 
restarted)
[pid 21708] +++ killed by SIGSEGV +++
+++ killed by SIGSEGV +++


Thanks in advance,
FliTTi aka Matthias

Quoting matthias.flittner@...:

> Hello developers,
> Hendrik supposed me to build the nagios source without the option '-O2'. I
> have done this, but nothing changed. The SIGSEGV also happens in Version
> 3.0a2 without the '-O2' option.
> Output from gdb:
> #0  0xb7e43e84 in mallopt () from /lib/tls/libc.so.6
> #1  0xb7e42dcb in free () from /lib/tls/libc.so.6
> #2  0x080774ac in my_free ()
> #3  0x08073d4f in clear_volatile_macros ()
> #4  0x0806c92c in log_host_event ()
> #5  0x080706a8 in handle_host_state ()
> #6  0x0805bfa8 in process_host_check_result_3x ()
> #7  0x0805bc42 in handle_async_host_check_result_3x ()
> #8  0x08057533 in reap_check_results ()
> #9  0x0806ac7d in handle_timed_event ()
> #10 0x0806aaea in event_execution_loop ()
> #11 0x080554f1 in main ()


I think this is a problem with glibc6 version (2.3.2.ds1-22sa) in =20
debian sarge and the NPTL. I confirm the situation on a Debian Sarge =20
machine. I got no errors with the following workaround:

export LD_ASSUME_KERNEL=3D2.4.15
/usr/local/nagios/bin/nagios /usr/local/nagios/etc/nagios.cfg

The /debian/FAQ from glibc6 contains the following section:
The application should continue to run for now, but when the warning message
is printed use of the NPTL threading library, and other new kernel-2.6
features, it's worth trying to set environment variable LD_ASSUME_KERNEL=3D2=
.4.1
(switch to LinuxThreads instead of NPTL).

Debian using glibc-2.3 Ubuntu 2.4, RHEL5 glib-2.5. Perhaps this could =20
be a reason? See the debian changelog of glibc6:
http://packages.debian.org/changelogs/pool/main/g/glibc/glibc_2.5-0exp6/chan=
gelog

Best regards
Herbert Straub

Herbert Straub wrote:
> Quoting matthias.flittner@...:
> 
>> Hello developers,
>> Hendrik supposed me to build the nagios source without the option '-O2'. I
>> have done this, but nothing changed. The SIGSEGV also happens in Version
>> 3.0a2 without the '-O2' option.
>> Output from gdb:
>> #0  0xb7e43e84 in mallopt () from /lib/tls/libc.so.6
>> #1  0xb7e42dcb in free () from /lib/tls/libc.so.6
>> #2  0x080774ac in my_free ()
>> #3  0x08073d4f in clear_volatile_macros ()
>> #4  0x0806c92c in log_host_event ()
>> #5  0x080706a8 in handle_host_state ()
>> #6  0x0805bfa8 in process_host_check_result_3x ()
>> #7  0x0805bc42 in handle_async_host_check_result_3x ()
>> #8  0x08057533 in reap_check_results ()
>> #9  0x0806ac7d in handle_timed_event ()
>> #10 0x0806aaea in event_execution_loop ()
>> #11 0x080554f1 in main ()
> 
> 
> I think this is a problem with glibc6 version (2.3.2.ds1-22sa) in  
> debian sarge and the NPTL. I confirm the situation on a Debian Sarge  
> machine. I got no errors with the following workaround:
> 

Well, since NPTL has been discontinued, widely regarded as a mistake with
a too complex implementation, I think it would be a good idea to just
recommend people not to use it.

> export LD_ASSUME_KERNEL=2.4.15
> /usr/local/nagios/bin/nagios /usr/local/nagios/etc/nagios.cfg
> 
> The /debian/FAQ from glibc6 contains the following section:
> The application should continue to run for now, but when the warning message
> is printed use of the NPTL threading library, and other new kernel-2.6
> features, it's worth trying to set environment variable LD_ASSUME_KERNEL=2.4.1
> (switch to LinuxThreads instead of NPTL).
> 

Yup. I've seen the same happen for some other applications. It never became much
of an issue for those, as they had a pretty big warning in loads of places with
this solution pasted all over the place.

-- 
Andreas Ericsson                   andreas.ericsson@...
OP5 AB                             http://www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Andreas Ericsson wrote:
> Herbert Straub wrote:
>   
> Well, since NPTL has been discontinued, widely regarded as a mistake with
> a too complex implementation, I think it would be a good idea to just
> recommend people not to use it.
>
>   
I'm not sure, if it is my misunderstanding, but i think NPTL is
completely integrated in glibc. See

http://en.wikipedia.org/wiki/Native_POSIX_Thread_Library

and

http://www-128.ibm.com/developerworks/linux/library/l-threading.html?ca=dgr-lnxw07LinuxThreadsAndNPTL

getconf GNU_LIBPTHREAD_VERSION shows on

Debian Sarge: NPTL 0.60  <-- looks very old
Ubuntu Edgy: NPTL 2.4
Ubuntu Feisty: NPTL 2.5
RHEL 5: NPTL 2.5


The debian changelog:


    2006


      glibc <http://packages.debian.org/src:glibc> (2.4-1) UNRELEASED;
      urgency=low

   * New upstream version 2.4:

    - NPTL being the default thread library when available, only one
       version of the thread library is available (either linuxthreads or
       NPTL).  Closes: #399035 <http://bugs.debian.org/399035>.

http://packages.debian.org/changelogs/pool/main/g/glibc/glibc_2.5-0exp6/changelog

Best Regards
Herbert Straub

Thanks for the research Hendrik - As mentioned in later posts, it would 
appear from your findings that the threading libraries may be a problem.

Hendrik Baecker wrote:
> <snip_all>
> 
> Hi,
> 
> I've started a discussion thread on the german nagios-portal.de to
> animate different users to do some testing against latest cvs code.
> 
> Following information where asked:
> 
> 1. Operatingsystem + Version
> 2. Kernelversion
> 3. gcc -v
> 4. ldd against nagios binary
> 5. state of nagios
> 
> Following results came back up to now but I am not able to see a single
> hint - "Crash Users" at the end of the list:
> 
> If there are more information we should supply, please feel free to tell
> us what we should provice more.
> 
> ##### User 1 (mess)- Nagios running fine without SIGSEGV #####
> 
>    1. OS: Ubuntu 6.10
>    2. Linux galadriela 2.6.17-11-386 #2 Thu Feb 1 19:50:13 UTC 2007 i686
>       GNU/Linux
>    3. gcc -v
>       Using built-in specs.
>       Target: i486-linux-gnu
>       Configured with: ../src/configure -v
>       --enable-languages=c,c++,fortran,objc,obj-c++,treelang
>       --prefix=/usr --enable-shared --with-system-zlib
>       --libexecdir=/usr/lib --without-included-gettext
>       --enable-threads=posix --enable-nls --program-suffix=-4.1
>       --enable-__cxa_atexit --enable-clocale=gnu
>       --enable-libstdcxx-debug --enable-mpfr --enable-checking=release
>       i486-linux-gnu
>       Thread model: posix
>       gcc version 4.1.2 20060928 (prerelease) (Ubuntu 4.1.1-13ubuntu5)
>    4. ldd /usr/local/nagios/bin/nagios
>       linux-gate.so.1 => (0xffffe000)
>       libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0xb7f05000)
>       libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 (0xb7ef2000)
>       libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0xb7eed000)
>       libglib-2.0.so.0 => /usr/lib/libglib-2.0.so.0 (0xb7e5b000)
>       libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7d27000)
>       /lib/ld-linux.so.2 (0xb7f3f000)
>       librt.so.1 => /lib/tls/i686/cmov/librt.so.1 (0xb7d1e000)
>    5. Nagios running fine!!!
> 
> ##### User 2 (myself)- Nagios running fine without SIGSEGV#####
> 
>    1. OS: Ubuntu 6.10 (Edgy Eft) (i686)
>    2. uname -a: Linux abby 2.6.20-custom #1 SMP Tue Feb 27 18:45:20 CET
>       2007 i686 GNU/Linux (kein Distrokernel)
>    3. gcc -v:Using built-in specs.
>       Target: i486-linux-gnu
>       Configured with: ../src/configure -v
>       --enable-languages=c,c++,fortran,objc,obj-c++,treelang
>       --prefix=/usr --enable-shared --with-system-zlib
>       --libexecdir=/usr/lib --without-included-gettext
>       --enable-threads=posix --enable-nls --program-suffix=-4.1
>       --enable-__cxa_atexit --enable-clocale=gnu
>       --enable-libstdcxx-debug --enable-mpfr --enable-checking=release
>       i486-linux-gnu
>       Thread model: posix
>       gcc version 4.1.2 20060928 (prerelease) (Ubuntu 4.1.1-13ubuntu5)
>    4. ldd:ldd /usr/local/nagios-clean/bin/nagios
>       linux-gate.so.1 => (0xffffe000)
>       libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0xb7f46000)
>       libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 (0xb7f33000)
>       libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0xb7f2e000)
>       libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7dfa000)
>       /lib/ld-linux.so.2 (0xb7f7d000)
>    5. Nagios running fine!!!
> 
> 
> ##### User 3 (cybit) - Nagios running fine without SIGSEGV#####
> 
>    1. OS: Ubuntu 6.10
>    2. Linux galadriela 2.6.17-11-386 #2 Thu Feb 1 19:50:13 UTC 2007 i686
>       GNU/Linux
>    3. gcc -v
>       Using built-in specs.
>       Target: i486-linux-gnu
>       Configured with: ../src/configure -v
>       --enable-languages=c,c++,fortran,objc,obj-c++,treelang
>       --prefix=/usr --enable-shared --with-system-zlib
>       --libexecdir=/usr/lib --without-included-gettext
>       --enable-threads=posix --enable-nls --program-suffix=-4.1
>       --enable-__cxa_atexit --enable-clocale=gnu
>       --enable-libstdcxx-debug --enable-mpfr --enable-checking=release
>       i486-linux-gnu
>       Thread model: posix
>       gcc version 4.1.2 20060928 (prerelease) (Ubuntu 4.1.1-13ubuntu5)
>    4. ldd /usr/local/nagios/bin/nagios
>       linux-gate.so.1 => (0xffffe000)
>       libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0xb7f05000)
>       libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 (0xb7ef2000)
>       libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0xb7eed000)
>       libglib-2.0.so.0 => /usr/lib/libglib-2.0.so.0 (0xb7e5b000)
>       libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7d27000)
>       /lib/ld-linux.so.2 (0xb7f3f000)
>       librt.so.1 => /lib/tls/i686/cmov/librt.so.1 (0xb7d1e000)
>    5. Nagios running fine!!!
> 
> 
> ##### User 4 (pitchfork) - Nagios running fine without SIGSEGV#####
> 
>    1. OS OpenSuSE 10.2 PPC
>    2. uname -a Linux lpa117 2.6.18.2-34-ppc64 #1 SMP Mon Nov 27 11:46:27
>       UTC 2006 ppc64 ppc64 ppc64 GNU/Linux
>    3. gcc -v Using built-in specs.
>       Target: powerpc64-suse-linux
>       Configured with: ../configure --enable-threads=posix --prefix=/usr
>       --with-local-prefix=/usr/local --infodir=/usr/share/info
>       --mandir=/usr/share/man --libdir=/usr/lib --libexecdir=/usr/lib
>       --enable-languages=c,c++,objc,fortran,obj-c++,java,ada
>       --enable-checking=release
>       --with-gxx-include-dir=/usr/include/c++/4.1.2 --enable-ssp
>       --disable-libssp --disable-libgcj --with-slibdir=/lib
>       --with-system-zlib --enable-shared --enable-__cxa_atexit
>       --enable-libstdcxx-allocator=new --program-suffix=-4.1
>       --enable-version-specific-runtime-libs --without-system-libunwind
>       --with-cpu=default32 --enable-secureplt --with-long-double-128
>       --host=powerpc64-suse-linux
>       Thread model: posix
>       gcc version 4.1.2 20061115 (prerelease) (SUSE Linux)
>    4. ldd /usr/local/nagios/bin/nagios
>       linux-vdso32.so.1 => (0x00100000)
>       libm.so.6 => /lib/power5/libm.so.6 (0x0ff35000)
>       libpthread.so.0 => /lib/power5/libpthread.so.0 (0x0fefc000)
>       libdl.so.2 => /lib/libdl.so.2 (0x0fed8000)
>       libc.so.6 => /lib/power5/libc.so.6 (0x0fd5d000)
>       /lib/ld.so.1 (0xf7fe1000)
>    5. Nagios running fine!!!
> 
> 
> ##### User 5 (asa) - Nagios running fine without SIGSEGV#####
> 
>    1. OS SuSe Linux Enterprise Server 9 (SLES9) x86
>    2. uname -a Linux <hostname> 2.6.5-7.244-default #1 Mon Dec 12
>       18:32:25 UTC 2005 i686 i686 i386 GNU/Linux
>    3. gcc -v
>       Reading specs from /usr/lib/gcc-lib/i586-suse-linux/3.3.3/specs
>       Configured with: ../configure --enable-threads=posix --prefix=/usr
>       --with-local-prefix=/usr/local --infodir=/usr/share/info
>       --mandir=/usr/share/man --enable-languages=c,c++,f77,objc,java,ada
>       --disable-checking --libdir=/usr/lib --enable-libgcj
>       --with-gxx-include-dir=/usr/include/g++ --with-slibdir=/lib
>       --with-system-zlib --enable-shared --enable-__cxa_atexit
>       i586-suse-linux
>       Thread model: posix
>       gcc version 3.3.3 (SuSE Linux)
>    4. ldd /usr/local/nagios3/bin/nagios
>       linux-gate.so.1 => (0xffffe000)
>       libm.so.6 => /lib/tls/libm.so.6 (0x40030000)
>       libpthread.so.0 => /lib/tls/libpthread.so.0 (0x40052000)
>       libltdl.so.3 => /usr/lib/libltdl.so.3 (0x40062000)
>       libglib-2.0.so.0 => /opt/gnome/lib/libglib-2.0.so.0 (0x40069000)
>       libc.so.6 => /lib/tls/libc.so.6 (0x400d6000)
>       /lib/ld-linux.so.2 (0x40000000)
>       libdl.so.2 => /lib/libdl.so.2 (0x401f0000)
>    5. Nagios running fine!!!
> 
> 
> ##### User 6 (mess - system nr 2) - Nagios running fine without SIGSEGV#####
> 
>    1. OS:Red Hat Enterprise Linux AS release 4 (Nahant Update 3)
>    2. uname -a: Linux xxxxxxxxxx 2.6.9-34.EL #1 Fri Feb 24 16:44:51 EST
>       2006 i686 i686 i386 GNU/Linux
>    3. gcc -v:
>       Reading specs from /usr/lib/gcc/i386-redhat-linux/3.4.5/specs
>       Configured with: ../configure --prefix=/usr
>       --mandir=/usr/share/man --infodir=/usr/share/info --enable-shared
>       --enable-threads=posix --disable-checking --with-system-zlib
>       --enable-__cxa_atexit --disable-libunwind-exceptions
>       --enable-java-awt=gtk --host=i386-redhat-linux
>       Thread model: posix
>       gcc version 3.4.5 20051201 (Red Hat 3.4.5-2)
>    4. ldd /usr/local/nagios/bin/nagios
>       linux-gate.so.1 => (0xffffe000)
>       libm.so.6 => /lib/libm.so.6 (0xb7f64000)
>       libpthread.so.0 => /lib/libpthread.so.0 (0xb7f4c000)
>       libltdl.so.3 => /usr/lib/libltdl.so.3 (0xb7f45000)
>       libglib-2.0.so.0 => /opt/gnome/lib/libglib-2.0.so.0 (0xb7eb0000)
>       libc.so.6 => /lib/libc.so.6 (0xb7d81000)
>       /lib/ld-linux.so.2 (0xb7fa9000)
>       libdl.so.2 => /lib/libdl.so.2 (0xb7d7d000)
>       librt.so.1 => /lib/librt.so.1 (0xb7d74000)
>    5. Nagios running fine!!!
> 
> 
> ##### User 7 (flitti) - SIGSEGV detected #####
> 
>    1. OS: Debian Sarge Netinstall (inkl. aptitude upgrade) [vergleichbar
>       mit 3.1rev5]
>    2. uname -a Linux nagios-test 2.6.20.1 #1 SMP PREEMPT Tue Feb 27
>       13:04:57 CET 2007 i686 GNU/Linux
>    3. gcc -v Reading specs from /usr/lib/gcc-lib/i486-linux/3.3.5/specs
>       Configured with: ../src/configure -v
>       --enable-languages=c,c++,java,f77,pascal,objc,ada,treelang
>       --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info
>       --with-gxx-include-dir=/usr/include/c++/3.3 --enable-shared
>       --enable-__cxa_atexit --with-system-zlib --enable-nls
>       --without-included-gettext --enable-clocale=gnu --enable-debug
>       --enable-java-gc=boehm --enable-java-awt=xlib --enable-objc-gc
>       i486-linux
>       Thread model: posix
>       gcc version 3.3.5 (Debian 1:3.3.5-13)
>    4. ldd /usr/local/nagios/bin/nagio libm.so.6 => /lib/tls/libm.so.6
>       (0xb7f62000)
>       libpthread.so.0 => /lib/tls/libpthread.so.0 (0xb7f52000)
>       libdl.so.2 => /lib/tls/libdl.so.2 (0xb7f4f000)
>       libc.so.6 => /lib/tls/libc.so.6 (0xb7e1a000)
>       /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0xb7f8a000)
>    5. Nagios crashes even when a check was started!!!
> 
> 
> ##### User 8 (fwe) - SIGSEGV detected #####
> 
>    1. OS: Debian 3.1
>    2. uname -a: Linux grp-nagios 2.6.8-3-386 #1 Wed Dec 6 00:38:53 UTC
>       2006 i686 GNU/Linux
>    3. gcc -v
>       Lese Spezifikationen von /usr/lib/gcc-lib/i486-linux/3.3.5/specs
>       Konfiguriert mit: ../src/configure -v
>       --enable-languages=c,c++,java,f77,pascal,objc,ada,treelang
>       --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info
>       --with-gxx-include-dir=/usr/include/c++/3.3 --enable-shared
>       --enable-__cxa_atexit --with-system-zlib --enable-nls
>       --without-included-gettext --enable-clocale=gnu --enable-debug
>       --enable-java-gc=boehm --enable-java-awt=xlib --enable-objc-gc
>       i486-linux
>       Thread-Modell: posix
>       gcc-Version 3.3.5 (Debian 1:3.3.5-13)
>    4. ldd /usr/local/nagios/bin/nagios
>       libm.so.6 => /lib/tls/libm.so.6 (0x4001c000)
>       libpthread.so.0 => /lib/tls/libpthread.so.0 (0x4003e000)
>       libdl.so.2 => /lib/tls/libdl.so.2 (0x4004e000)
>       libc.so.6 => /lib/tls/libc.so.6 (0x40051000)
>       /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000)
>    5. Nagios crashes even when a check was started!!!
> 
> End of List
> 
> -------------------------------------------------------------------------


Ethan Galstad,
Nagios Developer
---
Email: nagios@...
Website: http://www.nagios.org

Herbert Straub wrote:
> Quoting matthias.flittner@...:
> 
>> Hello developers,
>> Hendrik supposed me to build the nagios source without the option '-O2'. I
>> have done this, but nothing changed. The SIGSEGV also happens in Version
>> 3.0a2 without the '-O2' option.
>> Output from gdb:
>> #0  0xb7e43e84 in mallopt () from /lib/tls/libc.so.6
>> #1  0xb7e42dcb in free () from /lib/tls/libc.so.6
>> #2  0x080774ac in my_free ()
>> #3  0x08073d4f in clear_volatile_macros ()
>> #4  0x0806c92c in log_host_event ()
>> #5  0x080706a8 in handle_host_state ()
>> #6  0x0805bfa8 in process_host_check_result_3x ()
>> #7  0x0805bc42 in handle_async_host_check_result_3x ()
>> #8  0x08057533 in reap_check_results ()
>> #9  0x0806ac7d in handle_timed_event ()
>> #10 0x0806aaea in event_execution_loop ()
>> #11 0x080554f1 in main ()
> 
> 
> I think this is a problem with glibc6 version (2.3.2.ds1-22sa) in  
> debian sarge and the NPTL. I confirm the situation on a Debian Sarge  
> machine. I got no errors with the following workaround:
> 
> export LD_ASSUME_KERNEL=2.4.15
> /usr/local/nagios/bin/nagios /usr/local/nagios/etc/nagios.cfg
> 
> The /debian/FAQ from glibc6 contains the following section:
> The application should continue to run for now, but when the warning message
> is printed use of the NPTL threading library, and other new kernel-2.6
> features, it's worth trying to set environment variable LD_ASSUME_KERNEL=2.4.1
> (switch to LinuxThreads instead of NPTL).
> 
> Debian using glibc-2.3 Ubuntu 2.4, RHEL5 glib-2.5. Perhaps this could  
> be a reason? See the debian changelog of glibc6:
> http://packages.debian.org/changelogs/pool/main/g/glibc/glibc_2.5-0exp6/changelog
> 
> Best regards
> Herbert Straub
> 

This could be a bug in glibc6 or NPTL, but it could also be a problem 
with the thread code in Nagios.  Perhaps in the signal handling or 
thread startup or state information?  I'm not sure where the problem 
might lie, or where to go for pointers on finding the source of the 
problem.  Any threading gurus care to lend a hand?


Ethan Galstad,
Nagios Developer
---
Email: nagios@...
Website: http://www.nagios.org

Quoting Ethan Galstad <nagios@...>:

> Herbert Straub wrote:
>> Quoting matthias.flittner@...:
>>
>>> Hello developers,
>>> Hendrik supposed me to build the nagios source without the option '-O2'.=
 I
>>> have done this, but nothing changed. The SIGSEGV also happens in Version
>>> 3.0a2 without the '-O2' option.
>>> Output from gdb:
>>> #0  0xb7e43e84 in mallopt () from /lib/tls/libc.so.6
>>> #1  0xb7e42dcb in free () from /lib/tls/libc.so.6
>>> #2  0x080774ac in my_free ()
>>> #3  0x08073d4f in clear_volatile_macros ()
>>> #4  0x0806c92c in log_host_event ()
>>> #5  0x080706a8 in handle_host_state ()
>>> #6  0x0805bfa8 in process_host_check_result_3x ()
>>> #7  0x0805bc42 in handle_async_host_check_result_3x ()
>>> #8  0x08057533 in reap_check_results ()
>>> #9  0x0806ac7d in handle_timed_event ()
>>> #10 0x0806aaea in event_execution_loop ()
>>> #11 0x080554f1 in main ()
>>
>>
>> I think this is a problem with glibc6 version (2.3.2.ds1-22sa) in
>> debian sarge and the NPTL. I confirm the situation on a Debian Sarge
>> machine. I got no errors with the following workaround:
>>
>> export LD_ASSUME_KERNEL=3D2.4.15
>> /usr/local/nagios/bin/nagios /usr/local/nagios/etc/nagios.cfg
>>
>> The /debian/FAQ from glibc6 contains the following section:
>> The application should continue to run for now, but when the warning mess=
age
>> is printed use of the NPTL threading library, and other new kernel-2.6
>> features, it's worth trying to set environment variable  =20
>> LD_ASSUME_KERNEL=3D2.4.1
>> (switch to LinuxThreads instead of NPTL).
>>
>> Debian using glibc-2.3 Ubuntu 2.4, RHEL5 glib-2.5. Perhaps this could
>> be a reason? See the debian changelog of glibc6:
>> http://packages.debian.org/changelogs/pool/main/g/glibc/glibc_2.5-0exp6/c=
hangelog
>>
>
> This could be a bug in glibc6 or NPTL, but it could also be a problem
> with the thread code in Nagios.  Perhaps in the signal handling or
> thread startup or state information?  I'm not sure where the problem
> might lie, or where to go for pointers on finding the source of the
> problem.  Any threading gurus care to lend a hand?
>

Yesterday I installed Ubuntu Warty, which using the same glibc and =20
NPTL version as Debian Sarge and i can reproduce the segmentation =20
fault error. Next, I tried to configure nagios with --enable-DEBUGALL =20
and see one warning and one error in the compile phase:

utils.c: In function ?get_raw_command_line2?:
utils.c:2944: error: ?cmd? undeclared (first use in this function)
utils.c:2944: error: (Each undeclared identifier is reported only once
utils.c:2944: error: for each function it appears in.)
--> Ok

utils.c: In function ?generate_check_stats?:
utils.c:5984: warning: incompatible implicit declaration of built-in =20
function ?round?
utils.c:5989: warning: incompatible implicit declaration of built-in =20
function ?round?
--> this could the reason for the segfault!

After patching nagios (see patch attachment), i could'nt reproduce the =20
segfault under Debian Sarge! Looks good! I compared the 2.8 with 3.0 =20
and see, that 2.8 doesn't contains the round function call.

For me, the details of this phenomenon are unclear, but one hint can =20
give this little test programm:

#include <stdio.h>
#include <math.h>

main()
{
         int test1; int test2; int test3;

         test1=3D5; test2=3D10;
         test3=3D(int)floor((double)(test1*test2));
         printf("test3: %d\n", test3);
         test3=3D(int)round((double)(test1*test2));
         printf("test3: %d\n", test3);
}

Debian Sarge:
$ gcc -o test_round -lm test_round.c
$ ./test_round
test3: 50
test3: 0


Ubuntu Edgy:
$ gcc -o test_round -lm test_round.c
test_round.c: In function ?main?:
test_round.c:11: warning: incompatible implicit declaration of =20
built-in function ?round?
$ ./test_round
./test_round
test3: 50
test3: 50

Anyway, please test the situation and report the situation.

Best Regards
Herbert Straub

Quoting Herbert Straub <herbert@...>:

> Anyway, please test the situation and report the situation.

The patch works also on Ubuntu Warty and there are no more  
segmentation faults. My testresults for problematic systems Debian  
Sarge und Ubuntu Warty OK.  Matthias Flittner has posted a SIGSEGV  
problem in this thread and Hendrik has posted a list, which contains  
one other people, who has a SIGSEGV detected (also on Debian Sarge).  
Would it be possible do test the patch on the problematic systems?


HowTo test the patch (patch attached):

Download nagios-3.0a2.tar.gz and nagios-3.0a2.patch to the home directory
tar xzf nagios-3.0a2.tar.gz
cd nagios-3.0a2
patch patch -p0 < ../nagios-3.0a2.patch
./configure
make all
sudo make install install-config

root$ /usr/local/nagios/bin/nagios /usr/local/nagios/etc/nagios.cfg


Best regards
Herbert Straub

Herbert Straub wrote:
>> Anyway, please test the situation and report the situation.
> 
> The patch works also on Ubuntu Warty and there are no more segmentation 
> faults. My testresults for problematic systems Debian Sarge und Ubuntu 
> Warty OK.  Matthias Flittner has posted a SIGSEGV problem in this thread 
> and Hendrik has posted a list, which contains one other people, who has 
> a SIGSEGV detected (also on Debian Sarge). Would it be possible do test 
> the patch on the problematic systems?

Hello Herbert, hello List,

I am one of the guys from Hendriks list who could reproduce the problem on a 
RHEL3 box.

Good news: with your patch nagios-3.0a2 gets no more SIGSEGV on RHEL3!

Kind regards,
Matthias


OS: Red Hat Enterprise Linux AS release 3 (Taroon Update 8)

uname -a:
Linux xxxxxxxxxxx 2.4.21-47.0.1.ELsmp #1 SMP Fri Oct 13 17:56:20 EDT 2006 i686 
i686 i386 GNU/Linux

gcc -v:
Reading specs from /usr/lib/gcc-lib/i386-redhat-linux/3.2.3/specs
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man 
--infodir=/usr/share/info --enable-shared --enable-threads=posix 
--disable-checking --with-system-zlib --enable-__cxa_atexit 
--host=i386-redhat-linux
Thread model: posix
gcc version 3.2.3 20030502 (Red Hat Linux 3.2.3-58)

ldd /usr/local/nagios/bin/nagios:
         libm.so.6 => /lib/tls/libm.so.6 (0x00652000)
         libpthread.so.0 => /lib/tls/libpthread.so.0 (0x0096f000)
         libltdl.so.3 => /usr/lib/libltdl.so.3 (0x00fd7000)
         libglib-2.0.so.0 => /usr/lib/libglib-2.0.so.0 (0x0053e000)
         libc.so.6 => /lib/tls/libc.so.6 (0x001ed000)
         /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x003ec000)
         libdl.so.2 => /lib/libdl.so.2 (0x00139000)

Herbert Straub wrote:
> Quoting Herbert Straub <herbert@...>:
> 
>> Anyway, please test the situation and report the situation.
> 
> The patch works also on Ubuntu Warty and there are no more segmentation 
> faults. My testresults for problematic systems Debian Sarge und Ubuntu 
> Warty OK.  Matthias Flittner has posted a SIGSEGV problem in this thread 
> and Hendrik has posted a list, which contains one other people, who has 
> a SIGSEGV detected (also on Debian Sarge). Would it be possible do test 
> the patch on the problematic systems?
> 
> 
> HowTo test the patch (patch attached):
> 
> Download nagios-3.0a2.tar.gz and nagios-3.0a2.patch to the home directory
> tar xzf nagios-3.0a2.tar.gz
> cd nagios-3.0a2
> patch patch -p0 < ../nagios-3.0a2.patch
> ./configure
> make all
> sudo make install install-config
> 
> root$ /usr/local/nagios/bin/nagios /usr/local/nagios/etc/nagios.cfg
> 
> 
> Best regards
> Herbert Straub

Thanks Herbert!  Patch has been applied to CVS with one change - half of 
the floor() calls have been changed to ceil() to try and compensate for 
the missing round().

I find it bizarre that calling round() would cause a segfault, so it 
definitely sounds like a bug with the libraries.  Since there aren't any 
round() calls in Nagios 2.x, there must be something else wrong in that 
code...


Ethan Galstad,
Nagios Developer
---
Email: nagios@...
Website: http://www.nagios.org

[snip all]
Herbert Straub wrote:
>> Anyway, please test the situation and report the situation.
[snip]

Hello Herbert, Hello Ethan, Hello List,
Thanks for your efforts. The newest Nagios version runs without any 
SIGSEGV. Thanks for patching it.

Regards,
FliTTi aka Matthias

matthias.flittner@... wrote:
> 
> [snip all]
> Herbert Straub wrote:
>  >> Anyway, please test the situation and report the situation.
> [snip]
> 
> Hello Herbert, Hello Ethan, Hello List,
> Thanks for your efforts. The newest Nagios version runs without any 
> SIGSEGV. Thanks for patching it.
> 
> Regards,
> FliTTi aka Matthias
> 
> 

Good to hear! :-)


Ethan Galstad,
Nagios Developer
---
Email: nagios@...
Website: http://www.nagios.org