I had daily slowloris attacks and i found this module very helpful.
Yesterday one of my server became unresponsive (Load was 265). At first i thought it was some other kind of DDOS attack but later when i checked my error_log i saw many glibc errors.
Today also my other server got unresponsive in the same way. And has also these errors.
When i turn off the qos module there are no errors anymore.
I am running 2 dedicated servers. One of them have Redhat5 and other has Centos 5.4
- Both have Apache/2.2.3
- Centos has PHP 5.3.1
- Apache has PHP 5.2.2
- Both have latest kernel and all other updates (yum update)
- They are hosted by different providers.
child process 1025 still did not exit, sending a SIGTERM
child process 1027 still did not exit, sending a SIGTERM
child process 1029 still did not exit, sending a SIGTERM
child process 1032 still did not exit, sending a SIGTERM
child process 1034 still did not exit, sending a SIGTERM
child process 1038 still did not exit, sending a SIGTERM
child process 1042 still did not exit, sending a SIGTERM
child process 1044 still did not exit, sending a SIGTERM
child process 1046 still did not exit, sending a SIGTERM
child process 1048 still did not exit, sending a SIGTERM
child process 1050 still did not exit, sending a SIGTERM
child process 1051 still did not exit, sending a SIGTERM
child process 1053 still did not exit, sending a SIGTERM
child process 1061 still did not exit, sending a SIGTERM
child process 1063 still did not exit, sending a SIGTERM
child process 1068 still did not exit, sending a SIGTERM
child process 1070 still did not exit, sending a SIGTERM
child process 1072 still did not exit, sending a SIGTERM
child process 1074 still did not exit, sending a SIGTERM
child process 1076 still did not exit, sending a SIGTERM
child process 1078 still did not exit, sending a SIGTERM
child process 1080 still did not exit, sending a SIGTERM
child process 1082 still did not exit, sending a SIGTERM
child process 1084 still did not exit, sending a SIGTERM
child process 1086 still did not exit, sending a SIGTERM
child process 1088 still did not exit, sending a SIGTERM
child process 1090 still did not exit, sending a SIGTERM
child process 1092 still did not exit, sending a SIGTERM
child process 1094 still did not exit, sending a SIGTERM
child process 1098 still did not exit, sending a SIGTERM
child process 1100 still did not exit, sending a SIGTERM
child process 1102 still did not exit, sending a SIGTERM
child process 1104 still did not exit, sending a SIGTERM
child process 1106 still did not exit, sending a SIGTERM
child process 1108 still did not exit, sending a SIGTERM
child process 1110 still did not exit, sending a SIGTERM
child process 1113 still did not exit, sending a SIGTERM
child process 1115 still did not exit, sending a SIGTERM
child process 1117 still did not exit, sending a SIGTERM
child process 1119 still did not exit, sending a SIGTERM
child process 1123 still did not exit, sending a SIGTERM
child process 1125 still did not exit, sending a SIGTERM
child process 1127 still did not exit, sending a SIGTERM
child process 1129 still did not exit, sending a SIGTERM
child process 1131 still did not exit, sending a SIGTERM
child process 1133 still did not exit, sending a SIGTERM
child process 1135 still did not exit, sending a SIGTERM
child process 1137 still did not exit, sending a SIGTERM
child process 1139 still did not exit, sending a SIGTERM
child process 1142 still did not exit, sending a SIGTERM
child process 1144 still did not exit, sending a SIGTERM
child process 1146 still did not exit, sending a SIGTERM
child process 1148 still did not exit, sending a SIGTERM
child process 1150 still did not exit, sending a SIGTERM
child process 1152 still did not exit, sending a SIGTERM
Any idea what can I do, because i realy need this module.
Thank you in advance and hope to hear from you soon!
-matej
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
When does this error happen?
- at server start?
- restart (sighup)?
- greaceful restart (sigusr1)?
- normal operation?
- reaching MaxRequestsPerChild?
What is your configuration (especially the mod_qos and the base server settings like ThreadsPerChild. MaxRequestsPerChild, ServerLimit, MaxClients, ….)?
What does "ldd httpd" show? What's the output of "httpd -V"?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
At normal operation. It happens almost every second.
What is your configuration (especially the mod_qos and the base server settings like ThreadsPerChild. MaxRequestsPerChild, ServerLimit, MaxClients, ….)?
> At normal operation. It happens almost every second.
So this happens probably right after forking a new child.
I'm still not able to reproduce the error. Do you use virtual hosts within your configuration? Does your sever have "Native POSIX Threads Library" (you may check this by executing the GNU C librabry, /usr/libc-*.so)?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Yes both Red Hat and Centos server host 3 virtual hosts (3 different domains)
[root@server ~]# /lib/libc-2.5.so
GNU C Library stable release version 2.5, by Roland McGrath et al.
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 4.1.2 20080704 (Red Hat 4.1.2-46).
Compiled on a Linux 2.6.9 system on 2009-12-07.
Available extensions:
The C stubs add-on version 2.1.2.
crypt add-on version 2.1 by Michael Glad and others
GNU Libidn by Simon Josefsson
GNU libio by Per Bothner
NIS(YP)/NIS+ NSS modules 0.19 by Thorsten Kukuk
Native POSIX Threads Library by Ulrich Drepper et al
BIND-8.2.3-T5B
RT using linux kernel aio
Thread-local storage support included.
For bug reporting instructions, please see:
<http://www.gnu.org/software/libc/bugs.html>.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
[root@server ~]# /lib/libc.so.6
GNU C Library stable release version 2.5, by Roland McGrath et al.
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 4.1.2 20080704 (Red Hat 4.1.2-46).
Compiled on a Linux 2.6.9 system on 2009-12-07.
Available extensions:
The C stubs add-on version 2.1.2.
crypt add-on version 2.1 by Michael Glad and others
GNU Libidn by Simon Josefsson
GNU libio by Per Bothner
NIS(YP)/NIS+ NSS modules 0.19 by Thorsten Kukuk
Native POSIX Threads Library by Ulrich Drepper et al
BIND-8.2.3-T5B
RT using linux kernel aio
Thread-local storage support included.
For bug reporting instructions, please see: http://www.gnu.org/software/libc/bugs.html.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
NPTL is okay. glibc/kernel version should not matter (I've tested with new and old versions). Can't really see any reason for this behavior at the moment.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
It is realy weird that the same error happens on two totaly different servers with totaly different hardware configurations and different OS.
Is there a way i could compile the mod in some mode, so you get more info about the error ?
maybe this version of glibc has issues? older/newer don't ?
The other things that come to my mind:
both servers use mod_rewrite a lot
both servers are virtual hosts
both servers handle a lot of traffic
none of them use any kind of proxys
if i think of anything else i will let you know.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
It may help if we could determine which function causes the error. Please test the module using different configurations:
1) Delete the QS_ClientEntries directive (since it has not effect at all as long as no QS_Client* directive is used).
2) Test using per server limitation settings only (only set the QS_SrvMaxConnPerIP and QS_SrvMaxConnClose directive).
3) Test using the per connection bandwidth limitation settings only (only set the QS_SrvMinDataRate directive).
This should show us if the configuration 2) or 3) (or both) cause the error.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
ist this case solved?
i have exactly same problem. The moment i got this flood of errors is if i do a reload of apache. restart works, seems only to be the reload which causes this problem.
THis is the config iam using, you can find it in many tutorials:
i did the update but no other result. sometimes at reloading there are "only" segmentation faults, but in other case i can see the glic-errors again. I will deliver my data:
/lib/libc.so.6
GNU C Library development release version 2.4 (20060526), by Roland McGrath et al.
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Configured for i686-suse-linux.
Compiled by GNU CC version 4.1.0 (SUSE Linux).
Compiled on a Linux 2.6.16 system on 2006-05-26.
Available extensions:
crypt add-on version 2.1 by Michael Glad and others
GNU Libidn by Simon Josefsson
GNU libio by Per Bothner
NIS(YP)/NIS+ NSS modules 0.19 by Thorsten Kukuk
NoVersion patch for broken glibc 2.0 binaries
Native POSIX Threads Library by Ulrich Drepper et al
BIND-8.2.3-T5B
Thread-local storage support included.
For bug reporting instructions, please see:
<http://www.gnu.org/software/libc/bugs.html>.
I'm currently using Apache 2.2.15/16 for testing and prefer the MPM worker mode (which probably won't have those DSO module unload/load problems).
I've never tested a setup using a different run-time APR library from those used at compile time.
> Server version: Apache/2.2.10 (Unix)
> Server loaded: APR 1.3.8, APR-Util 1.3.9
> Compiled using: APR 1.3.3, APR-Util 1.3.4
I recommend:
- to use MPM worker binaries if applicable (depends on you application/loaded modules)
- up2date Apache server
- up2date APR/APR-util libraries (to compile and to run the server)
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
the apr-thing was a bad example from my side. situation from testserver, but problem was the same on other correct servers…
Apache-worker is not an option for me, but will try newer version next weeks and report back.
Hans
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hello!
I had daily slowloris attacks and i found this module very helpful.
Yesterday one of my server became unresponsive (Load was 265). At first i thought it was some other kind of DDOS attack but later when i checked my error_log i saw many glibc errors.
Today also my other server got unresponsive in the same way. And has also these errors.
When i turn off the qos module there are no errors anymore.
I am running 2 dedicated servers. One of them have Redhat5 and other has Centos 5.4
- Both have Apache/2.2.3
- Centos has PHP 5.3.1
- Apache has PHP 5.2.2
- Both have latest kernel and all other updates (yum update)
- They are hosted by different providers.
**one of the many glibc errors:**
*** glibc detected *** /usr/sbin/httpd: corrupted double-linked list: 0x09563960 ***
======= Backtrace: =========
/lib/libc.so.6
/lib/libc.so.6(cfree+0x59)
/usr/lib/libapr-1.so.0(apr_pool_destroy+0x155)
/usr/lib/libapr-1.so.0(apr_thread_exit+0x25)
/etc/httpd/modules/mod_qos.so
/usr/lib/libapr-1.so.0
/lib/libpthread.so.0
/lib/libc.so.6(clone+0x5e)
======= Memory map: ========
00110000-0012c000 r-xp 00000000 08:03 63224400 /usr/lib/libaprutil-1.so.0.2.7
0012c000-0012d000 rwxp 0001b000 08:03 63224400 /usr/lib/libaprutil-1.so.0.2.7
0012d000-0013a000 r-xp 00000000 08:03 63213457 /usr/lib/liblber-2.3.so.0.2.31
0013a000-0013b000 rwxp 0000c000 08:03 63213457 /usr/lib/liblber-2.3.so.0.2.31
0013b000-00176000 r-xp 00000000 08:03 102957956 /lib/libsepol.so.1
00176000-00177000 rwxp 0003b000 08:03 102957956 /lib/libsepol.so.1
00177000-00181000 rwxp 00177000 00:00 0
00181000-00184000 r-xp 00000000 08:03 102957951 /lib/libuuid.so.1.2
00184000-00185000 rwxp 00003000 08:03 102957951 /lib/libuuid.so.1.2
00185000-00194000 r-xp 00000000 08:03 102957963 /lib/libresolv-2.5.so
00194000-00195000 r-xp 0000e000 08:03 102957963 /lib/libresolv-2.5.so
00195000-00196000 rwxp 0000f000 08:03 102957963 /lib/libresolv-2.5.so
00196000-00198000 rwxp 00196000 00:00 0
00198000-001b0000 r-xp 00000000 08:03 63226047 /usr/lib/libsasl2.so.2.0.22
001b0000-001b1000 rwxp 00017000 08:03 63226047 /usr/lib/libsasl2.so.2.0.22
001b1000-001c4000 r-xp 00000000 08:03 102957950 /lib/libnsl-2.5.so
001c4000-001c5000 r-xp 00012000 08:03 102957950 /lib/libnsl-2.5.so
001c5000-001c6000 rwxp 00013000 08:03 102957950 /lib/libnsl-2.5.so
001c6000-001c8000 rwxp 001c6000 00:00 0
001c8000-001ca000 r-xp 00000000 08:03 102957964 /lib/libcom_err.so.2.1
001ca000-001cb000 rwxp 00001000 08:03 102957964 /lib/libcom_err.so.2.1
001cb000-001dd000 r-xp 00000000 08:03 63225997 /usr/lib/libz.so.1.2.3
001dd000-001de000 rwxp 00011000 08:03 63225997 /usr/lib/libz.so.1.2.3
001de000-001e0000 r-xp 00000000 08:03 102957126 /lib/libkeyutils-1.2.so
001e0000-001e1000 rwxp 00001000 08:03 102957126 /lib/libkeyutils-1.2.so
001e1000-001e3000 r-xp 00000000 08:03 63636038 /usr/lib/httpd/modules/mod_auth_basic.so
001e3000-001e5000 rwxp 00001000 08:03 63636038 /usr/lib/httpd/modules/mod_auth_basic.so
001e5000-001e6000 r-xp 00000000 08:03 63636116 /usr/lib/httpd/modules/mod_authn_default.so
001e6000-001e8000 rwxp 00000000 08:03 63636116 /usr/lib/httpd/modules/mod_authn_default.so
001e8000-001ea000 r-xp 00000000 08:03 63636122 /usr/lib/httpd/modules/mod_authz_host.so
001ea000-001ec000 rwxp 00001000 08:03 63636122 /usr/lib/httpd/modules/mod_authz_host.so
001ec000-001ed000 r-xp 00000000 08:03 63636123 /usr/lib/httpd/modules/mod_authz_owner.so
001ed000-001ef000 rwxp 00001000 08:03 63636123 /usr/lib/httpd/modules/mod_authz_owner.so
001ef000-001f5000 r-xp 00000000 08:03 63636118 /usr/lib/httpd/modules/mod_authnz_ldap.so
001f5000-001f7000 rwxp 00005000 08:03 63636118 /usr/lib/httpd/modules/mod_authnz_ldap.so
001f7000-001fc000 r-xp 00000000 08:03 63636155 /usr/lib/httpd/modules/mod_log_config.so
001fc000-001fe000 rwxp 00004000 08:03 63636155 /usr/lib/httpd/modules/mod_log_config.so
001fe000-001ff000 r-xp 00000000 08:03 63636161 /usr/lib/httpd/modules/mod_logio.so
001ff000-00201000 rwxp 00000000 08:03 63636161 /usr/lib/httpd/modules/mod_logio.so
00201000-00203000 r-xp 00000000 08:03 63636141 /usr/lib/httpd/modules/mod_expires.so
00203000-00205000 rwxp 00001000 08:03 63636141 /usr/lib/httpd/modules/mod_expires.so
00205000-00208000 r-xp 00000000 08:03 63636166 /usr/lib/httpd/modules/mod_mime.so
00208000-0020a000 rwxp 00003000 08:03 63636166 /usr/lib/httpd/modules/mod_mime.so
0020a000-00211000 r-xp 00000000 08:03 63636125 /usr/lib/httpd/modules/mod_autoindex.so
00211000-00212000 -xp 00007000 08:03 63636125 /usr/lib/httpd/modules/mod_autoindex.so
00212000-00214000 rwxp 00007000 08:03 63636125 /usr/lib/httpd/modules/mod_autoindex.so
00214000-00216000 r-xp 00000000 08:03 63636291 /usr/lib/httpd/modules/mod_vhost_alias.so
00216000-00218000 rwxp 00001000 08:03 63636291 /usr/lib/httpd/modules/mod_vhost_alias.so
00218000-00219000 r-xp 00000000 08:03 63636131 /usr/lib/httpd/modules/mod_dir.so
00219000-0021a000 -xp 00001000 08:03 63636131 /usr/lib/httpd/modules/mod_dir.so
0021a000-0021c000 rwxp 00001000 08:03 63636131 /usr/lib/httpd/modules/mod_dir.so
0021c000-0022a000 r-xp 00000000 08:03 63636152 /usr/lib/httpd/modules/mod_rewrite.so
0022a000-0022c000 rwxp 0000d000 08:03 63636152 /usr/lib/httpd/modules/mod_rewrite.so
0022c000-0022d000 r-xp 00000000 08:03 63636156 /usr/lib/httpd/modules/mod_suexec.so
0022d000-0022f000 rwxp 00000000 08:03 63636156 /usr/lib/httpd/modules/mod_suexec.so
00230000-00257000 r-xp 00000000 08:03 63220343 /usr/lib/libapr-1.so.0.2.7
00257000-00258000 rwxp 00026000 08:03 63220343 /usr/lib/libapr-1.so.0.2.7
00258000-00397000 r-xp 00000000 08:03 102957943 /lib/libc-2.5.so
00397000-00398000 -xp 0013f000 08:03 102957943 /lib/libc-2.5.so
00398000-0039a000 r-xp 0013f000 08:03 102957943 /lib/libc-2.5.so
0039a000-0039b000 rwxp 00141000 08:03 102957943 /lib/libc-2.5.so
0039b000-0039e000 rwxp 0039b000 00:00 0
0039e000-0039f000 r-xp 00000000 08:03 63636290 /usr/lib/httpd/modules/mod_version.so
0039f000-003a1000 rwxp 00000000 08:03 63636290 /usr/lib/httpd/modules/mod_version.so
003a1000-003be000 r-xp 00000000 08:03 63636146 /usr/lib/httpd/modules/mod_qos.so
003be000-003bf000 rwxp 0001d000 08:03 63636146 /usr/lib/httpd/modules/mod_qos.so
003bf000-003c0000 r-xp 00000000 08:03 63635910 /usr/lib/httpd/modules/mod_antiloris.so
003c0000-003c1000 rwxp 00000000 08:03 63635910 /usr/lib/httpd/modules/mod_antiloris.so
003c1000-003c4000 r-xp 00000000 08:03 102957944 /lib/libcap.so.1.10
003c4000-003c5000 rwxp 00002000 08:03 102957944 /lib/libcap.so.1.10
003c5000-003c9000 r-xp 00000000 08:03 63221377 /usr/lib/libdistcache.so.1.0.1
003c9000-003ca000 rwxp 00003000 08:03 63221377 /usr/lib/libdistcache.so.1.0.1
003d2000-003ff000 r-xp 00000000 08:03 63214750 /usr/lib/libgssapi_krb5.so.2.2
003ff000-00400000 rwxp 0002d000 08:03 63214750 /usr/lib/libgssapi_krb5.so.2.2
00400000-00410000 r-xp 00000000 08:03 63214788 /usr/lib/libbz2.so.1.0.3
00410000-00411000 rwxp 00010000 08:03 63214788 /usr/lib/libbz2.so.1.0.3
00411000-00426000 r-xp 00000000 08:03 63636104 /usr/lib/httpd/modules/mod_python.so
00426000-00429000 rwxp 00014000 08:03 63636104 /usr/lib/httpd/modules/mod_python.so
00429000-00448000 r-xp 00000000 08:03 102957967 /lib/libexpat.so.0.5.0
00448000-0044a000 rwxp 0001e000 08:03 102957967 /lib/libexpat.so.0.5.0
0044a000-0047d000 r-xp 00000000 08:03 63635565 /usr/lib/httpd/modules/mod_perl.so
0047d000-0047f000 rwxp 00033000 08:03 63635565 /usr/lib/httpd/modules/mod_perl.so
0047f000-00487000 r-xp 00000000 08:03 63215113 /usr/lib/libnal.so.1.0.1
00487000-00488000 rwxp 00007000 08:03 63215113 /usr/lib/libnal.so.1.0.1
00492000-00499000 r-xp 00000000 08:03 63636269 /usr/lib/httpd/modules/mod_negotiation.so
00499000-0049b000 rwxp 00006000 08:03 63636269 /usr/lib/httpd/modules/mod_negotiation.so
004a6000-004a8000 r-xp 00000000 08:03 63573867 /usr/lib/python2.4/lib-dynload/syslog.so
004a8000-004a9000 rwxp 00001000 08:03 63573867 /usr/lib/python2.4/lib-dynload/syslog.so
004a9000-004b2000 r-xp 00000000 08:03 102957255 /lib/libnss_files-2.5.so
004b2000-004b3000 r-xp 00008000 08:03 102957255 /lib/libnss_files-2.5.so
004b3000-004b4000 rwxp 00009000 08:03 102957255 /lib/libnss_files-2.5.so
004b4000-004bf000 r-xp 00000000 08:03 102957955 /lib/libgcc_s-4.1.2-20080825.so.1
004bf000-004c0000 rwxp 0000a000 08:03 102957955 /lib/libgcc_s-4.1.2-20080825.so.1
004c5000-00510000 r-xp 00000000 08:03 63223238 /usr/sbin/httpd
00510000-00512000 rwxp 0004b000 08:03 63223238 /usr/sbin/httpd
00512000-00515000 rwxp 00512000 00:00 0
00536000-00537000 r-xp 00536000 00:00 0
00537000-005ca000 r-xp 00000000 08:03 63214746 /usr/lib/libkrb5.so.3.3
005ca000-005cd000 rwxp 00092000 08:03 63214746 /usr/lib/libkrb5.so.3.3
005d5000-006ca000 r-xp 00000000 08:03 102957969 /lib/libdb-4.3.so
006ca000-006cd000 rwxp 000f4000 08:03 102957969 /lib/libdb-4.3.so
006ce000-006d7000 r-xp 00000000 08:03 102957308 /lib/libcrypt-2.5.so
006d7000-006d8000 r-xp 00008000 08:03 102957308 /lib/libcrypt-2.5.so
006d8000-006d9000 rwxp 00009000 08:03 102957308 /lib/libcrypt-2.5.so
006d9000-00700000 rwxp 006d9000 00:00 0
00704000-00761000 r-xp 00000000 08:03 63220326 /usr/lib/libsqlite3.so.0.8.6
00761000-00763000 rwxp 0005d000 08:03 63220326 /usr/lib/libsqlite3.so.0.8.6
00773000-00798000 r-xp 00000000 08:03 102957954 /lib/libm-2.5.so
00798000-00799000 r-xp 00024000 08:03 102957954 /lib/libm-2.5.so
00799000-0079a000 rwxp 00025000 08:03 102957954 /lib/libm-2.5.so
0079a000-008c4000 r-xp 00000000 08:03 102957965 /lib/libcrypto.so.0.9.8e
008c4000-008d7000 rwxp 00129000 08:03 102957965 /lib/libcrypto.so.0.9.8e
008d7000-008db000 rwxp 008d7000 00:00 0
00900000-00903000 r-xp 00000000 08:03 63573869 /usr/lib/python2.4/lib-dynload/timemodule.so
00903000-00905000 rwxp 00002000 08:03 63573869 /usr/lib/python2.4/lib-dynload/timemodule.so
00905000-00924000 r-xp 00000000 08:03 63212113 /usr/lib/libpq.so.4.1
00924000-00925000 rwxp 0001f000 08:03 63212113 /usr/lib/libpq.so.4.1
00935000-0093c000 r-xp 00000000 08:03 102957947 /lib/librt-2.5.so
0093c000-0093d000 r-xp 00006000 08:03 102957947 /lib/librt-2.5.so
0093d000-0093e000 rwxp 00007000 08:03 102957947 /lib/librt-2.5.so
0093e000-00942000 r-xp 00000000 08:03 63636165 /usr/lib/httpd/modules/mod_auth_mysql.so
00942000-00943000 rwxp 00004000 08:03 63636165 /usr/lib/httpd/modules/mod_auth_mysql.so
0094d000-00958000 r-xp 00000000 08:03 63636016 /usr/lib/httpd/modules/mod_ldap.so
00958000-0095a000 rwxp 0000a000 08:03 63636016 /usr/lib/httpd/modules/mod_ldap.so
00970000-00972000 r-xp 00000000 08:03 102957952 /lib/libdl-2.5.so
00972000-00973000 r-xp 00001000 08:03 102957952 /lib/libdl-2.5.so
00973000-00974000 rwxp 00002000 08:03 102957952 /lib/libdl-2.5.so
00974000-009a7000 r-xp 00000000 08:03 63222542 /usr/lib/sse2/libgmp.so.3.3.3
009a7000-009a8000 rwxp 00032000 08:03 63222542 /usr/lib/sse2/libgmp.so.3.3.3
009b0000-009f3000 r-xp 00000000 08:03 102957070 /lib/libssl.so.0.9.8e
009f3000-009f7000 rwxp 00042000 08:03 102957070 /lib/libssl.so.0.9.8e
00a02000-00a03000 r-xp 00000000 08:03 63636120 /usr/lib/httpd/modules/mod_authz_default.so
00a03000-00a05000 rwxp 00000000 08:03 63636120 /usr/lib/httpd/modules/mod_authz_default.so
00a05000-00a09000 r-xp 00000000 08:03 102957091 /lib/libnss_dns-2.5.so
00a09000-00a0a000 r-xp 00003000 08:03 102957091 /lib/libnss_dns-2.5.so
00a0a000-00a0b000 rwxp 00004000 08:03 102957091 /lib/libnss_dns-2.5.so
00a0b000-00a0c000 r-xp 00000000 08:03 63636133 /usr/lib/httpd/modules/mod_env.so
00a0c000-00a0e000 rwxp 00000000 08:03 63636133 /usr/lib/httpd/modules/mod_env.so
00a0e000-00a36000 r-xp 00000000 08:03 63636049 /usr/lib/httpd/modules/mod_ssl.so
00a36000-00a38000 rwxp 00027000 08:03 63636049 /usr/lib/httpd/modules/mod_ssl.so
00a38000-00a3a000 rwxp 00a38000 00:00 0
00aae000-00ab1000 r-xp 00000000 08:03 63573853 /usr/lib/python2.4/lib-dynload/cStringIO.so
00ab1000-00ab2000 rwxp 00003000 08:03 63573853 /usr/lib/python2.4/lib-dynload/cStringIO.so
00ab3000-00abb000 r-xp 00000000 08:03 63214036 /usr/lib/libkrb5support.so.0.1
00abb000-00abc000 rwxp 00007000 08:03 63214036 /usr/lib/libkrb5support.so.0.1
00abe000-00ac0000 r-xp 00000000 08:03 102957344 /lib/libutil-2.5.so
00ac0000-00ac1000 r-xp 00001000 08:03 102957344 /lib/libutil-2.5.so
00ac1000-00ac2000 rwxp 00002000 08:03 102957344 /lib/libutil-2.5.so
00ac2000-00ac6000 r-xp 00000000 08:03 63572809 /usr/lib/python2.4/lib-dynload/stropmodule.so
00ac6000-00ac8000 rwxp 00004000 08:03 63572809 /usr/lib/python2.4/lib-dynload/stropmodule.so
00acc000-00ae0000 r-xp 00000000 08:03 102957946 /lib/libpthread-2.5.so
00ae0000-00ae1000 r-xp 00013000 08:03 102957946 /lib/libpthread-2.5.so
00ae1000-00ae2000 rwxp 00014000 08:03 102957946 /lib/libpthread-2.5.so
00ae2000-00ae4000 rwxp 00ae2000 00:00 0
00ae8000-00aea000 r-xp 00000000 08:03 63636145 /usr/lib/httpd/modules/mod_file_cache.so
00aea000-00aec000 rwxp 00001000 08:03 63636145 /usr/lib/httpd/modules/mod_file_cache.so
00aec000-00af1000 r-xp 00000000 08:03 63572784 /usr/lib/python2.4/lib-dynload/itertoolsmodule.so
00af1000-00af3000 rwxp 00005000 08:03 63572784 /usr/lib/python2.4/lib-dynload/itertoolsmodule.so
00afc000-00b12000 r-xp 00000000 08:03 102957957 /lib/libselinux.so.1
00b12000-00b14000 rwxp 00015000 08:03 102957957 /lib/libselinux.so.1
00b1d000-00b1f000 r-xp 00000000 08:03 63636008 /usr/lib/httpd/modules/mod_alias.so
00b1f000-00b20000 -xp 00002000 08:03 63636008 /usr/lib/httpd/modules/mod_alias.so
00b20000-00b22000 rwxp 00002000 08:03 63636008 /usr/lib/httpd/modules/mod_alias.so
00baa000-00bac000 r-xp 00000000 08:03 63636121 /usr/lib/httpd/modules/mod_authz_groupfile.so
00bac000-00bae000 rwxp 00001000 08:03 63636121 /usr/lib/httpd/modules/mod_authz_groupfile.so
00c08000-00c0a000 r-xp 00000000 08:03 63636119 /usr/lib/httpd/modules/mod_authz_dbm.so
00c0a000-00c0c000 rwxp 00001000 08:03 63636119 /usr/lib/httpd/modules/mod_authz_dbm.so
00cb0000-00cca000 r-xp 00000000 08:03 102957092 /lib/ld-2.5.so
00cca000-00ccb000 r-xp 00019000 08:03 102957092 /lib/ld-2.5.so
00ccb000-00ccc000 rwxp 0001a000 08:03 102957092 /lib/ld-2.5.so
00cd0000-00cd1000 r-xp 00000000 08:03 63636124 /usr/lib/httpd/modules/mod_authz_user.so
00cd1000-00cd3000 rwxp 00000000 08:03 63636124 /usr/lib/httpd/modules/mod_authz_user.so
00cd3000-00dfe000 r-xp 00000000 08:03 63308067 /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so
00dfe000-00e03000 rwxp 0012a000 08:03 63308067 /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so
00e03000-00e05000 rwxp 00e03000 00:00 0
00e27000-00e29000 r-xp 00000000 08:03 63636282 /usr/lib/httpd/modules/mod_setenvif.so
00e29000-00e2b000 rwxp 00001000 08:03 63636282 /usr/lib/httpd/modules/mod_setenvif.so
00e71000-00e72000 r-xp 00000000 08:03 63636117 /usr/lib/httpd/modules/mod_authn_file.so
00e72000-00e74000 rwxp 00000000 08:03 63636117 /usr/lib/httpd/modules/mod_authn_file.so
00e7b000-00e7d000 r-xp 00000000 08:03 63636287 /usr/lib/httpd/modules/mod_userdir.so
00e7d000-00e7f000 rwxp 00001000 08:03 63636287 /usr/lib/httpd/modules/mod_userdir.so
00e8e000-00ec7000 r-xp 00000000 08:03 63215403 /usr/lib/libldap-2.3.so.0.2.31
00ec7000-00ec8000 rwxp 00039000 08:03 63215403 /usr/lib/libldap-2.3.so.0.2.31
00ecc000-00ef1000 r-xp 00000000 08:03 63214112 /usr/lib/libk5crypto.so.3.1
00ef1000-00ef2000 rwxp 00025000 08:03 63214112 /usr/lib/libk5crypto.so.3.1
00f10000-00f2c000 r-xp 00000000 08:03 102957972 /lib/libpcre.so.0.0.1
00f2c000-00f2d000 rwxp 0001c000 08:03 102957972 /lib/libpcre.so.0.0.1
00f2d000-0104e000 r-xp 00000000 08:03 63603196 /usr/lib/mysql/libmysqlclient.so.15.0.0
0104e000-01090000 rwxp 00120000 08:03 63603196 /usr/lib/mysql/libmysqlclient.so.15.0.0
01090000-01091000 rwxp 01090000 00:00 0
04ced000-04de1000 r-xp 00000000 08:03 63222904 /usr/lib/libpython2.4.so.1.0
04de1000-04e03000 rwxp 000f4000 08:03 63222904 /usr/lib/libpython2.4.so.1.0
04e03000-04e06000 rwxp 04e03000 00:00 0
05732000-0585e000 r-xp 00000000 08:03 63219945 /usr/lib/libxml2.so.2.6.26
0585e000-05863000 rwxp 0012c000 08:03 63219945 /usr/lib/libxml2.so.2.6.26
05863000-05864000 rwxp 05863000 00:00 0
08c35000-08f47000 r-xp 00000000 08:03 63636153 /usr/lib/httpd/modules/libphp5.so
08f47000-08f7b000 rwxp 00311000 08:03 63636153 /usr/lib/httpd/modules/libphp5.so
08f7b000-08f91000 rwxp 08f7b000 00:00 0
09117000-09660000 rw-p 09117000 00:00 0
b6300000-b6321000 rw-p b6300000 00:00 0
b6321000-b6400000 --p b6321000 00:00 0
b64cb000-b64cc000 --p b64cb000 00:00 0
b64cc000-b6ecc000 rw-p b64cc000 00:00 0
b6ecc000-b6edc000 rw-s 00000000 00:09 64529657 /dev/zero (deleted)
b6edc000-b6f5a000 rw-s 00000000 00:09 64529655 /dev/zero (deleted)
b6f5a000-b7f5a000 rw-s 00000000 00:09 294912 /SYSV00000000 (deleted)
b7fc3000-b7fdc000 rw-s 00000000 00:09 64529652 /dev/zero (deleted)
b7fdc000-b7fe4000 rw-p b7fdc000 00:00 0
b7fed000-b7fef000 rw-s 00000000 00:09 64529653 /dev/zero (deleted)
bfd11000-bfd26000 rw-p bffea000 00:00 0
child pid 14619 exit signal Aborted (6)
And when server gets halted i get these errors:
child process 1025 still did not exit, sending a SIGTERM
child process 1027 still did not exit, sending a SIGTERM
child process 1029 still did not exit, sending a SIGTERM
child process 1032 still did not exit, sending a SIGTERM
child process 1034 still did not exit, sending a SIGTERM
child process 1038 still did not exit, sending a SIGTERM
child process 1042 still did not exit, sending a SIGTERM
child process 1044 still did not exit, sending a SIGTERM
child process 1046 still did not exit, sending a SIGTERM
child process 1048 still did not exit, sending a SIGTERM
child process 1050 still did not exit, sending a SIGTERM
child process 1051 still did not exit, sending a SIGTERM
child process 1053 still did not exit, sending a SIGTERM
child process 1061 still did not exit, sending a SIGTERM
child process 1063 still did not exit, sending a SIGTERM
child process 1068 still did not exit, sending a SIGTERM
child process 1070 still did not exit, sending a SIGTERM
child process 1072 still did not exit, sending a SIGTERM
child process 1074 still did not exit, sending a SIGTERM
child process 1076 still did not exit, sending a SIGTERM
child process 1078 still did not exit, sending a SIGTERM
child process 1080 still did not exit, sending a SIGTERM
child process 1082 still did not exit, sending a SIGTERM
child process 1084 still did not exit, sending a SIGTERM
child process 1086 still did not exit, sending a SIGTERM
child process 1088 still did not exit, sending a SIGTERM
child process 1090 still did not exit, sending a SIGTERM
child process 1092 still did not exit, sending a SIGTERM
child process 1094 still did not exit, sending a SIGTERM
child process 1098 still did not exit, sending a SIGTERM
child process 1100 still did not exit, sending a SIGTERM
child process 1102 still did not exit, sending a SIGTERM
child process 1104 still did not exit, sending a SIGTERM
child process 1106 still did not exit, sending a SIGTERM
child process 1108 still did not exit, sending a SIGTERM
child process 1110 still did not exit, sending a SIGTERM
child process 1113 still did not exit, sending a SIGTERM
child process 1115 still did not exit, sending a SIGTERM
child process 1117 still did not exit, sending a SIGTERM
child process 1119 still did not exit, sending a SIGTERM
child process 1123 still did not exit, sending a SIGTERM
child process 1125 still did not exit, sending a SIGTERM
child process 1127 still did not exit, sending a SIGTERM
child process 1129 still did not exit, sending a SIGTERM
child process 1131 still did not exit, sending a SIGTERM
child process 1133 still did not exit, sending a SIGTERM
child process 1135 still did not exit, sending a SIGTERM
child process 1137 still did not exit, sending a SIGTERM
child process 1139 still did not exit, sending a SIGTERM
child process 1142 still did not exit, sending a SIGTERM
child process 1144 still did not exit, sending a SIGTERM
child process 1146 still did not exit, sending a SIGTERM
child process 1148 still did not exit, sending a SIGTERM
child process 1150 still did not exit, sending a SIGTERM
child process 1152 still did not exit, sending a SIGTERM
Any idea what can I do, because i realy need this module.
Thank you in advance and hope to hear from you soon!
-matej
Here are also all the httpd modules:
Loaded Modules:
core_module (static)
mpm_prefork_module (static)
http_module (static)
so_module (static)
auth_basic_module (shared)
authn_file_module (shared)
authn_default_module (shared)
authz_host_module (shared)
authz_user_module (shared)
authz_owner_module (shared)
authz_groupfile_module (shared)
authz_dbm_module (shared)
authz_default_module (shared)
ldap_module (shared)
authnz_ldap_module (shared)
log_config_module (shared)
logio_module (shared)
env_module (shared)
expires_module (shared)
setenvif_module (shared)
mime_module (shared)
autoindex_module (shared)
vhost_alias_module (shared)
negotiation_module (shared)
dir_module (shared)
userdir_module (shared)
alias_module (shared)
rewrite_module (shared)
suexec_module (shared)
file_cache_module (shared)
version_module (shared)
antiloris_module (shared)
mysql_auth_module (shared)
perl_module (shared)
php5_module (shared)
python_module (shared)
ssl_module (shared)
I've seen that you've added new version, but i still have the same problem.
I recommend to update the Apache server first.
I have the latest release i can get through yum updates from Centos
I have upgraded apache to the latest release:
Server version: Apache/2.2.14 (Unix)
I have rebuild mod_qos, but i STILL get the same error on all servers.
Then comes long memory map and at the end
When i disable the module the error is gone.
I forgot to add that the error is repeating for every child
Hi Matej
When does this error happen?
- at server start?
- restart (sighup)?
- greaceful restart (sigusr1)?
- normal operation?
- reaching MaxRequestsPerChild?
What is your configuration (especially the mod_qos and the base server settings like ThreadsPerChild. MaxRequestsPerChild, ServerLimit, MaxClients, ….)?
What does "ldd httpd" show? What's the output of "httpd -V"?
At normal operation. It happens almost every second.
> At normal operation. It happens almost every second.
So this happens probably right after forking a new child.
I'm still not able to reproduce the error. Do you use virtual hosts within your configuration? Does your sever have "Native POSIX Threads Library" (you may check this by executing the GNU C librabry, /usr/libc-*.so)?
Yes both Red Hat and Centos server host 3 virtual hosts (3 different domains)
/lib/libc.so.6 would probably be the correct libc to verify
[root@server ~]# /lib/libc.so.6
GNU C Library stable release version 2.5, by Roland McGrath et al.
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 4.1.2 20080704 (Red Hat 4.1.2-46).
Compiled on a Linux 2.6.9 system on 2009-12-07.
Available extensions:
The C stubs add-on version 2.1.2.
crypt add-on version 2.1 by Michael Glad and others
GNU Libidn by Simon Josefsson
GNU libio by Per Bothner
NIS(YP)/NIS+ NSS modules 0.19 by Thorsten Kukuk
Native POSIX Threads Library by Ulrich Drepper et al
BIND-8.2.3-T5B
RT using linux kernel aio
Thread-local storage support included.
For bug reporting instructions, please see:
http://www.gnu.org/software/libc/bugs.html.
/lib/libc.so.6 has symlink to /lib/libc-2.5.so
NPTL is okay. glibc/kernel version should not matter (I've tested with new and old versions). Can't really see any reason for this behavior at the moment.
It is realy weird that the same error happens on two totaly different servers with totaly different hardware configurations and different OS.
Is there a way i could compile the mod in some mode, so you get more info about the error ?
maybe this version of glibc has issues? older/newer don't ?
The other things that come to my mind:
both servers use mod_rewrite a lot
both servers are virtual hosts
both servers handle a lot of traffic
none of them use any kind of proxys
if i think of anything else i will let you know.
CentOS and Redhat are very similar (both use the same source). A stack trace of a core file taken with a debug binary would be helpful.
where do i get that / how do i do that?
It may help if we could determine which function causes the error. Please test the module using different configurations:
1) Delete the QS_ClientEntries directive (since it has not effect at all as long as no QS_Client* directive is used).
2) Test using per server limitation settings only (only set the QS_SrvMaxConnPerIP and QS_SrvMaxConnClose directive).
3) Test using the per connection bandwidth limitation settings only (only set the QS_SrvMinDataRate directive).
This should show us if the configuration 2) or 3) (or both) cause the error.
I tried with this:
No glibc error or any other error but got server load over 200 after 5 - 10 minutes of running
then i tried with
<IfModule mod_qos.c>
MaxClients 256
QS_SrvMinDataRate 150 1200
</IfModule>
and got glibc errors
i also noticed that i get httpd <defunct> process every few seconds…. you think this has anything to do with it ?
Hello,
ist this case solved?
i have exactly same problem. The moment i got this flood of errors is if i do a reload of apache. restart works, seems only to be the reload which causes this problem.
THis is the config iam using, you can find it in many tutorials:
LoadModule qos_module /usr/lib/apache/mod_qos.so
QS_ClientEntries 100000
QS_SrvMaxConnPerIP 50
MaxClients 150
QS_SrvMaxConnClose 100
QS_SrvMinDataRate 150 1200
after changing to this config also a reload was working:
LoadModule qos_module /usr/lib/apache/mod_qos.so
QS_SrvMaxConnPerIP 40
MaxClients 150
QS_SrvMaxConnClose 180
Apache is 2.2.10 and mod_qos 9.22
What to do now? Would like to use mod_qos, seems to be a good mod to fight against dos and stuff…
Thanks,
Hans
If you encounter this problem with an MPM prefork Apache, the answer is probably yes (using mod_qos 9.24). Let me know if not.
Hello,
i did the update but no other result. sometimes at reloading there are "only" segmentation faults, but in other case i can see the glic-errors again. I will deliver my data:
httpd -V
Server version: Apache/2.2.10 (Unix)
Server built: Aug 28 2009 08:54:09
Server's Module Magic Number: 20051115:18
Server loaded: APR 1.3.8, APR-Util 1.3.9
Compiled using: APR 1.3.3, APR-Util 1.3.4
Architecture: 32-bit
Server MPM: Prefork
threaded: no
forked: yes (variable process count)
Server compiled with….
-D APACHE_MPM_DIR="server/mpm/prefork"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/usr"
-D SUEXEC_BIN="/usr/sbin/suexec"
-D DEFAULT_PIDLOG="/www/logs/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_LOCKFILE="/www/logs/accept.lock"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="/etc/httpd/mime.types"
-D SERVER_CONFIG_FILE="/etc/httpd/httpd.conf"
ldd /usr/sbin/httpd
linux-gate.so.1 => (0xffffe000)
libm.so.6 => /lib/libm.so.6 (0xb7f1e000)
libaprutil-1.so.0 => /usr/lib/libaprutil-1.so.0 (0xb7eff000)
libexpat.so.1 => /usr/lib/libexpat.so.1 (0xb7ee0000)
libapr-1.so.0 => /usr/lib/libapr-1.so.0 (0xb7eb9000)
librt.so.1 => /lib/librt.so.1 (0xb7eaf000)
libcrypt.so.1 => /lib/libcrypt.so.1 (0xb7e7c000)
libpthread.so.0 => /lib/libpthread.so.0 (0xb7e68000)
libdl.so.2 => /lib/libdl.so.2 (0xb7e64000)
libc.so.6 => /lib/libc.so.6 (0xb7d44000)
/lib/ld-linux.so.2 (0xb7f58000)
libuuid.so.1 => /lib/libuuid.so.1 (0xb7d40000)
/lib/libc.so.6
GNU C Library development release version 2.4 (20060526), by Roland McGrath et al.
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Configured for i686-suse-linux.
Compiled by GNU CC version 4.1.0 (SUSE Linux).
Compiled on a Linux 2.6.16 system on 2006-05-26.
Available extensions:
crypt add-on version 2.1 by Michael Glad and others
GNU Libidn by Simon Josefsson
GNU libio by Per Bothner
NIS(YP)/NIS+ NSS modules 0.19 by Thorsten Kukuk
NoVersion patch for broken glibc 2.0 binaries
Native POSIX Threads Library by Ulrich Drepper et al
BIND-8.2.3-T5B
Thread-local storage support included.
For bug reporting instructions, please see:
<http://www.gnu.org/software/libc/bugs.html>.
(current kernel is 2.6.25.20-25.1-pae)
This is the error itself when reloading apache.
Apache/2.2.10 (Unix) DAV/2 PHP/5.2.12 with Suhosin-Patch mod_ssl/2.2.10 OpenSSL/0.9.8a mod_qos/9.24 configured - resuming normal operations
Graceful restart requested, doing restart
*** glibc detected *** /usr/sbin/httpd: double free or corruption (!prev): 0x08714b20 ***
======= Backtrace: =========
/lib/libc.so.6
/lib/libc.so.6(__libc_free+0x84)
/usr/lib/libapr-1.so.0(apr_allocator_destroy+0x2d)
/usr/lib/libapr-1.so.0(apr_pool_destroy+0x1ba)
/usr/sbin/httpd
/usr/sbin/httpd
/usr/sbin/httpd
/usr/sbin/httpd
/usr/sbin/httpd(ap_mpm_run+0x267)
/usr/sbin/httpd(main+0xb46)
/lib/libc.so.6(__libc_start_main+0xdc)
/usr/sbin/httpd(apr_os_proc_mutex_put+0x5d)
======= Memory map: ========
*** glibc detected *** /usr/sbin/httpd: double free or corruption (!prev): 0x08714b20 ***
*** glibc detected *** /usr/sbin/httpd: double free or corruption (!prev): 0x08714b20 ***
======= Backtrace: =========
======= Backtrace: =========
/lib/libc.so.6
/lib/libc.so.6
/lib/libc.so.6(__libc_free+0x84)
/lib/libc.so.6(__libc_free+0x84)
/usr/lib/libapr-1.so.0(apr_allocator_destroy+0x2d)
/usr/lib/libapr-1.so.0(apr_allocator_destroy+0x2d)
/usr/lib/libapr-1.so.0(apr_pool_destroy+0x1ba)
/usr/lib/libapr-1.so.0(apr_pool_destroy+0x1ba)
/usr/sbin/httpd
/usr/sbin/httpd
/usr/sbin/httpd
/usr/sbin/httpd
/usr/sbin/httpd
/usr/sbin/httpd
/usr/sbin/httpd
/usr/sbin/httpd
/usr/sbin/httpd(ap_mpm_run+0x267)
/usr/sbin/httpd(ap_mpm_run+0x267)
/usr/sbin/httpd(main+0xb46)
/usr/sbin/httpd(main+0xb46)
/lib/libc.so.6(__libc_start_main+0xdc)
/lib/libc.so.6(__libc_start_main+0xdc)
/usr/sbin/httpd(apr_os_proc_mutex_put+0x5d)
/usr/sbin/httpd(apr_os_proc_mutex_put+0x5d)
======= Memory map: ========
======= Memory map: ========
*** glibc detected *** /usr/sbin/httpd: double free or corruption (!prev): 0x08714b20 ***
======= Backtrace: =========
/lib/libc.so.6
/lib/libc.so.6(__libc_free+0x84)
/usr/lib/libapr-1.so.0(apr_allocator_destroy+0x2d)
/usr/lib/libapr-1.so.0(apr_pool_destroy+0x1ba)
/usr/sbin/httpd
/usr/sbin/httpd
/usr/sbin/httpd
/usr/sbin/httpd
/usr/sbin/httpd(ap_mpm_run+0x267)
/usr/sbin/httpd(main+0xb46)
/lib/libc.so.6(__libc_start_main+0xdc)
/usr/sbin/httpd(apr_os_proc_mutex_put+0x5d)
======= Memory map: ========
*** glibc detected *** /usr/sbin/httpd: double free or corruption (!prev): 0x08714b20 ***
======= Backtrace: =========
/lib/libc.so.6
/lib/libc.so.6(__libc_free+0x84)
/usr/lib/libapr-1.so.0(apr_allocator_destroy+0x2d)
/usr/lib/libapr-1.so.0(apr_pool_destroy+0x1ba)
/usr/sbin/httpd
/usr/sbin/httpd
/usr/sbin/httpd
/usr/sbin/httpd
/usr/sbin/httpd(ap_mpm_run+0x267)
/usr/sbin/httpd(main+0xb46)
/lib/libc.so.6(__libc_start_main+0xdc)
/usr/sbin/httpd(apr_os_proc_mutex_put+0x5d)
======= Memory map: ========
Digest: generating secret for digest authentication …
Digest: done
Apache/2.2.10 (Unix) DAV/2 PHP/5.2.12 with Suhosin-Patch mod_ssl/2.2.10 OpenSSL/0.9.8a mod_qos/9.24 configured - resuming normal operations
long lost child came home! (pid 25663)
child pid 25665 exit signal Aborted (6)
child pid 25667 exit signal Aborted (6)
child pid 25669 exit signal Aborted (6)
child pid 25671 exit signal Aborted (6)
child pid 25673 exit signal Aborted (6)
or also
Graceful restart requested, doing restart
*** glibc detected *** /usr/sbin/httpd: corrupted double-linked list: 0x08716b50 ***
======= Backtrace: =========
/lib/libc.so.6
/lib/libc.so.6
/lib/libc.so.6(__libc_free+0x84)
/usr/lib/libapr-1.so.0(apr_allocator_destroy+0x2d)
/usr/lib/libapr-1.so.0(apr_pool_destroy+0x1ba)
/usr/sbin/httpd
/usr/sbin/httpd
/usr/sbin/httpd
/usr/sbin/httpd
/usr/sbin/httpd(ap_mpm_run+0x267)
/usr/sbin/httpd(main+0xb46)
/lib/libc.so.6(__libc_start_main+0xdc)
/usr/sbin/httpd(apr_os_proc_mutex_put+0x5d)
======= Memory map: ========
Do you need more data? I want to help to solve the case. What should i do?
Thanks,
Hans
I'm currently using Apache 2.2.15/16 for testing and prefer the MPM worker mode (which probably won't have those DSO module unload/load problems).
I've never tested a setup using a different run-time APR library from those used at compile time.
> Server version: Apache/2.2.10 (Unix)
> Server loaded: APR 1.3.8, APR-Util 1.3.9
> Compiled using: APR 1.3.3, APR-Util 1.3.4
I recommend:
- to use MPM worker binaries if applicable (depends on you application/loaded modules)
- up2date Apache server
- up2date APR/APR-util libraries (to compile and to run the server)
the apr-thing was a bad example from my side. situation from testserver, but problem was the same on other correct servers…
Apache-worker is not an option for me, but will try newer version next weeks and report back.
Hans