mod_qos / Discussion / Open Discussion: exit signal Floating point exception

D Roberts - 2010-11-23

Anyone else seen this error with apache when using mod_qos ?

" child pid 29994 exit signal Floating point exception (8)"

I get this every so often and mod_qos seems to have problems tracking concurrent connections. I have tracked it to mod_qos via repeated testing with / without different modules. I have also tracked it to something in the mod_qos configuration.

I will update if I find what in the configuration causes the issue, but wondered if anyone else had already seen the issue.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

D Roberts - 2010-11-23

I cannot be 100% sure, but testing seems to strongly indicate that this problem only occurs when I have an "Event" setting such as the below:

##    BrowserMatchNoCase            "iPhone.*Mac.*AppleWebKit.*Mobile" iPhone
##    QS_EventRequestLimit          iPhone 20
##    QS_EventPerSecLimit           iPhone 15

I will test more, but this will hopefully let anyone that's more in the know, give some advice or suggestions as to the cause.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

D Roberts - 2010-11-23

Yep - pretty conclusive (IMHO) that it's the combination of QS_EventXXXXX that's causing this. I've used the firefox plugin to change my useragent, and when you use one that triggers event rules it occasionally (far too often for a real web site with that useragent) bombs out. If you don't hit that event trigger it potters along nicely. So it's something in that section of code I am guessing.

Sadly, my C / C++ is so, so old that I doubt I'd be able to hazzard a guess at where by looking at the code.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Pascal Buchbinder - 2010-11-23

Hi mrdaleroberts.
Thank you for reporting these issues. On what platform do you encounter these errors? A stack trace would be very helpful (enable core dumps and get a stack trace out of the core file written by Apache).

I guess this happens somewhere when calculating hits/second caused by different size of time variables (depending on the operating system). I'll try to make the code more portable in this area…

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Pascal Buchbinder - 2010-11-23

I can't reproduce the error but made some changes in mod_qos 9.34 which are probably worth to give a try.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

D Roberts - 2010-11-24

Hi Pascal,

I'm running on CentOS release 5.5 with Apache 2.2.17 (output from httpd -V) is:

Server version: Apache/2.2.17 (Unix)
Server built:   Nov 4 2010 14:33:01
Server's Module Magic Number: 20051115:25
Server loaded: APR 1.2.7, APR-Util 1.2.7
Compiled using: APR 1.2.7, APR-Util 1.2.7
Architecture:   32-bit
Server MPM:     Worker
threaded:     yes (fixed thread count)
    forked:     yes (variable process count)
Server compiled with….
-D APACHE_MPM_DIR="server/mpm/worker"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/usr/local/apache2.2.17"
-D SUEXEC_BIN="/usr/local/apache2.2.17/bin/suexec"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"

I'll continue looking into getting core files our and also at testing your new version of mod_qos.

Thanks

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

D Roberts - 2010-11-24

OK,

This is going to seem like an idiot questions, but how do I get apache to generate core dumps. I've tried adding ulimit -c to the service script and adding the CoreDumpDirectory directive to the config file and still I don't get coredumps. As far as I can tell this is all you do to enable coredumps (or have I missed something that will require me to go back and recompile apache) ?

Again, apologies for the idiot question. Oh, and I'm sorry to say, the problem still exists in mod_qos version 9.34.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

D Roberts - 2010-11-24

Don't know if this means anything, but after recompiling on our dev system, addin gin mod_qos 9.34 and mod_backtrace (still can't get core dumps) I get this:

pid 30358 mod_backtrace backtrace for sig 8 (thread "pid" 30358)
pid 30358 mod_backtrace main() is at 8062ab4
/usr/local/apache2.2.17/modules/mod_backtrace.so
/usr/local/apache2.2.17/bin/httpd(ap_run_fatal_exception+0x46)
/usr/local/apache2.2.17/bin/httpd
/usr/local/apache2.2.17/bin/httpd

/usr/local/apache2.2.17/modules/mod_qos.so
/usr/local/apache2.2.17/modules/mod_qos.so
/usr/local/apache2.2.17/bin/httpd(ap_run_log_transaction+0x46)
/usr/local/apache2.2.17/bin/httpd(ap_process_request+0xda)
/usr/local/apache2.2.17/bin/httpd
/usr/local/apache2.2.17/bin/httpd(ap_run_process_connection+0x46)
/usr/local/apache2.2.17/bin/httpd(ap_process_connection+0x58)
/usr/local/apache2.2.17/bin/httpd
/usr/local/apache2.2.17/bin/httpd
/usr/lib/libapr-1.so.0
/lib/libpthread.so.0
/lib/libc.so.6(clone+0x5e)
pid 30358 mod_backtrace end of backtrace
child pid 30358 exit signal Floating point exception (8)

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

D Roberts - 2010-11-24

If you wanted to be swamped by detail (as I can't get a core dump for love nor money) here are the compile / install options I used for the non debug apache:

Apache:: ./configure —prefix=/usr/local/apache2.2.17 —with-mpm=worker —enable-mods-shared=all —enable-ssl=shared —enable-ssl —with-ssl —enable-proxy —enable-proxy-connect —enable-proxy-ftp —enable-proxy-http —enable-file-cache —enable-cache —enable-mem-cache —enable-bucketeer —enable-reqtimeout —enable-substitute —enable-deflate —enable-proxy-ajp —enable-proxy-balancer —with-pcre –enable-nonportable-atomics=yes; make; make install

mod_jk:: cd native; ./configure —with-apxs=/usr/local/apache2.2.17/bin/apxs; make ; make install

mod_qos:: cd apache2; /usr/local/apache2.2.17/bin/apxs -i -c mod_qos.c

mod_security:: cd apache2; ./configure —with-apxs=/usr/local/apache2.2.17/bin/apxs; make; make install

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Pascal Buchbinder - 2010-11-24

ulimit -c unlimited and setting the CoreDumpDirectory: that's exactly what I would have tried to get a core file.

CentOS: it's a 32bit server, isn't it? I'll try to reproduce the error.

configure Apache: So you compiled the Apache server yourself on the host you encounter the problems with mod_qos, right?

QS_EventRequestLimit and QS_EventPerSecLimit: are the only mod_qos directives you are using at the moment? In sight or out sight a VirtualHost? Do you have any VirtualHosts?

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

D Roberts - 2010-11-24

Correct, it's 32 bit. and I have compiled apache myself in order to get the latest version of mod_qos.

As to the directives, I have a number of directives from mod_qos all included at the global apache level but only adding the specified directives causes issue. I would send you the config but I'm not able to access that systems at the moment.

I have a number of virtual hosts defined but the directives for mod_qos are "outside" at the global level as I say. If it's any help / difference I have ssl for one of those virtual hosts and the configs listens on 3 different ports in total.

P.S Thanks for all the help.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

D Roberts - 2010-11-25

Here's the qos definition I have:

<IfModule qos_module>
    SetEnvIf                      Remote_Addr 43.193. QS_VipRequest=yes
    QS_SrvMaxConnClose            70%
    QS_ErrorPage                  /static/maintenance/busy.html
    QS_ErrorResponseCode          414
    ErrorDocument                 414 /static/maintenance/busy.html
    ErrorDocument                 500 /static/maintenance/busy.html
    QS_VipHeaderName              mod-qos-nonIP
    QS_VipIPHeaderName            mod-qos-IP
    QS_SessionKey                 quai2feSusingicieXexa
    QS_ClientPrefer               90
    QS_ClientEntries              500000
    QS_SrvMinDataRate             150 1200
    LimitRequestFields            30
    QS_LimitRequestBody           102400
    QS_SrvMaxConnPerIP            30
    QS_SrvMaxConnExcludeIP        10.0.0.1
    QS_SrvMaxConnExcludeIP        10.1.1.1
    QS_LocRequestLimitMatch       "^/.*/$" 20
    QS_LocRequestPerSecLimitMatch "^/.*/$" 50
    QS_LocRequestLimitMatch       "^/(servlet/file|media)/.*$" 75
    QS_LocRequestPerSecLimitMatch "^/(servlet/file|media)/.*$" 150
    QS_LocRequestLimitMatch       "^/.*\.*$" 75
    QS_LocRequestPerSecLimitMatch "^/.*\.*$" 150
    QS_LocRequestLimitMatch       ^/psmove/$ 30
    QS_LocRequestPerSecLimitMatch ^/psmove/$ 15
    BrowserMatchNoCase            "PlayStation Portable" PSP
    QS_EventRequestLimit          PSP 20
    QS_EventPerSecLimit           PSP 15
    BrowserMatchNoCase            "iPhone.*Mac.*AppleWebKit.*Mobile" iPhone
    QS_EventRequestLimit          iPhone 20
    QS_EventPerSecLimit           iPhone 15
    SetEnvIf                      Request_Method POST Poster
    QS_EventRequestLimit          Poster 20
    QS_EventPerSecLimit           Poster 15
    BrowserMatchNoCase            "spider"      Some_Spider
    BrowserMatchNoCase            "robot"       Some_Spider
    BrowserMatchNoCase            "crawler"     Some_Spider
    BrowserMatchNoCase            "sleuth"      Some_Spider
    BrowserMatchNoCase            "slurp"       Some_Spider
    BrowserMatchNoCase            "bot.htm"     Some_Spider
    BrowserMatchNoCase            "libwww-perl" Some_Spider
    BrowserMatchNoCase            "larbin"      Some_Spider
    BrowserMatchNoCase            "grub-client" Some_Spider
    BrowserMatchNoCase            "bot "        Some_Spider
    BrowserMatchNoCase            "scooter"     Some_Spider
    BrowserMatchNoCase            "wget"        Some_Spider
    QS_EventRequestLimit          Some_Spider 10
    QS_EventPerSecLimit           Some_Spider 10
</IfModule>

The lines in bold are the ones that cause issue when you use either an iPhone or PSP to browse the page or Firefox with a user-agent switcher masquerading as an iPhone or PSP. Alternate user agents cause the same issue. If you don't use one of those user-agents for browsing everything seems fine, as it does if you remove those specific lines.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Pascal Buchbinder - 2010-11-25

Hm, this seems to be really difficult. I've installed CentOS 5 and compiled an Apache server on it as you have described.

I've installed to following libraries too:

[root@localhost apache2.2.17]# rpm -qa | grep -e pcre -e apr -e openssl apr-devel-1.2.7-11.el5_5.3 pcre-devel-6.6-2.el5_1.7 pcre-6.6-2.el5_1.7 openssl-0.9.8e-12.el5_4.6 apr-1.2.7-11.el5_3.1 apr-util-1.2.7-11.el5 apr-1.2.7-11.el5_5.3 apr-util-1.2.7-11.el5_5.1 apr-util-devel-1.2.7-11.el5_5.1 openssl-devel-0.9.8e-12.el5_4.6

I've not installed mod_security nor using mod_jk but have configured the directives you have listed but without success to reproduce the error.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

But I was able to create a core dump.

1) add the following lime to /etc/security/limits.conf

*               soft    core            0

2) add the following line to /etc/profile

ulimit -S -c unlimited > /dev/null 2>1

3) logout and login as user root again

4) add the CoreDumpDirectory directive to your configuration

CoreDumpDirectory /usr/local/apache2.2.17/logs

5) ensure the core dump directory is writable by the apache child process and restart the server

[root@localhost ~]# cd /usr/local/apache2.2.17/
[root@localhost apache2.2.17]# chmod 777 logs/
[root@localhost apache2.2.17]# ulimit -c unlimited
[root@localhost apache2.2.17]# ./bin/httpd -k restart

I was then able to get a core file…

[root@localhost apache2.2.17]# ps -ef | grep `cat logs/httpd.pid`
root     10375     1  0 13:14 ?        00:00:00 ./bin/httpd -k start
daemon   27696 10375  0 13:19 ?        00:00:00 ./bin/httpd -k start
daemon   27697 10375  0 13:19 ?        00:00:00 ./bin/httpd -k start
daemon   27699 10375  0 13:19 ?        00:00:00 ./bin/httpd -k start
daemon   27700 10375  0 13:19 ?        00:00:00 ./bin/httpd -k start
root     28494  9134  0 13:19 pts/2    00:00:00 grep 10375
[root@localhost apache2.2.17]# kill -11 27699
[root@localhost apache2.2.17]# ls -l logs/core.27699 
-rw------- 1 daemon daemon 335867904 Nov 25 13:19 logs/core.27699
[Thu Nov 25 13:19:46 2010] [notice] child pid 27699 exit signal Segmentation fault (11), possible coredump in /usr/local/apache2.2.17/logs

Pascal Buchbinder - 2010-11-25

and: it would be very great to get a stack trace of a core file from a debug binary

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

exit signal Floating point exception

Quality of service module for Apache httpd

Forums

Help

exit signal Floating point exception

exit signal Floating point exception

Quality of service module for Apache httpd

Forums

Help

exit signal Floating point exception document.SUBSCRIPTION_OPTIONS = { "thing": "topic", "subscribed": false, "url": "subscribe", "icon": { "css": "fa fa-envelope-o" } };

exit signal Floating point exception