Work at SourceForge, help us to make it a better place! We have an immediate need for a Support Technician in our San Francisco or Denver office.

Close

Busy server, Child returned a Fatal error

2012-05-11
2013-05-23
  • sf_user435214
    sf_user435214
    2012-05-11

    Hi pbuchbinder and thanks for this useful module.
    I've used it for quite some time on multiple machine and never had a problem so far.

    On a busy reverse proxy with mod_qos when there is a connection spike, apache is shutting down with "Child XXXXX returned a Fatal error… Apache is exiting!". Other interesting errors: " (22)Invalid argument: ap_queue_info_set_idle failed. Attempting to shutdown process gracefully." and  lots of " (22)Invalid argument: ap_queue_pop failed" (more logs below).

    First I didn't suspected mod_qos but some os limitation (due to " (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread" so I've dubled the stack size (ulimit -s 20480), halfed ThreadsPerChild and fine tuned ThreadLimit to no avail.
    On the other hand, there are several mod_qos errors (" mod_qos(008): could not create supervisor thread (create thread), disable request rate enforcement" ) and it never happend after disabling mod_qos.

    The machine is a linux 64bits centos 6.2 server running apache 2.2.22 with mod_qos 10.2, mod_security 2.6.3 and mod_disk_cache / mod_proxy / mod_proxy_http / mod_proxy_balancer .

    More info on pastebin (expireing in 3 weeks, password is mod_qos):
    full apache error log: http://pastebin.ca/J82y5Z-j
    mod_qos conf: http://pastebin.ca/HMmuFoWV
    other relevant apache conf: http://pastebin.ca/N8puMLsd
    machine info: http://pastebin.ca/SJMBGQaM
    other info available on request

    Please advise,
    Thanks.

     
  • The "apr_thread_create: unable to create worker thread" message sounds like a memory issue (the newly forked child process - created to due to the MaxRequestsPerChild settings - is not able to allocate the memory for all 32+ threads). Your setting allows up to 256 child processes. I don't know how much memory each of theses processes need, but it might be quite much…