Hello Curtis,

What is your apr library version ? I suppose it could be apr bug. I would like to check in apr project and see if i can find something related to apr_pool_cleanup_null.

Thanks

Breno

On Tue, Feb 12, 2013 at 7:26 PM, Curtis Wood <cwood@site5.com> wrote:
Hi All,

We seem to have found a potential issue with mod security - we are using
cPanel along with Apache 2.2.23/mod_security 2.7.1. We noticed a strange
issue with Apache last year where it would be getting caught in an
internal loop with the apr_pool_cleanup routines - essentially trying to
clear the same pool over and over. Initially it was thought to only be
with this customers particular website/setup - although recently we saw
the same issues on our production servers and have verified it is same
issue.

We have disabled modsec2 fleet wide (2500+ servers) and the problem has
ceased to exist at this time. Unfortunately we have no idea what
triggers this, if it's a particular URL being accessed or what.

Apache's 3rd level children consuming much of the CPU..

nobody   51805  1.1  2.2 837744 273500 ?       Sl   15:58   0:24  \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody    2884 46.1  2.1 837744 263912 ?       R    16:29   1:24  |   \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody    3955 47.3  2.1 837744 264148 ?       R    16:30   1:02  |   \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody    3981 45.3  2.1 837744 264164 ?       R    16:30   0:58  |   \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody    3986 47.1  2.1 837744 264164 ?       R    16:30   1:00  |   \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody    4024 47.2  2.1 837744 264168 ?       R    16:30   0:58  |   \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody    4031 51.2  2.1 837744 264176 ?       R    16:30   1:02  |   \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody    4052 46.6  2.1 837744 264176 ?       R    16:30   0:55  |   \_ /usr/local/apache/bin/httpd -k start -DSSL
...
...

2nd indication is an ltrace on the effected process, the apr_pool_cleanup_null is caught in a loop..

apr_pool_cleanup_null(0x2914ea0, 2, 0x42a9f0, 0, 0x7a62efdaa940)                                                                  = 0
apr_pool_cleanup_null(0x2914f10, 2, 0x42a9f0, 0, 0x7a62efdaa940)                                                                  = 0
apr_pool_cleanup_null(0x2914ed8, 2, 0x42a9f0, 0, 0x7a62efdaa940)                                                                  = 0
apr_pool_cleanup_null(0x2914ea0, 2, 0x42a9f0, 0, 0x7a62efdaa940)                                                                  = 0
apr_pool_cleanup_null(0x2914f10, 2, 0x42a9f0, 0, 0x7a62efdaa940)                                                                  = 0
apr_pool_cleanup_null(0x2914ed8, 2, 0x42a9f0, 0, 0x7a62efdaa940)                                                                  = 0
apr_pool_cleanup_null(0x2914ea0, 2, 0x42a9f0, 0, 0x7a62efdaa940)                                                                  = 0
...
...

If you try and strace the process, it just sits there as there are no system calls being made - its just caught in a loop of sorts.

On the main production servers, the load can escalate to 30-40 normally from this - and refusing to take any new connections, while on a smaller VPS the load can sky rocket to 300 in a matter of seconds and taking it down.

Has anyone else seen anything like this, and have any pointers as to how to hunt down what is really causing it or to even reproduce it? Unfortunately since we have disabled modsec - security issues are on the rise again, and obviously we would like to have this re-implemented.


--
Curtis Wood


------------------------------------------------------------------------------
Free Next-Gen Firewall Hardware Offer
Buy your Sophos next-gen firewall before the end March 2013
and get the hardware for free! Learn more.
http://p.sf.net/sfu/sophos-d2d-feb
_______________________________________________
mod-security-developers mailing list
mod-security-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mod-security-developers
ModSecurity Services from Trustwave's SpiderLabs:
https://www.trustwave.com/spiderLabs.php