|
From: Curtis W. <cw...@si...> - 2013-02-12 22:05:32
|
Hi All, We seem to have found a potential issue with mod security - we are using cPanel along with Apache 2.2.23/mod_security 2.7.1. We noticed a strange issue with Apache last year where it would be getting caught in an internal loop with the apr_pool_cleanup routines - essentially trying to clear the same pool over and over. Initially it was thought to only be with this customers particular website/setup - although recently we saw the same issues on our production servers and have verified it is same issue. We have disabled modsec2 fleet wide (2500+ servers) and the problem has ceased to exist at this time. Unfortunately we have no idea what triggers this, if it's a particular URL being accessed or what. Apache's 3rd level children consuming much of the CPU.. nobody 51805 1.1 2.2 837744 273500 ? Sl 15:58 0:24 \_ /usr/local/apache/bin/httpd -k start -DSSL nobody 2884 46.1 2.1 837744 263912 ? R 16:29 1:24 | \_ /usr/local/apache/bin/httpd -k start -DSSL nobody 3955 47.3 2.1 837744 264148 ? R 16:30 1:02 | \_ /usr/local/apache/bin/httpd -k start -DSSL nobody 3981 45.3 2.1 837744 264164 ? R 16:30 0:58 | \_ /usr/local/apache/bin/httpd -k start -DSSL nobody 3986 47.1 2.1 837744 264164 ? R 16:30 1:00 | \_ /usr/local/apache/bin/httpd -k start -DSSL nobody 4024 47.2 2.1 837744 264168 ? R 16:30 0:58 | \_ /usr/local/apache/bin/httpd -k start -DSSL nobody 4031 51.2 2.1 837744 264176 ? R 16:30 1:02 | \_ /usr/local/apache/bin/httpd -k start -DSSL nobody 4052 46.6 2.1 837744 264176 ? R 16:30 0:55 | \_ /usr/local/apache/bin/httpd -k start -DSSL ... ... 2nd indication is an ltrace on the effected process, the apr_pool_cleanup_null is caught in a loop.. apr_pool_cleanup_null(0x2914ea0, 2, 0x42a9f0, 0, 0x7a62efdaa940) = 0 apr_pool_cleanup_null(0x2914f10, 2, 0x42a9f0, 0, 0x7a62efdaa940) = 0 apr_pool_cleanup_null(0x2914ed8, 2, 0x42a9f0, 0, 0x7a62efdaa940) = 0 apr_pool_cleanup_null(0x2914ea0, 2, 0x42a9f0, 0, 0x7a62efdaa940) = 0 apr_pool_cleanup_null(0x2914f10, 2, 0x42a9f0, 0, 0x7a62efdaa940) = 0 apr_pool_cleanup_null(0x2914ed8, 2, 0x42a9f0, 0, 0x7a62efdaa940) = 0 apr_pool_cleanup_null(0x2914ea0, 2, 0x42a9f0, 0, 0x7a62efdaa940) = 0 ... ... If you try and strace the process, it just sits there as there are no system calls being made - its just caught in a loop of sorts. On the main production servers, the load can escalate to 30-40 normally from this - and refusing to take any new connections, while on a smaller VPS the load can sky rocket to 300 in a matter of seconds and taking it down. Has anyone else seen anything like this, and have any pointers as to how to hunt down what is really causing it or to even reproduce it? Unfortunately since we have disabled modsec - security issues are on the rise again, and obviously we would like to have this re-implemented. -- Curtis Wood |
