|
From: Curtis W. <cw...@si...> - 2013-02-12 22:05:32
|
Hi All, We seem to have found a potential issue with mod security - we are using cPanel along with Apache 2.2.23/mod_security 2.7.1. We noticed a strange issue with Apache last year where it would be getting caught in an internal loop with the apr_pool_cleanup routines - essentially trying to clear the same pool over and over. Initially it was thought to only be with this customers particular website/setup - although recently we saw the same issues on our production servers and have verified it is same issue. We have disabled modsec2 fleet wide (2500+ servers) and the problem has ceased to exist at this time. Unfortunately we have no idea what triggers this, if it's a particular URL being accessed or what. Apache's 3rd level children consuming much of the CPU.. nobody 51805 1.1 2.2 837744 273500 ? Sl 15:58 0:24 \_ /usr/local/apache/bin/httpd -k start -DSSL nobody 2884 46.1 2.1 837744 263912 ? R 16:29 1:24 | \_ /usr/local/apache/bin/httpd -k start -DSSL nobody 3955 47.3 2.1 837744 264148 ? R 16:30 1:02 | \_ /usr/local/apache/bin/httpd -k start -DSSL nobody 3981 45.3 2.1 837744 264164 ? R 16:30 0:58 | \_ /usr/local/apache/bin/httpd -k start -DSSL nobody 3986 47.1 2.1 837744 264164 ? R 16:30 1:00 | \_ /usr/local/apache/bin/httpd -k start -DSSL nobody 4024 47.2 2.1 837744 264168 ? R 16:30 0:58 | \_ /usr/local/apache/bin/httpd -k start -DSSL nobody 4031 51.2 2.1 837744 264176 ? R 16:30 1:02 | \_ /usr/local/apache/bin/httpd -k start -DSSL nobody 4052 46.6 2.1 837744 264176 ? R 16:30 0:55 | \_ /usr/local/apache/bin/httpd -k start -DSSL ... ... 2nd indication is an ltrace on the effected process, the apr_pool_cleanup_null is caught in a loop.. apr_pool_cleanup_null(0x2914ea0, 2, 0x42a9f0, 0, 0x7a62efdaa940) = 0 apr_pool_cleanup_null(0x2914f10, 2, 0x42a9f0, 0, 0x7a62efdaa940) = 0 apr_pool_cleanup_null(0x2914ed8, 2, 0x42a9f0, 0, 0x7a62efdaa940) = 0 apr_pool_cleanup_null(0x2914ea0, 2, 0x42a9f0, 0, 0x7a62efdaa940) = 0 apr_pool_cleanup_null(0x2914f10, 2, 0x42a9f0, 0, 0x7a62efdaa940) = 0 apr_pool_cleanup_null(0x2914ed8, 2, 0x42a9f0, 0, 0x7a62efdaa940) = 0 apr_pool_cleanup_null(0x2914ea0, 2, 0x42a9f0, 0, 0x7a62efdaa940) = 0 ... ... If you try and strace the process, it just sits there as there are no system calls being made - its just caught in a loop of sorts. On the main production servers, the load can escalate to 30-40 normally from this - and refusing to take any new connections, while on a smaller VPS the load can sky rocket to 300 in a matter of seconds and taking it down. Has anyone else seen anything like this, and have any pointers as to how to hunt down what is really causing it or to even reproduce it? Unfortunately since we have disabled modsec - security issues are on the rise again, and obviously we would like to have this re-implemented. -- Curtis Wood |
|
From: Breno S. <bre...@gm...> - 2013-02-12 22:36:35
|
Hello Curtis, What is your apr library version ? I suppose it could be apr bug. I would like to check in apr project and see if i can find something related to apr_pool_cleanup_null. Thanks Breno On Tue, Feb 12, 2013 at 7:26 PM, Curtis Wood <cw...@si...> wrote: > Hi All, > > We seem to have found a potential issue with mod security - we are using > cPanel along with Apache 2.2.23/mod_security 2.7.1. We noticed a strange > issue with Apache last year where it would be getting caught in an > internal loop with the apr_pool_cleanup routines - essentially trying to > clear the same pool over and over. Initially it was thought to only be > with this customers particular website/setup - although recently we saw > the same issues on our production servers and have verified it is same > issue. > > We have disabled modsec2 fleet wide (2500+ servers) and the problem has > ceased to exist at this time. Unfortunately we have no idea what > triggers this, if it's a particular URL being accessed or what. > > Apache's 3rd level children consuming much of the CPU.. > > nobody 51805 1.1 2.2 837744 273500 ? Sl 15:58 0:24 \_ > /usr/local/apache/bin/httpd -k start -DSSL > nobody 2884 46.1 2.1 837744 263912 ? R 16:29 1:24 | \_ > /usr/local/apache/bin/httpd -k start -DSSL > nobody 3955 47.3 2.1 837744 264148 ? R 16:30 1:02 | \_ > /usr/local/apache/bin/httpd -k start -DSSL > nobody 3981 45.3 2.1 837744 264164 ? R 16:30 0:58 | \_ > /usr/local/apache/bin/httpd -k start -DSSL > nobody 3986 47.1 2.1 837744 264164 ? R 16:30 1:00 | \_ > /usr/local/apache/bin/httpd -k start -DSSL > nobody 4024 47.2 2.1 837744 264168 ? R 16:30 0:58 | \_ > /usr/local/apache/bin/httpd -k start -DSSL > nobody 4031 51.2 2.1 837744 264176 ? R 16:30 1:02 | \_ > /usr/local/apache/bin/httpd -k start -DSSL > nobody 4052 46.6 2.1 837744 264176 ? R 16:30 0:55 | \_ > /usr/local/apache/bin/httpd -k start -DSSL > ... > ... > > 2nd indication is an ltrace on the effected process, the > apr_pool_cleanup_null is caught in a loop.. > > apr_pool_cleanup_null(0x2914ea0, 2, 0x42a9f0, 0, 0x7a62efdaa940) > = 0 > apr_pool_cleanup_null(0x2914f10, 2, 0x42a9f0, 0, 0x7a62efdaa940) > = 0 > apr_pool_cleanup_null(0x2914ed8, 2, 0x42a9f0, 0, 0x7a62efdaa940) > = 0 > apr_pool_cleanup_null(0x2914ea0, 2, 0x42a9f0, 0, 0x7a62efdaa940) > = 0 > apr_pool_cleanup_null(0x2914f10, 2, 0x42a9f0, 0, 0x7a62efdaa940) > = 0 > apr_pool_cleanup_null(0x2914ed8, 2, 0x42a9f0, 0, 0x7a62efdaa940) > = 0 > apr_pool_cleanup_null(0x2914ea0, 2, 0x42a9f0, 0, 0x7a62efdaa940) > = 0 > ... > ... > > If you try and strace the process, it just sits there as there are no > system calls being made - its just caught in a loop of sorts. > > On the main production servers, the load can escalate to 30-40 normally > from this - and refusing to take any new connections, while on a smaller > VPS the load can sky rocket to 300 in a matter of seconds and taking it > down. > > Has anyone else seen anything like this, and have any pointers as to how > to hunt down what is really causing it or to even reproduce it? > Unfortunately since we have disabled modsec - security issues are on the > rise again, and obviously we would like to have this re-implemented. > > > -- > Curtis Wood > > > > ------------------------------------------------------------------------------ > Free Next-Gen Firewall Hardware Offer > Buy your Sophos next-gen firewall before the end March 2013 > and get the hardware for free! Learn more. > http://p.sf.net/sfu/sophos-d2d-feb > _______________________________________________ > mod-security-developers mailing list > mod...@li... > https://lists.sourceforge.net/lists/listinfo/mod-security-developers > ModSecurity Services from Trustwave's SpiderLabs: > https://www.trustwave.com/spiderLabs.php > |
|
From: Curtis W. <cw...@si...> - 2013-02-12 22:52:06
|
Hi Breno, We have seen this with both apr 1.4.5 and 1.4.6.. libapr-1.so.0.4.6 libapr-1.so.0.4.5 I've submitted something to apache as well - asking if they have any insight on this also. Do you have any suggestions on how to possibly try and reproduce this? I'm assuming (if its related to modsec) it's some particular URL that it may be trying to parse - in which apache isnt going to log anything (to get the URL) until it has finished processing everything? On 02/12/2013 04:36 PM, Breno Silva wrote: > Hello Curtis, > > What is your apr library version ? I suppose it could be apr bug. I > would like to check in apr project and see if i can find something > related to apr_pool_cleanup_null. > > Thanks > > Breno > > On Tue, Feb 12, 2013 at 7:26 PM, Curtis Wood <cw...@si... > <mailto:cw...@si...>> wrote: > > Hi All, > > We seem to have found a potential issue with mod security - we are > using > cPanel along with Apache 2.2.23/mod_security 2.7.1. We noticed a > strange > issue with Apache last year where it would be getting caught in an > internal loop with the apr_pool_cleanup routines - essentially > trying to > clear the same pool over and over. Initially it was thought to only be > with this customers particular website/setup - although recently > we saw > the same issues on our production servers and have verified it is same > issue. > > We have disabled modsec2 fleet wide (2500+ servers) and the > problem has > ceased to exist at this time. Unfortunately we have no idea what > triggers this, if it's a particular URL being accessed or what. > > Apache's 3rd level children consuming much of the CPU.. > > nobody 51805 1.1 2.2 837744 273500 ? Sl 15:58 0:24 > \_ /usr/local/apache/bin/httpd -k start -DSSL > nobody 2884 46.1 2.1 837744 263912 ? R 16:29 1:24 > | \_ /usr/local/apache/bin/httpd -k start -DSSL > nobody 3955 47.3 2.1 837744 264148 ? R 16:30 1:02 > | \_ /usr/local/apache/bin/httpd -k start -DSSL > nobody 3981 45.3 2.1 837744 264164 ? R 16:30 0:58 > | \_ /usr/local/apache/bin/httpd -k start -DSSL > nobody 3986 47.1 2.1 837744 264164 ? R 16:30 1:00 > | \_ /usr/local/apache/bin/httpd -k start -DSSL > nobody 4024 47.2 2.1 837744 264168 ? R 16:30 0:58 > | \_ /usr/local/apache/bin/httpd -k start -DSSL > nobody 4031 51.2 2.1 837744 264176 ? R 16:30 1:02 > | \_ /usr/local/apache/bin/httpd -k start -DSSL > nobody 4052 46.6 2.1 837744 264176 ? R 16:30 0:55 > | \_ /usr/local/apache/bin/httpd -k start -DSSL > ... > ... > > 2nd indication is an ltrace on the effected process, the > apr_pool_cleanup_null is caught in a loop.. > > apr_pool_cleanup_null(0x2914ea0, 2, 0x42a9f0, 0, 0x7a62efdaa940) > = 0 > apr_pool_cleanup_null(0x2914f10, 2, 0x42a9f0, 0, 0x7a62efdaa940) > = 0 > apr_pool_cleanup_null(0x2914ed8, 2, 0x42a9f0, 0, 0x7a62efdaa940) > = 0 > apr_pool_cleanup_null(0x2914ea0, 2, 0x42a9f0, 0, 0x7a62efdaa940) > = 0 > apr_pool_cleanup_null(0x2914f10, 2, 0x42a9f0, 0, 0x7a62efdaa940) > = 0 > apr_pool_cleanup_null(0x2914ed8, 2, 0x42a9f0, 0, 0x7a62efdaa940) > = 0 > apr_pool_cleanup_null(0x2914ea0, 2, 0x42a9f0, 0, 0x7a62efdaa940) > = 0 > ... > ... > > If you try and strace the process, it just sits there as there are > no system calls being made - its just caught in a loop of sorts. > > On the main production servers, the load can escalate to 30-40 > normally from this - and refusing to take any new connections, > while on a smaller VPS the load can sky rocket to 300 in a matter > of seconds and taking it down. > > Has anyone else seen anything like this, and have any pointers as > to how to hunt down what is really causing it or to even reproduce > it? Unfortunately since we have disabled modsec - security issues > are on the rise again, and obviously we would like to have this > re-implemented. > > > -- > Curtis Wood > > > ------------------------------------------------------------------------------ > Free Next-Gen Firewall Hardware Offer > Buy your Sophos next-gen firewall before the end March 2013 > and get the hardware for free! Learn more. > http://p.sf.net/sfu/sophos-d2d-feb > _______________________________________________ > mod-security-developers mailing list > mod...@li... > <mailto:mod...@li...> > https://lists.sourceforge.net/lists/listinfo/mod-security-developers > ModSecurity Services from Trustwave's SpiderLabs: > https://www.trustwave.com/spiderLabs.php > > > > > ------------------------------------------------------------------------------ > Free Next-Gen Firewall Hardware Offer > Buy your Sophos next-gen firewall before the end March 2013 > and get the hardware for free! Learn more. > http://p.sf.net/sfu/sophos-d2d-feb > > > _______________________________________________ > mod-security-developers mailing list > mod...@li... > https://lists.sourceforge.net/lists/listinfo/mod-security-developers > ModSecurity Services from Trustwave's SpiderLabs: > https://www.trustwave.com/spiderLabs.php |
|
From: Rainer J. <rai...@ki...> - 2013-02-13 02:22:59
|
On 12.02.2013 22:26, Curtis Wood wrote: > Hi All, > > We seem to have found a potential issue with mod security - we are using > cPanel along with Apache 2.2.23/mod_security 2.7.1. We noticed a strange > issue with Apache last year where it would be getting caught in an > internal loop with the apr_pool_cleanup routines - essentially trying to > clear the same pool over and over. Initially it was thought to only be > with this customers particular website/setup - although recently we saw > the same issues on our production servers and have verified it is same > issue. > > We have disabled modsec2 fleet wide (2500+ servers) and the problem has > ceased to exist at this time. Unfortunately we have no idea what > triggers this, if it's a particular URL being accessed or what. Pool cleanup loops typically indicate a corruption in the pool data structures due to unsynchronized pool use by multiple threads. APR pools are not thread-safe. Regards, Rainer |
|
From: Breno S. <bre...@gm...> - 2013-02-13 11:55:45
|
Curtis, Are you using some custom ruleset ? or using only CRS ? Maybe i can copy your configuration/ruleset and try to reproduce, for better investigation Thanks Breno On Wed, Feb 13, 2013 at 12:05 AM, Rainer Jung <rai...@ki...>wrote: > On 12.02.2013 22:26, Curtis Wood wrote: > > Hi All, > > > > We seem to have found a potential issue with mod security - we are using > > cPanel along with Apache 2.2.23/mod_security 2.7.1. We noticed a strange > > issue with Apache last year where it would be getting caught in an > > internal loop with the apr_pool_cleanup routines - essentially trying to > > clear the same pool over and over. Initially it was thought to only be > > with this customers particular website/setup - although recently we saw > > the same issues on our production servers and have verified it is same > > issue. > > > > We have disabled modsec2 fleet wide (2500+ servers) and the problem has > > ceased to exist at this time. Unfortunately we have no idea what > > triggers this, if it's a particular URL being accessed or what. > > Pool cleanup loops typically indicate a corruption in the pool data > structures due to unsynchronized pool use by multiple threads. APR pools > are not thread-safe. > > Regards, > > Rainer > > > > ------------------------------------------------------------------------------ > Free Next-Gen Firewall Hardware Offer > Buy your Sophos next-gen firewall before the end March 2013 > and get the hardware for free! Learn more. > http://p.sf.net/sfu/sophos-d2d-feb > _______________________________________________ > mod-security-developers mailing list > mod...@li... > https://lists.sourceforge.net/lists/listinfo/mod-security-developers > ModSecurity Services from Trustwave's SpiderLabs: > https://www.trustwave.com/spiderLabs.php > |
|
From: Curtis W. <cw...@si...> - 2013-02-13 14:53:16
|
Hi Breno,
We are using the following rules - with the exception of excluding some
particular rule ID's or white listing IP's here and there, there isn't
anything really custom.
/var/asl/rules/10_asl_rules.conf
/var/asl/rules/20_asl_useragents.conf
/var/asl/rules/30_asl_antispam.conf
/var/asl/rules/50_asl_rootkits.conf
/var/asl/rules/60_asl_recons.conf
/var/asl/rules/99_asl_jitp.conf
We are using the mpm_event - as opposed to the mpm_prefork which I
understand is what is not threaded, could this be the underlying issue
as Rainer mentioned?
Following up with Rainers point, the APR pools are becoming corrupt - I
dont recall the particular function (there are a few of them) with the
following loop, all though the node in the linked list is basically
pointing to it self so, never gets freed up and consequently the loop
goes on for ever - the 'fix' we implemented originally in libapr was to
simply add a conditional break statement based on whether "c == c->next"..
(in run_child_cleanups())
while (c) {
*cref = c->next;
(*c->child_cleanup_fn)((void *)c->data);
c = *cref;
}
On 02/13/2013 05:55 AM, Breno Silva wrote:
> Curtis,
>
> Are you using some custom ruleset ? or using only CRS ?
> Maybe i can copy your configuration/ruleset and try to reproduce, for
> better investigation
>
> Thanks
>
> Breno
>
> On Wed, Feb 13, 2013 at 12:05 AM, Rainer Jung <rai...@ki...
> <mailto:rai...@ki...>> wrote:
>
> On 12.02.2013 22:26, Curtis Wood wrote:
> > Hi All,
> >
> > We seem to have found a potential issue with mod security - we
> are using
> > cPanel along with Apache 2.2.23/mod_security 2.7.1. We noticed a
> strange
> > issue with Apache last year where it would be getting caught in an
> > internal loop with the apr_pool_cleanup routines - essentially
> trying to
> > clear the same pool over and over. Initially it was thought to
> only be
> > with this customers particular website/setup - although recently
> we saw
> > the same issues on our production servers and have verified it
> is same
> > issue.
> >
> > We have disabled modsec2 fleet wide (2500+ servers) and the
> problem has
> > ceased to exist at this time. Unfortunately we have no idea what
> > triggers this, if it's a particular URL being accessed or what.
>
> Pool cleanup loops typically indicate a corruption in the pool data
> structures due to unsynchronized pool use by multiple threads. APR
> pools
> are not thread-safe.
>
> Regards,
>
> Rainer
>
>
> ------------------------------------------------------------------------------
> Free Next-Gen Firewall Hardware Offer
> Buy your Sophos next-gen firewall before the end March 2013
> and get the hardware for free! Learn more.
> http://p.sf.net/sfu/sophos-d2d-feb
> _______________________________________________
> mod-security-developers mailing list
> mod...@li...
> <mailto:mod...@li...>
> https://lists.sourceforge.net/lists/listinfo/mod-security-developers
> ModSecurity Services from Trustwave's SpiderLabs:
> https://www.trustwave.com/spiderLabs.php
>
>
>
>
> ------------------------------------------------------------------------------
> Free Next-Gen Firewall Hardware Offer
> Buy your Sophos next-gen firewall before the end March 2013
> and get the hardware for free! Learn more.
> http://p.sf.net/sfu/sophos-d2d-feb
>
>
> _______________________________________________
> mod-security-developers mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-developers
> ModSecurity Services from Trustwave's SpiderLabs:
> https://www.trustwave.com/spiderLabs.php
|
|
From: Breno S. <bre...@gm...> - 2013-02-13 15:51:27
|
Curtis,
Yes. APR pools are not thread safe. I think mpm-event is an experimental
code in Apache 2.2 right ? I think it is stable in Apache 2.4.
Any chance you try to setup one box with Apache 2.4 and see what happens ?
Also what platform are you using ? x86, x86_64 ? Are you using Linux ? Glib
has support to Epoll ?
Thanks
Breno
On Wed, Feb 13, 2013 at 12:53 PM, Curtis Wood <cw...@si...> wrote:
> Hi Breno,
>
> We are using the following rules - with the exception of excluding some
> particular rule ID's or white listing IP's here and there, there isn't
> anything really custom.
>
> /var/asl/rules/10_asl_rules.conf
> /var/asl/rules/20_asl_useragents.conf
> /var/asl/rules/30_asl_antispam.conf
> /var/asl/rules/50_asl_rootkits.conf
> /var/asl/rules/60_asl_recons.conf
> /var/asl/rules/99_asl_jitp.conf
>
> We are using the mpm_event - as opposed to the mpm_prefork which I
> understand is what is not threaded, could this be the underlying issue as
> Rainer mentioned?
>
> Following up with Rainers point, the APR pools are becoming corrupt - I
> dont recall the particular function (there are a few of them) with the
> following loop, all though the node in the linked list is basically
> pointing to it self so, never gets freed up and consequently the loop goes
> on for ever - the 'fix' we implemented originally in libapr was to simply
> add a conditional break statement based on whether "c == c->next"..
>
> (in run_child_cleanups())
>
> while (c) {
> *cref = c->next;
> (*c->child_cleanup_fn)((void *)c->data);
> c = *cref;
>
> }
>
>
> On 02/13/2013 05:55 AM, Breno Silva wrote:
>
> Curtis,
>
> Are you using some custom ruleset ? or using only CRS ?
> Maybe i can copy your configuration/ruleset and try to reproduce, for
> better investigation
>
> Thanks
>
> Breno
>
> On Wed, Feb 13, 2013 at 12:05 AM, Rainer Jung <rai...@ki...>wrote:
>
>> On 12.02.2013 22:26, Curtis Wood wrote:
>> > Hi All,
>> >
>> > We seem to have found a potential issue with mod security - we are using
>> > cPanel along with Apache 2.2.23/mod_security 2.7.1. We noticed a strange
>> > issue with Apache last year where it would be getting caught in an
>> > internal loop with the apr_pool_cleanup routines - essentially trying to
>> > clear the same pool over and over. Initially it was thought to only be
>> > with this customers particular website/setup - although recently we saw
>> > the same issues on our production servers and have verified it is same
>> > issue.
>> >
>> > We have disabled modsec2 fleet wide (2500+ servers) and the problem has
>> > ceased to exist at this time. Unfortunately we have no idea what
>> > triggers this, if it's a particular URL being accessed or what.
>>
>> Pool cleanup loops typically indicate a corruption in the pool data
>> structures due to unsynchronized pool use by multiple threads. APR pools
>> are not thread-safe.
>>
>> Regards,
>>
>> Rainer
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Free Next-Gen Firewall Hardware Offer
>> Buy your Sophos next-gen firewall before the end March 2013
>> and get the hardware for free! Learn more.
>> http://p.sf.net/sfu/sophos-d2d-feb
>> _______________________________________________
>> mod-security-developers mailing list
>> mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/mod-security-developers
>> ModSecurity Services from Trustwave's SpiderLabs:
>> https://www.trustwave.com/spiderLabs.php
>>
>
>
>
> ------------------------------------------------------------------------------
> Free Next-Gen Firewall Hardware Offer
> Buy your Sophos next-gen firewall before the end March 2013
> and get the hardware for free! Learn more.http://p.sf.net/sfu/sophos-d2d-feb
>
>
>
> _______________________________________________
> mod-security-developers mailing lis...@li...://lists.sourceforge.net/lists/listinfo/mod-security-developers
> ModSecurity Services from Trustwave's SpiderLabs:https://www.trustwave.com/spiderLabs.php
>
>
>
>
> ------------------------------------------------------------------------------
> Free Next-Gen Firewall Hardware Offer
> Buy your Sophos next-gen firewall before the end March 2013
> and get the hardware for free! Learn more.
> http://p.sf.net/sfu/sophos-d2d-feb
> _______________________________________________
> mod-security-developers mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-developers
> ModSecurity Services from Trustwave's SpiderLabs:
> https://www.trustwave.com/spiderLabs.php
>
|
|
From: Rainer J. <rai...@ki...> - 2013-02-13 17:17:18
|
On 13.02.2013 16:51, Breno Silva wrote: > Curtis, > > Yes. APR pools are not thread safe. I think mpm-event is an experimental > code in Apache 2.2 right ? I think it is stable in Apache 2.4. > Any chance you try to setup one box with Apache 2.4 and see what happens ? > > Also what platform are you using ? x86, x86_64 ? Are you using Linux ? > Glib has support to Epoll ? Adding to the question list: - do you have any other third party modules loaded, i.e. apart from the ones that come bundled with Apache or mod_security? - does it only happen during or after periods of high load/concurrency ? Regards, Rainer |
|
From: Curtis W. <cw...@si...> - 2013-02-13 18:21:40
|
Hi Guys, Yes, mpm-event is experimental - and now with everything coming to light, that may be the whole problem along with the apr not being thread safe it sounds like? After talking it over some we may just go with the prefork. On the actual system - sry about that :-P We are on Linux X86_64, CentOS 5.9 - On the extra modules, we have ssl,fcgi,passenger,bw_limited and qos On when it happens - it can be at any time, originally when this came to light on a single site VPS, you could randomly click links to trigger it after a few minutes - while, clicking the same link twice would not guarantee anything. On the main servers which get a lot more traffic, it can be anywhere from hours to days. On 02/13/2013 11:17 AM, Rainer Jung wrote: > On 13.02.2013 16:51, Breno Silva wrote: >> Curtis, >> >> Yes. APR pools are not thread safe. I think mpm-event is an experimental >> code in Apache 2.2 right ? I think it is stable in Apache 2.4. >> Any chance you try to setup one box with Apache 2.4 and see what happens ? >> >> Also what platform are you using ? x86, x86_64 ? Are you using Linux ? >> Glib has support to Epoll ? > Adding to the question list: > > - do you have any other third party modules loaded, i.e. apart from the > ones that come bundled with Apache or mod_security? > > - does it only happen during or after periods of high load/concurrency ? > > Regards, > > Rainer > > > ------------------------------------------------------------------------------ > Free Next-Gen Firewall Hardware Offer > Buy your Sophos next-gen firewall before the end March 2013 > and get the hardware for free! Learn more. > http://p.sf.net/sfu/sophos-d2d-feb > _______________________________________________ > mod-security-developers mailing list > mod...@li... > https://lists.sourceforge.net/lists/listinfo/mod-security-developers > ModSecurity Services from Trustwave's SpiderLabs: > https://www.trustwave.com/spiderLabs.php |
|
From: Rainer J. <rai...@ki...> - 2013-02-14 00:15:01
|
On 13.02.2013 19:21, Curtis Wood wrote: > Hi Guys, > > Yes, mpm-event is experimental - and now with everything coming to > light, that may be the whole problem along with the apr not being thread > safe it sounds like? The event MPM itself should not be the problem. The APR libraries were originally developed as a basis for Apache 2 and many of the Apache devs are also APR devs. But the threaded nature of the Apache processes when using the event or worker MPMs means that all modules must be programmed thread-safe too. Most of the popular ones are though. > After talking it over some we may just go with the prefork. If that is an option for you, chances are good the problem vanishes. > On the actual system - sry about that :-P We are on Linux X86_64, CentOS > 5.9 - On the extra modules, we have ssl,fcgi,passenger,bw_limited and qos Ah, lots of 3rd party. You could try to update to latest versions of those. I don't have an indication, that those have a problem though. > On when it happens - it can be at any time, originally when this came to > light on a single site VPS, you could randomly click links to trigger it > after a few minutes - while, clicking the same link twice would not > guarantee anything. On the main servers which get a lot more traffic, it > can be anywhere from hours to days. ACK. Rainer |
|
From: Breno S. <bre...@gm...> - 2013-02-14 00:50:05
|
We have a lot of users running Apache worker and prefork. However to be honest i don't remember anyone that contact me using event. So, as i never saw a report about this issue using worker and prefork... you can try both and send us a feedback. Thanks Breno On Wed, Feb 13, 2013 at 10:14 PM, Rainer Jung <rai...@ki...>wrote: > On 13.02.2013 19:21, Curtis Wood wrote: > > Hi Guys, > > > > Yes, mpm-event is experimental - and now with everything coming to > > light, that may be the whole problem along with the apr not being thread > > safe it sounds like? > > The event MPM itself should not be the problem. The APR libraries were > originally developed as a basis for Apache 2 and many of the Apache devs > are also APR devs. > > But the threaded nature of the Apache processes when using the event or > worker MPMs means that all modules must be programmed thread-safe too. > Most of the popular ones are though. > > > After talking it over some we may just go with the prefork. > > If that is an option for you, chances are good the problem vanishes. > > > On the actual system - sry about that :-P We are on Linux X86_64, CentOS > > 5.9 - On the extra modules, we have ssl,fcgi,passenger,bw_limited and qos > > Ah, lots of 3rd party. You could try to update to latest versions of > those. I don't have an indication, that those have a problem though. > > > On when it happens - it can be at any time, originally when this came to > > light on a single site VPS, you could randomly click links to trigger it > > after a few minutes - while, clicking the same link twice would not > > guarantee anything. On the main servers which get a lot more traffic, it > > can be anywhere from hours to days. > > ACK. > > Rainer > > > > ------------------------------------------------------------------------------ > Free Next-Gen Firewall Hardware Offer > Buy your Sophos next-gen firewall before the end March 2013 > and get the hardware for free! Learn more. > http://p.sf.net/sfu/sophos-d2d-feb > _______________________________________________ > mod-security-developers mailing list > mod...@li... > https://lists.sourceforge.net/lists/listinfo/mod-security-developers > ModSecurity Services from Trustwave's SpiderLabs: > https://www.trustwave.com/spiderLabs.php > |
|
From: Curtis W. <cw...@si...> - 2013-02-26 20:12:41
|
Hey guys, Sorry for the long delay in responding, we have ended up going with prefork and everything seems to be stable with it - a slightly higher load perhaps, but stable none the less :) We were never able to figure much out, even with 'guessing' and randomly disabling the extra modules :-/ On 02/13/2013 06:49 PM, Breno Silva wrote: > We have a lot of users running Apache worker and prefork. However to > be honest i don't remember anyone that contact me using event. So, as > i never saw a report about this issue using worker and prefork... you > can try both and send us a feedback. > > Thanks > > Breno > > On Wed, Feb 13, 2013 at 10:14 PM, Rainer Jung <rai...@ki... > <mailto:rai...@ki...>> wrote: > > On 13.02.2013 19:21, Curtis Wood wrote: > > Hi Guys, > > > > Yes, mpm-event is experimental - and now with everything coming to > > light, that may be the whole problem along with the apr not > being thread > > safe it sounds like? > > The event MPM itself should not be the problem. The APR libraries were > originally developed as a basis for Apache 2 and many of the > Apache devs > are also APR devs. > > But the threaded nature of the Apache processes when using the > event or > worker MPMs means that all modules must be programmed thread-safe too. > Most of the popular ones are though. > > > After talking it over some we may just go with the prefork. > > If that is an option for you, chances are good the problem vanishes. > > > On the actual system - sry about that :-P We are on Linux > X86_64, CentOS > > 5.9 - On the extra modules, we have > ssl,fcgi,passenger,bw_limited and qos > > Ah, lots of 3rd party. You could try to update to latest versions of > those. I don't have an indication, that those have a problem though. > > > On when it happens - it can be at any time, originally when this > came to > > light on a single site VPS, you could randomly click links to > trigger it > > after a few minutes - while, clicking the same link twice would not > > guarantee anything. On the main servers which get a lot more > traffic, it > > can be anywhere from hours to days. > > ACK. > > Rainer > > > ------------------------------------------------------------------------------ > Free Next-Gen Firewall Hardware Offer > Buy your Sophos next-gen firewall before the end March 2013 > and get the hardware for free! Learn more. > http://p.sf.net/sfu/sophos-d2d-feb > _______________________________________________ > mod-security-developers mailing list > mod...@li... > <mailto:mod...@li...> > https://lists.sourceforge.net/lists/listinfo/mod-security-developers > ModSecurity Services from Trustwave's SpiderLabs: > https://www.trustwave.com/spiderLabs.php > > > > > ------------------------------------------------------------------------------ > Free Next-Gen Firewall Hardware Offer > Buy your Sophos next-gen firewall before the end March 2013 > and get the hardware for free! Learn more. > http://p.sf.net/sfu/sophos-d2d-feb > > > _______________________________________________ > mod-security-developers mailing list > mod...@li... > https://lists.sourceforge.net/lists/listinfo/mod-security-developers > ModSecurity Services from Trustwave's SpiderLabs: > https://www.trustwave.com/spiderLabs.php |
