Hello,

Has anyone given a good answer on how to avoid this?  The creation of entries with /x00/x00/x00 keys doesn't look fixed in 2.8.0.

On 4/16/14, 3:41 AM, Winfried Neessen wrote:
Hi,

I've been struggling with a problem with mod_security for a while now, and
I have no idea what the
issue is and how to solve it.

Here a brief description of the problem…

I've built my own ruleset for mod_security, which is working fine. The
basic ruleset doesn't use any
persistent collections. As we were in need of blocking IPs which are
hitting us too hard, I introduced
an IP collection, so I can count the hits (and deprecate them as well) and
block if a specific count is
reached. This is where the problem starts.

Once the IP collection is enabled in the ruleset (in addtion to the
blocking rules), the server still runs
fine… at least for a couple of hours. But after approx. 5-12 hours the
logs begin to throw messages
like this:

[Tue Apr 15 17:28:43 2014] [error] [client xxx.xxx.xxx.xxx] ModSecurity:
collections_remove_stale: Failed deleting collection (name "ip", key
"xxx.xxx.xxx.xxx"): Internal error [hostname "some.domain.com"] [uri
"/some/path"] [unique_id "U01sSwoABzUAAH@Bf2wAAAD-"]
[Tue Apr 15 17:28:43 2014] [error] [client yyy.yyy.yyy.yyy] ModSecurity:
collections_remove_stale: Failed deleting collection (name "ip", key
"yyy.yyy.yyy.yyy"): Internal error [hostname "other.domain.com"] [uri
"/some/other/path"] [unique_id "U01sSwoABzUAAJb6E1kAAAAU"]
[Tue Apr 15 17:28:43 2014] [error] [client zzz.zzz.zzz.zzz] ModSecurity:
collections_remove_stale: Failed deleting collection (name "ip", key
"yyy.yyy.yyy.yyy"): Internal error [hostname "even.other.domain.com"] [uri
"/some/more/paths"] [unique_id "U01sSwoABzUAAKYLmMAAAAGQ"]
[Tue Apr 15 17:28:44 2014] [error] [client aaa.aaa.aaa.aaa] ModSecurity:
collections_remove_stale: Failed deleting collection (name "ip", key
"zzz.zzz.zzz.zzz"): Internal error [hostname "another.domain.com"] [uri
"/another/path"] [unique_id "U01sSwoABzUAAH-BxoEAAABf"]
[Tue Apr 15 17:28:44 2014] [error] [client bbb.bbb.bbb.bbb] ModSecurity:
collections_remove_stale: Failed deleting collection (name "ip", key
"zzz.zzz.zzz.zzz"): Internal error [hostname "yep.another.domain.com"]
[uri "/yep/another/path"] [unique_id "U01sSwoABzUAAH-n8Q8AAAF3"]
[Tue Apr 15 17:28:44 2014] [error] [client ccc.ccc.ccc.ccc] ModSecurity:
collections_remove_stale: Failed deleting collection (name "ip", key
"zzz.zzz.zzz.zzz"): Internal error [hostname "and.another.one.com"] [uri
"/a/specific/file.pdf"] [unique_id "U01sSwoABzUAAIJMZ5kAAADw"]
sAAACZ"]
[…]
(Please note, that the client IPs don't match the key values for the IP
collection)

At the same time, a couple of Apache processes start to use a lot of
system time. The load of the processes
go up to 100% each- the CPU usage shows almost no interrups, no user or
nice usage, but lots of system
usage. The Apache starts to become unresponsible and in the server-status
page, you can see lots of "L"
processes. The state of the processes that eat all the CPU, show some
locking state. I was once able to run
a "truss" on one of the procceses before the server died… it seemed to
read the IP collection file. Lots of
Null-bytes and every once and a while, one of the collected IPs was read.

First I used a combination of REMOTE_ADDR and MD5(User-Agent) as
identifier for the IP collection.
I noticed that the IP collection file grew very fast, given that we have a
lot of traffic. But as this is a sparse
file, "ls" and "du" showed different sizes- so the actual file didn't
really grow the shown size.

Still I thought this might be the issue. So I moved the file to be stored
onto a Ramdisk. This didn' fix the issue.
Again the server ran fine for a couple of hour and then started to act as
described above.

As next step I changed the IP collection to only collect the REMOTE_ADDR,
no User-Agent hash. This kept the
IP collection file pretty decent in size. But again after a couple of
hours, the same issue occurred.

I've tried several versions of mod_security. 2.7.2 thru 2.7.5 and
currently 2.7.7- all with the same result.
I have no idea what to do next. So any hint is highly appreciated.


Thanks
Winfried



------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/NeoTech


_______________________________________________
mod-security-users mailing list
mod-security-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mod-security-users
Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
http://www.modsecurity.org/projects/commercial/rules/
http://www.modsecurity.org/projects/commercial/support/