Re: [mod-security-users] Collection db file growing at alarming rate
Brought to you by:
victorhora,
zimmerletw
Menu
▾
▴
Re: [mod-security-users] Collection db file growing at alarming rate
|
From: Breno S. <bre...@gm...> - 2012-06-20 19:45:07
|
Mark,
Just two recommendations. You are running modsecurity with different
library versions from apache, it could cause crash.
Also i saw you are not using SESSION and USER collections. However would be
a good idea upgrade to a newer modsecurity version because a problem
related to that collections were fixed more recently.
You can also check SecCollectionTimeout to define a lower timeout value for
your collections.
Thanks
Breno
On Wed, Jun 20, 2012 at 2:30 PM, Ryan Barnett <RBa...@tr...>wrote:
>
> On 6/20/12 3:09 PM, "Mark Moseley" <mos...@gm...> wrote:
>
> >This is a "Just checking if this is normal" post. This may also be a
> >"You're doing it wrong" post :)
> >
> >I've got a collection on a fairly busy server, running apache 2.2.22
> >with modsec 2.6.3, Debian Squeeze 32-bit. I'm trying to track IP+URL
> >for ratelimiting. I'm constructing the IP collection like this:
> >
> >SecRule REQUEST_URI "^(.*)$"
> >"phase:1,pass,nolog,t:none,t:md5,t:hexEncode,capture,id:99000"
> >SecRule TX:1 "^(.{8})"
> >"phase:1,pass,nolog,t:none,capture,setvar:tx.requri_hash=%{TX.1},id:99001"
> >
> >SecRule REMOTE_ADDR "@rx ^(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})$"
> >"capture,setvar:tx.fwdhost=%{TX.1},phase:1,id:99910,nolog,pass"
> >SecRule REMOTE_ADDR "!@rx ^10\.\d{1,3}\.\d{1,3}\.\d{1,3}$"
> >"chain,initcol:IP=%{TX.fwdhost}:%{TX.requri_hash},phase:1,id:99911,nolog,p
> >ass"
> >SecRule REMOTE_ADDR "@rx ^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$"
> >
> >SecRule REMOTE_ADDR "!@rx ^10\.\d{1,3}\.\d{1,3}\.\d{1,3}$"
> >"chain,phase:1,setvar:IP.req_count_60s=+1,expirevar:IP.req_count_60s=60,se
> >tvar:IP.req_count_1800s=+1,expirevar:IP.req_count_1800s=1800,nolog,pass,id
> >:99930"
> >SecRule REMOTE_ADDR "@rx ^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$"
> >
> >(Leaving out the part where it checks against the collection and does a
> >'deny')
> >
> >So the IP collection key is a combination of the IP (with the 10./8
> >filtered out) +the first 8 chars in the MD5 hash of the URL (Saving
> >the whole URL seemed like it could be way too big). I'm expiring on
> >both a 60 second window and a 3600 second window. BTW, there's a good
> >but unrelated reason why I'm doing @rx against REMOTE_ADDR; I realize
> >it'd normally be unnecessary.
> >
> >First off, the collection seems to be working ok. I wrote a little
> >perl script to dump the SDBM file and it's collecting what looks
> >right. My issue is that the ip.pag file grows insanely quickly. For
> >example, in 9 minutes, it had grown to 4 gig. In an earlier run (with
> >the SDBM file deleted between all tests), it made it to 8 gig in about
> >15 minutes.
> >
> >In the 4 gig file, the perl script (which might consequently be wrong
> >itself) said there was 13359 entries. That's 360k per entry, which
> >seems a bit large. Is it storing a lot of unprintable data in each
> >record, like a detailed record of every request? This is admittedly
> >probably over-simplistic, but iterating over the tied hash from the
> >SDBM file in Perl and adding up the record sizes as well as the sizes
> >of the keys, I get a measly 4.7 meg. I'd expect a good deal of
> >per-entry overhead in the db, but that's off by almost 1000x.
> >
> >I saw in MODSEC-160 that 1800 is probably a bit high, so I can lower
> >that (or just have the 60s one). But if my file is hitting that size
> >in 9 minutes, then 1800 seconds is probably irrelevant here. I had
> >originally just did a 60 second expiration time and even with just
> >that (i.e. no additional 1800 second expiration), the file was growing
> >very large, very quickly, though obviously not as rapidly as with the
> >additional 1800s expiration, but growing quickly enough to see
> >unwarranted by the number of records in the SDBM file.
> >
> >So:
> >a) Am I doing something insanely stupid here? That'd be my preference.
> >This might also include a poor conceptual grasp of what goes into the
> >SDBM file.
> >b) Am I abusing the IP collection, and if so, is there somewhere else
> >I should be storing this?
> >c) Am I hitting a bug of some sort? The release notes for >=2.6.4
> >didn't mention any fixes relating to collections, so I haven't tried
> >upgrading yet.
> >d) Any hints on how I could be doing this better are highly appreciated.
>
> Mark,
> 1) The persistent storage files are sparse data files. Run a "du -b" vs.
> "ls -l" and see what the difference is.
> 2) Have you reviewed the DoS/Brute Force rules in the OWASP CRS? These
> may have the functionality you want.
> 3) You could also consider using the SecGuardianLog directive with the
> httpd-guardian.pl script -
> http://mod-security.svn.sourceforge.net/viewvc/mod-security/crs/trunk/util/
> httpd-guardian.pl?revision=1961&content-type=text%2Fplain
>
> This monitors the same data going to the apache logs and can fire off
> commands to IPTables to blacklist source IP addresses at a lower level.
>
> --
> Ryan Barnett
> Trustwave SpiderLabs
> ModSecurity Project Leader
> OWASP ModSecurity CRS Project Leader
>
>
> This transmission may contain information that is privileged,
> confidential, and/or exempt from disclosure under applicable law. If you
> are not the intended recipient, you are hereby notified that any
> disclosure, copying, distribution, or use of the information contained
> herein (including any reliance thereon) is STRICTLY PROHIBITED. If you
> received this transmission in error, please immediately contact the sender
> and destroy the material in its entirety, whether in electronic or hard
> copy format.
>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/
>
|