mod-security-users

[mod-security-users] Collections_remove_stale: Failed deleting collection

[Sophie L. <sop...@tr...>]

Hi,

     I installed some rules for rate limiting was concerned by a message 
in  modsec_audit.log.

My question: What does this error message really mean?

Message: collections_remove_stale: Failed deleting collection (name 
"ip", key "213.56.235.241_ef6e1e02a3981d38a7faf3db672aa4a4bf7cb53c"): 
Internal error

Some additional notes
I thought this may be related to /var/lib/mod_security/ip.pag or 
user.pag. So, I installed modsec-sdbm-util to see if this dB needed 
shrinking periodically, was fragmented or should be emptied ( e.g > 
ip.pag ) and still saw the messages.

  # modsec-sdbm-util -s ip.pag
Opening file: ip.pag
Database ready to be used.
  [\] 720 records so far.
Total of 726 elements processed.
0 elements removed.
Expired elements: 6, inconsistent items: 0
Fragmentation rate: 0.83% of the database is/was dirty data.

  # modsec-sdbm-util -s user.pag
Opening file: user.pag
Database ready to be used.
  [-] 790 records so far.
Total of 799 elements processed.
0 elements removed.
Expired elements: 12, inconsistent items: 0
Fragmentation rate: 1.50% of the database is/was dirty data.


Some version information

mod_security-2.7.3-2.el6.x86_64
modsec-sdbm-util v1.0

Activated rules,
/etc/https/conf.d/mod_security.conf
/etc/httpd/modsecurity.d/modsec_sophie.conf
/etc/httpd/modsecurity.d/modsecurity_crs_10_config.conf

# cat /etc/httpd/modsecurity.d/modsec_sophie.conf
   SecAction 
initcol:ip=%{REMOTE_ADDR},pass,nolog,id:10000001,msg:Sophie_10000001
   SecAction 
"phase:5,deprecatevar:ip.somepathcounter=1/1,pass,nolog,id:10000002,msg:Sophie_10000002"
   SecRule IP:SOMEPATHCOUNTER "@gt 60" 
"phase:2,pause:300,deny,status:509,setenv:RATELIMITED,skip:1,nolog,id:10000003,msg:Sophie_10000003"
   SecAction 
"phase:2,pass,setvar:ip.somepathcounter=+1,nolog,id:10000004,msg:Sophie_10000004"
   Header always set Retry-After "10" env=RATELIMITED
ErrorDocument 509 "Rate Limit Exceeded"



Kind regards,
Sophie

-- 
Sophie Loewenthal
System Engineer ITOPS / Trimble Transport & Logistics
GSM:+32.471.900703

[mod-security-users] RuleRemoveTargetById for a regexp'able list of ARGS

[Elmar K. B. <el...@no...>]

Guys,

I have a rather peculiar problem (who could have guessed)...

We're using the OWASP rules, as most people, I guess, and they do of course fire
on our "ARGS:translation0..." parameters, since those are designed to contain
"foreign characters". So far, so good. Or not. Sounds simple, right?

Now I need to explicitly whitelist those parameters. I was about to try

SecRule REQUEST_FILENAME "^/<whatever-endpoint>$" \
  "phase:2,id:'xxxxx',pass,ctl:RuleRemoveTargetById=900000-999999;ARGS:/translation.*/"

... but I got stopped by lightheartedly googling away and finding out that
regexps are not implemented for RuleRemoveTargetById.

Can anybody hint to how to get this done in a way that is more elegant than
having 50 (or 60 or 70) SecRule entries matching a REQUEST_FILENAME regexp and
then RuleRemoveTargetById'ing translationXX (which would also mean quite a
performance hit, I guess)?

One idea would have been to use a transformation function that would zap the
contents, but such transformation does not exist and thus would have to be
programmed, compiled to a module and included.

So if there is an easier way, please help me out.

Thank you,
        Elmar.

Re: [mod-security-users] Collections_remove_stale: Failed deleting collection

[Christian F. <chr...@ti...>]

Hello Sophie,

You are touching on an interesting subject or problem depending
on the perspective. 

The pag-Files are rarely discussed in the community and I would
not be surprised if you hit an issue with ModSec. I for one hardly 
ever look at these files as they just seem to work. I have seen the
said error message before, but never paid too much attention
as I did not have that growth problem of the file.

Our apache servers are gracefully restarted once a day. I do not
know if that benefits the pag-files at all. But it might be a 
noteworthy detail.

Sorry for not being able to help you. But I would be curious to
read anything about the topic.

Ahoj,

Christian


On Thu, Aug 20, 2015 at 02:26:14PM +0200, Sophie Loewenthal wrote:
> Hi,
> 
>      I installed some rules for rate limiting was concerned by a message 
> in  modsec_audit.log.
> 
> My question: What does this error message really mean?
> 
> Message: collections_remove_stale: Failed deleting collection (name 
> "ip", key "213.56.235.241_ef6e1e02a3981d38a7faf3db672aa4a4bf7cb53c"): 
> Internal error
> 
> Some additional notes
> I thought this may be related to /var/lib/mod_security/ip.pag or 
> user.pag. So, I installed modsec-sdbm-util to see if this dB needed 
> shrinking periodically, was fragmented or should be emptied ( e.g > 
> ip.pag ) and still saw the messages.
> 
>   # modsec-sdbm-util -s ip.pag
> Opening file: ip.pag
> Database ready to be used.
>   [\] 720 records so far.
> Total of 726 elements processed.
> 0 elements removed.
> Expired elements: 6, inconsistent items: 0
> Fragmentation rate: 0.83% of the database is/was dirty data.
> 
>   # modsec-sdbm-util -s user.pag
> Opening file: user.pag
> Database ready to be used.
>   [-] 790 records so far.
> Total of 799 elements processed.
> 0 elements removed.
> Expired elements: 12, inconsistent items: 0
> Fragmentation rate: 1.50% of the database is/was dirty data.
> 
> 
> Some version information
> 
> mod_security-2.7.3-2.el6.x86_64
> modsec-sdbm-util v1.0
> 
> Activated rules,
> /etc/https/conf.d/mod_security.conf
> /etc/httpd/modsecurity.d/modsec_sophie.conf
> /etc/httpd/modsecurity.d/modsecurity_crs_10_config.conf
> 
> # cat /etc/httpd/modsecurity.d/modsec_sophie.conf
>    SecAction 
> initcol:ip=%{REMOTE_ADDR},pass,nolog,id:10000001,msg:Sophie_10000001
>    SecAction 
> "phase:5,deprecatevar:ip.somepathcounter=1/1,pass,nolog,id:10000002,msg:Sophie_10000002"
>    SecRule IP:SOMEPATHCOUNTER "@gt 60" 
> "phase:2,pause:300,deny,status:509,setenv:RATELIMITED,skip:1,nolog,id:10000003,msg:Sophie_10000003"
>    SecAction 
> "phase:2,pass,setvar:ip.somepathcounter=+1,nolog,id:10000004,msg:Sophie_10000004"
>    Header always set Retry-After "10" env=RATELIMITED
> ErrorDocument 509 "Rate Limit Exceeded"
> 
> 
> 
> Kind regards,
> Sophie
> 
> -- 
> Sophie Loewenthal
> System Engineer ITOPS / Trimble Transport & Logistics
> GSM:+32.471.900703
> 
> ------------------------------------------------------------------------------
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/

-- 
Christian Folini
Ringstrasse 2
CH-3639 Kiesen
+41 (0)31 301 60 71 (H)
+41 (0)79 220 23 76 (M)
mailto:chr...@ne... (Business)
mailto:chr...@ti... (Private)
http://www.christian-folini.ch

Re: [mod-security-users] Collections_remove_stale: Failed deleting collection

[Barry P. <bar...@ho...>]

Ultimately I don't think storing collection data in a disk based file is ever going to work particularly well for any site with any volume. You will get contention on that file and sometimes the clean up job which is supposed to delete old collections fails when trying to access the file (which I think is what that message means). I'm also not sure it tries again to clean up that particular record when this happens as I've seen this file grow and grow until it starts to cause performance problems - as well as taking up a lot of disk space.

There is an experimental build of ModSecurity which stores this in shared memory which seems like a much more sensible place to me but not had a look at that yet.

Here's one way to handle the problem:

1) Reduce usage of these collection files to the minimum.

For example if you only use them for Brute Force checks of login pages, then only register collections for these resources and don't use them for static page requests which make up the majority of your requests.

Similarly if you want to use them for DoS checks then decide what do you want to protect. Everything ideally of course, but I question the effectiveness of DoS protection for Apache itself within Mod Security and think this is best handled outside it. So maybe only use DoS protection rules for application resources that take extra resources and typically have much lower DoS limits. So again only initialise collections for those resources.

This means moving the initialisation of these collections out of the modsecurity_crs_10_setup.conf file (where they are done near the beginning of phase 1 and so basically done for every request), and writing your own similar, rules to initialise them only for the requests that you need. I've had some success with moving them to a file which is processed at the end of phase 1, after basic header checks are done, and after rules white whitelist static requests kicked it.

Note also, while we're on the subject,  that the CRS whitelisting of static page rules are not the useful IMHO (they run at the beginning of phase 2 after collections have been initialised and the whitelisting only works when the rule engine is turned On, rather than also in DetectionOnly mode which you might want to run in initially) so I'd suggest writing your own versions to address these two issues.

This does help to reduce the growth of the collections drastically (also a good performance benefit presumably!) but may still not solve it completely as the files can still grow depending on your traffic.

So you could also pull out the sledge hammer approach:

2) Set up a cron to delete these collection files periodically (e.g. once a day). Probably not the best practice, but not seen any negative affects of this and doesn't even require a webserver bounce to recreate them.

This doesn't resolve the occasional error message about deleting stale requests l, but does ensure at least that the file doesn't get out of control and start to severely impact performance of your webserver, so you can just ignore those error messages.

As I alluded to at the start I do think collection handling is one of the weakest parts of ModSecurity and do hope the team get a chance to spend some time on this for the version 3 rewrite. In meantime I think it's probably best for most users to turn off collections if not really using them, and for those that want them to consider steps like above.

Hope that helps and definitely interested to hear what others have to say and any comments on above approach.

Thanks,
Barry

> On 21 Aug 2015, at 05:15, Christian Folini <chr...@ti...> wrote:
> 
> Hello Sophie,
> 
> You are touching on an interesting subject or problem depending
> on the perspective. 
> 
> The pag-Files are rarely discussed in the community and I would
> not be surprised if you hit an issue with ModSec. I for one hardly 
> ever look at these files as they just seem to work. I have seen the
> said error message before, but never paid too much attention
> as I did not have that growth problem of the file.
> 
> Our apache servers are gracefully restarted once a day. I do not
> know if that benefits the pag-files at all. But it might be a 
> noteworthy detail.
> 
> Sorry for not being able to help you. But I would be curious to
> read anything about the topic.
> 
> Ahoj,
> 
> Christian
> 
> 
>> On Thu, Aug 20, 2015 at 02:26:14PM +0200, Sophie Loewenthal wrote:
>> Hi,
>> 
>>     I installed some rules for rate limiting was concerned by a message 
>> in  modsec_audit.log.
>> 
>> My question: What does this error message really mean?
>> 
>> Message: collections_remove_stale: Failed deleting collection (name 
>> "ip", key "213.56.235.241_ef6e1e02a3981d38a7faf3db672aa4a4bf7cb53c"): 
>> Internal error
>> 
>> Some additional notes
>> I thought this may be related to /var/lib/mod_security/ip.pag or 
>> user.pag. So, I installed modsec-sdbm-util to see if this dB needed 
>> shrinking periodically, was fragmented or should be emptied ( e.g > 
>> ip.pag ) and still saw the messages.
>> 
>>  # modsec-sdbm-util -s ip.pag
>> Opening file: ip.pag
>> Database ready to be used.
>>  [\] 720 records so far.
>> Total of 726 elements processed.
>> 0 elements removed.
>> Expired elements: 6, inconsistent items: 0
>> Fragmentation rate: 0.83% of the database is/was dirty data.
>> 
>>  # modsec-sdbm-util -s user.pag
>> Opening file: user.pag
>> Database ready to be used.
>>  [-] 790 records so far.
>> Total of 799 elements processed.
>> 0 elements removed.
>> Expired elements: 12, inconsistent items: 0
>> Fragmentation rate: 1.50% of the database is/was dirty data.
>> 
>> 
>> Some version information
>> 
>> mod_security-2.7.3-2.el6.x86_64
>> modsec-sdbm-util v1.0
>> 
>> Activated rules,
>> /etc/https/conf.d/mod_security.conf
>> /etc/httpd/modsecurity.d/modsec_sophie.conf
>> /etc/httpd/modsecurity.d/modsecurity_crs_10_config.conf
>> 
>> # cat /etc/httpd/modsecurity.d/modsec_sophie.conf
>>   SecAction 
>> initcol:ip=%{REMOTE_ADDR},pass,nolog,id:10000001,msg:Sophie_10000001
>>   SecAction 
>> "phase:5,deprecatevar:ip.somepathcounter=1/1,pass,nolog,id:10000002,msg:Sophie_10000002"
>>   SecRule IP:SOMEPATHCOUNTER "@gt 60" 
>> "phase:2,pause:300,deny,status:509,setenv:RATELIMITED,skip:1,nolog,id:10000003,msg:Sophie_10000003"
>>   SecAction 
>> "phase:2,pass,setvar:ip.somepathcounter=+1,nolog,id:10000004,msg:Sophie_10000004"
>>   Header always set Retry-After "10" env=RATELIMITED
>> ErrorDocument 509 "Rate Limit Exceeded"
>> 
>> 
>> 
>> Kind regards,
>> Sophie
>> 
>> -- 
>> Sophie Loewenthal
>> System Engineer ITOPS / Trimble Transport & Logistics
>> GSM:+32.471.900703
>> 
>> ------------------------------------------------------------------------------
>> _______________________________________________
>> mod-security-users mailing list
>> mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> http://www.modsecurity.org/projects/commercial/rules/
>> http://www.modsecurity.org/projects/commercial/support/
> 
> -- 
> Christian Folini
> Ringstrasse 2
> CH-3639 Kiesen
> +41 (0)31 301 60 71 (H)
> +41 (0)79 220 23 76 (M)
> mailto:chr...@ne... (Business)
> mailto:chr...@ti... (Private)
> http://www.christian-folini.ch
> 
> 
> ------------------------------------------------------------------------------
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/

Re: [mod-security-users] Collections_remove_stale: Failed deleting collection

[Christian F. <chr...@ti...>]

Barry,

thanks for a very good response on the topic.

Christian

On Fri, Aug 21, 2015 at 08:04:20AM +0100, Barry Pollard wrote:
> Ultimately I don't think storing collection data in a disk based file is ever going to work particularly well for any site with any volume. You will get contention on that file and sometimes the clean up job which is supposed to delete old collections fails when trying to access the file (which I think is what that message means). I'm also not sure it tries again to clean up that particular record when this happens as I've seen this file grow and grow until it starts to cause performance problems - as well as taking up a lot of disk space.


> 
> There is an experimental build of ModSecurity which stores this in shared memory which seems like a much more sensible place to me but not had a look at that yet.
> 
> Here's one way to handle the problem:
> 
> 1) Reduce usage of these collection files to the minimum.
> 
> For example if you only use them for Brute Force checks of login pages, then only register collections for these resources and don't use them for static page requests which make up the majority of your requests.
> 
> Similarly if you want to use them for DoS checks then decide what do you want to protect. Everything ideally of course, but I question the effectiveness of DoS protection for Apache itself within Mod Security and think this is best handled outside it. So maybe only use DoS protection rules for application resources that take extra resources and typically have much lower DoS limits. So again only initialise collections for those resources.
> 
> This means moving the initialisation of these collections out of the modsecurity_crs_10_setup.conf file (where they are done near the beginning of phase 1 and so basically done for every request), and writing your own similar, rules to initialise them only for the requests that you need. I've had some success with moving them to a file which is processed at the end of phase 1, after basic header checks are done, and after rules white whitelist static requests kicked it.
> 
> Note also, while we're on the subject,  that the CRS whitelisting of static page rules are not the useful IMHO (they run at the beginning of phase 2 after collections have been initialised and the whitelisting only works when the rule engine is turned On, rather than also in DetectionOnly mode which you might want to run in initially) so I'd suggest writing your own versions to address these two issues.
> 
> This does help to reduce the growth of the collections drastically (also a good performance benefit presumably!) but may still not solve it completely as the files can still grow depending on your traffic.
> 
> So you could also pull out the sledge hammer approach:
> 
> 2) Set up a cron to delete these collection files periodically (e.g. once a day). Probably not the best practice, but not seen any negative affects of this and doesn't even require a webserver bounce to recreate them.
> 
> This doesn't resolve the occasional error message about deleting stale requests l, but does ensure at least that the file doesn't get out of control and start to severely impact performance of your webserver, so you can just ignore those error messages.
> 
> As I alluded to at the start I do think collection handling is one of the weakest parts of ModSecurity and do hope the team get a chance to spend some time on this for the version 3 rewrite. In meantime I think it's probably best for most users to turn off collections if not really using them, and for those that want them to consider steps like above.
> 
> Hope that helps and definitely interested to hear what others have to say and any comments on above approach.
> 
> Thanks,
> Barry
> 
> > On 21 Aug 2015, at 05:15, Christian Folini <chr...@ti...> wrote:
> > 
> > Hello Sophie,
> > 
> > You are touching on an interesting subject or problem depending
> > on the perspective. 
> > 
> > The pag-Files are rarely discussed in the community and I would
> > not be surprised if you hit an issue with ModSec. I for one hardly 
> > ever look at these files as they just seem to work. I have seen the
> > said error message before, but never paid too much attention
> > as I did not have that growth problem of the file.
> > 
> > Our apache servers are gracefully restarted once a day. I do not
> > know if that benefits the pag-files at all. But it might be a 
> > noteworthy detail.
> > 
> > Sorry for not being able to help you. But I would be curious to
> > read anything about the topic.
> > 
> > Ahoj,
> > 
> > Christian
> > 
> > 
> >> On Thu, Aug 20, 2015 at 02:26:14PM +0200, Sophie Loewenthal wrote:
> >> Hi,
> >> 
> >>     I installed some rules for rate limiting was concerned by a message 
> >> in  modsec_audit.log.
> >> 
> >> My question: What does this error message really mean?
> >> 
> >> Message: collections_remove_stale: Failed deleting collection (name 
> >> "ip", key "213.56.235.241_ef6e1e02a3981d38a7faf3db672aa4a4bf7cb53c"): 
> >> Internal error
> >> 
> >> Some additional notes
> >> I thought this may be related to /var/lib/mod_security/ip.pag or 
> >> user.pag. So, I installed modsec-sdbm-util to see if this dB needed 
> >> shrinking periodically, was fragmented or should be emptied ( e.g > 
> >> ip.pag ) and still saw the messages.
> >> 
> >>  # modsec-sdbm-util -s ip.pag
> >> Opening file: ip.pag
> >> Database ready to be used.
> >>  [\] 720 records so far.
> >> Total of 726 elements processed.
> >> 0 elements removed.
> >> Expired elements: 6, inconsistent items: 0
> >> Fragmentation rate: 0.83% of the database is/was dirty data.
> >> 
> >>  # modsec-sdbm-util -s user.pag
> >> Opening file: user.pag
> >> Database ready to be used.
> >>  [-] 790 records so far.
> >> Total of 799 elements processed.
> >> 0 elements removed.
> >> Expired elements: 12, inconsistent items: 0
> >> Fragmentation rate: 1.50% of the database is/was dirty data.
> >> 
> >> 
> >> Some version information
> >> 
> >> mod_security-2.7.3-2.el6.x86_64
> >> modsec-sdbm-util v1.0
> >> 
> >> Activated rules,
> >> /etc/https/conf.d/mod_security.conf
> >> /etc/httpd/modsecurity.d/modsec_sophie.conf
> >> /etc/httpd/modsecurity.d/modsecurity_crs_10_config.conf
> >> 
> >> # cat /etc/httpd/modsecurity.d/modsec_sophie.conf
> >>   SecAction 
> >> initcol:ip=%{REMOTE_ADDR},pass,nolog,id:10000001,msg:Sophie_10000001
> >>   SecAction 
> >> "phase:5,deprecatevar:ip.somepathcounter=1/1,pass,nolog,id:10000002,msg:Sophie_10000002"
> >>   SecRule IP:SOMEPATHCOUNTER "@gt 60" 
> >> "phase:2,pause:300,deny,status:509,setenv:RATELIMITED,skip:1,nolog,id:10000003,msg:Sophie_10000003"
> >>   SecAction 
> >> "phase:2,pass,setvar:ip.somepathcounter=+1,nolog,id:10000004,msg:Sophie_10000004"
> >>   Header always set Retry-After "10" env=RATELIMITED
> >> ErrorDocument 509 "Rate Limit Exceeded"
> >> 
> >> 
> >> 
> >> Kind regards,
> >> Sophie
> >> 
> >> -- 
> >> Sophie Loewenthal
> >> System Engineer ITOPS / Trimble Transport & Logistics
> >> GSM:+32.471.900703
> >> 
> >> ------------------------------------------------------------------------------
> >> _______________________________________________
> >> mod-security-users mailing list
> >> mod...@li...
> >> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> >> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> >> http://www.modsecurity.org/projects/commercial/rules/
> >> http://www.modsecurity.org/projects/commercial/support/
> > 
> > -- 
> > Christian Folini
> > Ringstrasse 2
> > CH-3639 Kiesen
> > +41 (0)31 301 60 71 (H)
> > +41 (0)79 220 23 76 (M)
> > mailto:chr...@ne... (Business)
> > mailto:chr...@ti... (Private)
> > http://www.christian-folini.ch
> > 
> > 
> > ------------------------------------------------------------------------------
> > _______________________________________________
> > mod-security-users mailing list
> > mod...@li...
> > https://lists.sourceforge.net/lists/listinfo/mod-security-users
> > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> > http://www.modsecurity.org/projects/commercial/rules/
> > http://www.modsecurity.org/projects/commercial/support/
> 
> ------------------------------------------------------------------------------
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/

Re: [mod-security-users] Collections_remove_stale: Failed deleting collection

[Sophie L. <sop...@tr...>]

Hi Barry,

     Thank you for your well penned reply.

     For an quick fix, I have put the directory into a ram disc, and 
shall run some pruning methods from cron.

     Your other suggestion require I spend more time on how I should 
differentiate between static and dynamic content, although I doubt any 
static is requested because this is a soap gateway. Looking anyway  :)

    How else could I reduce modsec traffic?  If I could wrap this code 
up into a LocationMatch and place inside a vhost entry, maybe this could 
help.

Management would like a list of potential offenders by IP.
How could I adapt this code to add logging of IP and or request into a 
file?  I looked at SecAuditLogParts and enabling everything I could not 
see of it hit a rule.  Currently this runs in  detection mode.
I have logging enabled on this rule:
   SecRule IP:SOMEPATHCOUNTER "@gt 120" 
"phase:2,pause:100,deny,status:509,setenv:RATELIMITED,skip:1,log,id:10000003,msg:Sophie_10000003"
But would like an IP address logged when it was sent a 509 status message.
Still reading 
https://www.feistyduck.com/library/modsecurity-handbook-free/online/ch04-logging.html 
!

Kind regards,
Sophie

-- 
On 8/21/2015 9:04 AM, Barry Pollard wrote:
> Ultimately I don't think storing collection data in a disk based file is ever going to work particularly well for any site with any volume. You will get contention on that file and sometimes the clean up job which is supposed to delete old collections fails when trying to access the file (which I think is what that message means). I'm also not sure it tries again to clean up that particular record when this happens as I've seen this file grow and grow until it starts to cause performance problems - as well as taking up a lot of disk space.
>
> There is an experimental build of ModSecurity which stores this in shared memory which seems like a much more sensible place to me but not had a look at that yet.
>
> Here's one way to handle the problem:
>
> 1) Reduce usage of these collection files to the minimum.
>
> For example if you only use them for Brute Force checks of login pages, then only register collections for these resources and don't use them for static page requests which make up the majority of your requests.
>
> Similarly if you want to use them for DoS checks then decide what do you want to protect. Everything ideally of course, but I question the effectiveness of DoS protection for Apache itself within Mod Security and think this is best handled outside it. So maybe only use DoS protection rules for application resources that take extra resources and typically have much lower DoS limits. So again only initialise collections for those resources.
>
> This means moving the initialisation of these collections out of the modsecurity_crs_10_setup.conf file (where they are done near the beginning of phase 1 and so basically done for every request), and writing your own similar, rules to initialise them only for the requests that you need. I've had some success with moving them to a file which is processed at the end of phase 1, after basic header checks are done, and after rules white whitelist static requests kicked it.
>
> Note also, while we're on the subject,  that the CRS whitelisting of static page rules are not the useful IMHO (they run at the beginning of phase 2 after collections have been initialised and the whitelisting only works when the rule engine is turned On, rather than also in DetectionOnly mode which you might want to run in initially) so I'd suggest writing your own versions to address these two issues.
>
> This does help to reduce the growth of the collections drastically (also a good performance benefit presumably!) but may still not solve it completely as the files can still grow depending on your traffic.
>
> So you could also pull out the sledge hammer approach:
>
> 2) Set up a cron to delete these collection files periodically (e.g. once a day). Probably not the best practice, but not seen any negative affects of this and doesn't even require a webserver bounce to recreate them.
>
> This doesn't resolve the occasional error message about deleting stale requests l, but does ensure at least that the file doesn't get out of control and start to severely impact performance of your webserver, so you can just ignore those error messages.
>
> As I alluded to at the start I do think collection handling is one of the weakest parts of ModSecurity and do hope the team get a chance to spend some time on this for the version 3 rewrite. In meantime I think it's probably best for most users to turn off collections if not really using them, and for those that want them to consider steps like above.
>
> Hope that helps and definitely interested to hear what others have to say and any comments on above approach.
>
> Thanks,
> Barry
>
>> On 21 Aug 2015, at 05:15, Christian Folini <chr...@ti...> wrote:
>>
>> Hello Sophie,
>>
>> You are touching on an interesting subject or problem depending
>> on the perspective.
>>
>> The pag-Files are rarely discussed in the community and I would
>> not be surprised if you hit an issue with ModSec. I for one hardly
>> ever look at these files as they just seem to work. I have seen the
>> said error message before, but never paid too much attention
>> as I did not have that growth problem of the file.
>>
>> Our apache servers are gracefully restarted once a day. I do not
>> know if that benefits the pag-files at all. But it might be a
>> noteworthy detail.
>>
>> Sorry for not being able to help you. But I would be curious to
>> read anything about the topic.
>>
>> Ahoj,
>>
>> Christian
>>
>>
>>> On Thu, Aug 20, 2015 at 02:26:14PM +0200, Sophie Loewenthal wrote:
>>> Hi,
>>>
>>>      I installed some rules for rate limiting was concerned by a message
>>> in  modsec_audit.log.
>>>
>>> My question: What does this error message really mean?
>>>
>>> Message: collections_remove_stale: Failed deleting collection (name
>>> "ip", key "213.56.235.241_ef6e1e02a3981d38a7faf3db672aa4a4bf7cb53c"):
>>> Internal error
>>>
>>> Some additional notes
>>> I thought this may be related to /var/lib/mod_security/ip.pag or
>>> user.pag. So, I installed modsec-sdbm-util to see if this dB needed
>>> shrinking periodically, was fragmented or should be emptied ( e.g >
>>> ip.pag ) and still saw the messages.
>>>
>>>   # modsec-sdbm-util -s ip.pag
>>> Opening file: ip.pag
>>> Database ready to be used.
>>>   [\] 720 records so far.
>>> Total of 726 elements processed.
>>> 0 elements removed.
>>> Expired elements: 6, inconsistent items: 0
>>> Fragmentation rate: 0.83% of the database is/was dirty data.
>>>
>>>   # modsec-sdbm-util -s user.pag
>>> Opening file: user.pag
>>> Database ready to be used.
>>>   [-] 790 records so far.
>>> Total of 799 elements processed.
>>> 0 elements removed.
>>> Expired elements: 12, inconsistent items: 0
>>> Fragmentation rate: 1.50% of the database is/was dirty data.
>>>
>>>
>>> Some version information
>>>
>>> mod_security-2.7.3-2.el6.x86_64
>>> modsec-sdbm-util v1.0
>>>
>>> Activated rules,
>>> /etc/https/conf.d/mod_security.conf
>>> /etc/httpd/modsecurity.d/modsec_sophie.conf
>>> /etc/httpd/modsecurity.d/modsecurity_crs_10_config.conf
>>>
>>> # cat /etc/httpd/modsecurity.d/modsec_sophie.conf
>>>    SecAction
>>> initcol:ip=%{REMOTE_ADDR},pass,nolog,id:10000001,msg:Sophie_10000001
>>>    SecAction
>>> "phase:5,deprecatevar:ip.somepathcounter=1/1,pass,nolog,id:10000002,msg:Sophie_10000002"
>>>    SecRule IP:SOMEPATHCOUNTER "@gt 60"
>>> "phase:2,pause:300,deny,status:509,setenv:RATELIMITED,skip:1,nolog,id:10000003,msg:Sophie_10000003"
>>>    SecAction
>>> "phase:2,pass,setvar:ip.somepathcounter=+1,nolog,id:10000004,msg:Sophie_10000004"
>>>    Header always set Retry-After "10" env=RATELIMITED
>>> ErrorDocument 509 "Rate Limit Exceeded"
>>>
>>>
>>>
>>> Kind regards,
>>> Sophie
>>>
>>> -- 
>>> Sophie Loewenthal
>>> System Engineer ITOPS / Trimble Transport & Logistics
>>> GSM:+32.471.900703
>>>
>>> ------------------------------------------------------------------------------
>>> _______________________________________________
>>> mod-security-users mailing list
>>> mod...@li...
>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>> http://www.modsecurity.org/projects/commercial/rules/
>>> http://www.modsecurity.org/projects/commercial/support/
>> -- 
>> Christian Folini
>> Ringstrasse 2
>> CH-3639 Kiesen
>> +41 (0)31 301 60 71 (H)
>> +41 (0)79 220 23 76 (M)
>> mailto:chr...@ne... (Business)
>> mailto:chr...@ti... (Private)
>> http://www.christian-folini.ch
>>
>>
>> ------------------------------------------------------------------------------
>> _______________________________________________
>> mod-security-users mailing list
>> mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> http://www.modsecurity.org/projects/commercial/rules/
>> http://www.modsecurity.org/projects/commercial/support/
> ------------------------------------------------------------------------------
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/

Re: [mod-security-users] Collections_remove_stale: Failed deleting collection

[Rainer J. <rai...@ki...>]

Am 25.08.2015 um 10:47 schrieb Sophie Loewenthal:
> Hi Barry,
>
>       Thank you for your well penned reply.
>
>       For an quick fix, I have put the directory into a ram disc, and
> shall run some pruning methods from cron.
>
>       Your other suggestion require I spend more time on how I should
> differentiate between static and dynamic content, although I doubt any
> static is requested because this is a soap gateway. Looking anyway  :)
>
>      How else could I reduce modsec traffic?  If I could wrap this code
> up into a LocationMatch and place inside a vhost entry, maybe this could
> help.
>
> Management would like a list of potential offenders by IP.
> How could I adapt this code to add logging of IP and or request into a
> file?  I looked at SecAuditLogParts and enabling everything I could not
> see of it hit a rule.  Currently this runs in  detection mode.
> I have logging enabled on this rule:
>     SecRule IP:SOMEPATHCOUNTER "@gt 120"
> "phase:2,pause:100,deny,status:509,setenv:RATELIMITED,skip:1,log,id:10000003,msg:Sophie_10000003"
> But would like an IP address logged when it was sent a 509 status message.
> Still reading
> https://www.feistyduck.com/library/modsecurity-handbook-free/online/ch04-logging.html
> !

Since 509 is very specific, why not taking the IP from the normal access 
log of the web server?

Regards,

Rainer

Re: [mod-security-users] Collections_remove_stale: Failed deleting collection

[Sophie L. <sop...@tr...>]

Hi Rainer ,

  >> Since 509 is very specific, why not taking the IP from the normal 
access log of the web server?
Mod_Sec is running in Detection Only mode. Would this be logged in the 
access log?

Sophie

On 8/25/2015 11:27 AM, Rainer Jung wrote:
> Am 25.08.2015 um 10:47 schrieb Sophie Loewenthal:
>> Hi Barry,
>>
>>        Thank you for your well penned reply.
>>
>>        For an quick fix, I have put the directory into a ram disc, and
>> shall run some pruning methods from cron.
>>
>>        Your other suggestion require I spend more time on how I should
>> differentiate between static and dynamic content, although I doubt any
>> static is requested because this is a soap gateway. Looking anyway  :)
>>
>>       How else could I reduce modsec traffic?  If I could wrap this code
>> up into a LocationMatch and place inside a vhost entry, maybe this could
>> help.
>>
>> Management would like a list of potential offenders by IP.
>> How could I adapt this code to add logging of IP and or request into a
>> file?  I looked at SecAuditLogParts and enabling everything I could not
>> see of it hit a rule.  Currently this runs in  detection mode.
>> I have logging enabled on this rule:
>>      SecRule IP:SOMEPATHCOUNTER "@gt 120"
>> "phase:2,pause:100,deny,status:509,setenv:RATELIMITED,skip:1,log,id:10000003,msg:Sophie_10000003"
>> But would like an IP address logged when it was sent a 509 status message.
>> Still reading
>> https://www.feistyduck.com/library/modsecurity-handbook-free/online/ch04-logging.html
>> !
> Since 509 is very specific, why not taking the IP from the normal access
> log of the web server?
>
> Regards,
>
> Rainer
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/

Re: [mod-security-users] Collections_remove_stale: Failed deleting collection

[Rainer J. <rai...@ki...>]

Am 25.08.2015 um 12:02 schrieb Sophie Loewenthal:
> Hi Rainer ,
>
>    >> Since 509 is very specific, why not taking the IP from the normal
> access log of the web server?
> Mod_Sec is running in Detection Only mode. Would this be logged in the
> access log?

Any web server allows to activate an access log. Typically it is already 
active by default. The contents of the log are adjustable but again the 
IP and the HTTP status code should be there by default.

The access log contains one line for each http request that was handled 
by the web server. So you'd have to find the log or identify how to 
activate it for your web server and identify the configured or default 
format of its content. It will then be easy to filter the log for all 
requests that were answered with a 509 and extract the IP addresses.

All of this access log stuff is independent of your mod_security config.

Regards,

Rainer

> On 8/25/2015 11:27 AM, Rainer Jung wrote:
>> Am 25.08.2015 um 10:47 schrieb Sophie Loewenthal:
>>> Hi Barry,
>>>
>>>         Thank you for your well penned reply.
>>>
>>>         For an quick fix, I have put the directory into a ram disc, and
>>> shall run some pruning methods from cron.
>>>
>>>         Your other suggestion require I spend more time on how I should
>>> differentiate between static and dynamic content, although I doubt any
>>> static is requested because this is a soap gateway. Looking anyway  :)
>>>
>>>        How else could I reduce modsec traffic?  If I could wrap this code
>>> up into a LocationMatch and place inside a vhost entry, maybe this could
>>> help.
>>>
>>> Management would like a list of potential offenders by IP.
>>> How could I adapt this code to add logging of IP and or request into a
>>> file?  I looked at SecAuditLogParts and enabling everything I could not
>>> see of it hit a rule.  Currently this runs in  detection mode.
>>> I have logging enabled on this rule:
>>>       SecRule IP:SOMEPATHCOUNTER "@gt 120"
>>> "phase:2,pause:100,deny,status:509,setenv:RATELIMITED,skip:1,log,id:10000003,msg:Sophie_10000003"
>>> But would like an IP address logged when it was sent a 509 status message.
>>> Still reading
>>> https://www.feistyduck.com/library/modsecurity-handbook-free/online/ch04-logging.html
>>> !
>> Since 509 is very specific, why not taking the IP from the normal access
>> log of the web server?
>>
>> Regardsco
>>
>> Rainer

Re: [mod-security-users] Collections_remove_stale: Failed deleting collection

[Barry P. <bar...@ho...>]

There's a much better solution than either the access log or the audit log.

ModSecurity rules which are fired, are logged in the Apache error log (on one line) as well as in the Audit Log (in multiple lines so more difficult to report on and more meant to help you investigate). The error log by default shows the client IP address, though you can configure it as much as you want in Apache.

Here's two example errors from my test system (with some data changed to remove details of my system):

SecRuleEngine On:
[Tue Aug 25 18:04:15.767701 2015] [:error] [pid 123:tid 13456] [client 10.40.123.456] ModSecurity: Access denied with code 403 (phase 2). Invalid URL Encoding: Not enough characters at the end of input at REQUEST_URI. [file "blahblah/base_rules/modsecurity_crs_20_protocol_violations.conf"] [line "461"] [id "950107"] [rev "2"] [msg "URL Encoding Abuse Attack Attempt"] [severity "WARNING"] [ver "OWASP_CRS/2.2.9"] [maturity "6"] [accuracy "8"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/EVASION"] [hostname "server.example.com"] [uri "/index.html"] [unique_id "VdygDwooaRUAAEU8FUYAAACW"]

SecRuleEngine DetectionOnly:
[Tue Aug 25 18:05:26.156386 2015] [:error] [pid 124:tid 123456] [client 10.40.123.456] ModSecurity: Warning. Invalid URL Encoding: Not enough characters at the end of input at REQUEST_URI. [file "blahblah/base_rules/modsecurity_crs_20_protocol_violations.conf"] [line "461"] [id "950107"] [rev "2"] [msg "URL Encoding Abuse Attack Attempt"] [severity "WARNING"] [ver "OWASP_CRS/2.2.9"] [maturity "6"] [accuracy "8"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/EVASION"] [hostname "www.example.com"] [uri "/index.html"] [unique_id "VdygVgooaRUAAASs0ZoAAACD"]

I believe that gives all the information you need (the client ip address, the rule that fired, whether access was denied (and with what return code) or it's just a Warning...etc.). Now the exact logging depends on the rule definition, and as you can see the OWASP rules have lots of information.

As to your other questions:

Be vary of LocationMatch for ModSecurity rule filtering as that runs AFTER phase 1 rules, so any filters in there won't work for phase 1 rules. See my answer here for more details on that: http://stackoverflow.com/questions/32041482/apache-locationmatch-wildcard-for-modsecurity-on-wordpress-site/32067850#32067850

In regards to reducing requests that need logged into collections, if you're are SOAP requests then probably not much you can do here, since they will all be application requests and so shouldn't bypass ModSecurity as presumably you put this in place to protect that application. If you were running a web server, then there are rules you can use to weed out static requests and reduce load on ModSecurity and some rules are in modsecurity_crs10_ignore_static.conf but, as I mentioned in my first reply, I think they could use a little tweaking and happy to explain how if anyone wants to know but don't want to divert from this thread too much.

Thanks,
Barry



----------------------------------------
> To: mod...@li...
> From: rai...@ki...
> Date: Tue, 25 Aug 2015 14:38:47 +0200
> Subject: Re: [mod-security-users] Collections_remove_stale: Failed deleting collection
>
> Am 25.08.2015 um 12:02 schrieb Sophie Loewenthal:
>> Hi Rainer ,
>>
>>>> Since 509 is very specific, why not taking the IP from the normal
>> access log of the web server?
>> Mod_Sec is running in Detection Only mode. Would this be logged in the
>> access log?
>
> Any web server allows to activate an access log. Typically it is already
> active by default. The contents of the log are adjustable but again the
> IP and the HTTP status code should be there by default.
>
> The access log contains one line for each http request that was handled
> by the web server. So you'd have to find the log or identify how to
> activate it for your web server and identify the configured or default
> format of its content. It will then be easy to filter the log for all
> requests that were answered with a 509 and extract the IP addresses.
>
> All of this access log stuff is independent of your mod_security config.
>
> Regards,
>
> Rainer
>
>> On 8/25/2015 11:27 AM, Rainer Jung wrote:
>>> Am 25.08.2015 um 10:47 schrieb Sophie Loewenthal:
>>>> Hi Barry,
>>>>
>>>> Thank you for your well penned reply.
>>>>
>>>> For an quick fix, I have put the directory into a ram disc, and
>>>> shall run some pruning methods from cron.
>>>>
>>>> Your other suggestion require I spend more time on how I should
>>>> differentiate between static and dynamic content, although I doubt any
>>>> static is requested because this is a soap gateway. Looking anyway :)
>>>>
>>>> How else could I reduce modsec traffic? If I could wrap this code
>>>> up into a LocationMatch and place inside a vhost entry, maybe this could
>>>> help.
>>>>
>>>> Management would like a list of potential offenders by IP.
>>>> How could I adapt this code to add logging of IP and or request into a
>>>> file? I looked at SecAuditLogParts and enabling everything I could not
>>>> see of it hit a rule. Currently this runs in detection mode.
>>>> I have logging enabled on this rule:
>>>> SecRule IP:SOMEPATHCOUNTER "@gt 120"
>>>> "phase:2,pause:100,deny,status:509,setenv:RATELIMITED,skip:1,log,id:10000003,msg:Sophie_10000003"
>>>> But would like an IP address logged when it was sent a 509 status message.
>>>> Still reading
>>>> https://www.feistyduck.com/library/modsecurity-handbook-free/online/ch04-logging.html
>>>> !
>>> Since 509 is very specific, why not taking the IP from the normal access
>>> log of the web server?
>>>
>>> Regardsco
>>>
>>> Rainer
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/