Hi Winfreid,

Having the topic back to the list.

We have an issue open to track the progress on the memcached collection implementation which is available here:
https://github.com/SpiderLabs/ModSecurity/issues/378

As you can see in the issue, we have a experimental version of it available here:
https://github.com/SpiderLabs/ModSecurity/tree/memcache_collections

Br.,
Felipe "Zimmerle" Costa
Security Researcher, SpiderLabs

Trustwave | SMART SECURITY ON DEMAND
www.trustwave.com



On May 6, 2014, at 8:05 AM, Winfried Neessen <neessen@cleverbridge.com> wrote:

Hi Felipe,
 
I was just able to enable the persistent collection for IPs on one of our servers for testing purpose and
the issue happened again. I ran the sdbm-util on the ip.pag file and compared to the functioning file
I worked on yesterday, I can already see a difference.
 
The file I worked on yesterday was about 400MiB big and had approx. 75k entries- as the file was a little
older, the entries were all expired already. The file I have now is 1GiB big but when I run the sdbm-util
on it, (e. g. with the s parameter) it only shows 4 elements. The file from yesterday had a fragmentation
rate of 100% whereas the current file has 25% FR. The webserver ran from this morning to this noon, so
there should be at least 50k elements in the collection.
 
Here is the output of the sdbm-util with s v parameters:
 
$ LD_LIBRARY_PATH=/www/lib ./modsec-sdbm-util -v -s ./ip.pag                                                                                                                                                                    
Opening file: ./ip.pag                                                                                                                                                                                                                                                             
modsec-sdbm-util.c:57:open_sdbm(): Trying to open: ./ip.pag                                                                                                                                                                                                                       
modsec-sdbm-util.c:70:open_sdbm(): Trying to open: ./ip                                                                                                                                                                                                                           
modsec-sdbm-util.c:84:open_sdbm(): File opened.                                                                                                                                                                                                                                   
Database ready to be used.                                                                                                                                                                                                                                                        
modsec-sdbm-util.c:200:dump_database(): Showing some status about the databases...                                                                                                                                                                                                
modsec-sdbm-util.c:157:modsec_unpack(): Expired: 1399373780, 1399373695 delta: -85                                                                                                                                                                                                
modsec-sdbm-util.c:157:modsec_unpack(): Expired: 1399373797, 1399373695 delta: -102                                                                                                                                                                                                
modsec-sdbm-util.c:157:modsec_unpack(): Expired: 1399373024, 1399373695 delta: 671                                                                                                                                                                                                
modsec-sdbm-util.c:157:modsec_unpack(): Expired: 1399373815, 1399373695 delta: -120                                                                                                                                                                                               
modsec-sdbm-util.c:297:dump_database(): Failed to retrieve the next key.                 
 
I saw that "Failed to retrieve the next key." notice for the file I worked with yesterday as
well, but only after it showed at least 75k elements.
 
Does this information help you to help me troubleshoot this issue?
 
 
Thanks
Winfried
 
 
From: Felipe Costa [mailto:FCosta@trustwave.com] 
Sent: Wednesday, April 16, 2014 7:39 PM
To: <mod-security-users@lists.sourceforge.net>; Winfried Neessen
Subject: Re: [mod-security-users] Locking issue with enabled persistent collection
 
Hi Winfried,
 
On Apr 16, 2014, at 5:41 AM, Winfried Neessen <neessen@cleverbridge.com> wrote:
 Once the IP collection is enabled in the ruleset (in addtion to the blocking rules), the server still runs
fine at least for a couple of hours. But after approx. 5-12 hours the logs begin to throw messages
 
There are some issues opened on GitHub related to similar problems. It seems that this problem is trigged in specific scenarios, so thanks for your detailed report, it is valuable. 
 
Low values for SecCollectionTimeout (https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual#SecCollectionTimeout) may help you to minimize it.
 
While debugging those issues we have created the modsec-sdbm-util, which is able to open the collection file and interpret its content as ModSecurity does. 
The utility can be downloaded here: https://github.com/SpiderLabs/modsec-sdbm-util. The full list of functionalities: https://github.com/SpiderLabs/modsec-sdbm-util/blob/master/README.md
 
Similar problems have been reported in the following issues:
 
We are also working in alternatives to SDBM:
 
Br.,
Felipe "Zimmerle" Costa
Security Researcher, SpiderLabs
 
Trustwave | SMART SECURITY ON DEMAND
 
 
 


This transmission may contain information that is privileged, confidential, and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is strictly prohibited. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format.




This transmission may contain information that is privileged, confidential, and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is strictly prohibited. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format.