Re: [mod-security-users] (no subject)

[Christian B. <ch...@jw...>]

Am 10.07.2009 um 17:26 schrieb clau .:

> Hello
>
> I just got this in my modsec-audit. It is stopping google bot? If  
> not why does it appears here?
>
> thanks.
>
>
> --400c0234-B--
> GET /robots.txt HTTP/1.1
> Host: www.******.***
> Connection: Keep-alive
> Accept: text/plain,text/html
> From: googlebot(at)googlebot.com
> User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
> Accept-Encoding: gzip,deflate
>
> --400c0234-F--
> HTTP/1.1 200 OK
> Last-Modified: Wed, 04 Mar 2009 06:29:30 GMT
> Accept-Ranges: bytes
> Vary: Accept-Encoding
> Content-Encoding: gzip
> Content-Length: 85
> Keep-Alive: timeout=15, max=100
> Connection: Keep-Alive
> Content-Type: text/plain; charset=UTF-8
>
> --400c0234-H--
> Message: Warning. Pattern match "(?:(?:gsa-crawler \(enterprise; s4- 
> e9lj2b82fjjaa; me\@mycompany\.com|adsbot-google \(\+http:\/\/www 
> \.google\.com\/adsbot\.html)\)|\b(?:google(?:-sitemaps|bot)| 
> mediapartners-google)\b)" at REQUEST_HEADERS:User-Agent. [file "/etc/ 
> apache2/conf.d/modsecurity/optional_rules/ 
> modsecurity_crs_55_marketing.conf"] [line "21"] [id "910006"] [msg  
> "Google robot activity"] [severity "NOTICE"]
> Message: Unable to retrieve collection (name "global", key  
> "global"). Use SecDataDir to define data directory first.
> Message: Warning. Operator EQ matched 0 at GLOBAL. [file "/******/ 
> modsecurity_crs_30_http_policy.conf"] [line "120"] [id "960903"]  
> [msg "ModSecurity does not support content encodings"] [severity  
> "WARNING"]
> Stopwatch: 1246842095214958 5339 (515 4182 5143)
> Producer: ModSecurity for Apache/2.5.9 (http:// 
> www.modsecurity.org/); core ruleset/1.6.1; core ruleset/1.6.1.
> Server: Apache/2.2.9 (Ubuntu)
>


It does not stop your bot. You can depict this from the response part,  
which has a status of 200
being sent to the client. This usually comes from apache properly  
handling the request (at least,
if you do not explicitly provide a rule for blocking, which delivers a  
response with status 200).

The interesting part of the message appears at the end, saying  
"ModSecurity does not support
content encodings". The client says it will accept compressed  
responses (Accept-Encoding) and
Apache delivers the gzip-compressed page to the client.
When ModSecurity tries to process the response of Apache before it  
finally gets delivered to the
browser, it cannot match its regular expressions, as there is no  
support for un-compressing the
response within ModSecurity.


Best regards,
     Chris