Thread: [Mod-security-developers] CRS DoS protection & x-forwarded-for header

Brought to you by: victorhora, zimmerletw

mod-security-developers

[Mod-security-developers] CRS DoS protection & x-forwarded-for header

From: Oleg G. <ole...@ya...> - 2011-04-28 23:33:03

Attachments: modsecurity_crs_10_config.frd.conf modsecurity_crs_11_dos_protection.frd.conf

I'm not sure if I can discuss CRS rules here. If not, please let me know what 
the right place is. I want to suggest an improvement to DoS protection in CRS 
2.1.2. The problem is that enterprise applications usually run behind load 
balancers, so relying on remote_addr doesn't make too much sense, because you'll 
always have an LB's IP in there. 


My improved rules (attached)  check for x-forwarded-for header and if it's 
present, this IP will be used to initialize IP collection. If it's not then the 
old logic will be used.

It would be great if we can include this improvement to the next CRS release.

Thanks,
Oleg.

Re: [Mod-security-developers] CRS DoS protection & x-forwarded-for header

From: Ryan B. <RBa...@tr...> - 2011-04-28 23:41:27

Thanks for the updates Oleg!  This will certainly be a useful update to
not only the DoS rules buy any rules that will be based on the client IP.
I will actually go back to check other uses of REMOTE_ADDR and see if we
can swap it for tx.real_ip instead.

I will add this to the CRS v2.2.0 that I am working on.

For future reference - here is the OWASP CRS mail-list -

https://lists.owasp.org/mailman/listinfo/owasp-modsecurity-core-rule-set

--
Ryan Barnett
Senior Security Researcher
Trustwave - SpiderLabs




On 4/28/11 7:32 PM, "Oleg Gryb" <ole...@ya...> wrote:

>I'm not sure if I can discuss CRS rules here. If not, please let me know
>what 
>the right place is. I want to suggest an improvement to DoS protection in
>CRS 
>2.1.2. The problem is that enterprise applications usually run behind
>load 
>balancers, so relying on remote_addr doesn't make too much sense, because
>you'll 
>always have an LB's IP in there.
>
>
>My improved rules (attached)  check for x-forwarded-for header and if
>it's 
>present, this IP will be used to initialize IP collection. If it's not
>then the 
>old logic will be used.
>
>It would be great if we can include this improvement to the next CRS
>release.
>
>Thanks,
>Oleg.

Re: [Mod-security-developers] CRS DoS protection & x-forwarded-for header

From: Oleg G. <ole...@ya...> - 2011-04-29 00:19:46

I've just realized that there might be a problem with relying on that header: if 
an attacker intentionally sends different random IPs in there, DoS protection 
can be efficiently by-passed. In my case it should not happen, because an LB is 
the one who adds the header, but in general we should warn engineers about the 
possible exploit.


Actually, even in LB case: if a request has already had the header, LB will 
create a new one with the existing value appended to the client IP:

x-forwarded-for: real-client-ip, whatever-client-sent-to-LB

It means that we would need to rely on the the first IP in the list, everything 
else should be considered as untrusted.

Thanks,
Oleg.


----- Original Message ----
> From: Ryan Barnett <RBa...@tr...>
> To: Oleg Gryb <ol...@gr...>; "mod...@li..." 
><mod...@li...>
> Cc: "owa...@li..." 
><owa...@li...>
> Sent: Thu, April 28, 2011 4:41:14 PM
> Subject: Re: [Mod-security-developers] CRS DoS protection & x-forwarded-for 
>header
> 
> Thanks for the updates Oleg!  This will certainly be a useful update  to
> not only the DoS rules buy any rules that will be based on the client  IP.
> I will actually go back to check other uses of REMOTE_ADDR and see if  we
> can swap it for tx.real_ip instead.
> 
> I will add this to the CRS  v2.2.0 that I am working on.
> 
> For future reference - here is the OWASP CRS  mail-list -
> 
> https://lists.owasp.org/mailman/listinfo/owasp-modsecurity-core-rule-set
> 
> --
> Ryan  Barnett
> Senior Security Researcher
> Trustwave -  SpiderLabs
> 
> 
> 
> 
> On 4/28/11 7:32 PM, "Oleg Gryb" <ole...@ya...>  wrote:
> 
> >I'm not sure if I can discuss CRS rules here. If not, please  let me know
> >what 
> >the right place is. I want to suggest an  improvement to DoS protection in
> >CRS 
> >2.1.2. The problem is that  enterprise applications usually run behind
> >load 
> >balancers, so  relying on remote_addr doesn't make too much sense, because
> >you'll 
> >always have an LB's IP in there.
> >
> >
> >My improved  rules (attached)  check for x-forwarded-for header and if
> >it's 
> >present, this IP will be used to initialize IP collection. If it's  not
> >then the 
> >old logic will be used.
> >
> >It would be  great if we can include this improvement to the next  CRS
> >release.
> >
> >Thanks,
> >Oleg.
> 
> 
> 
> ------------------------------------------------------------------------------
> WhatsUp  Gold - Download Free Network Management Software
> The most intuitive,  comprehensive, and cost-effective network 
> management toolset available  today.  Delivers lowest initial 
> acquisition cost and overall TCO of any  competing  solution.
> http://p.sf.net/sfu/whatsupgold-sd
> _______________________________________________
> mod-security-developers  mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-developers
> ModSecurity  Services from Trustwave's SpiderLabs:
> https://www.trustwave.com/spiderLabs.php
>

Re: [Mod-security-developers] CRS DoS protection & x-forwarded-for header

From: Ryan B. <RBa...@tr...> - 2011-05-05 14:46:29

FYI - updated the CRS 10 config file to add in this logic and uploaded it to SVN -
http://mod-security.svn.sourceforge.net/viewvc/mod-security/crs/trunk/modsecurity_crs_10_config.conf.example?revision=1772

#
# -=[ Global and IP Collections ]=-
#
# Create both Global and IP collections for rules to use
# There are some CRS rules that assume that these two collections
# have already been initiated.
#
SecRule REQUEST_HEADERS:User-Agent "^(.*)$" "phase:1,id:'981217',t:none,pass,nolog,t:sha1,t:hexEncode,setvar:tx.ua_hash=%{matched_var}"
SecRule REQUEST_HEADERS:x-forwarded-for "^\b(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\b" "phase:1,id:'981225',t:none,pass,nolog,capture,setvar:tx.real_ip=%{tx.1}"
SecRule &TX:REAL_IP "!@eq 0" "phase:1,id:'981226',t:none,pass,nolog,initcol:global=global,initcol:ip=%{tx.real_ip}_%{tx.ua_hash}"
SecRule &TX:REAL_IP "@eq 0"  "phase:1,id:'981218',t:none,pass,nolog,initcol:global=global,initcol:ip=%{remote_addr}_%{tx.ua_hash}"

The new rules will grab the first IP address listed in an X-Forwared-For header and use that for the IP collection key.  If X-Forwarded-For is not present, then it will use REMOTE_ADDR.

Thanks for the suggestion!

-Ryan

On 4/28/11 8:19 PM, "Oleg Gryb" <ole...@ya...<mailto:ole...@ya...>> wrote:

I've just realized that there might be a problem with relying on that header: if
an attacker intentionally sends different random IPs in there, DoS protection
can be efficiently by-passed. In my case it should not happen, because an LB is
the one who adds the header, but in general we should warn engineers about the
possible exploit.

Actually, even in LB case: if a request has already had the header, LB will
create a new one with the existing value appended to the client IP:

x-forwarded-for: real-client-ip, whatever-client-sent-to-LB

It means that we would need to rely on the the first IP in the list, everything
else should be considered as untrusted.

Thanks,
Oleg.

----- Original Message ----
From: Ryan Barnett <RBa...@tr...<mailto:RBa...@tr...>>
To: Oleg Gryb <ol...@gr...<mailto:ol...@gr...>>; "mod...@li...<mailto:mod...@li...>"
<mod...@li...<mailto:mod...@li...>>
Cc: "owa...@li...<mailto:owa...@li...>"
<owa...@li...<mailto:owa...@li...>>
Sent: Thu, April 28, 2011 4:41:14 PM
Subject: Re: [Mod-security-developers] CRS DoS protection & x-forwarded-for
header
Thanks for the updates Oleg!  This will certainly be a useful update  to
not only the DoS rules buy any rules that will be based on the client  IP.
I will actually go back to check other uses of REMOTE_ADDR and see if  we
can swap it for tx.real_ip instead.
I will add this to the CRS  v2.2.0 that I am working on.
For future reference - here is the OWASP CRS  mail-list -
https://lists.owasp.org/mailman/listinfo/owasp-modsecurity-core-rule-set
--
Ryan  Barnett
Senior Security Researcher
Trustwave -  SpiderLabs
On 4/28/11 7:32 PM, "Oleg Gryb" <ole...@ya...<mailto:ole...@ya...>>  wrote:
>I'm not sure if I can discuss CRS rules here. If not, please  let me know
>what
>the right place is. I want to suggest an  improvement to DoS protection in
>CRS
>2.1.2. The problem is that  enterprise applications usually run behind
>load
>balancers, so  relying on remote_addr doesn't make too much sense, because
>you'll
>always have an LB's IP in there.
>
>
>My improved  rules (attached)  check for x-forwarded-for header and if
>it's
>present, this IP will be used to initialize IP collection. If it's  not
>then the
>old logic will be used.
>
>It would be  great if we can include this improvement to the next  CRS
>release.
>
>Thanks,
>Oleg.
------------------------------------------------------------------------------
WhatsUp  Gold - Download Free Network Management Software
The most intuitive,  comprehensive, and cost-effective network
management toolset available  today.  Delivers lowest initial
acquisition cost and overall TCO of any  competing  solution.
http://p.sf.net/sfu/whatsupgold-sd
_______________________________________________
mod-security-developers  mailing list
mod...@li...<mailto:mod...@li...>
https://lists.sourceforge.net/lists/listinfo/mod-security-developers
ModSecurity  Services from Trustwave's SpiderLabs:
https://www.trustwave.com/spiderLabs.php

------------------------------------------------------------------------------
WhatsUp Gold - Download Free Network Management Software
The most intuitive, comprehensive, and cost-effective network
management toolset available today.  Delivers lowest initial
acquisition cost and overall TCO of any competing solution.
http://p.sf.net/sfu/whatsupgold-sd
_______________________________________________
mod-security-developers mailing list
mod...@li...<mailto:mod...@li...>
https://lists.sourceforge.net/lists/listinfo/mod-security-developers
ModSecurity Services from Trustwave's SpiderLabs:
https://www.trustwave.com/spiderLabs.php

________________________________
This transmission may contain information that is privileged, confidential, and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format.

Re: [Mod-security-developers] CRS DoS protection & x-forwarded-for header

From: Oleg G. <ole...@ya...> - 2011-05-06 17:16:15

Ryan,

Thanks for the update. It might have problems with IPv6 though. Here are my rules: they search for ',' and disregard the IP structure:


SecRule REQUEST_HEADERS:User-Agent "^(.*)$" "phase:1,t:none,pass,nolog,t:sha1,t:hexEncode,setvar:tx.ua_hash=%{matched_var}"
SecRule REQUEST_HEADERS:x-forwarded-for "^(.*)$" "phase:1,t:none,pass,nolog,setvar:tx.real_ip=%{matched_var}"
SecRule TX:REAL_IP "^(.*?),.*$" "phase:1,t:none,pass,nolog,capture,setvar:tx.real_ip=%{TX.1}"

SecRule &TX:REAL_IP "@eq 0"  "phase:1,t:none,pass,nolog,initcol:global=global,initcol:ip=%{remote_addr}_%{tx.ua_hash},,setvar:tx.real_ip=%{remote_addr}"
SecRule &TX:REAL_IP "!@eq 0" "phase:1,t:none,pass,nolog,initcol:global=global,initcol:ip=%{tx.real_ip}_%{tx.ua_hash}"

Thanks,
Oleg.

--- On Thu, 5/5/11, Ryan Barnett <RBa...@tr...> wrote:

> From: Ryan Barnett <RBa...@tr...>
> Subject: Re: [Mod-security-developers] CRS DoS protection & x-forwarded-for header
> To: "Oleg Gryb" <ol...@gr...>, "mod...@li..." <mod...@li...>
> Cc: "owa...@li..." <owa...@li...>
> Date: Thursday, May 5, 2011, 10:46 AM
> FYI - updated the CRS 10 config file
> to add in this logic and uploaded it to SVN -
> http://mod-security.svn.sourceforge.net/viewvc/mod-security/crs/trunk/modsecurity_crs_10_config.conf.example?revision=1772
> 
> #
> # -=[ Global and IP Collections ]=-
> #
> # Create both Global and IP collections for rules to use
> # There are some CRS rules that assume that these two
> collections
> # have already been initiated.
> #
> SecRule REQUEST_HEADERS:User-Agent "^(.*)$"
> "phase:1,id:'981217',t:none,pass,nolog,t:sha1,t:hexEncode,setvar:tx.ua_hash=%{matched_var}"
> SecRule REQUEST_HEADERS:x-forwarded-for
> "^\b(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\b"
> "phase:1,id:'981225',t:none,pass,nolog,capture,setvar:tx.real_ip=%{tx.1}"
> SecRule &TX:REAL_IP "!@eq 0"
> "phase:1,id:'981226',t:none,pass,nolog,initcol:global=global,initcol:ip=%{tx.real_ip}_%{tx.ua_hash}"
> SecRule &TX:REAL_IP "@eq 0" 
> "phase:1,id:'981218',t:none,pass,nolog,initcol:global=global,initcol:ip=%{remote_addr}_%{tx.ua_hash}"
> 
> The new rules will grab the first IP address listed in an
> X-Forwared-For header and use that for the IP collection
> key.  If X-Forwarded-For is not present, then it will
> use REMOTE_ADDR.
> 
> Thanks for the suggestion!
> 
> -Ryan
> 
> 
> On 4/28/11 8:19 PM, "Oleg Gryb" <ole...@ya...<mailto:ole...@ya...>>
> wrote:
> 
> I've just realized that there might be a problem with
> relying on that header: if
> an attacker intentionally sends different random IPs in
> there, DoS protection
> can be efficiently by-passed. In my case it should not
> happen, because an LB is
> the one who adds the header, but in general we should warn
> engineers about the
> possible exploit.
> 
> 
> Actually, even in LB case: if a request has already had the
> header, LB will
> create a new one with the existing value appended to the
> client IP:
> 
> x-forwarded-for: real-client-ip,
> whatever-client-sent-to-LB
> 
> It means that we would need to rely on the the first IP in
> the list, everything
> else should be considered as untrusted.
> 
> Thanks,
> Oleg.
> 
> 
> ----- Original Message ----
> From: Ryan Barnett <RBa...@tr...<mailto:RBa...@tr...>>
> To: Oleg Gryb <ol...@gr...<mailto:ol...@gr...>>;
> "mod...@li...<mailto:mod...@li...>"
> <mod...@li...<mailto:mod...@li...>>
> Cc: "owa...@li...<mailto:owa...@li...>"
> <owa...@li...<mailto:owa...@li...>>
> Sent: Thu, April 28, 2011 4:41:14 PM
> Subject: Re: [Mod-security-developers] CRS DoS protection
> & x-forwarded-for
> header
> Thanks for the updates Oleg!  This will certainly be a
> useful update  to
> not only the DoS rules buy any rules that will be based on
> the client  IP.
> I will actually go back to check other uses of REMOTE_ADDR
> and see if  we
> can swap it for tx.real_ip instead.
> I will add this to the CRS  v2.2.0 that I am working
> on.
> For future reference - here is the OWASP CRS 
> mail-list -
> https://lists.owasp.org/mailman/listinfo/owasp-modsecurity-core-rule-set
> --
> Ryan  Barnett
> Senior Security Researcher
> Trustwave -  SpiderLabs
> On 4/28/11 7:32 PM, "Oleg Gryb" <ole...@ya...<mailto:ole...@ya...>> 
> wrote:
> >I'm not sure if I can discuss CRS rules here. If not,
> please  let me know
> >what
> >the right place is. I want to suggest an 
> improvement to DoS protection in
> >CRS
> >2.1.2. The problem is that  enterprise
> applications usually run behind
> >load
> >balancers, so  relying on remote_addr doesn't make
> too much sense, because
> >you'll
> >always have an LB's IP in there.
> >
> >
> >My improved  rules (attached)  check for
> x-forwarded-for header and if
> >it's
> >present, this IP will be used to initialize IP
> collection. If it's  not
> >then the
> >old logic will be used.
> >
> >It would be  great if we can include this
> improvement to the next  CRS
> >release.
> >
> >Thanks,
> >Oleg.
> ------------------------------------------------------------------------------
> WhatsUp  Gold - Download Free Network Management
> Software
> The most intuitive,  comprehensive, and cost-effective
> network
> management toolset available  today.  Delivers
> lowest initial
> acquisition cost and overall TCO of any 
> competing  solution.
> http://p.sf.net/sfu/whatsupgold-sd
> _______________________________________________
> mod-security-developers  mailing list
> mod...@li...<mailto:mod...@li...>
> https://lists.sourceforge.net/lists/listinfo/mod-security-developers
> ModSecurity  Services from Trustwave's SpiderLabs:
> https://www.trustwave.com/spiderLabs.php
> 
> ------------------------------------------------------------------------------
> WhatsUp Gold - Download Free Network Management Software
> The most intuitive, comprehensive, and cost-effective
> network
> management toolset available today.  Delivers lowest
> initial
> acquisition cost and overall TCO of any competing
> solution.
> http://p.sf.net/sfu/whatsupgold-sd
> _______________________________________________
> mod-security-developers mailing list
> mod...@li...<mailto:mod...@li...>
> https://lists.sourceforge.net/lists/listinfo/mod-security-developers
> ModSecurity Services from Trustwave's SpiderLabs:
> https://www.trustwave.com/spiderLabs.php
> 
> 
> 
> ________________________________
> This transmission may contain information that is
> privileged, confidential, and/or exempt from disclosure
> under applicable law. If you are not the intended recipient,
> you are hereby notified that any disclosure, copying,
> distribution, or use of the information contained herein
> (including any reliance thereon) is STRICTLY PROHIBITED. If
> you received this transmission in error, please immediately
> contact the sender and destroy the material in its entirety,
> whether in electronic or hard copy format.
> 
>

Re: [Mod-security-developers] CRS DoS protection & x-forwarded-for header

From: Ryan B. <RBa...@tr...> - 2011-05-06 17:30:06

Good point about IPv6.  Looks like we could still do this using the same #
of SecRules and just not using an IP regex like this -

SecRule REQUEST_HEADERS:User-Agent "^(.*)$"
"phase:1,t:none,pass,nolog,t:sha1,t:hexEncode,setvar:tx.ua_hash=%{matched_v
ar}"
SecRule REQUEST_HEADERS:x-forwarded-for "^(.*?),?.*$"
"phase:1,t:none,pass,nolog,setvar:tx.real_ip=%{matched_var}"

SecRule &TX:REAL_IP "@eq 0"
"phase:1,t:none,pass,nolog,initcol:global=global,initcol:ip=%{remote_addr}_
%{tx.ua_hash},,setvar:tx.real_ip=%{remote_addr}"
SecRule &TX:REAL_IP "!@eq 0"
"phase:1,t:none,pass,nolog,initcol:global=global,initcol:ip=%{tx.real_ip}_%
{tx.ua_hash}"



--
Ryan



On 5/6/11 1:16 PM, "Oleg Gryb" <ole...@ya...> wrote:

>Ryan,
>
>Thanks for the update. It might have problems with IPv6 though. Here are
>my rules: they search for ',' and disregard the IP structure:
>
>
>SecRule REQUEST_HEADERS:User-Agent "^(.*)$"
>"phase:1,t:none,pass,nolog,t:sha1,t:hexEncode,setvar:tx.ua_hash=%{matched_
>var}"
>SecRule REQUEST_HEADERS:x-forwarded-for "^(.*)$"
>"phase:1,t:none,pass,nolog,setvar:tx.real_ip=%{matched_var}"
>SecRule TX:REAL_IP "^(.*?),.*$"
>"phase:1,t:none,pass,nolog,capture,setvar:tx.real_ip=%{TX.1}"
>
>SecRule &TX:REAL_IP "@eq 0"
>"phase:1,t:none,pass,nolog,initcol:global=global,initcol:ip=%{remote_addr}
>_%{tx.ua_hash},,setvar:tx.real_ip=%{remote_addr}"
>SecRule &TX:REAL_IP "!@eq 0"
>"phase:1,t:none,pass,nolog,initcol:global=global,initcol:ip=%{tx.real_ip}_
>%{tx.ua_hash}"
>
>Thanks,
>Oleg.
>
>--- On Thu, 5/5/11, Ryan Barnett <RBa...@tr...> wrote:
>
>> From: Ryan Barnett <RBa...@tr...>
>> Subject: Re: [Mod-security-developers] CRS DoS protection &
>>x-forwarded-for header
>> To: "Oleg Gryb" <ol...@gr...>,
>>"mod...@li..."
>><mod...@li...>
>> Cc: "owa...@li..."
>><owa...@li...>
>> Date: Thursday, May 5, 2011, 10:46 AM
>> FYI - updated the CRS 10 config file
>> to add in this logic and uploaded it to SVN -
>>
>>http://mod-security.svn.sourceforge.net/viewvc/mod-security/crs/trunk/mod
>>security_crs_10_config.conf.example?revision=1772
>>
>> #
>> # -=[ Global and IP Collections ]=-
>> #
>> # Create both Global and IP collections for rules to use
>> # There are some CRS rules that assume that these two
>> collections
>> # have already been initiated.
>> #
>> SecRule REQUEST_HEADERS:User-Agent "^(.*)$"
>>
>>"phase:1,id:'981217',t:none,pass,nolog,t:sha1,t:hexEncode,setvar:tx.ua_ha
>>sh=%{matched_var}"
>> SecRule REQUEST_HEADERS:x-forwarded-for
>> "^\b(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\b"
>>
>>"phase:1,id:'981225',t:none,pass,nolog,capture,setvar:tx.real_ip=%{tx.1}"
>> SecRule &TX:REAL_IP "!@eq 0"
>>
>>"phase:1,id:'981226',t:none,pass,nolog,initcol:global=global,initcol:ip=%
>>{tx.real_ip}_%{tx.ua_hash}"
>> SecRule &TX:REAL_IP "@eq 0"
>>
>>"phase:1,id:'981218',t:none,pass,nolog,initcol:global=global,initcol:ip=%
>>{remote_addr}_%{tx.ua_hash}"
>>
>> The new rules will grab the first IP address listed in an
>> X-Forwared-For header and use that for the IP collection
>> key.  If X-Forwarded-For is not present, then it will
>> use REMOTE_ADDR.
>>
>> Thanks for the suggestion!
>>
>> -Ryan
>>
>>
>> On 4/28/11 8:19 PM, "Oleg Gryb"
>><ole...@ya...<mailto:ole...@ya...>>
>> wrote:
>>
>> I've just realized that there might be a problem with
>> relying on that header: if
>> an attacker intentionally sends different random IPs in
>> there, DoS protection
>> can be efficiently by-passed. In my case it should not
>> happen, because an LB is
>> the one who adds the header, but in general we should warn
>> engineers about the
>> possible exploit.
>>
>>
>> Actually, even in LB case: if a request has already had the
>> header, LB will
>> create a new one with the existing value appended to the
>> client IP:
>>
>> x-forwarded-for: real-client-ip,
>> whatever-client-sent-to-LB
>>
>> It means that we would need to rely on the the first IP in
>> the list, everything
>> else should be considered as untrusted.
>>
>> Thanks,
>> Oleg.
>>
>>
>> ----- Original Message ----
>> From: Ryan Barnett
>><RBa...@tr...<mailto:RBa...@tr...>>
>> To: Oleg Gryb <ol...@gr...<mailto:ol...@gr...>>;
>>
>>"mod...@li...<mailto:mod-security-develo
>>pe...@li...>"
>>
>><mod...@li...<mailto:mod-security-develo
>>pe...@li...>>
>> Cc:
>>"owa...@li...<mailto:owasp-modsecurity
>>-cor...@li...>"
>>
>><owa...@li...<mailto:owasp-modsecurity
>>-cor...@li...>>
>> Sent: Thu, April 28, 2011 4:41:14 PM
>> Subject: Re: [Mod-security-developers] CRS DoS protection
>> & x-forwarded-for
>> header
>> Thanks for the updates Oleg!  This will certainly be a
>> useful update  to
>> not only the DoS rules buy any rules that will be based on
>> the client  IP.
>> I will actually go back to check other uses of REMOTE_ADDR
>> and see if  we
>> can swap it for tx.real_ip instead.
>> I will add this to the CRS  v2.2.0 that I am working
>> on.
>> For future reference - here is the OWASP CRS
>> mail-list -
>> https://lists.owasp.org/mailman/listinfo/owasp-modsecurity-core-rule-set
>> --
>> Ryan  Barnett
>> Senior Security Researcher
>> Trustwave -  SpiderLabs
>> On 4/28/11 7:32 PM, "Oleg Gryb"
>><ole...@ya...<mailto:ole...@ya...>>
>> wrote:
>> >I'm not sure if I can discuss CRS rules here. If not,
>> please  let me know
>> >what
>> >the right place is. I want to suggest an
>> improvement to DoS protection in
>> >CRS
>> >2.1.2. The problem is that  enterprise
>> applications usually run behind
>> >load
>> >balancers, so  relying on remote_addr doesn't make
>> too much sense, because
>> >you'll
>> >always have an LB's IP in there.
>> >
>> >
>> >My improved  rules (attached)  check for
>> x-forwarded-for header and if
>> >it's
>> >present, this IP will be used to initialize IP
>> collection. If it's  not
>> >then the
>> >old logic will be used.
>> >
>> >It would be  great if we can include this
>> improvement to the next  CRS
>> >release.
>> >
>> >Thanks,
>> >Oleg.
>>
>>-------------------------------------------------------------------------
>>-----
>> WhatsUp  Gold - Download Free Network Management
>> Software
>> The most intuitive,  comprehensive, and cost-effective
>> network
>> management toolset available  today.  Delivers
>> lowest initial
>> acquisition cost and overall TCO of any
>> competing  solution.
>> http://p.sf.net/sfu/whatsupgold-sd
>> _______________________________________________
>> mod-security-developers  mailing list
>>
>>mod...@li...<mailto:mod-security-develop
>>er...@li...>
>> https://lists.sourceforge.net/lists/listinfo/mod-security-developers
>> ModSecurity  Services from Trustwave's SpiderLabs:
>> https://www.trustwave.com/spiderLabs.php
>>
>>
>>-------------------------------------------------------------------------
>>-----
>> WhatsUp Gold - Download Free Network Management Software
>> The most intuitive, comprehensive, and cost-effective
>> network
>> management toolset available today.  Delivers lowest
>> initial
>> acquisition cost and overall TCO of any competing
>> solution.
>> http://p.sf.net/sfu/whatsupgold-sd
>> _______________________________________________
>> mod-security-developers mailing list
>>
>>mod...@li...<mailto:mod-security-develop
>>er...@li...>
>> https://lists.sourceforge.net/lists/listinfo/mod-security-developers
>> ModSecurity Services from Trustwave's SpiderLabs:
>> https://www.trustwave.com/spiderLabs.php
>>
>>
>>
>> ________________________________
>> This transmission may contain information that is
>> privileged, confidential, and/or exempt from disclosure
>> under applicable law. If you are not the intended recipient,
>> you are hereby notified that any disclosure, copying,
>> distribution, or use of the information contained herein
>> (including any reliance thereon) is STRICTLY PROHIBITED. If
>> you received this transmission in error, please immediately
>> contact the sender and destroy the material in its entirety,
>> whether in electronic or hard copy format.
>>
>>
>


This transmission may contain information that is privileged, confidential, and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format.

Re: [Mod-security-developers] CRS DoS protection & x-forwarded-for header

From: Oleg G. <ole...@ya...> - 2011-05-06 21:41:17

Ryan,
That might not work for a single IP in x-forwarded-for header, becuase 
non-greedy match for (.*?) will return empty string.

Try:

perl -e "'1.2.3.4' =~ /^(.*?),?.*$/; print $1;"

and you'll see what I mean. I'm assuming here that modsecurity uses PCRE.

Thanks,
Oleg.






----- Original Message ----
> From: Ryan Barnett <RBa...@tr...>
> To: "ol...@gr..." <ol...@gr...>; 
>"mod...@li..." 
><mod...@li...>
> Cc: "owa...@li..." 
><owa...@li...>
> Sent: Fri, May 6, 2011 10:29:57 AM
> Subject: Re: [Mod-security-developers] CRS DoS protection & x-forwarded-for 
>header
> 
> Good point about IPv6.  Looks like we could still do this using the same  #
> of SecRules and just not using an IP regex like this -
> 
> SecRule  REQUEST_HEADERS:User-Agent  "^(.*)$"
> "phase:1,t:none,pass,nolog,t:sha1,t:hexEncode,setvar:tx.ua_hash=%{matched_v
> ar}"
> SecRule  REQUEST_HEADERS:x-forwarded-for  "^(.*?),?.*$"
> "phase:1,t:none,pass,nolog,setvar:tx.real_ip=%{matched_var}"
> 
> SecRule  &TX:REAL_IP "@eq  0"
> "phase:1,t:none,pass,nolog,initcol:global=global,initcol:ip=%{remote_addr}_
> %{tx.ua_hash},,setvar:tx.real_ip=%{remote_addr}"
> SecRule  &TX:REAL_IP "!@eq  0"
> "phase:1,t:none,pass,nolog,initcol:global=global,initcol:ip=%{tx.real_ip}_%
> {tx.ua_hash}"
> 
> 
> 
> --
> Ryan
> 
> 
> 
> On  5/6/11 1:16 PM, "Oleg Gryb" <ole...@ya...>  wrote:
> 
> >Ryan,
> >
> >Thanks for the update. It might have  problems with IPv6 though. Here are
> >my rules: they search for ',' and  disregard the IP structure:
> >
> >
> >SecRule  REQUEST_HEADERS:User-Agent  "^(.*)$"
> >"phase:1,t:none,pass,nolog,t:sha1,t:hexEncode,setvar:tx.ua_hash=%{matched_
> >var}"
> >SecRule  REQUEST_HEADERS:x-forwarded-for  "^(.*)$"
> >"phase:1,t:none,pass,nolog,setvar:tx.real_ip=%{matched_var}"
> >SecRule  TX:REAL_IP  "^(.*?),.*$"
> >"phase:1,t:none,pass,nolog,capture,setvar:tx.real_ip=%{TX.1}"
> >
> >SecRule  &TX:REAL_IP "@eq  0"
> >"phase:1,t:none,pass,nolog,initcol:global=global,initcol:ip=%{remote_addr}
> >_%{tx.ua_hash},,setvar:tx.real_ip=%{remote_addr}"
> >SecRule  &TX:REAL_IP "!@eq  0"
> >"phase:1,t:none,pass,nolog,initcol:global=global,initcol:ip=%{tx.real_ip}_
> >%{tx.ua_hash}"
> >
> >Thanks,
> >Oleg.
> >
> >---  On Thu, 5/5/11, Ryan Barnett <RBa...@tr...>  wrote:
> >
> >> From: Ryan Barnett <RBa...@tr...>
> >>  Subject: Re: [Mod-security-developers] CRS DoS protection  &
> >>x-forwarded-for header
> >> To: "Oleg Gryb" <ol...@gr...>,
> >>"mod...@li..."
> >><mod...@li...>
> >>  Cc: "owa...@li..."
> >><owa...@li...>
> >>  Date: Thursday, May 5, 2011, 10:46 AM
> >> FYI - updated the CRS 10  config file
> >> to add in this logic and uploaded it to SVN  -
> >>
> >>http://mod-security.svn.sourceforge.net/viewvc/mod-security/crs/trunk/mod
> >>security_crs_10_config.conf.example?revision=1772
> >>
> >>  #
> >> # -=[ Global and IP Collections ]=-
> >> #
> >> #  Create both Global and IP collections for rules to use
> >> # There are  some CRS rules that assume that these two
> >> collections
> >> #  have already been initiated.
> >> #
> >> SecRule  REQUEST_HEADERS:User-Agent  "^(.*)$"
> >>
> >>"phase:1,id:'981217',t:none,pass,nolog,t:sha1,t:hexEncode,setvar:tx.ua_ha
> >>sh=%{matched_var}"
> >>  SecRule REQUEST_HEADERS:x-forwarded-for
> >>  "^\b(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\b"
> >>
> >>"phase:1,id:'981225',t:none,pass,nolog,capture,setvar:tx.real_ip=%{tx.1}"
> >>  SecRule &TX:REAL_IP "!@eq  0"
> >>
> >>"phase:1,id:'981226',t:none,pass,nolog,initcol:global=global,initcol:ip=%
> >>{tx.real_ip}_%{tx.ua_hash}"
> >>  SecRule &TX:REAL_IP "@eq  0"
> >>
> >>"phase:1,id:'981218',t:none,pass,nolog,initcol:global=global,initcol:ip=%
> >>{remote_addr}_%{tx.ua_hash}"
> >>
> >>  The new rules will grab the first IP address listed in an
> >>  X-Forwared-For header and use that for the IP collection
> >> key.   If X-Forwarded-For is not present, then it will
> >> use  REMOTE_ADDR.
> >>
> >> Thanks for the  suggestion!
> >>
> >> -Ryan
> >>
> >>
> >> On  4/28/11 8:19 PM, "Oleg Gryb"
> >><ole...@ya...<mailto:ole...@ya...>>
> >>  wrote:
> >>
> >> I've just realized that there might be a problem  with
> >> relying on that header: if
> >> an attacker  intentionally sends different random IPs in
> >> there, DoS  protection
> >> can be efficiently by-passed. In my case it should  not
> >> happen, because an LB is
> >> the one who adds the  header, but in general we should warn
> >> engineers about  the
> >> possible exploit.
> >>
> >>
> >> Actually,  even in LB case: if a request has already had the
> >> header, LB  will
> >> create a new one with the existing value appended to  the
> >> client IP:
> >>
> >> x-forwarded-for:  real-client-ip,
> >> whatever-client-sent-to-LB
> >>
> >>  It means that we would need to rely on the the first IP in
> >> the list,  everything
> >> else should be considered as  untrusted.
> >>
> >> Thanks,
> >>  Oleg.
> >>
> >>
> >> ----- Original Message  ----
> >> From: Ryan Barnett
> >><RBa...@tr...<mailto:RBa...@tr...>>
> >>  To: Oleg Gryb <ol...@gr...<mailto:ol...@gr...>>;
> >>
> >>"mod...@li...<mailto:mod-security-develo
> >>pe...@li...>"
> >>
> >><mod...@li...<mailto:mod-security-develo
> >>pe...@li...>>
> >>  Cc:
> >>"owa...@li...<mailto:owasp-modsecurity
> >>-cor...@li...>"
> >>
> >><owa...@li...<mailto:owasp-modsecurity
> >>-cor...@li...>>
> >>  Sent: Thu, April 28, 2011 4:41:14 PM
> >> Subject: Re:  [Mod-security-developers] CRS DoS protection
> >> &  x-forwarded-for
> >> header
> >> Thanks for the updates  Oleg!  This will certainly be a
> >> useful update   to
> >> not only the DoS rules buy any rules that will be based  on
> >> the client  IP.
> >> I will actually go back to check  other uses of REMOTE_ADDR
> >> and see if  we
> >> can swap  it for tx.real_ip instead.
> >> I will add this to the CRS  v2.2.0  that I am working
> >> on.
> >> For future reference - here is the  OWASP CRS
> >> mail-list -
> >> https://lists.owasp.org/mailman/listinfo/owasp-modsecurity-core-rule-set
> >>  --
> >> Ryan  Barnett
> >> Senior Security  Researcher
> >> Trustwave -  SpiderLabs
> >> On 4/28/11 7:32  PM, "Oleg Gryb"
> >><ole...@ya...<mailto:ole...@ya...>>
> >>  wrote:
> >> >I'm not sure if I can discuss CRS rules here. If  not,
> >> please  let me know
> >> >what
> >>  >the right place is. I want to suggest an
> >> improvement to DoS  protection in
> >> >CRS
> >> >2.1.2. The problem is  that  enterprise
> >> applications usually run behind
> >>  >load
> >> >balancers, so  relying on remote_addr doesn't  make
> >> too much sense, because
> >> >you'll
> >>  >always have an LB's IP in there.
> >> >
> >>  >
> >> >My improved  rules (attached)  check  for
> >> x-forwarded-for header and if
> >> >it's
> >>  >present, this IP will be used to initialize IP
> >> collection. If  it's  not
> >> >then the
> >> >old logic will be  used.
> >> >
> >> >It would be  great if we can include  this
> >> improvement to the next  CRS
> >>  >release.
> >> >
> >> >Thanks,
> >>  >Oleg.
> >>
> >>-------------------------------------------------------------------------
> >>-----
> >>  WhatsUp  Gold - Download Free Network Management
> >>  Software
> >> The most intuitive,  comprehensive, and  cost-effective
> >> network
> >> management toolset  available  today.  Delivers
> >> lowest initial
> >>  acquisition cost and overall TCO of any
> >> competing   solution.
> >> http://p.sf.net/sfu/whatsupgold-sd
> >>  _______________________________________________
> >>  mod-security-developers  mailing list
> >>
> >>mod...@li...<mailto:mod-security-develop
> >>er...@li...>
> >> https://lists.sourceforge.net/lists/listinfo/mod-security-developers
> >>  ModSecurity  Services from Trustwave's SpiderLabs:
> >> https://www.trustwave.com/spiderLabs.php
> >>
> >>
> >>-------------------------------------------------------------------------
> >>-----
> >>  WhatsUp Gold - Download Free Network Management Software
> >> The most  intuitive, comprehensive, and cost-effective
> >> network
> >>  management toolset available today.  Delivers lowest
> >>  initial
> >> acquisition cost and overall TCO of any  competing
> >> solution.
> >> http://p.sf.net/sfu/whatsupgold-sd
> >>  _______________________________________________
> >>  mod-security-developers mailing list
> >>
> >>mod...@li...<mailto:mod-security-develop
> >>er...@li...>
> >> https://lists.sourceforge.net/lists/listinfo/mod-security-developers
> >>  ModSecurity Services from Trustwave's SpiderLabs:
> >> https://www.trustwave.com/spiderLabs.php
> >>
> >>
> >>
> >>  ________________________________
> >> This transmission may contain  information that is
> >> privileged, confidential, and/or exempt from  disclosure
> >> under applicable law. If you are not the intended  recipient,
> >> you are hereby notified that any disclosure,  copying,
> >> distribution, or use of the information contained  herein
> >> (including any reliance thereon) is STRICTLY PROHIBITED.  If
> >> you received this transmission in error, please  immediately
> >> contact the sender and destroy the material in its  entirety,
> >> whether in electronic or hard copy  format.
> >>
> >>
> >
> 
> 
> This transmission may contain  information that is privileged, confidential, 
>and/or exempt from disclosure  under applicable law. If you are not the intended 
>recipient, you are hereby  notified that any disclosure, copying, distribution, 
>or use of the information  contained herein (including any reliance thereon) is 
>STRICTLY PROHIBITED. If you  received this transmission in error, please 
>immediately contact the sender and  destroy the material in its entirety, 
>whether in electronic or hard copy  format.
> 
>

Re: [Mod-security-developers] CRS DoS protection & x-forwarded-for header

From: Ryan B. <RBa...@tr...> - 2011-05-07 00:26:59

This one seems to work in ModSecurity (verified with the debug log).  I
matches a single IP, or the 1st one when there a multiple and it handles
IPv6 addresses as well -

SecRule  REQUEST_HEADERS:User-Agent  "^(.*)$" \
        "phase:1,t:none,pass,nolog,t:sha1,t:hexEncode,setvar:tx.ua_hash=%{matched_
var}"
SecRule  REQUEST_HEADERS:x-forwarded-for "^([0-9a-fA-F\.\:]*),?.*$" \
        "phase:1,t:none,pass,nolog,capture,setvar:tx.real_ip=%{tx.1}
SecRule  &TX:REAL_IP "@eq  0" \
        "phase:1,t:none,pass,nolog,initcol:global=global,initcol:ip=%{remote_addr}
_%{tx.ua_hash},,setvar:tx.real_ip=%{remote_addr}"
SecRule  &TX:REAL_IP "!@eq  0" \
        "phase:1,t:none,pass,nolog,initcol:global=global,initcol:ip=%{tx.real_ip}_
%{tx.ua_hash}"



By the way - with regards to the pcre matching, switch your single/double
quotes around to test -

perl -e '"1.2.3.4" =~ m/^([0-9a-fA-F\.\:]*),?.*$/; print "$1\n";'

--
Ryan Barnett
Senior Security Researcher
Trustwave - SpiderLabs
Email: rba...@tr...
Phone: (703) 794-2248
Cell: (571) 382-0476
www.trustwave.com

This transmission may contain information that is privileged,
confidential, and/or exempt from disclosure under applicable law. If you
are not the intended recipient, you are hereby notified that any
disclosure, copying, distribution, or use of the information contained
herein (including any reliance thereon) is STRICTLY PROHIBITED. If you
received this transmission in error, please immediately contact the sender
and destroy the material in its entirety, whether in electronic or hard
copy format. Thank you.



On 5/6/11 5:41 PM, "Oleg Gryb" <ole...@ya...> wrote:

>Ryan,
>That might not work for a single IP in x-forwarded-for header, becuase
>non-greedy match for (.*?) will return empty string.
>
>Try:
>
>perl -e "'1.2.3.4' =~ /^(.*?),?.*$/; print $1;"
>
>and you'll see what I mean. I'm assuming here that modsecurity uses PCRE.
>
>Thanks,
>Oleg.
>
>
>
>
>
>
>----- Original Message ----
>> From: Ryan Barnett <RBa...@tr...>
>> To: "ol...@gr..." <ol...@gr...>;
>>"mod...@li..."
>><mod...@li...>
>> Cc: "owa...@li..."
>><owa...@li...>
>> Sent: Fri, May 6, 2011 10:29:57 AM
>> Subject: Re: [Mod-security-developers] CRS DoS protection &
>>x-forwarded-for
>>header
>>
>> Good point about IPv6.  Looks like we could still do this using the
>>same  #
>> of SecRules and just not using an IP regex like this -
>>
>> SecRule  REQUEST_HEADERS:User-Agent  "^(.*)$"
>>
>>"phase:1,t:none,pass,nolog,t:sha1,t:hexEncode,setvar:tx.ua_hash=%{matched
>>_v
>> ar}"
>> SecRule  REQUEST_HEADERS:x-forwarded-for  "^(.*?),?.*$"
>> "phase:1,t:none,pass,nolog,setvar:tx.real_ip=%{matched_var}"
>>
>> SecRule  &TX:REAL_IP "@eq  0"
>>
>>"phase:1,t:none,pass,nolog,initcol:global=global,initcol:ip=%{remote_addr
>>}_
>> %{tx.ua_hash},,setvar:tx.real_ip=%{remote_addr}"
>> SecRule  &TX:REAL_IP "!@eq  0"
>>
>>"phase:1,t:none,pass,nolog,initcol:global=global,initcol:ip=%{tx.real_ip}
>>_%
>> {tx.ua_hash}"
>>
>>
>>
>> --
>> Ryan
>>
>>
>>
>> On  5/6/11 1:16 PM, "Oleg Gryb" <ole...@ya...>  wrote:
>>
>> >Ryan,
>> >
>> >Thanks for the update. It might have  problems with IPv6 though. Here
>>are
>> >my rules: they search for ',' and  disregard the IP structure:
>> >
>> >
>> >SecRule  REQUEST_HEADERS:User-Agent  "^(.*)$"
>>
>>>"phase:1,t:none,pass,nolog,t:sha1,t:hexEncode,setvar:tx.ua_hash=%{matche
>>>d_
>> >var}"
>> >SecRule  REQUEST_HEADERS:x-forwarded-for  "^(.*)$"
>> >"phase:1,t:none,pass,nolog,setvar:tx.real_ip=%{matched_var}"
>> >SecRule  TX:REAL_IP  "^(.*?),.*$"
>> >"phase:1,t:none,pass,nolog,capture,setvar:tx.real_ip=%{TX.1}"
>> >
>> >SecRule  &TX:REAL_IP "@eq  0"
>>
>>>"phase:1,t:none,pass,nolog,initcol:global=global,initcol:ip=%{remote_add
>>>r}
>> >_%{tx.ua_hash},,setvar:tx.real_ip=%{remote_addr}"
>> >SecRule  &TX:REAL_IP "!@eq  0"
>>
>>>"phase:1,t:none,pass,nolog,initcol:global=global,initcol:ip=%{tx.real_ip
>>>}_
>> >%{tx.ua_hash}"
>> >
>> >Thanks,
>> >Oleg.
>> >
>> >---  On Thu, 5/5/11, Ryan Barnett <RBa...@tr...>  wrote:
>> >
>> >> From: Ryan Barnett <RBa...@tr...>
>> >>  Subject: Re: [Mod-security-developers] CRS DoS protection  &
>> >>x-forwarded-for header
>> >> To: "Oleg Gryb" <ol...@gr...>,
>> >>"mod...@li..."
>> >><mod...@li...>
>> >>  Cc: "owa...@li..."
>> >><owa...@li...>
>> >>  Date: Thursday, May 5, 2011, 10:46 AM
>> >> FYI - updated the CRS 10  config file
>> >> to add in this logic and uploaded it to SVN  -
>> >>
>>
>>>>http://mod-security.svn.sourceforge.net/viewvc/mod-security/crs/trunk/m
>>>>od
>> >>security_crs_10_config.conf.example?revision=1772
>> >>
>> >>  #
>> >> # -=[ Global and IP Collections ]=-
>> >> #
>> >> #  Create both Global and IP collections for rules to use
>> >> # There are  some CRS rules that assume that these two
>> >> collections
>> >> #  have already been initiated.
>> >> #
>> >> SecRule  REQUEST_HEADERS:User-Agent  "^(.*)$"
>> >>
>>
>>>>"phase:1,id:'981217',t:none,pass,nolog,t:sha1,t:hexEncode,setvar:tx.ua_
>>>>ha
>> >>sh=%{matched_var}"
>> >>  SecRule REQUEST_HEADERS:x-forwarded-for
>> >>  "^\b(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\b"
>> >>
>>
>>>>"phase:1,id:'981225',t:none,pass,nolog,capture,setvar:tx.real_ip=%{tx.1
>>>>}"
>> >>  SecRule &TX:REAL_IP "!@eq  0"
>> >>
>>
>>>>"phase:1,id:'981226',t:none,pass,nolog,initcol:global=global,initcol:ip
>>>>=%
>> >>{tx.real_ip}_%{tx.ua_hash}"
>> >>  SecRule &TX:REAL_IP "@eq  0"
>> >>
>>
>>>>"phase:1,id:'981218',t:none,pass,nolog,initcol:global=global,initcol:ip
>>>>=%
>> >>{remote_addr}_%{tx.ua_hash}"
>> >>
>> >>  The new rules will grab the first IP address listed in an
>> >>  X-Forwared-For header and use that for the IP collection
>> >> key.   If X-Forwarded-For is not present, then it will
>> >> use  REMOTE_ADDR.
>> >>
>> >> Thanks for the  suggestion!
>> >>
>> >> -Ryan
>> >>
>> >>
>> >> On  4/28/11 8:19 PM, "Oleg Gryb"
>> >><ole...@ya...<mailto:ole...@ya...>>
>> >>  wrote:
>> >>
>> >> I've just realized that there might be a problem  with
>> >> relying on that header: if
>> >> an attacker  intentionally sends different random IPs in
>> >> there, DoS  protection
>> >> can be efficiently by-passed. In my case it should  not
>> >> happen, because an LB is
>> >> the one who adds the  header, but in general we should warn
>> >> engineers about  the
>> >> possible exploit.
>> >>
>> >>
>> >> Actually,  even in LB case: if a request has already had the
>> >> header, LB  will
>> >> create a new one with the existing value appended to  the
>> >> client IP:
>> >>
>> >> x-forwarded-for:  real-client-ip,
>> >> whatever-client-sent-to-LB
>> >>
>> >>  It means that we would need to rely on the the first IP in
>> >> the list,  everything
>> >> else should be considered as  untrusted.
>> >>
>> >> Thanks,
>> >>  Oleg.
>> >>
>> >>
>> >> ----- Original Message  ----
>> >> From: Ryan Barnett
>> >><RBa...@tr...<mailto:RBa...@tr...>>
>> >>  To: Oleg Gryb <ol...@gr...<mailto:ol...@gr...>>;
>> >>
>>
>>>>"mod...@li...<mailto:mod-security-deve
>>>>lo
>> >>pe...@li...>"
>> >>
>>
>>>><mod...@li...<mailto:mod-security-deve
>>>>lo
>> >>pe...@li...>>
>> >>  Cc:
>>
>>>>"owa...@li...<mailto:owasp-modsecuri
>>>>ty
>> >>-cor...@li...>"
>> >>
>>
>>>><owa...@li...<mailto:owasp-modsecuri
>>>>ty
>> >>-cor...@li...>>
>> >>  Sent: Thu, April 28, 2011 4:41:14 PM
>> >> Subject: Re:  [Mod-security-developers] CRS DoS protection
>> >> &  x-forwarded-for
>> >> header
>> >> Thanks for the updates  Oleg!  This will certainly be a
>> >> useful update   to
>> >> not only the DoS rules buy any rules that will be based  on
>> >> the client  IP.
>> >> I will actually go back to check  other uses of REMOTE_ADDR
>> >> and see if  we
>> >> can swap  it for tx.real_ip instead.
>> >> I will add this to the CRS  v2.2.0  that I am working
>> >> on.
>> >> For future reference - here is the  OWASP CRS
>> >> mail-list -
>> >>
>>https://lists.owasp.org/mailman/listinfo/owasp-modsecurity-core-rule-set
>> >>  --
>> >> Ryan  Barnett
>> >> Senior Security  Researcher
>> >> Trustwave -  SpiderLabs
>> >> On 4/28/11 7:32  PM, "Oleg Gryb"
>> >><ole...@ya...<mailto:ole...@ya...>>
>> >>  wrote:
>> >> >I'm not sure if I can discuss CRS rules here. If  not,
>> >> please  let me know
>> >> >what
>> >>  >the right place is. I want to suggest an
>> >> improvement to DoS  protection in
>> >> >CRS
>> >> >2.1.2. The problem is  that  enterprise
>> >> applications usually run behind
>> >>  >load
>> >> >balancers, so  relying on remote_addr doesn't  make
>> >> too much sense, because
>> >> >you'll
>> >>  >always have an LB's IP in there.
>> >> >
>> >>  >
>> >> >My improved  rules (attached)  check  for
>> >> x-forwarded-for header and if
>> >> >it's
>> >>  >present, this IP will be used to initialize IP
>> >> collection. If  it's  not
>> >> >then the
>> >> >old logic will be  used.
>> >> >
>> >> >It would be  great if we can include  this
>> >> improvement to the next  CRS
>> >>  >release.
>> >> >
>> >> >Thanks,
>> >>  >Oleg.
>> >>
>>
>>>>-----------------------------------------------------------------------
>>>>--
>> >>-----
>> >>  WhatsUp  Gold - Download Free Network Management
>> >>  Software
>> >> The most intuitive,  comprehensive, and  cost-effective
>> >> network
>> >> management toolset  available  today.  Delivers
>> >> lowest initial
>> >>  acquisition cost and overall TCO of any
>> >> competing   solution.
>> >> http://p.sf.net/sfu/whatsupgold-sd
>> >>  _______________________________________________
>> >>  mod-security-developers  mailing list
>> >>
>>
>>>>mod...@li...<mailto:mod-security-devel
>>>>op
>> >>er...@li...>
>> >> https://lists.sourceforge.net/lists/listinfo/mod-security-developers
>> >>  ModSecurity  Services from Trustwave's SpiderLabs:
>> >> https://www.trustwave.com/spiderLabs.php
>> >>
>> >>
>>
>>>>-----------------------------------------------------------------------
>>>>--
>> >>-----
>> >>  WhatsUp Gold - Download Free Network Management Software
>> >> The most  intuitive, comprehensive, and cost-effective
>> >> network
>> >>  management toolset available today.  Delivers lowest
>> >>  initial
>> >> acquisition cost and overall TCO of any  competing
>> >> solution.
>> >> http://p.sf.net/sfu/whatsupgold-sd
>> >>  _______________________________________________
>> >>  mod-security-developers mailing list
>> >>
>>
>>>>mod...@li...<mailto:mod-security-devel
>>>>op
>> >>er...@li...>
>> >> https://lists.sourceforge.net/lists/listinfo/mod-security-developers
>> >>  ModSecurity Services from Trustwave's SpiderLabs:
>> >> https://www.trustwave.com/spiderLabs.php
>> >>
>> >>
>> >>
>> >>  ________________________________
>> >> This transmission may contain  information that is
>> >> privileged, confidential, and/or exempt from  disclosure
>> >> under applicable law. If you are not the intended  recipient,
>> >> you are hereby notified that any disclosure,  copying,
>> >> distribution, or use of the information contained  herein
>> >> (including any reliance thereon) is STRICTLY PROHIBITED.  If
>> >> you received this transmission in error, please  immediately
>> >> contact the sender and destroy the material in its  entirety,
>> >> whether in electronic or hard copy  format.
>> >>
>> >>
>> >
>>
>>
>> This transmission may contain  information that is privileged,
>>confidential,
>>and/or exempt from disclosure  under applicable law. If you are not the
>>intended
>>recipient, you are hereby  notified that any disclosure, copying,
>>distribution,
>>or use of the information  contained herein (including any reliance
>>thereon) is
>>STRICTLY PROHIBITED. If you  received this transmission in error, please
>>immediately contact the sender and  destroy the material in its entirety,
>>whether in electronic or hard copy  format.
>>
>>
>


This transmission may contain information that is privileged, confidential, and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format.

Re: [Mod-security-developers] CRS DoS protection & x-forwarded-for header

From: Oleg G. <ole...@ya...> - 2011-05-07 15:13:46

> By  the way - with regards to the pcre matching, switch your  single/double
> quotes around to test -

Here is the difference between greedy and non-greedy match:
ogryb@debian:~$ perl -e '"1.2.3.4" =~ /^(.*),?.*$/; print $1;'
1.2.3.4ogryb@debian:~$ perl -e '"1.2.3.4" =~ /^(.*?),?.*$/; print $1;'
ogryb@debian:~$ 

But your new rule should work because you've enforced the IP characters that 
exclude ',' and got rid of non-greedy match.The most importantly, you've 
achieved your goal of reducing the # of SecRulles :)

One tiny suggestion though:
^([0-9a-fA-F\.\:]+),?.*$

I think, we should treat empty values in the same way as cases when no 
x-forwarded-for header is present.



----- Original Message ----
> From: Ryan Barnett <RBa...@tr...>
> To: Oleg Gryb <ol...@gr...>; "mod...@li..." 
><mod...@li...>
> Cc: "owa...@li..." 
><owa...@li...>
> Sent: Fri, May 6, 2011 5:26:45 PM
> Subject: Re: [Mod-security-developers] CRS DoS protection & x-forwarded-for 
>header
> 
> This one seems to work in ModSecurity (verified with the debug log).   I
> matches a single IP, or the 1st one when there a multiple and it  handles
> IPv6 addresses as well -
> 
> SecRule   REQUEST_HEADERS:User-Agent  "^(.*)$" \
>          
>"phase:1,t:none,pass,nolog,t:sha1,t:hexEncode,setvar:tx.ua_hash=%{matched_
> var}"
> SecRule   REQUEST_HEADERS:x-forwarded-for "^([0-9a-fA-F\.\:]*),?.*$" \
>           "phase:1,t:none,pass,nolog,capture,setvar:tx.real_ip=%{tx.1}
> SecRule   &TX:REAL_IP "@eq  0" \
>          
>"phase:1,t:none,pass,nolog,initcol:global=global,initcol:ip=%{remote_addr}
> _%{tx.ua_hash},,setvar:tx.real_ip=%{remote_addr}"
> SecRule   &TX:REAL_IP "!@eq  0" \
>          
>"phase:1,t:none,pass,nolog,initcol:global=global,initcol:ip=%{tx.real_ip}_
> %{tx.ua_hash}"
> 
> 
> 
> By  the way - with regards to the pcre matching, switch your  single/double
> quotes around to test -
> 
> perl -e '"1.2.3.4" =~  m/^([0-9a-fA-F\.\:]*),?.*$/; print "$1\n";'
> 
> --
> Ryan Barnett
> Senior  Security Researcher
> Trustwave - SpiderLabs
> Email: rba...@tr...
> Phone: (703)  794-2248
> Cell: (571) 382-0476
> www.trustwave.com
> 
> This transmission  may contain information that is privileged,
> confidential, and/or exempt from  disclosure under applicable law. If you
> are not the intended recipient, you  are hereby notified that any
> disclosure, copying, distribution, or use of the  information contained
> herein (including any reliance thereon) is STRICTLY  PROHIBITED. If you
> received this transmission in error, please immediately  contact the sender
> and destroy the material in its entirety, whether in  electronic or hard
> copy format. Thank you.
> 
> 
> 
> On 5/6/11 5:41 PM,  "Oleg Gryb" <ole...@ya...>  wrote:
> 
> >Ryan,
> >That might not work for a single IP in  x-forwarded-for header, becuase
> >non-greedy match for (.*?) will return  empty string.
> >
> >Try:
> >
> >perl -e "'1.2.3.4' =~  /^(.*?),?.*$/; print $1;"
> >
> >and you'll see what I mean. I'm  assuming here that modsecurity uses  PCRE.
> >
> >Thanks,
> >Oleg.
> >
> >
> >
> >
> >
> >
> >-----  Original Message ----
> >> From: Ryan Barnett <RBa...@tr...>
> >>  To: "ol...@gr..." <ol...@gr...>;
> >>"mod...@li..."
> >><mod...@li...>
> >>  Cc: "owa...@li..."
> >><owa...@li...>
> >>  Sent: Fri, May 6, 2011 10:29:57 AM
> >> Subject: Re:  [Mod-security-developers] CRS DoS protection  &
> >>x-forwarded-for
> >>header
> >>
> >> Good  point about IPv6.  Looks like we could still do this using  the
> >>same  #
> >> of SecRules and just not using an IP  regex like this -
> >>
> >> SecRule   REQUEST_HEADERS:User-Agent   "^(.*)$"
> >>
> >>"phase:1,t:none,pass,nolog,t:sha1,t:hexEncode,setvar:tx.ua_hash=%{matched
> >>_v
> >>  ar}"
> >> SecRule  REQUEST_HEADERS:x-forwarded-for   "^(.*?),?.*$"
> >>  "phase:1,t:none,pass,nolog,setvar:tx.real_ip=%{matched_var}"
> >>
> >>  SecRule  &TX:REAL_IP "@eq   0"
> >>
> >>"phase:1,t:none,pass,nolog,initcol:global=global,initcol:ip=%{remote_addr
> >>}_
> >>  %{tx.ua_hash},,setvar:tx.real_ip=%{remote_addr}"
> >> SecRule   &TX:REAL_IP "!@eq   0"
> >>
> >>"phase:1,t:none,pass,nolog,initcol:global=global,initcol:ip=%{tx.real_ip}
> >>_%
> >>  {tx.ua_hash}"
> >>
> >>
> >>
> >> --
> >>  Ryan
> >>
> >>
> >>
> >> On  5/6/11 1:16 PM,  "Oleg Gryb" <ole...@ya...>   wrote:
> >>
> >> >Ryan,
> >> >
> >> >Thanks  for the update. It might have  problems with IPv6 though.  Here
> >>are
> >> >my rules: they search for ',' and   disregard the IP structure:
> >> >
> >> >
> >>  >SecRule  REQUEST_HEADERS:User-Agent   "^(.*)$"
> >>
> >>>"phase:1,t:none,pass,nolog,t:sha1,t:hexEncode,setvar:tx.ua_hash=%{matche
> >>>d_
> >>  >var}"
> >> >SecRule  REQUEST_HEADERS:x-forwarded-for   "^(.*)$"
> >>  >"phase:1,t:none,pass,nolog,setvar:tx.real_ip=%{matched_var}"
> >>  >SecRule  TX:REAL_IP  "^(.*?),.*$"
> >>  >"phase:1,t:none,pass,nolog,capture,setvar:tx.real_ip=%{TX.1}"
> >>  >
> >> >SecRule  &TX:REAL_IP "@eq   0"
> >>
> >>>"phase:1,t:none,pass,nolog,initcol:global=global,initcol:ip=%{remote_add
> >>>r}
> >>  >_%{tx.ua_hash},,setvar:tx.real_ip=%{remote_addr}"
> >>  >SecRule  &TX:REAL_IP "!@eq   0"
> >>
> >>>"phase:1,t:none,pass,nolog,initcol:global=global,initcol:ip=%{tx.real_ip
> >>>}_
> >>  >%{tx.ua_hash}"
> >> >
> >> >Thanks,
> >>  >Oleg.
> >> >
> >> >---  On Thu, 5/5/11, Ryan  Barnett <RBa...@tr...>   wrote:
> >> >
> >> >> From: Ryan Barnett <RBa...@tr...>
> >>  >>  Subject: Re: [Mod-security-developers] CRS DoS protection   &
> >> >>x-forwarded-for header
> >> >> To: "Oleg  Gryb" <ol...@gr...>,
> >> >>"mod...@li..."
> >>  >><mod...@li...>
> >>  >>  Cc: "owa...@li..."
> >>  >><owa...@li...>
> >>  >>  Date: Thursday, May 5, 2011, 10:46 AM
> >> >> FYI -  updated the CRS 10  config file
> >> >> to add in this logic  and uploaded it to SVN  -
> >>  >>
> >>
> >>>>http://mod-security.svn.sourceforge.net/viewvc/mod-security/crs/trunk/m
> >>>>od
> >>  >>security_crs_10_config.conf.example?revision=1772
> >>  >>
> >> >>  #
> >> >> # -=[ Global and IP  Collections ]=-
> >> >> #
> >> >> #  Create both  Global and IP collections for rules to use
> >> >> # There  are  some CRS rules that assume that these two
> >> >>  collections
> >> >> #  have already been  initiated.
> >> >> #
> >> >> SecRule   REQUEST_HEADERS:User-Agent  "^(.*)$"
> >>  >>
> >>
> >>>>"phase:1,id:'981217',t:none,pass,nolog,t:sha1,t:hexEncode,setvar:tx.ua_
> >>>>ha
> >>  >>sh=%{matched_var}"
> >> >>  SecRule  REQUEST_HEADERS:x-forwarded-for
> >> >>   "^\b(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\b"
> >>  >>
> >>
> >>>>"phase:1,id:'981225',t:none,pass,nolog,capture,setvar:tx.real_ip=%{tx.1
> >>>>}"
> >>  >>  SecRule &TX:REAL_IP "!@eq  0"
> >>  >>
> >>
> >>>>"phase:1,id:'981226',t:none,pass,nolog,initcol:global=global,initcol:ip
> >>>>=%
> >>  >>{tx.real_ip}_%{tx.ua_hash}"
> >> >>  SecRule  &TX:REAL_IP "@eq  0"
> >>  >>
> >>
> >>>>"phase:1,id:'981218',t:none,pass,nolog,initcol:global=global,initcol:ip
> >>>>=%
> >>  >>{remote_addr}_%{tx.ua_hash}"
> >> >>
> >>  >>  The new rules will grab the first IP address listed in  an
> >> >>  X-Forwared-For header and use that for the IP  collection
> >> >> key.   If X-Forwarded-For is not present,  then it will
> >> >> use  REMOTE_ADDR.
> >>  >>
> >> >> Thanks for the  suggestion!
> >>  >>
> >> >> -Ryan
> >> >>
> >>  >>
> >> >> On  4/28/11 8:19 PM, "Oleg Gryb"
> >>  >><ole...@ya...<mailto:ole...@ya...>>
> >>  >>  wrote:
> >> >>
> >> >> I've just  realized that there might be a problem  with
> >> >> relying  on that header: if
> >> >> an attacker  intentionally sends  different random IPs in
> >> >> there, DoS   protection
> >> >> can be efficiently by-passed. In my case it  should  not
> >> >> happen, because an LB is
> >>  >> the one who adds the  header, but in general we should  warn
> >> >> engineers about  the
> >> >>  possible exploit.
> >> >>
> >> >>
> >> >>  Actually,  even in LB case: if a request has already had the
> >>  >> header, LB  will
> >> >> create a new one with the  existing value appended to  the
> >> >> client IP:
> >>  >>
> >> >> x-forwarded-for:  real-client-ip,
> >>  >> whatever-client-sent-to-LB
> >> >>
> >>  >>  It means that we would need to rely on the the first IP  in
> >> >> the list,  everything
> >> >> else  should be considered as  untrusted.
> >> >>
> >>  >> Thanks,
> >> >>  Oleg.
> >>  >>
> >> >>
> >> >> ----- Original Message   ----
> >> >> From: Ryan Barnett
> >> >><RBa...@tr...<mailto:RBa...@tr...>>
> >>  >>  To: Oleg Gryb <ol...@gr...<mailto:ol...@gr...>>;
> >>  >>
> >>
> >>>>"mod...@li...<mailto:mod-security-deve
> >>>>lo
> >>  >>pe...@li...>"
> >>  >>
> >>
> >>>><mod...@li...<mailto:mod-security-deve
> >>>>lo
> >>  >>pe...@li...>>
> >>  >>  Cc:
> >>
> >>>>"owa...@li...<mailto:owasp-modsecuri
> >>>>ty
> >>  >>-cor...@li...>"
> >>  >>
> >>
> >>>><owa...@li...<mailto:owasp-modsecuri
> >>>>ty
> >>  >>-cor...@li...>>
> >>  >>  Sent: Thu, April 28, 2011 4:41:14 PM
> >> >>  Subject: Re:  [Mod-security-developers] CRS DoS protection
> >>  >> &  x-forwarded-for
> >> >> header
> >>  >> Thanks for the updates  Oleg!  This will certainly be  a
> >> >> useful update   to
> >> >> not only the  DoS rules buy any rules that will be based  on
> >> >> the  client  IP.
> >> >> I will actually go back to check   other uses of REMOTE_ADDR
> >> >> and see if  we
> >>  >> can swap  it for tx.real_ip instead.
> >> >> I will  add this to the CRS  v2.2.0  that I am working
> >> >>  on.
> >> >> For future reference - here is the  OWASP  CRS
> >> >> mail-list -
> >> >>
> >>https://lists.owasp.org/mailman/listinfo/owasp-modsecurity-core-rule-set
> >>  >>  --
> >> >> Ryan  Barnett
> >> >>  Senior Security  Researcher
> >> >> Trustwave -   SpiderLabs
> >> >> On 4/28/11 7:32  PM, "Oleg  Gryb"
> >> >><ole...@ya...<mailto:ole...@ya...>>
> >>  >>  wrote:
> >> >> >I'm not sure if I can discuss CRS  rules here. If  not,
> >> >> please  let me  know
> >> >> >what
> >> >>  >the right  place is. I want to suggest an
> >> >> improvement to DoS   protection in
> >> >> >CRS
> >> >> >2.1.2. The  problem is  that  enterprise
> >> >> applications usually  run behind
> >> >>  >load
> >> >>  >balancers, so  relying on remote_addr doesn't  make
> >>  >> too much sense, because
> >> >> >you'll
> >>  >>  >always have an LB's IP in there.
> >> >>  >
> >> >>  >
> >> >> >My improved   rules (attached)  check  for
> >> >> x-forwarded-for  header and if
> >> >> >it's
> >> >>   >present, this IP will be used to initialize IP
> >> >>  collection. If  it's  not
> >> >> >then  the
> >> >> >old logic will be  used.
> >> >>  >
> >> >> >It would be  great if we can include   this
> >> >> improvement to the next  CRS
> >>  >>  >release.
> >> >> >
> >> >>  >Thanks,
> >> >>  >Oleg.
> >>  >>
> >>
> >>>>-----------------------------------------------------------------------
> >>>>--
> >>  >>-----
> >> >>  WhatsUp  Gold - Download Free  Network Management
> >> >>  Software
> >> >> The  most intuitive,  comprehensive, and  cost-effective
> >>  >> network
> >> >> management toolset  available   today.  Delivers
> >> >> lowest initial
> >>  >>  acquisition cost and overall TCO of any
> >> >>  competing   solution.
> >> >>  http://p.sf.net/sfu/whatsupgold-sd
> >> >>   _______________________________________________
> >> >>   mod-security-developers  mailing list
> >>  >>
> >>
> >>>>mod...@li...<mailto:mod-security-devel
> >>>>op
> >>  >>er...@li...>
> >>  >> https://lists.sourceforge.net/lists/listinfo/mod-security-developers
> >>  >>  ModSecurity  Services from Trustwave's  SpiderLabs:
> >> >> https://www.trustwave.com/spiderLabs.php
> >> >>
> >>  >>
> >>
> >>>>-----------------------------------------------------------------------
> >>>>--
> >>  >>-----
> >> >>  WhatsUp Gold - Download Free Network  Management Software
> >> >> The most  intuitive,  comprehensive, and cost-effective
> >> >> network
> >>  >>  management toolset available today.  Delivers  lowest
> >> >>  initial
> >> >> acquisition cost  and overall TCO of any  competing
> >> >>  solution.
> >> >> http://p.sf.net/sfu/whatsupgold-sd
> >> >>   _______________________________________________
> >> >>   mod-security-developers mailing list
> >>  >>
> >>
> >>>>mod...@li...<mailto:mod-security-devel
> >>>>op
> >>  >>er...@li...>
> >>  >> https://lists.sourceforge.net/lists/listinfo/mod-security-developers
> >>  >>  ModSecurity Services from Trustwave's SpiderLabs:
> >>  >> https://www.trustwave.com/spiderLabs.php
> >> >>
> >>  >>
> >> >>
> >> >>   ________________________________
> >> >> This transmission may  contain  information that is
> >> >> privileged, confidential,  and/or exempt from  disclosure
> >> >> under applicable law.  If you are not the intended  recipient,
> >> >> you are hereby  notified that any disclosure,  copying,
> >> >> distribution,  or use of the information contained  herein
> >> >> (including  any reliance thereon) is STRICTLY PROHIBITED.  If
> >> >> you  received this transmission in error, please  immediately
> >>  >> contact the sender and destroy the material in its   entirety,
> >> >> whether in electronic or hard copy   format.
> >> >>
> >> >>
> >>  >
> >>
> >>
> >> This transmission may contain   information that is privileged,
> >>confidential,
> >>and/or  exempt from disclosure  under applicable law. If you are not  the
> >>intended
> >>recipient, you are hereby  notified that  any disclosure, copying,
> >>distribution,
> >>or use of the  information  contained herein (including any reliance
> >>thereon)  is
> >>STRICTLY PROHIBITED. If you  received this transmission in  error, please
> >>immediately contact the sender and  destroy the  material in its entirety,
> >>whether in electronic or hard copy   format.
> >>
> >>
> >
> 
> 
> This transmission may contain  information that is privileged, confidential, 
>and/or exempt from disclosure  under applicable law. If you are not the intended 
>recipient, you are hereby  notified that any disclosure, copying, distribution, 
>or use of the information  contained herein (including any reliance thereon) is 
>STRICTLY PROHIBITED. If you  received this transmission in error, please 
>immediately contact the sender and  destroy the material in its entirety, 
>whether in electronic or hard copy  format.
> 
> 
> ------------------------------------------------------------------------------
> WhatsUp  Gold - Download Free Network Management Software
> The most intuitive,  comprehensive, and cost-effective network 
> management toolset available  today.  Delivers lowest initial 
> acquisition cost and overall TCO of any  competing solution.
> http://p.sf.net/sfu/whatsupgold-sd
> _______________________________________________
> mod-security-developers  mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-developers
> ModSecurity  Services from Trustwave's SpiderLabs:
> https://www.trustwave.com/spiderLabs.php
>