Thread: [mod-security-users] How to limit access rate by header?

Brought to you by: victorhora, zimmerletw

mod-security-users

[mod-security-users] How to limit access rate by header?

From: Luciano G. F. <luc...@gm...> - 2018-12-05 15:58:05

First of all, I'm new here so I'm not sure this is the right place for
asking for help (free modsec version). If it's not, I'll really appreciate
it if you can tell me where should I go.

I'm trying to limit hit rate by:

1. Request's header (like "facebookexternalhit").
2. (All hits to non static resources)

And then return a friendly "429 Too Many Requests" and "Retry-After: 3"
(seconds).
I know I can read a file of headers like:

SecRule REQUEST_HEADERS:User-Agent "@pmFromFile ratelimit-bots.txt"

But I'm getting trouble building the entire rule.

Any help would be really appreciated. Thank you!

Re: [mod-security-users] How to limit access rate by header?

From: Reindl H. <h.r...@th...> - 2018-12-05 19:14:02


Am 05.12.18 um 16:57 schrieb Luciano Guillermo Fantuzzi:
> First of all, I'm new here so I'm not sure this is the right place for
> asking for help (free modsec version). If it's not, I'll really
> appreciate it if you can tell me where should I go.
> 
> I'm trying to limit hit rate by:
> 
> 1. Request's header (like "facebookexternalhit").
> 2. (All hits to non static resources)
> 
> And then return a friendly "429 Too Many Requests" and "Retry-After: 3"
> (seconds).
> I know I can read a file of headers like:
> 
> SecRule REQUEST_HEADERS:User-Agent "@pmFromFile ratelimit-bots.txt"
> 
> But I'm getting trouble building the entire rule.
> 
> Any help would be really appreciated. Thank you!

this a non-iusse

normally you have rate-limits per IP in place and they should not be
within the application layer at all and in the best case not even on the
same machine

that below is from a firewall-vm on a complete /24 network before any
packet reaches a server at all, and for the individual servers are
simimlar rules with lower values per 2 seconds in place

when the request reachs the webserver damage is long done and if no
damage is done you are wasting expensive ressources with the rules

Chain INBOUND (2 references)
 pkts bytes target     prot opt in     out     source
destination
 1914  183K IPST_ALL   all  --  *      *       0.0.0.0/0
0.0.0.0/0            recent: UPDATE seconds: 2 hit_count: 250 TTL-Match
name: limit_all_global side: source mask: 255.255.255.255
 149K   15M DROP_ALL   all  --  *      *       0.0.0.0/0
0.0.0.0/0            recent: UPDATE seconds: 2 reap hit_count: 150
TTL-Match name: limit_all_global side: source mask: 255.255.255.255

Re: [mod-security-users] How to limit access rate by header?

From: Luciano G. F. <luc...@gm...> - 2018-12-05 21:26:54

Thank you for your answer, but maybe I'm not asking it the right way or
this is not the right place to ask(?).

I need a Modsecurity rule (I'm using it through Apache) to be able to
control hits from clients with a specific header, like
"facebookexternalhit/1.1".
Ie. to stop some agressive bots hitting too often my webservers and taking
them down eventually. I don't want to block them at all because I need some
of them (like Facebook bot to parse shared content), but I need a way to
tell them "stop, retry in some seconds".

Thanks.

El mié., 5 de dic. de 2018 a la(s) 16:16, Reindl Harald (
h.r...@th...) escribió:

>
>
> Am 05.12.18 um 16:57 schrieb Luciano Guillermo Fantuzzi:
> > First of all, I'm new here so I'm not sure this is the right place for
> > asking for help (free modsec version). If it's not, I'll really
> > appreciate it if you can tell me where should I go.
> >
> > I'm trying to limit hit rate by:
> >
> > 1. Request's header (like "facebookexternalhit").
> > 2. (All hits to non static resources)
> >
> > And then return a friendly "429 Too Many Requests" and "Retry-After: 3"
> > (seconds).
> > I know I can read a file of headers like:
> >
> > SecRule REQUEST_HEADERS:User-Agent "@pmFromFile ratelimit-bots.txt"
> >
> > But I'm getting trouble building the entire rule.
> >
> > Any help would be really appreciated. Thank you!
>
> this a non-iusse
>
> normally you have rate-limits per IP in place and they should not be
> within the application layer at all and in the best case not even on the
> same machine
>
> that below is from a firewall-vm on a complete /24 network before any
> packet reaches a server at all, and for the individual servers are
> simimlar rules with lower values per 2 seconds in place
>
> when the request reachs the webserver damage is long done and if no
> damage is done you are wasting expensive ressources with the rules
>
> Chain INBOUND (2 references)
>  pkts bytes target     prot opt in     out     source
> destination
>  1914  183K IPST_ALL   all  --  *      *       0.0.0.0/0
> 0.0.0.0/0            recent: UPDATE seconds: 2 hit_count: 250 TTL-Match
> name: limit_all_global side: source mask: 255.255.255.255
>  149K   15M DROP_ALL   all  --  *      *       0.0.0.0/0
> 0.0.0.0/0            recent: UPDATE seconds: 2 reap hit_count: 150
> TTL-Match name: limit_all_global side: source mask: 255.255.255.255
>
>
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/
>

Re: [mod-security-users] How to limit access rate by header?

From: Christian F. <chr...@ne...> - 2018-12-06 13:28:20

Hello Luciano,

You have a peculiar use case, but I see your thinking.

There are examples in the ModSecurity books that are really close to your
plan. They should be easy to adopt.

Other than that, you may want to look into mod_qos. It has functionality
that might be useful in your case.

Best,

Christian


On Wed, Dec 05, 2018 at 06:26:03PM -0300, Luciano Guillermo Fantuzzi wrote:
> Thank you for your answer, but maybe I'm not asking it the right way or
> this is not the right place to ask(?).
> 
> I need a Modsecurity rule (I'm using it through Apache) to be able to
> control hits from clients with a specific header, like
> "facebookexternalhit/1.1".
> Ie. to stop some agressive bots hitting too often my webservers and taking
> them down eventually. I don't want to block them at all because I need some
> of them (like Facebook bot to parse shared content), but I need a way to
> tell them "stop, retry in some seconds".
> 
> Thanks.
> 
> El mié., 5 de dic. de 2018 a la(s) 16:16, Reindl Harald (
> h.r...@th...) escribió:
> 
> >
> >
> > Am 05.12.18 um 16:57 schrieb Luciano Guillermo Fantuzzi:
> > > First of all, I'm new here so I'm not sure this is the right place for
> > > asking for help (free modsec version). If it's not, I'll really
> > > appreciate it if you can tell me where should I go.
> > >
> > > I'm trying to limit hit rate by:
> > >
> > > 1. Request's header (like "facebookexternalhit").
> > > 2. (All hits to non static resources)
> > >
> > > And then return a friendly "429 Too Many Requests" and "Retry-After: 3"
> > > (seconds).
> > > I know I can read a file of headers like:
> > >
> > > SecRule REQUEST_HEADERS:User-Agent "@pmFromFile ratelimit-bots.txt"
> > >
> > > But I'm getting trouble building the entire rule.
> > >
> > > Any help would be really appreciated. Thank you!
> >
> > this a non-iusse
> >
> > normally you have rate-limits per IP in place and they should not be
> > within the application layer at all and in the best case not even on the
> > same machine
> >
> > that below is from a firewall-vm on a complete /24 network before any
> > packet reaches a server at all, and for the individual servers are
> > simimlar rules with lower values per 2 seconds in place
> >
> > when the request reachs the webserver damage is long done and if no
> > damage is done you are wasting expensive ressources with the rules
> >
> > Chain INBOUND (2 references)
> >  pkts bytes target     prot opt in     out     source
> > destination
> >  1914  183K IPST_ALL   all  --  *      *       0.0.0.0/0
> > 0.0.0.0/0            recent: UPDATE seconds: 2 hit_count: 250 TTL-Match
> > name: limit_all_global side: source mask: 255.255.255.255
> >  149K   15M DROP_ALL   all  --  *      *       0.0.0.0/0
> > 0.0.0.0/0            recent: UPDATE seconds: 2 reap hit_count: 150
> > TTL-Match name: limit_all_global side: source mask: 255.255.255.255
> >
> >
> > _______________________________________________
> > mod-security-users mailing list
> > mod...@li...
> > https://lists.sourceforge.net/lists/listinfo/mod-security-users
> > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> > http://www.modsecurity.org/projects/commercial/rules/
> > http://www.modsecurity.org/projects/commercial/support/
> >


> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/

Re: [mod-security-users] How to limit access rate by header?

From: Luciano G. F. <luc...@gm...> - 2018-12-06 23:39:07

Thank you for your answer, Christian. Do you think it's possible for you to
just build the first part of the rule (in Modsec)? I'm trying but I'm not
understanding how variables work with the global scope. I was be able to
build some basic rules like:

# Banned Bots and Crawlers
SecRule REQUEST_HEADERS:User-Agent "@pmFromFile blacklist-bots.data" \
    "id:350001,phase:1,t:none,deny,log,msg:'BANNED BOT'"

# Specific IPs
SecRule REMOTE_ADDR "@pmFromFile blacklist-ip.data" \
    "id:350002,phase:1,t:none,deny,log,msg:'BANNED IP'"

I'm trying to understand examples from stackoverflow and different places,
but they are all intended to limit by IP and for specific resources (the
scope of the rule). Eg.:
https://gist.github.com/josnidhin/91d1ea9cd71fde386c27a9228476834e

I'm not asking for the entire rule, just an example of how var counters
work in the global scope (directly in /etc/modsecurity/modsecurity.conf)
and how can I connect them to sum by header instead of IP.

Thank you!

El jue., 6 de dic. de 2018 a la(s) 10:30, Christian Folini (
chr...@ne...) escribió:

> Hello Luciano,
>
> You have a peculiar use case, but I see your thinking.
>
> There are examples in the ModSecurity books that are really close to your
> plan. They should be easy to adopt.
>
> Other than that, you may want to look into mod_qos. It has functionality
> that might be useful in your case.
>
> Best,
>
> Christian
>
>
> On Wed, Dec 05, 2018 at 06:26:03PM -0300, Luciano Guillermo Fantuzzi wrote:
> > Thank you for your answer, but maybe I'm not asking it the right way or
> > this is not the right place to ask(?).
> >
> > I need a Modsecurity rule (I'm using it through Apache) to be able to
> > control hits from clients with a specific header, like
> > "facebookexternalhit/1.1".
> > Ie. to stop some agressive bots hitting too often my webservers and
> taking
> > them down eventually. I don't want to block them at all because I need
> some
> > of them (like Facebook bot to parse shared content), but I need a way to
> > tell them "stop, retry in some seconds".
> >
> > Thanks.
> >
> > El mié., 5 de dic. de 2018 a la(s) 16:16, Reindl Harald (
> > h.r...@th...) escribió:
> >
> > >
> > >
> > > Am 05.12.18 um 16:57 schrieb Luciano Guillermo Fantuzzi:
> > > > First of all, I'm new here so I'm not sure this is the right place
> for
> > > > asking for help (free modsec version). If it's not, I'll really
> > > > appreciate it if you can tell me where should I go.
> > > >
> > > > I'm trying to limit hit rate by:
> > > >
> > > > 1. Request's header (like "facebookexternalhit").
> > > > 2. (All hits to non static resources)
> > > >
> > > > And then return a friendly "429 Too Many Requests" and "Retry-After:
> 3"
> > > > (seconds).
> > > > I know I can read a file of headers like:
> > > >
> > > > SecRule REQUEST_HEADERS:User-Agent "@pmFromFile ratelimit-bots.txt"
> > > >
> > > > But I'm getting trouble building the entire rule.
> > > >
> > > > Any help would be really appreciated. Thank you!
> > >
> > > this a non-iusse
> > >
> > > normally you have rate-limits per IP in place and they should not be
> > > within the application layer at all and in the best case not even on
> the
> > > same machine
> > >
> > > that below is from a firewall-vm on a complete /24 network before any
> > > packet reaches a server at all, and for the individual servers are
> > > simimlar rules with lower values per 2 seconds in place
> > >
> > > when the request reachs the webserver damage is long done and if no
> > > damage is done you are wasting expensive ressources with the rules
> > >
> > > Chain INBOUND (2 references)
> > >  pkts bytes target     prot opt in     out     source
> > > destination
> > >  1914  183K IPST_ALL   all  --  *      *       0.0.0.0/0
> > > 0.0.0.0/0            recent: UPDATE seconds: 2 hit_count: 250
> TTL-Match
> > > name: limit_all_global side: source mask: 255.255.255.255
> > >  149K   15M DROP_ALL   all  --  *      *       0.0.0.0/0
> > > 0.0.0.0/0            recent: UPDATE seconds: 2 reap hit_count: 150
> > > TTL-Match name: limit_all_global side: source mask: 255.255.255.255
> > >
> > >
> > > _______________________________________________
> > > mod-security-users mailing list
> > > mod...@li...
> > > https://lists.sourceforge.net/lists/listinfo/mod-security-users
> > > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> > > http://www.modsecurity.org/projects/commercial/rules/
> > > http://www.modsecurity.org/projects/commercial/support/
> > >
>
>
> > _______________________________________________
> > mod-security-users mailing list
> > mod...@li...
> > https://lists.sourceforge.net/lists/listinfo/mod-security-users
> > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> > http://www.modsecurity.org/projects/commercial/rules/
> > http://www.modsecurity.org/projects/commercial/support/
>
>
>
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/
>

Re: [mod-security-users] How to limit access rate by header?

From: Luciano G. F. <luc...@gm...> - 2018-12-07 02:51:27

I've very frustrated... I can't make it work, even for IP control. What am
I doing wrong here? It always returns:
Could not set variable "IP.access_count" as the collection does not exist.

105 <LocationMatch "^/.*">
109   SecRule REQUEST_HEADERS:CF-Connecting-IP "@unconditionalMatch"
"phase:2,initcol:IP=%{MATCHED_VAR},pass,nolog,id:35003"
112   SecRule IP:ACCESS_COUNT "@gt 1"
"phase:2,pause:300,deny,status:503,setenv:RATELIMITED,skip:1,nolog,id:35004"
116   SecAction "phase:2,setvar:IP.access_count=+1,pass,nolog,id:35005"


119   SecAction
"phase:5,deprecatevar:IP.access_count=1/10,pass,nolog,id:35006"
122   Header always set Retry-After "10" env=RATELIMITED
123 </LocationMatch>
124
125 ErrorDocument 503 "Service Unavailable"


El jue., 6 de dic. de 2018 a la(s) 20:38, Luciano Guillermo Fantuzzi (
luc...@gm...) escribió:

> Thank you for your answer, Christian. Do you think it's possible for you
> to just build the first part of the rule (in Modsec)? I'm trying but I'm
> not understanding how variables work with the global scope. I was be able
> to build some basic rules like:
>
> # Banned Bots and Crawlers
> SecRule REQUEST_HEADERS:User-Agent "@pmFromFile blacklist-bots.data" \
>     "id:350001,phase:1,t:none,deny,log,msg:'BANNED BOT'"
>
> # Specific IPs
> SecRule REMOTE_ADDR "@pmFromFile blacklist-ip.data" \
>     "id:350002,phase:1,t:none,deny,log,msg:'BANNED IP'"
>
> I'm trying to understand examples from stackoverflow and different places,
> but they are all intended to limit by IP and for specific resources (the
> scope of the rule). Eg.:
> https://gist.github.com/josnidhin/91d1ea9cd71fde386c27a9228476834e
>
> I'm not asking for the entire rule, just an example of how var counters
> work in the global scope (directly in /etc/modsecurity/modsecurity.conf)
> and how can I connect them to sum by header instead of IP.
>
> Thank you!
>
> El jue., 6 de dic. de 2018 a la(s) 10:30, Christian Folini (
> chr...@ne...) escribió:
>
>> Hello Luciano,
>>
>> You have a peculiar use case, but I see your thinking.
>>
>> There are examples in the ModSecurity books that are really close to your
>> plan. They should be easy to adopt.
>>
>> Other than that, you may want to look into mod_qos. It has functionality
>> that might be useful in your case.
>>
>> Best,
>>
>> Christian
>>
>>
>> On Wed, Dec 05, 2018 at 06:26:03PM -0300, Luciano Guillermo Fantuzzi
>> wrote:
>> > Thank you for your answer, but maybe I'm not asking it the right way or
>> > this is not the right place to ask(?).
>> >
>> > I need a Modsecurity rule (I'm using it through Apache) to be able to
>> > control hits from clients with a specific header, like
>> > "facebookexternalhit/1.1".
>> > Ie. to stop some agressive bots hitting too often my webservers and
>> taking
>> > them down eventually. I don't want to block them at all because I need
>> some
>> > of them (like Facebook bot to parse shared content), but I need a way to
>> > tell them "stop, retry in some seconds".
>> >
>> > Thanks.
>> >
>> > El mié., 5 de dic. de 2018 a la(s) 16:16, Reindl Harald (
>> > h.r...@th...) escribió:
>> >
>> > >
>> > >
>> > > Am 05.12.18 um 16:57 schrieb Luciano Guillermo Fantuzzi:
>> > > > First of all, I'm new here so I'm not sure this is the right place
>> for
>> > > > asking for help (free modsec version). If it's not, I'll really
>> > > > appreciate it if you can tell me where should I go.
>> > > >
>> > > > I'm trying to limit hit rate by:
>> > > >
>> > > > 1. Request's header (like "facebookexternalhit").
>> > > > 2. (All hits to non static resources)
>> > > >
>> > > > And then return a friendly "429 Too Many Requests" and
>> "Retry-After: 3"
>> > > > (seconds).
>> > > > I know I can read a file of headers like:
>> > > >
>> > > > SecRule REQUEST_HEADERS:User-Agent "@pmFromFile ratelimit-bots.txt"
>> > > >
>> > > > But I'm getting trouble building the entire rule.
>> > > >
>> > > > Any help would be really appreciated. Thank you!
>> > >
>> > > this a non-iusse
>> > >
>> > > normally you have rate-limits per IP in place and they should not be
>> > > within the application layer at all and in the best case not even on
>> the
>> > > same machine
>> > >
>> > > that below is from a firewall-vm on a complete /24 network before any
>> > > packet reaches a server at all, and for the individual servers are
>> > > simimlar rules with lower values per 2 seconds in place
>> > >
>> > > when the request reachs the webserver damage is long done and if no
>> > > damage is done you are wasting expensive ressources with the rules
>> > >
>> > > Chain INBOUND (2 references)
>> > >  pkts bytes target     prot opt in     out     source
>> > > destination
>> > >  1914  183K IPST_ALL   all  --  *      *       0.0.0.0/0
>> > > 0.0.0.0/0            recent: UPDATE seconds: 2 hit_count: 250
>> TTL-Match
>> > > name: limit_all_global side: source mask: 255.255.255.255
>> > >  149K   15M DROP_ALL   all  --  *      *       0.0.0.0/0
>> > > 0.0.0.0/0            recent: UPDATE seconds: 2 reap hit_count: 150
>> > > TTL-Match name: limit_all_global side: source mask: 255.255.255.255
>> > >
>> > >
>> > > _______________________________________________
>> > > mod-security-users mailing list
>> > > mod...@li...
>> > > https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> > > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> > > http://www.modsecurity.org/projects/commercial/rules/
>> > > http://www.modsecurity.org/projects/commercial/support/
>> > >
>>
>>
>> > _______________________________________________
>> > mod-security-users mailing list
>> > mod...@li...
>> > https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> > http://www.modsecurity.org/projects/commercial/rules/
>> > http://www.modsecurity.org/projects/commercial/support/
>>
>>
>>
>> _______________________________________________
>> mod-security-users mailing list
>> mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> http://www.modsecurity.org/projects/commercial/rules/
>> http://www.modsecurity.org/projects/commercial/support/
>>
>

Re: [mod-security-users] How to limit access rate by header?

From: Scheblein, A. <ada...@ma...> - 2018-12-07 03:54:28

I had a similar problem.  You need to initialize the collection with something like this:

                SecAction id:'2000000',phase:1,nolog,pass,initcol:IP=%{REMOTE_ADDR}

From: Luciano Guillermo Fantuzzi <luc...@gm...>
Reply-To: "mod...@li..." <mod...@li...>
Date: Thursday, December 6, 2018 at 8:51 PM
To: "mod...@li..." <mod...@li...>
Subject: Re: [mod-security-users] How to limit access rate by header?

I've very frustrated... I can't make it work, even for IP control. What am I doing wrong here? It always returns:
Could not set variable "IP.access_count" as the collection does not exist.

105 <LocationMatch "^/.*">
109   SecRule REQUEST_HEADERS:CF-Connecting-IP "@unconditionalMatch" "phase:2,initcol:IP=%{MATCHED_VAR},pass,nolog,id:35003"
112   SecRule IP:ACCESS_COUNT "@gt 1" "phase:2,pause:300,deny,status:503,setenv:RATELIMITED,skip:1,nolog,id:35004"
116   SecAction "phase:2,setvar:IP.access_count=+1,pass,nolog,id:35005"
119   SecAction "phase:5,deprecatevar:IP.access_count=1/10,pass,nolog,id:35006"
122   Header always set Retry-After "10" env=RATELIMITED
123 </LocationMatch>
124
125 ErrorDocument 503 "Service Unavailable"


El jue., 6 de dic. de 2018 a la(s) 20:38, Luciano Guillermo Fantuzzi (luc...@gm...<mailto:luc...@gm...>) escribió:
Thank you for your answer, Christian. Do you think it's possible for you to just build the first part of the rule (in Modsec)? I'm trying but I'm not understanding how variables work with the global scope. I was be able to build some basic rules like:

# Banned Bots and Crawlers
SecRule REQUEST_HEADERS:User-Agent "@pmFromFile blacklist-bots.data" \
    "id:350001,phase:1,t:none,deny,log,msg:'BANNED BOT'"

# Specific IPs
SecRule REMOTE_ADDR "@pmFromFile blacklist-ip.data" \
    "id:350002,phase:1,t:none,deny,log,msg:'BANNED IP'"

I'm trying to understand examples from stackoverflow and different places, but they are all intended to limit by IP and for specific resources (the scope of the rule). Eg.:
https://gist.github.com/josnidhin/91d1ea9cd71fde386c27a9228476834e<https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_josnidhin_91d1ea9cd71fde386c27a9228476834e&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=L7iKlxwUA3exA-ByaKl7gyvQkoOevQwuEjv4ZKC6hOY&e=>

I'm not asking for the entire rule, just an example of how var counters work in the global scope (directly in /etc/modsecurity/modsecurity.conf) and how can I connect them to sum by header instead of IP.

Thank you!

El jue., 6 de dic. de 2018 a la(s) 10:30, Christian Folini (chr...@ne...<mailto:chr...@ne...>) escribió:
Hello Luciano,

You have a peculiar use case, but I see your thinking.

There are examples in the ModSecurity books that are really close to your
plan. They should be easy to adopt.

Other than that, you may want to look into mod_qos. It has functionality
that might be useful in your case.

Best,

Christian


On Wed, Dec 05, 2018 at 06:26:03PM -0300, Luciano Guillermo Fantuzzi wrote:
> Thank you for your answer, but maybe I'm not asking it the right way or
> this is not the right place to ask(?).
>
> I need a Modsecurity rule (I'm using it through Apache) to be able to
> control hits from clients with a specific header, like
> "facebookexternalhit/1.1".
> Ie. to stop some agressive bots hitting too often my webservers and taking
> them down eventually. I don't want to block them at all because I need some
> of them (like Facebook bot to parse shared content), but I need a way to
> tell them "stop, retry in some seconds".
>
> Thanks.
>
> El mié., 5 de dic. de 2018 a la(s) 16:16, Reindl Harald (
> h.r...@th...<mailto:h.r...@th...>) escribió:
>
> >
> >
> > Am 05.12.18 um 16:57 schrieb Luciano Guillermo Fantuzzi:
> > > First of all, I'm new here so I'm not sure this is the right place for
> > > asking for help (free modsec version). If it's not, I'll really
> > > appreciate it if you can tell me where should I go.
> > >
> > > I'm trying to limit hit rate by:
> > >
> > > 1. Request's header (like "facebookexternalhit").
> > > 2. (All hits to non static resources)
> > >
> > > And then return a friendly "429 Too Many Requests" and "Retry-After: 3"
> > > (seconds).
> > > I know I can read a file of headers like:
> > >
> > > SecRule REQUEST_HEADERS:User-Agent "@pmFromFile ratelimit-bots.txt"
> > >
> > > But I'm getting trouble building the entire rule.
> > >
> > > Any help would be really appreciated. Thank you!
> >
> > this a non-iusse
> >
> > normally you have rate-limits per IP in place and they should not be
> > within the application layer at all and in the best case not even on the
> > same machine
> >
> > that below is from a firewall-vm on a complete /24 network before any
> > packet reaches a server at all, and for the individual servers are
> > simimlar rules with lower values per 2 seconds in place
> >
> > when the request reachs the webserver damage is long done and if no
> > damage is done you are wasting expensive ressources with the rules
> >
> > Chain INBOUND (2 references)
> >  pkts bytes target     prot opt in     out     source
> > destination
> >  1914  183K IPST_ALL   all  --  *      *       0.0.0.0/0<https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=>
> > 0.0.0.0/0<https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=>            recent: UPDATE seconds: 2 hit_count: 250 TTL-Match
> > name: limit_all_global side: source mask: 255.255.255.255
> >  149K   15M DROP_ALL   all  --  *      *       0.0.0.0/0<https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=>
> > 0.0.0.0/0<https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=>            recent: UPDATE seconds: 2 reap hit_count: 150
> > TTL-Match name: limit_all_global side: source mask: 255.255.255.255
> >
> >
> > _______________________________________________
> > mod-security-users mailing list
> > mod...@li...<mailto:mod...@li...>
> > https://lists.sourceforge.net/lists/listinfo/mod-security-users<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_mod-2Dsecurity-2Dusers&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=RUDsPA0iq5WVkW20NWQOl8suSJ4RvNfYZ6TM3FXNtdM&e=>
> > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> > http://www.modsecurity.org/projects/commercial/rules/<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_rules_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=rQF299h7fZhdJbhudnhzjEcP4e3Aa8qCG0KvKi4CKiM&e=>
> > http://www.modsecurity.org/projects/commercial/support/<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_support_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=ofF4OfFPsr3nKEMOH7j-CQmBqLgK_51fvOnQTavYK3c&e=>
> >


> _______________________________________________
> mod-security-users mailing list
> mod...@li...<mailto:mod...@li...>
> https://lists.sourceforge.net/lists/listinfo/mod-security-users<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_mod-2Dsecurity-2Dusers&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=RUDsPA0iq5WVkW20NWQOl8suSJ4RvNfYZ6TM3FXNtdM&e=>
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_rules_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=rQF299h7fZhdJbhudnhzjEcP4e3Aa8qCG0KvKi4CKiM&e=>
> http://www.modsecurity.org/projects/commercial/support/<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_support_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=ofF4OfFPsr3nKEMOH7j-CQmBqLgK_51fvOnQTavYK3c&e=>



_______________________________________________
mod-security-users mailing list
mod...@li...<mailto:mod...@li...>
https://lists.sourceforge.net/lists/listinfo/mod-security-users<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_mod-2Dsecurity-2Dusers&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=RUDsPA0iq5WVkW20NWQOl8suSJ4RvNfYZ6TM3FXNtdM&e=>
Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
http://www.modsecurity.org/projects/commercial/rules/<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_rules_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=rQF299h7fZhdJbhudnhzjEcP4e3Aa8qCG0KvKi4CKiM&e=>
http://www.modsecurity.org/projects/commercial/support/<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_support_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=ofF4OfFPsr3nKEMOH7j-CQmBqLgK_51fvOnQTavYK3c&e=>

Re: [mod-security-users] How to limit access rate by header?

From: Luciano G. F. <luc...@gm...> - 2018-12-07 12:18:08

Already tried that, but still the same message in debug log. Anyway, I'm
behind Cloudflare so I need to access that var (that contains the real IP)
from header. Moreover, I tried with global collection and same luck. Am I
missing some initialization step in modsec?

Thanks.

El vie., 7 dic. 2018 00:56, Scheblein, Adam <ada...@ma...>
escribió:

> I had a similar problem.  You need to initialize the collection with
> something like this:
>
>
>
>                 SecAction
> id:'2000000',phase:1,nolog,pass,initcol:IP=%{REMOTE_ADDR}
>
>
>
> *From: *Luciano Guillermo Fantuzzi <luc...@gm...>
> *Reply-To: *"mod...@li..." <
> mod...@li...>
> *Date: *Thursday, December 6, 2018 at 8:51 PM
> *To: *"mod...@li..." <
> mod...@li...>
> *Subject: *Re: [mod-security-users] How to limit access rate by header?
>
>
>
> I've very frustrated... I can't make it work, even for IP control. What am
> I doing wrong here? It always returns:
>
> Could not set variable "IP.access_count" as the collection does not exist.
>
>
>
> 105 <LocationMatch "^/.*">
>
> 109   SecRule REQUEST_HEADERS:CF-Connecting-IP "@unconditionalMatch"
> "phase:2,initcol:IP=%{MATCHED_VAR},pass,nolog,id:35003"
>
> 112   SecRule IP:ACCESS_COUNT "@gt 1"
> "phase:2,pause:300,deny,status:503,setenv:RATELIMITED,skip:1,nolog,id:35004"
>
> 116   SecAction "phase:2,setvar:IP.access_count=+1,pass,nolog,id:35005"
>
>
>
> 119   SecAction
> "phase:5,deprecatevar:IP.access_count=1/10,pass,nolog,id:35006"
>
> 122   Header always set Retry-After "10" env=RATELIMITED
>
> 123 </LocationMatch>
>
> 124
>
> 125 ErrorDocument 503 "Service Unavailable"
>
>
>
>
>
> El jue., 6 de dic. de 2018 a la(s) 20:38, Luciano Guillermo Fantuzzi (
> luc...@gm...) escribió:
>
> Thank you for your answer, Christian. Do you think it's possible for you
> to just build the first part of the rule (in Modsec)? I'm trying but I'm
> not understanding how variables work with the global scope. I was be able
> to build some basic rules like:
>
>
>
> # Banned Bots and Crawlers
>
> SecRule REQUEST_HEADERS:User-Agent "@pmFromFile blacklist-bots.data" \
>
>     "id:350001,phase:1,t:none,deny,log,msg:'BANNED BOT'"
>
>
>
> # Specific IPs
>
> SecRule REMOTE_ADDR "@pmFromFile blacklist-ip.data" \
>
>     "id:350002,phase:1,t:none,deny,log,msg:'BANNED IP'"
>
>
>
> I'm trying to understand examples from stackoverflow and different places,
> but they are all intended to limit by IP and for specific resources (the
> scope of the rule). Eg.:
>
> https://gist.github.com/josnidhin/91d1ea9cd71fde386c27a9228476834e
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_josnidhin_91d1ea9cd71fde386c27a9228476834e&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=L7iKlxwUA3exA-ByaKl7gyvQkoOevQwuEjv4ZKC6hOY&e=>
>
>
>
> I'm not asking for the entire rule, just an example of how var counters
> work in the global scope (directly in /etc/modsecurity/modsecurity.conf)
> and how can I connect them to sum by header instead of IP.
>
>
>
> Thank you!
>
>
>
> El jue., 6 de dic. de 2018 a la(s) 10:30, Christian Folini (
> chr...@ne...) escribió:
>
> Hello Luciano,
>
> You have a peculiar use case, but I see your thinking.
>
> There are examples in the ModSecurity books that are really close to your
> plan. They should be easy to adopt.
>
> Other than that, you may want to look into mod_qos. It has functionality
> that might be useful in your case.
>
> Best,
>
> Christian
>
>
> On Wed, Dec 05, 2018 at 06:26:03PM -0300, Luciano Guillermo Fantuzzi wrote:
> > Thank you for your answer, but maybe I'm not asking it the right way or
> > this is not the right place to ask(?).
> >
> > I need a Modsecurity rule (I'm using it through Apache) to be able to
> > control hits from clients with a specific header, like
> > "facebookexternalhit/1.1".
> > Ie. to stop some agressive bots hitting too often my webservers and
> taking
> > them down eventually. I don't want to block them at all because I need
> some
> > of them (like Facebook bot to parse shared content), but I need a way to
> > tell them "stop, retry in some seconds".
> >
> > Thanks.
> >
> > El mié., 5 de dic. de 2018 a la(s) 16:16, Reindl Harald (
> > h.r...@th...) escribió:
> >
> > >
> > >
> > > Am 05.12.18 um 16:57 schrieb Luciano Guillermo Fantuzzi:
> > > > First of all, I'm new here so I'm not sure this is the right place
> for
> > > > asking for help (free modsec version). If it's not, I'll really
> > > > appreciate it if you can tell me where should I go.
> > > >
> > > > I'm trying to limit hit rate by:
> > > >
> > > > 1. Request's header (like "facebookexternalhit").
> > > > 2. (All hits to non static resources)
> > > >
> > > > And then return a friendly "429 Too Many Requests" and "Retry-After:
> 3"
> > > > (seconds).
> > > > I know I can read a file of headers like:
> > > >
> > > > SecRule REQUEST_HEADERS:User-Agent "@pmFromFile ratelimit-bots.txt"
> > > >
> > > > But I'm getting trouble building the entire rule.
> > > >
> > > > Any help would be really appreciated. Thank you!
> > >
> > > this a non-iusse
> > >
> > > normally you have rate-limits per IP in place and they should not be
> > > within the application layer at all and in the best case not even on
> the
> > > same machine
> > >
> > > that below is from a firewall-vm on a complete /24 network before any
> > > packet reaches a server at all, and for the individual servers are
> > > simimlar rules with lower values per 2 seconds in place
> > >
> > > when the request reachs the webserver damage is long done and if no
> > > damage is done you are wasting expensive ressources with the rules
> > >
> > > Chain INBOUND (2 references)
> > >  pkts bytes target     prot opt in     out     source
> > > destination
> > >  1914  183K IPST_ALL   all  --  *      *       0.0.0.0/0
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=>
> > > 0.0.0.0/0
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=>
>           recent: UPDATE seconds: 2 hit_count: 250 TTL-Match
> > > name: limit_all_global side: source mask: 255.255.255.255
> > >  149K   15M DROP_ALL   all  --  *      *       0.0.0.0/0
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=>
> > > 0.0.0.0/0
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=>
>           recent: UPDATE seconds: 2 reap hit_count: 150
> > > TTL-Match name: limit_all_global side: source mask: 255.255.255.255
> > >
> > >
> > > _______________________________________________
> > > mod-security-users mailing list
> > > mod...@li...
> > > https://lists.sourceforge.net/lists/listinfo/mod-security-users
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_mod-2Dsecurity-2Dusers&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=RUDsPA0iq5WVkW20NWQOl8suSJ4RvNfYZ6TM3FXNtdM&e=>
> > > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> > > http://www.modsecurity.org/projects/commercial/rules/
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_rules_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=rQF299h7fZhdJbhudnhzjEcP4e3Aa8qCG0KvKi4CKiM&e=>
> > > http://www.modsecurity.org/projects/commercial/support/
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_support_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=ofF4OfFPsr3nKEMOH7j-CQmBqLgK_51fvOnQTavYK3c&e=>
> > >
>
>
> > _______________________________________________
> > mod-security-users mailing list
> > mod...@li...
> > https://lists.sourceforge.net/lists/listinfo/mod-security-users
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_mod-2Dsecurity-2Dusers&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=RUDsPA0iq5WVkW20NWQOl8suSJ4RvNfYZ6TM3FXNtdM&e=>
> > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> > http://www.modsecurity.org/projects/commercial/rules/
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_rules_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=rQF299h7fZhdJbhudnhzjEcP4e3Aa8qCG0KvKi4CKiM&e=>
> > http://www.modsecurity.org/projects/commercial/support/
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_support_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=ofF4OfFPsr3nKEMOH7j-CQmBqLgK_51fvOnQTavYK3c&e=>
>
>
>
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_mod-2Dsecurity-2Dusers&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=RUDsPA0iq5WVkW20NWQOl8suSJ4RvNfYZ6TM3FXNtdM&e=>
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_rules_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=rQF299h7fZhdJbhudnhzjEcP4e3Aa8qCG0KvKi4CKiM&e=>
> http://www.modsecurity.org/projects/commercial/support/
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_support_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=ofF4OfFPsr3nKEMOH7j-CQmBqLgK_51fvOnQTavYK3c&e=>
>
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/
>

Re: [mod-security-users] How to limit access rate by header?

From: Eero V. <eer...@ik...> - 2018-12-07 12:23:59

are you using nginx or apache?

Luciano Guillermo Fantuzzi <luc...@gm...> kirjoitti pe 7.
jouluk. 2018 klo 14.19:

> Already tried that, but still the same message in debug log. Anyway, I'm
> behind Cloudflare so I need to access that var (that contains the real IP)
> from header. Moreover, I tried with global collection and same luck. Am I
> missing some initialization step in modsec?
>
> Thanks.
>
> El vie., 7 dic. 2018 00:56, Scheblein, Adam <ada...@ma...>
> escribió:
>
>> I had a similar problem.  You need to initialize the collection with
>> something like this:
>>
>>
>>
>>                 SecAction
>> id:'2000000',phase:1,nolog,pass,initcol:IP=%{REMOTE_ADDR}
>>
>>
>>
>> *From: *Luciano Guillermo Fantuzzi <luc...@gm...>
>> *Reply-To: *"mod...@li..." <
>> mod...@li...>
>> *Date: *Thursday, December 6, 2018 at 8:51 PM
>> *To: *"mod...@li..." <
>> mod...@li...>
>> *Subject: *Re: [mod-security-users] How to limit access rate by header?
>>
>>
>>
>> I've very frustrated... I can't make it work, even for IP control. What
>> am I doing wrong here? It always returns:
>>
>> Could not set variable "IP.access_count" as the collection does not exist.
>>
>>
>>
>> 105 <LocationMatch "^/.*">
>>
>> 109   SecRule REQUEST_HEADERS:CF-Connecting-IP "@unconditionalMatch"
>> "phase:2,initcol:IP=%{MATCHED_VAR},pass,nolog,id:35003"
>>
>> 112   SecRule IP:ACCESS_COUNT "@gt 1"
>> "phase:2,pause:300,deny,status:503,setenv:RATELIMITED,skip:1,nolog,id:35004"
>>
>> 116   SecAction "phase:2,setvar:IP.access_count=+1,pass,nolog,id:35005"
>>
>>
>>
>> 119   SecAction
>> "phase:5,deprecatevar:IP.access_count=1/10,pass,nolog,id:35006"
>>
>> 122   Header always set Retry-After "10" env=RATELIMITED
>>
>> 123 </LocationMatch>
>>
>> 124
>>
>> 125 ErrorDocument 503 "Service Unavailable"
>>
>>
>>
>>
>>
>> El jue., 6 de dic. de 2018 a la(s) 20:38, Luciano Guillermo Fantuzzi (
>> luc...@gm...) escribió:
>>
>> Thank you for your answer, Christian. Do you think it's possible for you
>> to just build the first part of the rule (in Modsec)? I'm trying but I'm
>> not understanding how variables work with the global scope. I was be able
>> to build some basic rules like:
>>
>>
>>
>> # Banned Bots and Crawlers
>>
>> SecRule REQUEST_HEADERS:User-Agent "@pmFromFile blacklist-bots.data" \
>>
>>     "id:350001,phase:1,t:none,deny,log,msg:'BANNED BOT'"
>>
>>
>>
>> # Specific IPs
>>
>> SecRule REMOTE_ADDR "@pmFromFile blacklist-ip.data" \
>>
>>     "id:350002,phase:1,t:none,deny,log,msg:'BANNED IP'"
>>
>>
>>
>> I'm trying to understand examples from stackoverflow and different
>> places, but they are all intended to limit by IP and for specific resources
>> (the scope of the rule). Eg.:
>>
>> https://gist.github.com/josnidhin/91d1ea9cd71fde386c27a9228476834e
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_josnidhin_91d1ea9cd71fde386c27a9228476834e&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=L7iKlxwUA3exA-ByaKl7gyvQkoOevQwuEjv4ZKC6hOY&e=>
>>
>>
>>
>> I'm not asking for the entire rule, just an example of how var counters
>> work in the global scope (directly in /etc/modsecurity/modsecurity.conf)
>> and how can I connect them to sum by header instead of IP.
>>
>>
>>
>> Thank you!
>>
>>
>>
>> El jue., 6 de dic. de 2018 a la(s) 10:30, Christian Folini (
>> chr...@ne...) escribió:
>>
>> Hello Luciano,
>>
>> You have a peculiar use case, but I see your thinking.
>>
>> There are examples in the ModSecurity books that are really close to your
>> plan. They should be easy to adopt.
>>
>> Other than that, you may want to look into mod_qos. It has functionality
>> that might be useful in your case.
>>
>> Best,
>>
>> Christian
>>
>>
>> On Wed, Dec 05, 2018 at 06:26:03PM -0300, Luciano Guillermo Fantuzzi
>> wrote:
>> > Thank you for your answer, but maybe I'm not asking it the right way or
>> > this is not the right place to ask(?).
>> >
>> > I need a Modsecurity rule (I'm using it through Apache) to be able to
>> > control hits from clients with a specific header, like
>> > "facebookexternalhit/1.1".
>> > Ie. to stop some agressive bots hitting too often my webservers and
>> taking
>> > them down eventually. I don't want to block them at all because I need
>> some
>> > of them (like Facebook bot to parse shared content), but I need a way to
>> > tell them "stop, retry in some seconds".
>> >
>> > Thanks.
>> >
>> > El mié., 5 de dic. de 2018 a la(s) 16:16, Reindl Harald (
>> > h.r...@th...) escribió:
>> >
>> > >
>> > >
>> > > Am 05.12.18 um 16:57 schrieb Luciano Guillermo Fantuzzi:
>> > > > First of all, I'm new here so I'm not sure this is the right place
>> for
>> > > > asking for help (free modsec version). If it's not, I'll really
>> > > > appreciate it if you can tell me where should I go.
>> > > >
>> > > > I'm trying to limit hit rate by:
>> > > >
>> > > > 1. Request's header (like "facebookexternalhit").
>> > > > 2. (All hits to non static resources)
>> > > >
>> > > > And then return a friendly "429 Too Many Requests" and
>> "Retry-After: 3"
>> > > > (seconds).
>> > > > I know I can read a file of headers like:
>> > > >
>> > > > SecRule REQUEST_HEADERS:User-Agent "@pmFromFile ratelimit-bots.txt"
>> > > >
>> > > > But I'm getting trouble building the entire rule.
>> > > >
>> > > > Any help would be really appreciated. Thank you!
>> > >
>> > > this a non-iusse
>> > >
>> > > normally you have rate-limits per IP in place and they should not be
>> > > within the application layer at all and in the best case not even on
>> the
>> > > same machine
>> > >
>> > > that below is from a firewall-vm on a complete /24 network before any
>> > > packet reaches a server at all, and for the individual servers are
>> > > simimlar rules with lower values per 2 seconds in place
>> > >
>> > > when the request reachs the webserver damage is long done and if no
>> > > damage is done you are wasting expensive ressources with the rules
>> > >
>> > > Chain INBOUND (2 references)
>> > >  pkts bytes target     prot opt in     out     source
>> > > destination
>> > >  1914  183K IPST_ALL   all  --  *      *       0.0.0.0/0
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=>
>> > > 0.0.0.0/0
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=>
>>           recent: UPDATE seconds: 2 hit_count: 250 TTL-Match
>> > > name: limit_all_global side: source mask: 255.255.255.255
>> > >  149K   15M DROP_ALL   all  --  *      *       0.0.0.0/0
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=>
>> > > 0.0.0.0/0
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=>
>>           recent: UPDATE seconds: 2 reap hit_count: 150
>> > > TTL-Match name: limit_all_global side: source mask: 255.255.255.255
>> > >
>> > >
>> > > _______________________________________________
>> > > mod-security-users mailing list
>> > > mod...@li...
>> > > https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_mod-2Dsecurity-2Dusers&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=RUDsPA0iq5WVkW20NWQOl8suSJ4RvNfYZ6TM3FXNtdM&e=>
>> > > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> > > http://www.modsecurity.org/projects/commercial/rules/
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_rules_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=rQF299h7fZhdJbhudnhzjEcP4e3Aa8qCG0KvKi4CKiM&e=>
>> > > http://www.modsecurity.org/projects/commercial/support/
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_support_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=ofF4OfFPsr3nKEMOH7j-CQmBqLgK_51fvOnQTavYK3c&e=>
>> > >
>>
>>
>> > _______________________________________________
>> > mod-security-users mailing list
>> > mod...@li...
>> > https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_mod-2Dsecurity-2Dusers&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=RUDsPA0iq5WVkW20NWQOl8suSJ4RvNfYZ6TM3FXNtdM&e=>
>> > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> > http://www.modsecurity.org/projects/commercial/rules/
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_rules_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=rQF299h7fZhdJbhudnhzjEcP4e3Aa8qCG0KvKi4CKiM&e=>
>> > http://www.modsecurity.org/projects/commercial/support/
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_support_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=ofF4OfFPsr3nKEMOH7j-CQmBqLgK_51fvOnQTavYK3c&e=>
>>
>>
>>
>> _______________________________________________
>> mod-security-users mailing list
>> mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_mod-2Dsecurity-2Dusers&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=RUDsPA0iq5WVkW20NWQOl8suSJ4RvNfYZ6TM3FXNtdM&e=>
>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> http://www.modsecurity.org/projects/commercial/rules/
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_rules_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=rQF299h7fZhdJbhudnhzjEcP4e3Aa8qCG0KvKi4CKiM&e=>
>> http://www.modsecurity.org/projects/commercial/support/
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_support_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=ofF4OfFPsr3nKEMOH7j-CQmBqLgK_51fvOnQTavYK3c&e=>
>>
>> _______________________________________________
>> mod-security-users mailing list
>> mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> http://www.modsecurity.org/projects/commercial/rules/
>> http://www.modsecurity.org/projects/commercial/support/
>>
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/
>

Re: [mod-security-users] How to limit access rate by header?

From: Luciano G. F. <luc...@gm...> - 2018-12-07 12:28:49

Apache 2.4.x

El vie., 7 dic. 2018 09:25, Eero Volotinen <eer...@ik...> escribió:

> are you using nginx or apache?
>
> Luciano Guillermo Fantuzzi <luc...@gm...> kirjoitti pe 7.
> jouluk. 2018 klo 14.19:
>
>> Already tried that, but still the same message in debug log. Anyway, I'm
>> behind Cloudflare so I need to access that var (that contains the real IP)
>> from header. Moreover, I tried with global collection and same luck. Am I
>> missing some initialization step in modsec?
>>
>> Thanks.
>>
>> El vie., 7 dic. 2018 00:56, Scheblein, Adam <ada...@ma...>
>> escribió:
>>
>>> I had a similar problem.  You need to initialize the collection with
>>> something like this:
>>>
>>>
>>>
>>>                 SecAction
>>> id:'2000000',phase:1,nolog,pass,initcol:IP=%{REMOTE_ADDR}
>>>
>>>
>>>
>>> *From: *Luciano Guillermo Fantuzzi <luc...@gm...>
>>> *Reply-To: *"mod...@li..." <
>>> mod...@li...>
>>> *Date: *Thursday, December 6, 2018 at 8:51 PM
>>> *To: *"mod...@li..." <
>>> mod...@li...>
>>> *Subject: *Re: [mod-security-users] How to limit access rate by header?
>>>
>>>
>>>
>>> I've very frustrated... I can't make it work, even for IP control. What
>>> am I doing wrong here? It always returns:
>>>
>>> Could not set variable "IP.access_count" as the collection does not
>>> exist.
>>>
>>>
>>>
>>> 105 <LocationMatch "^/.*">
>>>
>>> 109   SecRule REQUEST_HEADERS:CF-Connecting-IP "@unconditionalMatch"
>>> "phase:2,initcol:IP=%{MATCHED_VAR},pass,nolog,id:35003"
>>>
>>> 112   SecRule IP:ACCESS_COUNT "@gt 1"
>>> "phase:2,pause:300,deny,status:503,setenv:RATELIMITED,skip:1,nolog,id:35004"
>>>
>>> 116   SecAction "phase:2,setvar:IP.access_count=+1,pass,nolog,id:35005"
>>>
>>>
>>>
>>> 119   SecAction
>>> "phase:5,deprecatevar:IP.access_count=1/10,pass,nolog,id:35006"
>>>
>>> 122   Header always set Retry-After "10" env=RATELIMITED
>>>
>>> 123 </LocationMatch>
>>>
>>> 124
>>>
>>> 125 ErrorDocument 503 "Service Unavailable"
>>>
>>>
>>>
>>>
>>>
>>> El jue., 6 de dic. de 2018 a la(s) 20:38, Luciano Guillermo Fantuzzi (
>>> luc...@gm...) escribió:
>>>
>>> Thank you for your answer, Christian. Do you think it's possible for you
>>> to just build the first part of the rule (in Modsec)? I'm trying but I'm
>>> not understanding how variables work with the global scope. I was be able
>>> to build some basic rules like:
>>>
>>>
>>>
>>> # Banned Bots and Crawlers
>>>
>>> SecRule REQUEST_HEADERS:User-Agent "@pmFromFile blacklist-bots.data" \
>>>
>>>     "id:350001,phase:1,t:none,deny,log,msg:'BANNED BOT'"
>>>
>>>
>>>
>>> # Specific IPs
>>>
>>> SecRule REMOTE_ADDR "@pmFromFile blacklist-ip.data" \
>>>
>>>     "id:350002,phase:1,t:none,deny,log,msg:'BANNED IP'"
>>>
>>>
>>>
>>> I'm trying to understand examples from stackoverflow and different
>>> places, but they are all intended to limit by IP and for specific resources
>>> (the scope of the rule). Eg.:
>>>
>>> https://gist.github.com/josnidhin/91d1ea9cd71fde386c27a9228476834e
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_josnidhin_91d1ea9cd71fde386c27a9228476834e&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=L7iKlxwUA3exA-ByaKl7gyvQkoOevQwuEjv4ZKC6hOY&e=>
>>>
>>>
>>>
>>> I'm not asking for the entire rule, just an example of how var counters
>>> work in the global scope (directly in /etc/modsecurity/modsecurity.conf)
>>> and how can I connect them to sum by header instead of IP.
>>>
>>>
>>>
>>> Thank you!
>>>
>>>
>>>
>>> El jue., 6 de dic. de 2018 a la(s) 10:30, Christian Folini (
>>> chr...@ne...) escribió:
>>>
>>> Hello Luciano,
>>>
>>> You have a peculiar use case, but I see your thinking.
>>>
>>> There are examples in the ModSecurity books that are really close to your
>>> plan. They should be easy to adopt.
>>>
>>> Other than that, you may want to look into mod_qos. It has functionality
>>> that might be useful in your case.
>>>
>>> Best,
>>>
>>> Christian
>>>
>>>
>>> On Wed, Dec 05, 2018 at 06:26:03PM -0300, Luciano Guillermo Fantuzzi
>>> wrote:
>>> > Thank you for your answer, but maybe I'm not asking it the right way or
>>> > this is not the right place to ask(?).
>>> >
>>> > I need a Modsecurity rule (I'm using it through Apache) to be able to
>>> > control hits from clients with a specific header, like
>>> > "facebookexternalhit/1.1".
>>> > Ie. to stop some agressive bots hitting too often my webservers and
>>> taking
>>> > them down eventually. I don't want to block them at all because I need
>>> some
>>> > of them (like Facebook bot to parse shared content), but I need a way
>>> to
>>> > tell them "stop, retry in some seconds".
>>> >
>>> > Thanks.
>>> >
>>> > El mié., 5 de dic. de 2018 a la(s) 16:16, Reindl Harald (
>>> > h.r...@th...) escribió:
>>> >
>>> > >
>>> > >
>>> > > Am 05.12.18 um 16:57 schrieb Luciano Guillermo Fantuzzi:
>>> > > > First of all, I'm new here so I'm not sure this is the right place
>>> for
>>> > > > asking for help (free modsec version). If it's not, I'll really
>>> > > > appreciate it if you can tell me where should I go.
>>> > > >
>>> > > > I'm trying to limit hit rate by:
>>> > > >
>>> > > > 1. Request's header (like "facebookexternalhit").
>>> > > > 2. (All hits to non static resources)
>>> > > >
>>> > > > And then return a friendly "429 Too Many Requests" and
>>> "Retry-After: 3"
>>> > > > (seconds).
>>> > > > I know I can read a file of headers like:
>>> > > >
>>> > > > SecRule REQUEST_HEADERS:User-Agent "@pmFromFile ratelimit-bots.txt"
>>> > > >
>>> > > > But I'm getting trouble building the entire rule.
>>> > > >
>>> > > > Any help would be really appreciated. Thank you!
>>> > >
>>> > > this a non-iusse
>>> > >
>>> > > normally you have rate-limits per IP in place and they should not be
>>> > > within the application layer at all and in the best case not even on
>>> the
>>> > > same machine
>>> > >
>>> > > that below is from a firewall-vm on a complete /24 network before any
>>> > > packet reaches a server at all, and for the individual servers are
>>> > > simimlar rules with lower values per 2 seconds in place
>>> > >
>>> > > when the request reachs the webserver damage is long done and if no
>>> > > damage is done you are wasting expensive ressources with the rules
>>> > >
>>> > > Chain INBOUND (2 references)
>>> > >  pkts bytes target     prot opt in     out     source
>>> > > destination
>>> > >  1914  183K IPST_ALL   all  --  *      *       0.0.0.0/0
>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=>
>>> > > 0.0.0.0/0
>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=>
>>>           recent: UPDATE seconds: 2 hit_count: 250 TTL-Match
>>> > > name: limit_all_global side: source mask: 255.255.255.255
>>> > >  149K   15M DROP_ALL   all  --  *      *       0.0.0.0/0
>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=>
>>> > > 0.0.0.0/0
>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=>
>>>           recent: UPDATE seconds: 2 reap hit_count: 150
>>> > > TTL-Match name: limit_all_global side: source mask: 255.255.255.255
>>> > >
>>> > >
>>> > > _______________________________________________
>>> > > mod-security-users mailing list
>>> > > mod...@li...
>>> > > https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_mod-2Dsecurity-2Dusers&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=RUDsPA0iq5WVkW20NWQOl8suSJ4RvNfYZ6TM3FXNtdM&e=>
>>> > > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>> > > http://www.modsecurity.org/projects/commercial/rules/
>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_rules_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=rQF299h7fZhdJbhudnhzjEcP4e3Aa8qCG0KvKi4CKiM&e=>
>>> > > http://www.modsecurity.org/projects/commercial/support/
>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_support_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=ofF4OfFPsr3nKEMOH7j-CQmBqLgK_51fvOnQTavYK3c&e=>
>>> > >
>>>
>>>
>>> > _______________________________________________
>>> > mod-security-users mailing list
>>> > mod...@li...
>>> > https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_mod-2Dsecurity-2Dusers&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=RUDsPA0iq5WVkW20NWQOl8suSJ4RvNfYZ6TM3FXNtdM&e=>
>>> > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>> > http://www.modsecurity.org/projects/commercial/rules/
>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_rules_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=rQF299h7fZhdJbhudnhzjEcP4e3Aa8qCG0KvKi4CKiM&e=>
>>> > http://www.modsecurity.org/projects/commercial/support/
>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_support_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=ofF4OfFPsr3nKEMOH7j-CQmBqLgK_51fvOnQTavYK3c&e=>
>>>
>>>
>>>
>>> _______________________________________________
>>> mod-security-users mailing list
>>> mod...@li...
>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_mod-2Dsecurity-2Dusers&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=RUDsPA0iq5WVkW20NWQOl8suSJ4RvNfYZ6TM3FXNtdM&e=>
>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>> http://www.modsecurity.org/projects/commercial/rules/
>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_rules_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=rQF299h7fZhdJbhudnhzjEcP4e3Aa8qCG0KvKi4CKiM&e=>
>>> http://www.modsecurity.org/projects/commercial/support/
>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_support_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=ofF4OfFPsr3nKEMOH7j-CQmBqLgK_51fvOnQTavYK3c&e=>
>>>
>>> _______________________________________________
>>> mod-security-users mailing list
>>> mod...@li...
>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>> http://www.modsecurity.org/projects/commercial/rules/
>>> http://www.modsecurity.org/projects/commercial/support/
>>>
>> _______________________________________________
>> mod-security-users mailing list
>> mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> http://www.modsecurity.org/projects/commercial/rules/
>> http://www.modsecurity.org/projects/commercial/support/
>>
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/
>

Re: [mod-security-users] How to limit access rate by header?

From: Christian F. <chr...@ne...> - 2018-12-07 12:29:38

Luciano,

I understand your troubles. But if you are behind cloudflare and you are
giving cloudflare money to solve your problems, why are you coming here
asking for free support?

You are facing a special problem and you need a special solution. We have
given you pointers and hints but it seems it is not enough, so it is
likely you need to dig deeper and learn more - or you pay somebody
to do that for you.

What I am not going to do - and I doubt somebody else is willing to do that -
is investing 1-2-3 hours into developing a solution for you in my spare
time. And it is very likely this would take 1-2-3 hours for me, so do not
expect a quick win.

Just my 2 cents. Good luck,

Christian

On Fri, Dec 07, 2018 at 09:17:45AM -0300, Luciano Guillermo Fantuzzi wrote:
> Already tried that, but still the same message in debug log. Anyway, I'm
> behind Cloudflare so I need to access that var (that contains the real IP)
> from header. Moreover, I tried with global collection and same luck. Am I
> missing some initialization step in modsec?
> 
> Thanks.
> 
> El vie., 7 dic. 2018 00:56, Scheblein, Adam <ada...@ma...>
> escribió:
> 
> > I had a similar problem.  You need to initialize the collection with
> > something like this:
> >
> >
> >
> >                 SecAction
> > id:'2000000',phase:1,nolog,pass,initcol:IP=%{REMOTE_ADDR}
> >
> >
> >
> > *From: *Luciano Guillermo Fantuzzi <luc...@gm...>
> > *Reply-To: *"mod...@li..." <
> > mod...@li...>
> > *Date: *Thursday, December 6, 2018 at 8:51 PM
> > *To: *"mod...@li..." <
> > mod...@li...>
> > *Subject: *Re: [mod-security-users] How to limit access rate by header?
> >
> >
> >
> > I've very frustrated... I can't make it work, even for IP control. What am
> > I doing wrong here? It always returns:
> >
> > Could not set variable "IP.access_count" as the collection does not exist.
> >
> >
> >
> > 105 <LocationMatch "^/.*">
> >
> > 109   SecRule REQUEST_HEADERS:CF-Connecting-IP "@unconditionalMatch"
> > "phase:2,initcol:IP=%{MATCHED_VAR},pass,nolog,id:35003"
> >
> > 112   SecRule IP:ACCESS_COUNT "@gt 1"
> > "phase:2,pause:300,deny,status:503,setenv:RATELIMITED,skip:1,nolog,id:35004"
> >
> > 116   SecAction "phase:2,setvar:IP.access_count=+1,pass,nolog,id:35005"
> >
> >
> >
> > 119   SecAction
> > "phase:5,deprecatevar:IP.access_count=1/10,pass,nolog,id:35006"
> >
> > 122   Header always set Retry-After "10" env=RATELIMITED
> >
> > 123 </LocationMatch>
> >
> > 124
> >
> > 125 ErrorDocument 503 "Service Unavailable"
> >
> >
> >
> >
> >
> > El jue., 6 de dic. de 2018 a la(s) 20:38, Luciano Guillermo Fantuzzi (
> > luc...@gm...) escribió:
> >
> > Thank you for your answer, Christian. Do you think it's possible for you
> > to just build the first part of the rule (in Modsec)? I'm trying but I'm
> > not understanding how variables work with the global scope. I was be able
> > to build some basic rules like:
> >
> >
> >
> > # Banned Bots and Crawlers
> >
> > SecRule REQUEST_HEADERS:User-Agent "@pmFromFile blacklist-bots.data" \
> >
> >     "id:350001,phase:1,t:none,deny,log,msg:'BANNED BOT'"
> >
> >
> >
> > # Specific IPs
> >
> > SecRule REMOTE_ADDR "@pmFromFile blacklist-ip.data" \
> >
> >     "id:350002,phase:1,t:none,deny,log,msg:'BANNED IP'"
> >
> >
> >
> > I'm trying to understand examples from stackoverflow and different places,
> > but they are all intended to limit by IP and for specific resources (the
> > scope of the rule). Eg.:
> >
> > https://gist.github.com/josnidhin/91d1ea9cd71fde386c27a9228476834e
> > <https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_josnidhin_91d1ea9cd71fde386c27a9228476834e&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=L7iKlxwUA3exA-ByaKl7gyvQkoOevQwuEjv4ZKC6hOY&e=>
> >
> >
> >
> > I'm not asking for the entire rule, just an example of how var counters
> > work in the global scope (directly in /etc/modsecurity/modsecurity.conf)
> > and how can I connect them to sum by header instead of IP.
> >
> >
> >
> > Thank you!
> >
> >
> >
> > El jue., 6 de dic. de 2018 a la(s) 10:30, Christian Folini (
> > chr...@ne...) escribió:
> >
> > Hello Luciano,
> >
> > You have a peculiar use case, but I see your thinking.
> >
> > There are examples in the ModSecurity books that are really close to your
> > plan. They should be easy to adopt.
> >
> > Other than that, you may want to look into mod_qos. It has functionality
> > that might be useful in your case.
> >
> > Best,
> >
> > Christian
> >
> >
> > On Wed, Dec 05, 2018 at 06:26:03PM -0300, Luciano Guillermo Fantuzzi wrote:
> > > Thank you for your answer, but maybe I'm not asking it the right way or
> > > this is not the right place to ask(?).
> > >
> > > I need a Modsecurity rule (I'm using it through Apache) to be able to
> > > control hits from clients with a specific header, like
> > > "facebookexternalhit/1.1".
> > > Ie. to stop some agressive bots hitting too often my webservers and
> > taking
> > > them down eventually. I don't want to block them at all because I need
> > some
> > > of them (like Facebook bot to parse shared content), but I need a way to
> > > tell them "stop, retry in some seconds".
> > >
> > > Thanks.
> > >
> > > El mié., 5 de dic. de 2018 a la(s) 16:16, Reindl Harald (
> > > h.r...@th...) escribió:
> > >
> > > >
> > > >
> > > > Am 05.12.18 um 16:57 schrieb Luciano Guillermo Fantuzzi:
> > > > > First of all, I'm new here so I'm not sure this is the right place
> > for
> > > > > asking for help (free modsec version). If it's not, I'll really
> > > > > appreciate it if you can tell me where should I go.
> > > > >
> > > > > I'm trying to limit hit rate by:
> > > > >
> > > > > 1. Request's header (like "facebookexternalhit").
> > > > > 2. (All hits to non static resources)
> > > > >
> > > > > And then return a friendly "429 Too Many Requests" and "Retry-After:
> > 3"
> > > > > (seconds).
> > > > > I know I can read a file of headers like:
> > > > >
> > > > > SecRule REQUEST_HEADERS:User-Agent "@pmFromFile ratelimit-bots.txt"
> > > > >
> > > > > But I'm getting trouble building the entire rule.
> > > > >
> > > > > Any help would be really appreciated. Thank you!
> > > >
> > > > this a non-iusse
> > > >
> > > > normally you have rate-limits per IP in place and they should not be
> > > > within the application layer at all and in the best case not even on
> > the
> > > > same machine
> > > >
> > > > that below is from a firewall-vm on a complete /24 network before any
> > > > packet reaches a server at all, and for the individual servers are
> > > > simimlar rules with lower values per 2 seconds in place
> > > >
> > > > when the request reachs the webserver damage is long done and if no
> > > > damage is done you are wasting expensive ressources with the rules
> > > >
> > > > Chain INBOUND (2 references)
> > > >  pkts bytes target     prot opt in     out     source
> > > > destination
> > > >  1914  183K IPST_ALL   all  --  *      *       0.0.0.0/0
> > <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=>
> > > > 0.0.0.0/0
> > <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=>
> >           recent: UPDATE seconds: 2 hit_count: 250 TTL-Match
> > > > name: limit_all_global side: source mask: 255.255.255.255
> > > >  149K   15M DROP_ALL   all  --  *      *       0.0.0.0/0
> > <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=>
> > > > 0.0.0.0/0
> > <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=>
> >           recent: UPDATE seconds: 2 reap hit_count: 150
> > > > TTL-Match name: limit_all_global side: source mask: 255.255.255.255
> > > >
> > > >
> > > > _______________________________________________
> > > > mod-security-users mailing list
> > > > mod...@li...
> > > > https://lists.sourceforge.net/lists/listinfo/mod-security-users
> > <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_mod-2Dsecurity-2Dusers&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=RUDsPA0iq5WVkW20NWQOl8suSJ4RvNfYZ6TM3FXNtdM&e=>
> > > > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> > > > http://www.modsecurity.org/projects/commercial/rules/
> > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_rules_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=rQF299h7fZhdJbhudnhzjEcP4e3Aa8qCG0KvKi4CKiM&e=>
> > > > http://www.modsecurity.org/projects/commercial/support/
> > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_support_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=ofF4OfFPsr3nKEMOH7j-CQmBqLgK_51fvOnQTavYK3c&e=>
> > > >
> >
> >
> > > _______________________________________________
> > > mod-security-users mailing list
> > > mod...@li...
> > > https://lists.sourceforge.net/lists/listinfo/mod-security-users
> > <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_mod-2Dsecurity-2Dusers&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=RUDsPA0iq5WVkW20NWQOl8suSJ4RvNfYZ6TM3FXNtdM&e=>
> > > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> > > http://www.modsecurity.org/projects/commercial/rules/
> > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_rules_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=rQF299h7fZhdJbhudnhzjEcP4e3Aa8qCG0KvKi4CKiM&e=>
> > > http://www.modsecurity.org/projects/commercial/support/
> > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_support_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=ofF4OfFPsr3nKEMOH7j-CQmBqLgK_51fvOnQTavYK3c&e=>
> >
> >
> >
> > _______________________________________________
> > mod-security-users mailing list
> > mod...@li...
> > https://lists.sourceforge.net/lists/listinfo/mod-security-users
> > <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_mod-2Dsecurity-2Dusers&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=RUDsPA0iq5WVkW20NWQOl8suSJ4RvNfYZ6TM3FXNtdM&e=>
> > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> > http://www.modsecurity.org/projects/commercial/rules/
> > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_rules_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=rQF299h7fZhdJbhudnhzjEcP4e3Aa8qCG0KvKi4CKiM&e=>
> > http://www.modsecurity.org/projects/commercial/support/
> > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_support_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=ofF4OfFPsr3nKEMOH7j-CQmBqLgK_51fvOnQTavYK3c&e=>
> >
> > _______________________________________________
> > mod-security-users mailing list
> > mod...@li...
> > https://lists.sourceforge.net/lists/listinfo/mod-security-users
> > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> > http://www.modsecurity.org/projects/commercial/rules/
> > http://www.modsecurity.org/projects/commercial/support/
> >


> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/

Re: [mod-security-users] How to limit access rate by header?

From: Eero V. <eer...@ik...> - 2018-12-07 12:31:07

maybe you need to use this module

https://github.com/gnif/mod_rpaf

Eero

Luciano Guillermo Fantuzzi <luc...@gm...> kirjoitti pe 7.
jouluk. 2018 klo 14.29:

> Apache 2.4.x
>
> El vie., 7 dic. 2018 09:25, Eero Volotinen <eer...@ik...>
> escribió:
>
>> are you using nginx or apache?
>>
>> Luciano Guillermo Fantuzzi <luc...@gm...> kirjoitti pe 7.
>> jouluk. 2018 klo 14.19:
>>
>>> Already tried that, but still the same message in debug log. Anyway, I'm
>>> behind Cloudflare so I need to access that var (that contains the real IP)
>>> from header. Moreover, I tried with global collection and same luck. Am I
>>> missing some initialization step in modsec?
>>>
>>> Thanks.
>>>
>>> El vie., 7 dic. 2018 00:56, Scheblein, Adam <
>>> ada...@ma...> escribió:
>>>
>>>> I had a similar problem.  You need to initialize the collection with
>>>> something like this:
>>>>
>>>>
>>>>
>>>>                 SecAction
>>>> id:'2000000',phase:1,nolog,pass,initcol:IP=%{REMOTE_ADDR}
>>>>
>>>>
>>>>
>>>> *From: *Luciano Guillermo Fantuzzi <luc...@gm...>
>>>> *Reply-To: *"mod...@li..." <
>>>> mod...@li...>
>>>> *Date: *Thursday, December 6, 2018 at 8:51 PM
>>>> *To: *"mod...@li..." <
>>>> mod...@li...>
>>>> *Subject: *Re: [mod-security-users] How to limit access rate by header?
>>>>
>>>>
>>>>
>>>> I've very frustrated... I can't make it work, even for IP control. What
>>>> am I doing wrong here? It always returns:
>>>>
>>>> Could not set variable "IP.access_count" as the collection does not
>>>> exist.
>>>>
>>>>
>>>>
>>>> 105 <LocationMatch "^/.*">
>>>>
>>>> 109   SecRule REQUEST_HEADERS:CF-Connecting-IP "@unconditionalMatch"
>>>> "phase:2,initcol:IP=%{MATCHED_VAR},pass,nolog,id:35003"
>>>>
>>>> 112   SecRule IP:ACCESS_COUNT "@gt 1"
>>>> "phase:2,pause:300,deny,status:503,setenv:RATELIMITED,skip:1,nolog,id:35004"
>>>>
>>>> 116   SecAction
>>>> "phase:2,setvar:IP.access_count=+1,pass,nolog,id:35005"
>>>>
>>>>
>>>>
>>>> 119   SecAction
>>>> "phase:5,deprecatevar:IP.access_count=1/10,pass,nolog,id:35006"
>>>>
>>>> 122   Header always set Retry-After "10" env=RATELIMITED
>>>>
>>>> 123 </LocationMatch>
>>>>
>>>> 124
>>>>
>>>> 125 ErrorDocument 503 "Service Unavailable"
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> El jue., 6 de dic. de 2018 a la(s) 20:38, Luciano Guillermo Fantuzzi (
>>>> luc...@gm...) escribió:
>>>>
>>>> Thank you for your answer, Christian. Do you think it's possible for
>>>> you to just build the first part of the rule (in Modsec)? I'm trying but
>>>> I'm not understanding how variables work with the global scope. I was be
>>>> able to build some basic rules like:
>>>>
>>>>
>>>>
>>>> # Banned Bots and Crawlers
>>>>
>>>> SecRule REQUEST_HEADERS:User-Agent "@pmFromFile blacklist-bots.data" \
>>>>
>>>>     "id:350001,phase:1,t:none,deny,log,msg:'BANNED BOT'"
>>>>
>>>>
>>>>
>>>> # Specific IPs
>>>>
>>>> SecRule REMOTE_ADDR "@pmFromFile blacklist-ip.data" \
>>>>
>>>>     "id:350002,phase:1,t:none,deny,log,msg:'BANNED IP'"
>>>>
>>>>
>>>>
>>>> I'm trying to understand examples from stackoverflow and different
>>>> places, but they are all intended to limit by IP and for specific resources
>>>> (the scope of the rule). Eg.:
>>>>
>>>> https://gist.github.com/josnidhin/91d1ea9cd71fde386c27a9228476834e
>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_josnidhin_91d1ea9cd71fde386c27a9228476834e&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=L7iKlxwUA3exA-ByaKl7gyvQkoOevQwuEjv4ZKC6hOY&e=>
>>>>
>>>>
>>>>
>>>> I'm not asking for the entire rule, just an example of how var counters
>>>> work in the global scope (directly in /etc/modsecurity/modsecurity.conf)
>>>> and how can I connect them to sum by header instead of IP.
>>>>
>>>>
>>>>
>>>> Thank you!
>>>>
>>>>
>>>>
>>>> El jue., 6 de dic. de 2018 a la(s) 10:30, Christian Folini (
>>>> chr...@ne...) escribió:
>>>>
>>>> Hello Luciano,
>>>>
>>>> You have a peculiar use case, but I see your thinking.
>>>>
>>>> There are examples in the ModSecurity books that are really close to
>>>> your
>>>> plan. They should be easy to adopt.
>>>>
>>>> Other than that, you may want to look into mod_qos. It has functionality
>>>> that might be useful in your case.
>>>>
>>>> Best,
>>>>
>>>> Christian
>>>>
>>>>
>>>> On Wed, Dec 05, 2018 at 06:26:03PM -0300, Luciano Guillermo Fantuzzi
>>>> wrote:
>>>> > Thank you for your answer, but maybe I'm not asking it the right way
>>>> or
>>>> > this is not the right place to ask(?).
>>>> >
>>>> > I need a Modsecurity rule (I'm using it through Apache) to be able to
>>>> > control hits from clients with a specific header, like
>>>> > "facebookexternalhit/1.1".
>>>> > Ie. to stop some agressive bots hitting too often my webservers and
>>>> taking
>>>> > them down eventually. I don't want to block them at all because I
>>>> need some
>>>> > of them (like Facebook bot to parse shared content), but I need a way
>>>> to
>>>> > tell them "stop, retry in some seconds".
>>>> >
>>>> > Thanks.
>>>> >
>>>> > El mié., 5 de dic. de 2018 a la(s) 16:16, Reindl Harald (
>>>> > h.r...@th...) escribió:
>>>> >
>>>> > >
>>>> > >
>>>> > > Am 05.12.18 um 16:57 schrieb Luciano Guillermo Fantuzzi:
>>>> > > > First of all, I'm new here so I'm not sure this is the right
>>>> place for
>>>> > > > asking for help (free modsec version). If it's not, I'll really
>>>> > > > appreciate it if you can tell me where should I go.
>>>> > > >
>>>> > > > I'm trying to limit hit rate by:
>>>> > > >
>>>> > > > 1. Request's header (like "facebookexternalhit").
>>>> > > > 2. (All hits to non static resources)
>>>> > > >
>>>> > > > And then return a friendly "429 Too Many Requests" and
>>>> "Retry-After: 3"
>>>> > > > (seconds).
>>>> > > > I know I can read a file of headers like:
>>>> > > >
>>>> > > > SecRule REQUEST_HEADERS:User-Agent "@pmFromFile
>>>> ratelimit-bots.txt"
>>>> > > >
>>>> > > > But I'm getting trouble building the entire rule.
>>>> > > >
>>>> > > > Any help would be really appreciated. Thank you!
>>>> > >
>>>> > > this a non-iusse
>>>> > >
>>>> > > normally you have rate-limits per IP in place and they should not be
>>>> > > within the application layer at all and in the best case not even
>>>> on the
>>>> > > same machine
>>>> > >
>>>> > > that below is from a firewall-vm on a complete /24 network before
>>>> any
>>>> > > packet reaches a server at all, and for the individual servers are
>>>> > > simimlar rules with lower values per 2 seconds in place
>>>> > >
>>>> > > when the request reachs the webserver damage is long done and if no
>>>> > > damage is done you are wasting expensive ressources with the rules
>>>> > >
>>>> > > Chain INBOUND (2 references)
>>>> > >  pkts bytes target     prot opt in     out     source
>>>> > > destination
>>>> > >  1914  183K IPST_ALL   all  --  *      *       0.0.0.0/0
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=>
>>>> > > 0.0.0.0/0
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=>
>>>>           recent: UPDATE seconds: 2 hit_count: 250 TTL-Match
>>>> > > name: limit_all_global side: source mask: 255.255.255.255
>>>> > >  149K   15M DROP_ALL   all  --  *      *       0.0.0.0/0
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=>
>>>> > > 0.0.0.0/0
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=>
>>>>           recent: UPDATE seconds: 2 reap hit_count: 150
>>>> > > TTL-Match name: limit_all_global side: source mask: 255.255.255.255
>>>> > >
>>>> > >
>>>> > > _______________________________________________
>>>> > > mod-security-users mailing list
>>>> > > mod...@li...
>>>> > > https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_mod-2Dsecurity-2Dusers&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=RUDsPA0iq5WVkW20NWQOl8suSJ4RvNfYZ6TM3FXNtdM&e=>
>>>> > > Commercial ModSecurity Rules and Support from Trustwave's
>>>> SpiderLabs:
>>>> > > http://www.modsecurity.org/projects/commercial/rules/
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_rules_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=rQF299h7fZhdJbhudnhzjEcP4e3Aa8qCG0KvKi4CKiM&e=>
>>>> > > http://www.modsecurity.org/projects/commercial/support/
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_support_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=ofF4OfFPsr3nKEMOH7j-CQmBqLgK_51fvOnQTavYK3c&e=>
>>>> > >
>>>>
>>>>
>>>> > _______________________________________________
>>>> > mod-security-users mailing list
>>>> > mod...@li...
>>>> > https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_mod-2Dsecurity-2Dusers&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=RUDsPA0iq5WVkW20NWQOl8suSJ4RvNfYZ6TM3FXNtdM&e=>
>>>> > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>>> > http://www.modsecurity.org/projects/commercial/rules/
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_rules_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=rQF299h7fZhdJbhudnhzjEcP4e3Aa8qCG0KvKi4CKiM&e=>
>>>> > http://www.modsecurity.org/projects/commercial/support/
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_support_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=ofF4OfFPsr3nKEMOH7j-CQmBqLgK_51fvOnQTavYK3c&e=>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> mod-security-users mailing list
>>>> mod...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_mod-2Dsecurity-2Dusers&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=RUDsPA0iq5WVkW20NWQOl8suSJ4RvNfYZ6TM3FXNtdM&e=>
>>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>>> http://www.modsecurity.org/projects/commercial/rules/
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_rules_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=rQF299h7fZhdJbhudnhzjEcP4e3Aa8qCG0KvKi4CKiM&e=>
>>>> http://www.modsecurity.org/projects/commercial/support/
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_support_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=ofF4OfFPsr3nKEMOH7j-CQmBqLgK_51fvOnQTavYK3c&e=>
>>>>
>>>> _______________________________________________
>>>> mod-security-users mailing list
>>>> mod...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>>> http://www.modsecurity.org/projects/commercial/rules/
>>>> http://www.modsecurity.org/projects/commercial/support/
>>>>
>>> _______________________________________________
>>> mod-security-users mailing list
>>> mod...@li...
>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>> http://www.modsecurity.org/projects/commercial/rules/
>>> http://www.modsecurity.org/projects/commercial/support/
>>>
>> _______________________________________________
>> mod-security-users mailing list
>> mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> http://www.modsecurity.org/projects/commercial/rules/
>> http://www.modsecurity.org/projects/commercial/support/
>>
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/
>

Re: [mod-security-users] How to limit access rate by header?

From: Luciano G. F. <luc...@gm...> - 2018-12-07 15:39:01

@Eero Thank you for the advice. I understand you recommend this mod to be
able to load the value of CF-Connecting-IP into REMOTE_ADDR and use that.
Sadly, even using SecAction with REMOTE_ADDR, the var is not being set and
the error I receive is:

"Could not set variable "ip.access_count" as the collection does not exist."

I tried with other collections and loading different values, but I always
end up receiving the same error. Can I use the "global" collection to set
some custom value? Maybe I'm not understanding how collections work. Let's
say:

SecAction id:'2000000',phase:1,nolog,pass,initcol:global='some'

In the docs for modsec v2 they say collections must be initialized only
once per transaction. I'm not using all the rules from modsec (I didn't
even downloaded repo), I'm just using some basic rules I created
in /etc/modsecurity/modsecurity.conf That's all I need for now. Can modsec
work like that or is there some other initialization required? The othe
rules I created work good.

Thanks!

El vie., 7 de dic. de 2018 a la(s) 09:34, Eero Volotinen (
eer...@ik...) escribió:

> maybe you need to use this module
>
> https://github.com/gnif/mod_rpaf
>
> Eero
>
> Luciano Guillermo Fantuzzi <luc...@gm...> kirjoitti pe 7.
> jouluk. 2018 klo 14.29:
>
>> Apache 2.4.x
>>
>> El vie., 7 dic. 2018 09:25, Eero Volotinen <eer...@ik...>
>> escribió:
>>
>>> are you using nginx or apache?
>>>
>>> Luciano Guillermo Fantuzzi <luc...@gm...> kirjoitti pe 7.
>>> jouluk. 2018 klo 14.19:
>>>
>>>> Already tried that, but still the same message in debug log. Anyway,
>>>> I'm behind Cloudflare so I need to access that var (that contains the real
>>>> IP) from header. Moreover, I tried with global collection and same luck. Am
>>>> I missing some initialization step in modsec?
>>>>
>>>> Thanks.
>>>>
>>>> El vie., 7 dic. 2018 00:56, Scheblein, Adam <
>>>> ada...@ma...> escribió:
>>>>
>>>>> I had a similar problem.  You need to initialize the collection with
>>>>> something like this:
>>>>>
>>>>>
>>>>>
>>>>>                 SecAction
>>>>> id:'2000000',phase:1,nolog,pass,initcol:IP=%{REMOTE_ADDR}
>>>>>
>>>>>
>>>>>
>>>>> *From: *Luciano Guillermo Fantuzzi <luc...@gm...>
>>>>> *Reply-To: *"mod...@li..." <
>>>>> mod...@li...>
>>>>> *Date: *Thursday, December 6, 2018 at 8:51 PM
>>>>> *To: *"mod...@li..." <
>>>>> mod...@li...>
>>>>> *Subject: *Re: [mod-security-users] How to limit access rate by
>>>>> header?
>>>>>
>>>>>
>>>>>
>>>>> I've very frustrated... I can't make it work, even for IP control.
>>>>> What am I doing wrong here? It always returns:
>>>>>
>>>>> Could not set variable "IP.access_count" as the collection does not
>>>>> exist.
>>>>>
>>>>>
>>>>>
>>>>> 105 <LocationMatch "^/.*">
>>>>>
>>>>> 109   SecRule REQUEST_HEADERS:CF-Connecting-IP "@unconditionalMatch"
>>>>> "phase:2,initcol:IP=%{MATCHED_VAR},pass,nolog,id:35003"
>>>>>
>>>>> 112   SecRule IP:ACCESS_COUNT "@gt 1"
>>>>> "phase:2,pause:300,deny,status:503,setenv:RATELIMITED,skip:1,nolog,id:35004"
>>>>>
>>>>> 116   SecAction
>>>>> "phase:2,setvar:IP.access_count=+1,pass,nolog,id:35005"
>>>>>
>>>>>
>>>>>
>>>>> 119   SecAction
>>>>> "phase:5,deprecatevar:IP.access_count=1/10,pass,nolog,id:35006"
>>>>>
>>>>> 122   Header always set Retry-After "10" env=RATELIMITED
>>>>>
>>>>> 123 </LocationMatch>
>>>>>
>>>>> 124
>>>>>
>>>>> 125 ErrorDocument 503 "Service Unavailable"
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> El jue., 6 de dic. de 2018 a la(s) 20:38, Luciano Guillermo Fantuzzi (
>>>>> luc...@gm...) escribió:
>>>>>
>>>>> Thank you for your answer, Christian. Do you think it's possible for
>>>>> you to just build the first part of the rule (in Modsec)? I'm trying but
>>>>> I'm not understanding how variables work with the global scope. I was be
>>>>> able to build some basic rules like:
>>>>>
>>>>>
>>>>>
>>>>> # Banned Bots and Crawlers
>>>>>
>>>>> SecRule REQUEST_HEADERS:User-Agent "@pmFromFile blacklist-bots.data" \
>>>>>
>>>>>     "id:350001,phase:1,t:none,deny,log,msg:'BANNED BOT'"
>>>>>
>>>>>
>>>>>
>>>>> # Specific IPs
>>>>>
>>>>> SecRule REMOTE_ADDR "@pmFromFile blacklist-ip.data" \
>>>>>
>>>>>     "id:350002,phase:1,t:none,deny,log,msg:'BANNED IP'"
>>>>>
>>>>>
>>>>>
>>>>> I'm trying to understand examples from stackoverflow and different
>>>>> places, but they are all intended to limit by IP and for specific resources
>>>>> (the scope of the rule). Eg.:
>>>>>
>>>>> https://gist.github.com/josnidhin/91d1ea9cd71fde386c27a9228476834e
>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_josnidhin_91d1ea9cd71fde386c27a9228476834e&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=L7iKlxwUA3exA-ByaKl7gyvQkoOevQwuEjv4ZKC6hOY&e=>
>>>>>
>>>>>
>>>>>
>>>>> I'm not asking for the entire rule, just an example of how var
>>>>> counters work in the global scope (directly
>>>>> in /etc/modsecurity/modsecurity.conf) and how can I connect them to sum by
>>>>> header instead of IP.
>>>>>
>>>>>
>>>>>
>>>>> Thank you!
>>>>>
>>>>>
>>>>>
>>>>> El jue., 6 de dic. de 2018 a la(s) 10:30, Christian Folini (
>>>>> chr...@ne...) escribió:
>>>>>
>>>>> Hello Luciano,
>>>>>
>>>>> You have a peculiar use case, but I see your thinking.
>>>>>
>>>>> There are examples in the ModSecurity books that are really close to
>>>>> your
>>>>> plan. They should be easy to adopt.
>>>>>
>>>>> Other than that, you may want to look into mod_qos. It has
>>>>> functionality
>>>>> that might be useful in your case.
>>>>>
>>>>> Best,
>>>>>
>>>>> Christian
>>>>>
>>>>>
>>>>> On Wed, Dec 05, 2018 at 06:26:03PM -0300, Luciano Guillermo Fantuzzi
>>>>> wrote:
>>>>> > Thank you for your answer, but maybe I'm not asking it the right way
>>>>> or
>>>>> > this is not the right place to ask(?).
>>>>> >
>>>>> > I need a Modsecurity rule (I'm using it through Apache) to be able to
>>>>> > control hits from clients with a specific header, like
>>>>> > "facebookexternalhit/1.1".
>>>>> > Ie. to stop some agressive bots hitting too often my webservers and
>>>>> taking
>>>>> > them down eventually. I don't want to block them at all because I
>>>>> need some
>>>>> > of them (like Facebook bot to parse shared content), but I need a
>>>>> way to
>>>>> > tell them "stop, retry in some seconds".
>>>>> >
>>>>> > Thanks.
>>>>> >
>>>>> > El mié., 5 de dic. de 2018 a la(s) 16:16, Reindl Harald (
>>>>> > h.r...@th...) escribió:
>>>>> >
>>>>> > >
>>>>> > >
>>>>> > > Am 05.12.18 um 16:57 schrieb Luciano Guillermo Fantuzzi:
>>>>> > > > First of all, I'm new here so I'm not sure this is the right
>>>>> place for
>>>>> > > > asking for help (free modsec version). If it's not, I'll really
>>>>> > > > appreciate it if you can tell me where should I go.
>>>>> > > >
>>>>> > > > I'm trying to limit hit rate by:
>>>>> > > >
>>>>> > > > 1. Request's header (like "facebookexternalhit").
>>>>> > > > 2. (All hits to non static resources)
>>>>> > > >
>>>>> > > > And then return a friendly "429 Too Many Requests" and
>>>>> "Retry-After: 3"
>>>>> > > > (seconds).
>>>>> > > > I know I can read a file of headers like:
>>>>> > > >
>>>>> > > > SecRule REQUEST_HEADERS:User-Agent "@pmFromFile
>>>>> ratelimit-bots.txt"
>>>>> > > >
>>>>> > > > But I'm getting trouble building the entire rule.
>>>>> > > >
>>>>> > > > Any help would be really appreciated. Thank you!
>>>>> > >
>>>>> > > this a non-iusse
>>>>> > >
>>>>> > > normally you have rate-limits per IP in place and they should not
>>>>> be
>>>>> > > within the application layer at all and in the best case not even
>>>>> on the
>>>>> > > same machine
>>>>> > >
>>>>> > > that below is from a firewall-vm on a complete /24 network before
>>>>> any
>>>>> > > packet reaches a server at all, and for the individual servers are
>>>>> > > simimlar rules with lower values per 2 seconds in place
>>>>> > >
>>>>> > > when the request reachs the webserver damage is long done and if no
>>>>> > > damage is done you are wasting expensive ressources with the rules
>>>>> > >
>>>>> > > Chain INBOUND (2 references)
>>>>> > >  pkts bytes target     prot opt in     out     source
>>>>> > > destination
>>>>> > >  1914  183K IPST_ALL   all  --  *      *       0.0.0.0/0
>>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=>
>>>>> > > 0.0.0.0/0
>>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=>
>>>>>           recent: UPDATE seconds: 2 hit_count: 250 TTL-Match
>>>>> > > name: limit_all_global side: source mask: 255.255.255.255
>>>>> > >  149K   15M DROP_ALL   all  --  *      *       0.0.0.0/0
>>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=>
>>>>> > > 0.0.0.0/0
>>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=>
>>>>>           recent: UPDATE seconds: 2 reap hit_count: 150
>>>>> > > TTL-Match name: limit_all_global side: source mask: 255.255.255.255
>>>>> > >
>>>>> > >
>>>>> > > _______________________________________________
>>>>> > > mod-security-users mailing list
>>>>> > > mod...@li...
>>>>> > > https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_mod-2Dsecurity-2Dusers&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=RUDsPA0iq5WVkW20NWQOl8suSJ4RvNfYZ6TM3FXNtdM&e=>
>>>>> > > Commercial ModSecurity Rules and Support from Trustwave's
>>>>> SpiderLabs:
>>>>> > > http://www.modsecurity.org/projects/commercial/rules/
>>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_rules_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=rQF299h7fZhdJbhudnhzjEcP4e3Aa8qCG0KvKi4CKiM&e=>
>>>>> > > http://www.modsecurity.org/projects/commercial/support/
>>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_support_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=ofF4OfFPsr3nKEMOH7j-CQmBqLgK_51fvOnQTavYK3c&e=>
>>>>> > >
>>>>>
>>>>>
>>>>> > _______________________________________________
>>>>> > mod-security-users mailing list
>>>>> > mod...@li...
>>>>> > https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_mod-2Dsecurity-2Dusers&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=RUDsPA0iq5WVkW20NWQOl8suSJ4RvNfYZ6TM3FXNtdM&e=>
>>>>> > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>>>> > http://www.modsecurity.org/projects/commercial/rules/
>>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_rules_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=rQF299h7fZhdJbhudnhzjEcP4e3Aa8qCG0KvKi4CKiM&e=>
>>>>> > http://www.modsecurity.org/projects/commercial/support/
>>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_support_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=ofF4OfFPsr3nKEMOH7j-CQmBqLgK_51fvOnQTavYK3c&e=>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> mod-security-users mailing list
>>>>> mod...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_mod-2Dsecurity-2Dusers&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=RUDsPA0iq5WVkW20NWQOl8suSJ4RvNfYZ6TM3FXNtdM&e=>
>>>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>>>> http://www.modsecurity.org/projects/commercial/rules/
>>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_rules_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=rQF299h7fZhdJbhudnhzjEcP4e3Aa8qCG0KvKi4CKiM&e=>
>>>>> http://www.modsecurity.org/projects/commercial/support/
>>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_support_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=ofF4OfFPsr3nKEMOH7j-CQmBqLgK_51fvOnQTavYK3c&e=>
>>>>>
>>>>> _______________________________________________
>>>>> mod-security-users mailing list
>>>>> mod...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>>>> http://www.modsecurity.org/projects/commercial/rules/
>>>>> http://www.modsecurity.org/projects/commercial/support/
>>>>>
>>>> _______________________________________________
>>>> mod-security-users mailing list
>>>> mod...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>>> http://www.modsecurity.org/projects/commercial/rules/
>>>> http://www.modsecurity.org/projects/commercial/support/
>>>>
>>> _______________________________________________
>>> mod-security-users mailing list
>>> mod...@li...
>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>> http://www.modsecurity.org/projects/commercial/rules/
>>> http://www.modsecurity.org/projects/commercial/support/
>>>
>> _______________________________________________
>> mod-security-users mailing list
>> mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> http://www.modsecurity.org/projects/commercial/rules/
>> http://www.modsecurity.org/projects/commercial/support/
>>
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/
>

Re: [mod-security-users] How to limit access rate by header?

From: Luciano G. F. <luc...@gm...> - 2018-12-12 21:17:25

Oh, I didn't realize we were not anymore in the main mailing thread. I'm
re-joining it from here. Just in case I've documented all I could in my own
question here:
https://stackoverflow.com/questions/53620557/modsecurity-apache-how-to-limit-access-rate-by-header

BTW, I'm following the advice of removing the "pause" part. I'm not sure
about why it was needed in the first place, but I left it there because I
saw it in other example snippets I found (like
https://gist.github.com/josnidhin/91d1ea9cd71fde386c27a9228476834e). I
thought it was something related to stop successive hits (like trying to
delay the origin request), but I'm not sure it makes a lot of sense here...

> Hmm. Do not know, but %{matched_var} is generally a good friend. Did you
> try it with capture? I mostly stick to matched_var.
I'm not sure I understand the "capture" part. The rule was entirely the
same, just replaced tx.0 with matched_var. @pm is the capture part you are
refering to? I'm using @pmf so I think it's the same mechanism. I'm not
seeing anything else in the docs for TX usage:
https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual-(v2.x)#TX

> And I'd do an empty
> line between 11 and 12, but nothing else springs to mind.
I'm kind obsessive with coding standards. In this case, I didn't add an
empty line there just to make it look like a single rule (I have other
custom rules in that file), but I wasn't sure it was ok. For what I see,
indentation with no empty lines in main directives are only for chained
rules. On the other hand, rules helper blocks follow the format:

 76 #
 77 # [ description ]
 78 #
 79 # - reference X
 80 #

I don't know if that helper block is following some docs standard so it can
be parsed by some tool (eg. Javadoc).

Have a nice day.

El mié., 12 de dic. de 2018 a la(s) 17:11, Christian Folini (
chr...@ne...) escribió:

> Hey, hey,
>
> On Wed, Dec 12, 2018 at 02:18:47PM -0300, Luciano Guillermo Fantuzzi wrote:
> > Well, it finally worked with %{matched_var} instead of %{tx.0}. I don't
> > know why, because according to docs, tx.0 should contain the matching
> value
> > of @pm (I assume @pmf works the same way):
>
> Hmm. Do not know, but %{matched_var} is generally a good friend. Did you
> try it with capture? I mostly stick to matched_var.
>
> > Following your tip of t:sha1 in the same line didn't work for some
> reason.
>
> Oops. Forgot to tell you about the t:hexEncode. t:sha1 is binary, it takes
> the encoding to become useful as key. Sorry.
>
> > So I replaced %{matched_var} with %{tx.ua_hash} and it still works and I
> > think it's a more consistent way in case I need a more complex UA match.
>
> Very good.
>
> >   9 # Limit client hits by user agent
> >  10 SecRule REQUEST_HEADERS:User-Agent "@pmf
> data/ratelimit-clients.data" \
> >  11
> >
> "id:400009,phase:2,nolog,pass,setuid:%{tx.ua_hash},setvar:user.ratelimit_client=+1,expirevar:user.ratelimit_client=3"
> >  12 SecRule USER:RATELIMIT_CLIENT "@gt 1" \
> >  13
> >
> "chain,id:4000010,phase:2,pause:300,deny,status:429,setenv:RATELIMITED,log,msg:'RATELIMITED
> > BOT'"
> >  14     SecRule REQUEST_HEADERS:User-Agent "@pmf
> > data/ratelimit-clients.data"
> >  15 Header always set Retry-After "3" env=RATELIMITED
> >  16 ErrorDocument 429 "Too Many Requests"
>
> That looks quite good. I'd move into the rule range below 100000, though.
>
> The pause, deny combination actually blocks the server too. You could
> simply issue a drop. That's lighter on the server.
>
> > Is there some coding standards to write these rules?
>
> CRS has a coding guideline on the wiki on github. But that's just a
> convention. Personally, I put chain at the end of a rule. And I'd do an
> empty
> line between 11 and 12, but nothing else springs to mind.
>
> > Thank you for your tips, Christian.
>
> You're welcome. Glad it worked out for your in the end. It would be nice if
> you could post your recipe to the mailinglist. It might help other people
> in a
> similar situation in the future.
>
> Ahoj,
>
> Christian
>
>
>
> --
> A political leader must keep looking over his shoulder all the
> time to see if the boys are still there. If they aren’t still there,
> he’s no longer a political leader.
> -- Bernard Baruch
>

Re: [mod-security-users] How to limit access rate by header?

From: Luciano G. F. <luc...@gm...> - 2018-12-12 21:24:07

And the final rule is:

# Limit client hits by user agent
SecRule REQUEST_HEADERS:User-Agent "@pmf data/ratelimit-clients.data" \

"id:100008,phase:2,nolog,pass,setuid:%{tx.ua_hash},setvar:user.ratelimit_client=+1,expirevar:user.ratelimit_client=3"

SecRule USER:RATELIMIT_CLIENT "@gt 1" \

"chain,id:1000009,phase:2,deny,status:429,setenv:RATELIMITED,log,msg:'RATELIMITED
BOT'"

    SecRule REQUEST_HEADERS:User-Agent "@pmf data/ratelimit-clients.data"

Header always set Retry-After "3" env=RATELIMITED

ErrorDocument 429 "Too Many Requests"

El mié., 12 de dic. de 2018 a la(s) 18:16, Luciano Guillermo Fantuzzi (
luc...@gm...) escribió:

> Oh, I didn't realize we were not anymore in the main mailing thread. I'm
> re-joining it from here. Just in case I've documented all I could in my own
> question here:
> https://stackoverflow.com/questions/53620557/modsecurity-apache-how-to-limit-access-rate-by-header
>
> BTW, I'm following the advice of removing the "pause" part. I'm not sure
> about why it was needed in the first place, but I left it there because I
> saw it in other example snippets I found (like
> https://gist.github.com/josnidhin/91d1ea9cd71fde386c27a9228476834e). I
> thought it was something related to stop successive hits (like trying to
> delay the origin request), but I'm not sure it makes a lot of sense here...
>
> > Hmm. Do not know, but %{matched_var} is generally a good friend. Did you
> > try it with capture? I mostly stick to matched_var.
> I'm not sure I understand the "capture" part. The rule was entirely the
> same, just replaced tx.0 with matched_var. @pm is the capture part you are
> refering to? I'm using @pmf so I think it's the same mechanism. I'm not
> seeing anything else in the docs for TX usage:
> https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual-(v2.x)#TX
>
> > And I'd do an empty
> > line between 11 and 12, but nothing else springs to mind.
> I'm kind obsessive with coding standards. In this case, I didn't add an
> empty line there just to make it look like a single rule (I have other
> custom rules in that file), but I wasn't sure it was ok. For what I see,
> indentation with no empty lines in main directives are only for chained
> rules. On the other hand, rules helper blocks follow the format:
>
>  76 #
>  77 # [ description ]
>  78 #
>  79 # - reference X
>  80 #
>
> I don't know if that helper block is following some docs standard so it
> can be parsed by some tool (eg. Javadoc).
>
> Have a nice day.
>
> El mié., 12 de dic. de 2018 a la(s) 17:11, Christian Folini (
> chr...@ne...) escribió:
>
>> Hey, hey,
>>
>> On Wed, Dec 12, 2018 at 02:18:47PM -0300, Luciano Guillermo Fantuzzi
>> wrote:
>> > Well, it finally worked with %{matched_var} instead of %{tx.0}. I don't
>> > know why, because according to docs, tx.0 should contain the matching
>> value
>> > of @pm (I assume @pmf works the same way):
>>
>> Hmm. Do not know, but %{matched_var} is generally a good friend. Did you
>> try it with capture? I mostly stick to matched_var.
>>
>> > Following your tip of t:sha1 in the same line didn't work for some
>> reason.
>>
>> Oops. Forgot to tell you about the t:hexEncode. t:sha1 is binary, it takes
>> the encoding to become useful as key. Sorry.
>>
>> > So I replaced %{matched_var} with %{tx.ua_hash} and it still works and I
>> > think it's a more consistent way in case I need a more complex UA match.
>>
>> Very good.
>>
>> >   9 # Limit client hits by user agent
>> >  10 SecRule REQUEST_HEADERS:User-Agent "@pmf
>> data/ratelimit-clients.data" \
>> >  11
>> >
>> "id:400009,phase:2,nolog,pass,setuid:%{tx.ua_hash},setvar:user.ratelimit_client=+1,expirevar:user.ratelimit_client=3"
>> >  12 SecRule USER:RATELIMIT_CLIENT "@gt 1" \
>> >  13
>> >
>> "chain,id:4000010,phase:2,pause:300,deny,status:429,setenv:RATELIMITED,log,msg:'RATELIMITED
>> > BOT'"
>> >  14     SecRule REQUEST_HEADERS:User-Agent "@pmf
>> > data/ratelimit-clients.data"
>> >  15 Header always set Retry-After "3" env=RATELIMITED
>> >  16 ErrorDocument 429 "Too Many Requests"
>>
>> That looks quite good. I'd move into the rule range below 100000, though.
>>
>> The pause, deny combination actually blocks the server too. You could
>> simply issue a drop. That's lighter on the server.
>>
>> > Is there some coding standards to write these rules?
>>
>> CRS has a coding guideline on the wiki on github. But that's just a
>> convention. Personally, I put chain at the end of a rule. And I'd do an
>> empty
>> line between 11 and 12, but nothing else springs to mind.
>>
>> > Thank you for your tips, Christian.
>>
>> You're welcome. Glad it worked out for your in the end. It would be nice
>> if
>> you could post your recipe to the mailinglist. It might help other people
>> in a
>> similar situation in the future.
>>
>> Ahoj,
>>
>> Christian
>>
>>
>>
>> --
>> A political leader must keep looking over his shoulder all the
>> time to see if the boys are still there. If they aren’t still there,
>> he’s no longer a political leader.
>> -- Bernard Baruch
>>
>

Re: [mod-security-users] How to limit access rate by header?

From: Christian F. <chr...@ne...> - 2018-12-12 21:30:52

On Wed, Dec 12, 2018 at 06:16:49PM -0300, Luciano Guillermo Fantuzzi wrote:
> Oh, I didn't realize we were not anymore in the main mailing thread. I'm
> re-joining it from here.

Yes, I took it private after things turned sour following my comment.

Glad it worked out for your in the end.

Ahoj,

Christian

-- 
If liberty means anything at all, it means the right to tell people
what they do not want to hear.
-- George Orwell

Re: [mod-security-users] How to limit access rate by header?

From: Luciano G. F. <luc...@gm...> - 2018-12-12 23:16:38

Something I couldn't find in the docs. Is it possible to avoid logging in
the access log? With nolog action I can avoid logging it in error log (and
audit log, but it's turned off), but I couldn't find a way to avoid
displaying a message in the access log on every rule match. I find this
important because my idea was to sepparate logs and keep the access log as
clean as possible so I can analyze bots/crawlers not being catched in my
rules.

Thanks.

El mié., 12 de dic. de 2018 a la(s) 18:32, Christian Folini (
chr...@ne...) escribió:

> On Wed, Dec 12, 2018 at 06:16:49PM -0300, Luciano Guillermo Fantuzzi wrote:
> > Oh, I didn't realize we were not anymore in the main mailing thread. I'm
> > re-joining it from here.
>
> Yes, I took it private after things turned sour following my comment.
>
> Glad it worked out for your in the end.
>
> Ahoj,
>
> Christian
>
> --
> If liberty means anything at all, it means the right to tell people
> what they do not want to hear.
> -- George Orwell
>
>
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/
>

Re: [mod-security-users] How to limit access rate by header?

From: Reindl H. <h.r...@th...> - 2018-12-12 23:20:11


Am 13.12.18 um 00:16 schrieb Luciano Guillermo Fantuzzi:
> Something I couldn't find in the docs. Is it possible to avoid logging
> in the access log? With nolog action I can avoid logging it in error log
> (and audit log, but it's turned off), but I couldn't find a way to avoid
> displaying a message in the access log on every rule match. I find this
> important because my idea was to sepparate logs and keep the access log
> as clean as possible so I can analyze bots/crawlers not being catched in
> my rules.

the access log is not written by modsec at all

what you normally see is just the typical access entry with the repsonse
code be it triggered by modsec or a php-script returning a error code
and that's it

Re: [mod-security-users] How to limit access rate by header?

From: Manuel S. <spa...@gm...> - 2018-12-13 00:37:35

Hi Luciano, you can achieve that with environment variables and a custom log format with an env condition in Apache.
Modsec writes error and audit log only, access log tricks can be useful to prevent IP/personal/financial leaks.

Cheers!

Sent from my iPhone

> On 12 Dec 2018, at 18:16, Luciano Guillermo Fantuzzi <luc...@gm...> wrote:
> 
> Something I couldn't find in the docs. Is it possible to avoid logging in the access log? With nolog action I can avoid logging it in error log (and audit log, but it's turned off), but I couldn't find a way to avoid displaying a message in the access log on every rule match. I find this important because my idea was to sepparate logs and keep the access log as clean as possible so I can analyze bots/crawlers not being catched in my rules.
> 
> Thanks.
> 
> El mié., 12 de dic. de 2018 a la(s) 18:32, Christian Folini (chr...@ne...) escribió:
>> On Wed, Dec 12, 2018 at 06:16:49PM -0300, Luciano Guillermo Fantuzzi wrote:
>> > Oh, I didn't realize we were not anymore in the main mailing thread. I'm
>> > re-joining it from here.
>> 
>> Yes, I took it private after things turned sour following my comment.
>> 
>> Glad it worked out for your in the end.
>> 
>> Ahoj,
>> 
>> Christian
>> 
>> -- 
>> If liberty means anything at all, it means the right to tell people
>> what they do not want to hear.
>> -- George Orwell
>> 
>> 
>> _______________________________________________
>> mod-security-users mailing list
>> mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> http://www.modsecurity.org/projects/commercial/rules/
>> http://www.modsecurity.org/projects/commercial/support/
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/

Re: [mod-security-users] How to limit access rate by header?

From: Luciano G. F. <luc...@gm...> - 2018-12-13 01:05:48

Yes, I had to do it that way. Just in case:

---
# Local messages
SetEnvIf Remote_Addr "127\.0\.0\.1" dontlog
SetEnvIf Remote_Addr "::1" dontlog
SetEnvIfNoCase User-Agent "internal dummy connection" dontlog

# Not interesting
SetEnvIfNoCase Request_URI "^/robots\.txt$" dontlog

# Only relevant for modsec log (it will output to error log eventually)
# Note: Keep this list updated with /etc/modsecurity/data/*.data
SetEnvIfNoCase User-Agent (Googlebot|bingbot\


|AhrefsBot|MJ12bot|trovitBot|AwarioRssBot|Semrush|DotBot|BLEXBot|YandexBot|YandexMobileBot|PaperLiBot|Baidu|ZoominfoBot\
|facebookexternalhit) dontlog

CustomLog ${APACHE_LOG_DIR}/other_vhosts_access.log vhost_combined
env=!dontlog
---

The only thing that I don't like of this approach is that I can't pass
SetEnvIf a file to read, so I'll have to keep both (*.data and .conf files)
in sync.


El mié., 12 de dic. de 2018 a la(s) 21:39, Manuel Spartan (
spa...@gm...) escribió:

> Hi Luciano, you can achieve that with environment variables and a custom
> log format with an env condition in Apache.
> Modsec writes error and audit log only, access log tricks can be useful to
> prevent IP/personal/financial leaks.
>
> Cheers!
>
> Sent from my iPhone
>
> On 12 Dec 2018, at 18:16, Luciano Guillermo Fantuzzi <
> luc...@gm...> wrote:
>
> Something I couldn't find in the docs. Is it possible to avoid logging in
> the access log? With nolog action I can avoid logging it in error log (and
> audit log, but it's turned off), but I couldn't find a way to avoid
> displaying a message in the access log on every rule match. I find this
> important because my idea was to sepparate logs and keep the access log as
> clean as possible so I can analyze bots/crawlers not being catched in my
> rules.
>
> Thanks.
>
> El mié., 12 de dic. de 2018 a la(s) 18:32, Christian Folini (
> chr...@ne...) escribió:
>
>> On Wed, Dec 12, 2018 at 06:16:49PM -0300, Luciano Guillermo Fantuzzi
>> wrote:
>> > Oh, I didn't realize we were not anymore in the main mailing thread. I'm
>> > re-joining it from here.
>>
>> Yes, I took it private after things turned sour following my comment.
>>
>> Glad it worked out for your in the end.
>>
>> Ahoj,
>>
>> Christian
>>
>> --
>> If liberty means anything at all, it means the right to tell people
>> what they do not want to hear.
>> -- George Orwell
>>
>>
>> _______________________________________________
>> mod-security-users mailing list
>> mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> http://www.modsecurity.org/projects/commercial/rules/
>> http://www.modsecurity.org/projects/commercial/support/
>>
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/
>
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/
>

Re: [mod-security-users] How to limit access rate by header?

From: Manuel S. <spa...@gm...> - 2018-12-13 01:25:34

Try a modsec with ctl:setenv with pmf to the bot file for request_headers:user-agent. So if the header is in the file then set the nolog env.
Having two update point is painful on the long run.
Cheers!

Sent from my iPhone

> On 12 Dec 2018, at 20:05, Luciano Guillermo Fantuzzi <luc...@gm...> wrote:
> 
> Yes, I had to do it that way. Just in case:
> 
> ---
> # Local messages
> SetEnvIf Remote_Addr "127\.0\.0\.1" dontlog
> SetEnvIf Remote_Addr "::1" dontlog
> SetEnvIfNoCase User-Agent "internal dummy connection" dontlog
> 
> # Not interesting
> SetEnvIfNoCase Request_URI "^/robots\.txt$" dontlog
> 
> # Only relevant for modsec log (it will output to error log eventually)
> # Note: Keep this list updated with /etc/modsecurity/data/*.data
> SetEnvIfNoCase User-Agent (Googlebot|bingbot\                                                                                                                                  
> |AhrefsBot|MJ12bot|trovitBot|AwarioRssBot|Semrush|DotBot|BLEXBot|YandexBot|YandexMobileBot|PaperLiBot|Baidu|ZoominfoBot\
> |facebookexternalhit) dontlog
> 
> CustomLog ${APACHE_LOG_DIR}/other_vhosts_access.log vhost_combined env=!dontlog
> ---
> 
> The only thing that I don't like of this approach is that I can't pass SetEnvIf a file to read, so I'll have to keep both (*.data and .conf files) in sync.
> 
> 
> El mié., 12 de dic. de 2018 a la(s) 21:39, Manuel Spartan (spa...@gm...) escribió:
>> Hi Luciano, you can achieve that with environment variables and a custom log format with an env condition in Apache.
>> Modsec writes error and audit log only, access log tricks can be useful to prevent IP/personal/financial leaks.
>> 
>> Cheers!
>> 
>> Sent from my iPhone
>> 
>>> On 12 Dec 2018, at 18:16, Luciano Guillermo Fantuzzi <luc...@gm...> wrote:
>>> 
>>> Something I couldn't find in the docs. Is it possible to avoid logging in the access log? With nolog action I can avoid logging it in error log (and audit log, but it's turned off), but I couldn't find a way to avoid displaying a message in the access log on every rule match. I find this important because my idea was to sepparate logs and keep the access log as clean as possible so I can analyze bots/crawlers not being catched in my rules.
>>> 
>>> Thanks.
>>> 
>>> El mié., 12 de dic. de 2018 a la(s) 18:32, Christian Folini (chr...@ne...) escribió:
>>>> On Wed, Dec 12, 2018 at 06:16:49PM -0300, Luciano Guillermo Fantuzzi wrote:
>>>> > Oh, I didn't realize we were not anymore in the main mailing thread. I'm
>>>> > re-joining it from here.
>>>> 
>>>> Yes, I took it private after things turned sour following my comment.
>>>> 
>>>> Glad it worked out for your in the end.
>>>> 
>>>> Ahoj,
>>>> 
>>>> Christian
>>>> 
>>>> -- 
>>>> If liberty means anything at all, it means the right to tell people
>>>> what they do not want to hear.
>>>> -- George Orwell
>>>> 
>>>> 
>>>> _______________________________________________
>>>> mod-security-users mailing list
>>>> mod...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>>> http://www.modsecurity.org/projects/commercial/rules/
>>>> http://www.modsecurity.org/projects/commercial/support/
>>> _______________________________________________
>>> mod-security-users mailing list
>>> mod...@li...
>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>> http://www.modsecurity.org/projects/commercial/rules/
>>> http://www.modsecurity.org/projects/commercial/support/
>> _______________________________________________
>> mod-security-users mailing list
>> mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> http://www.modsecurity.org/projects/commercial/rules/
>> http://www.modsecurity.org/projects/commercial/support/
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/

Re: [mod-security-users] How to limit access rate by header?

From: Luciano G. F. <luc...@gm...> - 2018-12-13 02:01:14

It perfectly worked! I've just combined setenv:dontlog in all rules I
needed and that's it. This really helps me to keep an eye in the logs.

Thank you!

El mié., 12 de dic. de 2018 a la(s) 22:26, Manuel Spartan (
spa...@gm...) escribió:

> Try a modsec with ctl:setenv with pmf to the bot file for
> request_headers:user-agent. So if the header is in the file then set the
> nolog env.
> Having two update point is painful on the long run.
> Cheers!
>
> Sent from my iPhone
>
> On 12 Dec 2018, at 20:05, Luciano Guillermo Fantuzzi <
> luc...@gm...> wrote:
>
> Yes, I had to do it that way. Just in case:
>
> ---
> # Local messages
> SetEnvIf Remote_Addr "127\.0\.0\.1" dontlog
> SetEnvIf Remote_Addr "::1" dontlog
> SetEnvIfNoCase User-Agent "internal dummy connection" dontlog
>
> # Not interesting
> SetEnvIfNoCase Request_URI "^/robots\.txt$" dontlog
>
> # Only relevant for modsec log (it will output to error log eventually)
> # Note: Keep this list updated with /etc/modsecurity/data/*.data
> SetEnvIfNoCase User-Agent (Googlebot|bingbot\
>
>
>
> |AhrefsBot|MJ12bot|trovitBot|AwarioRssBot|Semrush|DotBot|BLEXBot|YandexBot|YandexMobileBot|PaperLiBot|Baidu|ZoominfoBot\
> |facebookexternalhit) dontlog
>
> CustomLog ${APACHE_LOG_DIR}/other_vhosts_access.log vhost_combined
> env=!dontlog
> ---
>
> The only thing that I don't like of this approach is that I can't pass
> SetEnvIf a file to read, so I'll have to keep both (*.data and .conf files)
> in sync.
>
>
> El mié., 12 de dic. de 2018 a la(s) 21:39, Manuel Spartan (
> spa...@gm...) escribió:
>
>> Hi Luciano, you can achieve that with environment variables and a custom
>> log format with an env condition in Apache.
>> Modsec writes error and audit log only, access log tricks can be useful
>> to prevent IP/personal/financial leaks.
>>
>> Cheers!
>>
>> Sent from my iPhone
>>
>> On 12 Dec 2018, at 18:16, Luciano Guillermo Fantuzzi <
>> luc...@gm...> wrote:
>>
>> Something I couldn't find in the docs. Is it possible to avoid logging in
>> the access log? With nolog action I can avoid logging it in error log (and
>> audit log, but it's turned off), but I couldn't find a way to avoid
>> displaying a message in the access log on every rule match. I find this
>> important because my idea was to sepparate logs and keep the access log as
>> clean as possible so I can analyze bots/crawlers not being catched in my
>> rules.
>>
>> Thanks.
>>
>> El mié., 12 de dic. de 2018 a la(s) 18:32, Christian Folini (
>> chr...@ne...) escribió:
>>
>>> On Wed, Dec 12, 2018 at 06:16:49PM -0300, Luciano Guillermo Fantuzzi
>>> wrote:
>>> > Oh, I didn't realize we were not anymore in the main mailing thread.
>>> I'm
>>> > re-joining it from here.
>>>
>>> Yes, I took it private after things turned sour following my comment.
>>>
>>> Glad it worked out for your in the end.
>>>
>>> Ahoj,
>>>
>>> Christian
>>>
>>> --
>>> If liberty means anything at all, it means the right to tell people
>>> what they do not want to hear.
>>> -- George Orwell
>>>
>>>
>>> _______________________________________________
>>> mod-security-users mailing list
>>> mod...@li...
>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>> http://www.modsecurity.org/projects/commercial/rules/
>>> http://www.modsecurity.org/projects/commercial/support/
>>>
>> _______________________________________________
>> mod-security-users mailing list
>> mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> http://www.modsecurity.org/projects/commercial/rules/
>> http://www.modsecurity.org/projects/commercial/support/
>>
>> _______________________________________________
>> mod-security-users mailing list
>> mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> http://www.modsecurity.org/projects/commercial/rules/
>> http://www.modsecurity.org/projects/commercial/support/
>>
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/
>
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/
>

Re: [mod-security-users] How to limit access rate by header?

From: Luciano G. F. <luc...@gm...> - 2018-12-07 15:22:36

@Christian First of all, you are completly wrong. Let me explain:

1. I'm using Cloudflare (free plan) just to save traffic. So no, I'm not a
rich guy and I'm looking for a solution server side, because since I'm in
the free plan I can't use their WAF solution.
2. I'm not a guy looking for someone to do my job. Maybe you didn't see my
last email, where I wrote the rule I created and that is not working for
some reason. Before spending my time creating an account here and writing
emails, I googled the entire day and tried different approaches, but no one
worked. I didn't find a single piece of code doing what I need to do, so I
tried with different IP rate limit snippets, but after debugging some time
I'm facing an error that no one else seem to see. This is why I'm stuck
here.
3. I'm not paying anyone to do something like this. Maybe you are the kind
of people that solve your problems with money and in the easiest way.
Sorry, I'm not that kind of people.

And last, I don't want the help of arrogant guys like you. I don't know the
reason you think you can talk to people the way you do, specially when they
are asking for help. Like if you were born knowing about everything... I
can only thank I don't know you and I don't have people like you in my
life. Fortunately, you are part of a minority.

Don't write me back again.

El vie., 7 de dic. de 2018 a la(s) 09:31, Christian Folini (
chr...@ne...) escribió:

> Luciano,
>
> I understand your troubles. But if you are behind cloudflare and you are
> giving cloudflare money to solve your problems, why are you coming here
> asking for free support?
>
> You are facing a special problem and you need a special solution. We have
> given you pointers and hints but it seems it is not enough, so it is
> likely you need to dig deeper and learn more - or you pay somebody
> to do that for you.
>
> What I am not going to do - and I doubt somebody else is willing to do
> that -
> is investing 1-2-3 hours into developing a solution for you in my spare
> time. And it is very likely this would take 1-2-3 hours for me, so do not
> expect a quick win.
>
> Just my 2 cents. Good luck,
>
> Christian
>
> On Fri, Dec 07, 2018 at 09:17:45AM -0300, Luciano Guillermo Fantuzzi wrote:
> > Already tried that, but still the same message in debug log. Anyway, I'm
> > behind Cloudflare so I need to access that var (that contains the real
> IP)
> > from header. Moreover, I tried with global collection and same luck. Am I
> > missing some initialization step in modsec?
> >
> > Thanks.
> >
> > El vie., 7 dic. 2018 00:56, Scheblein, Adam <
> ada...@ma...>
> > escribió:
> >
> > > I had a similar problem.  You need to initialize the collection with
> > > something like this:
> > >
> > >
> > >
> > >                 SecAction
> > > id:'2000000',phase:1,nolog,pass,initcol:IP=%{REMOTE_ADDR}
> > >
> > >
> > >
> > > *From: *Luciano Guillermo Fantuzzi <luc...@gm...>
> > > *Reply-To: *"mod...@li..." <
> > > mod...@li...>
> > > *Date: *Thursday, December 6, 2018 at 8:51 PM
> > > *To: *"mod...@li..." <
> > > mod...@li...>
> > > *Subject: *Re: [mod-security-users] How to limit access rate by header?
> > >
> > >
> > >
> > > I've very frustrated... I can't make it work, even for IP control.
> What am
> > > I doing wrong here? It always returns:
> > >
> > > Could not set variable "IP.access_count" as the collection does not
> exist.
> > >
> > >
> > >
> > > 105 <LocationMatch "^/.*">
> > >
> > > 109   SecRule REQUEST_HEADERS:CF-Connecting-IP "@unconditionalMatch"
> > > "phase:2,initcol:IP=%{MATCHED_VAR},pass,nolog,id:35003"
> > >
> > > 112   SecRule IP:ACCESS_COUNT "@gt 1"
> > >
> "phase:2,pause:300,deny,status:503,setenv:RATELIMITED,skip:1,nolog,id:35004"
> > >
> > > 116   SecAction "phase:2,setvar:IP.access_count=+1,pass,nolog,id:35005"
> > >
> > >
> > >
> > > 119   SecAction
> > > "phase:5,deprecatevar:IP.access_count=1/10,pass,nolog,id:35006"
> > >
> > > 122   Header always set Retry-After "10" env=RATELIMITED
> > >
> > > 123 </LocationMatch>
> > >
> > > 124
> > >
> > > 125 ErrorDocument 503 "Service Unavailable"
> > >
> > >
> > >
> > >
> > >
> > > El jue., 6 de dic. de 2018 a la(s) 20:38, Luciano Guillermo Fantuzzi (
> > > luc...@gm...) escribió:
> > >
> > > Thank you for your answer, Christian. Do you think it's possible for
> you
> > > to just build the first part of the rule (in Modsec)? I'm trying but
> I'm
> > > not understanding how variables work with the global scope. I was be
> able
> > > to build some basic rules like:
> > >
> > >
> > >
> > > # Banned Bots and Crawlers
> > >
> > > SecRule REQUEST_HEADERS:User-Agent "@pmFromFile blacklist-bots.data" \
> > >
> > >     "id:350001,phase:1,t:none,deny,log,msg:'BANNED BOT'"
> > >
> > >
> > >
> > > # Specific IPs
> > >
> > > SecRule REMOTE_ADDR "@pmFromFile blacklist-ip.data" \
> > >
> > >     "id:350002,phase:1,t:none,deny,log,msg:'BANNED IP'"
> > >
> > >
> > >
> > > I'm trying to understand examples from stackoverflow and different
> places,
> > > but they are all intended to limit by IP and for specific resources
> (the
> > > scope of the rule). Eg.:
> > >
> > > https://gist.github.com/josnidhin/91d1ea9cd71fde386c27a9228476834e
> > > <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_josnidhin_91d1ea9cd71fde386c27a9228476834e&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=L7iKlxwUA3exA-ByaKl7gyvQkoOevQwuEjv4ZKC6hOY&e=
> >
> > >
> > >
> > >
> > > I'm not asking for the entire rule, just an example of how var counters
> > > work in the global scope (directly in
> /etc/modsecurity/modsecurity.conf)
> > > and how can I connect them to sum by header instead of IP.
> > >
> > >
> > >
> > > Thank you!
> > >
> > >
> > >
> > > El jue., 6 de dic. de 2018 a la(s) 10:30, Christian Folini (
> > > chr...@ne...) escribió:
> > >
> > > Hello Luciano,
> > >
> > > You have a peculiar use case, but I see your thinking.
> > >
> > > There are examples in the ModSecurity books that are really close to
> your
> > > plan. They should be easy to adopt.
> > >
> > > Other than that, you may want to look into mod_qos. It has
> functionality
> > > that might be useful in your case.
> > >
> > > Best,
> > >
> > > Christian
> > >
> > >
> > > On Wed, Dec 05, 2018 at 06:26:03PM -0300, Luciano Guillermo Fantuzzi
> wrote:
> > > > Thank you for your answer, but maybe I'm not asking it the right way
> or
> > > > this is not the right place to ask(?).
> > > >
> > > > I need a Modsecurity rule (I'm using it through Apache) to be able to
> > > > control hits from clients with a specific header, like
> > > > "facebookexternalhit/1.1".
> > > > Ie. to stop some agressive bots hitting too often my webservers and
> > > taking
> > > > them down eventually. I don't want to block them at all because I
> need
> > > some
> > > > of them (like Facebook bot to parse shared content), but I need a
> way to
> > > > tell them "stop, retry in some seconds".
> > > >
> > > > Thanks.
> > > >
> > > > El mié., 5 de dic. de 2018 a la(s) 16:16, Reindl Harald (
> > > > h.r...@th...) escribió:
> > > >
> > > > >
> > > > >
> > > > > Am 05.12.18 um 16:57 schrieb Luciano Guillermo Fantuzzi:
> > > > > > First of all, I'm new here so I'm not sure this is the right
> place
> > > for
> > > > > > asking for help (free modsec version). If it's not, I'll really
> > > > > > appreciate it if you can tell me where should I go.
> > > > > >
> > > > > > I'm trying to limit hit rate by:
> > > > > >
> > > > > > 1. Request's header (like "facebookexternalhit").
> > > > > > 2. (All hits to non static resources)
> > > > > >
> > > > > > And then return a friendly "429 Too Many Requests" and
> "Retry-After:
> > > 3"
> > > > > > (seconds).
> > > > > > I know I can read a file of headers like:
> > > > > >
> > > > > > SecRule REQUEST_HEADERS:User-Agent "@pmFromFile
> ratelimit-bots.txt"
> > > > > >
> > > > > > But I'm getting trouble building the entire rule.
> > > > > >
> > > > > > Any help would be really appreciated. Thank you!
> > > > >
> > > > > this a non-iusse
> > > > >
> > > > > normally you have rate-limits per IP in place and they should not
> be
> > > > > within the application layer at all and in the best case not even
> on
> > > the
> > > > > same machine
> > > > >
> > > > > that below is from a firewall-vm on a complete /24 network before
> any
> > > > > packet reaches a server at all, and for the individual servers are
> > > > > simimlar rules with lower values per 2 seconds in place
> > > > >
> > > > > when the request reachs the webserver damage is long done and if no
> > > > > damage is done you are wasting expensive ressources with the rules
> > > > >
> > > > > Chain INBOUND (2 references)
> > > > >  pkts bytes target     prot opt in     out     source
> > > > > destination
> > > > >  1914  183K IPST_ALL   all  --  *      *       0.0.0.0/0
> > > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=
> >
> > > > > 0.0.0.0/0
> > > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=
> >
> > >           recent: UPDATE seconds: 2 hit_count: 250 TTL-Match
> > > > > name: limit_all_global side: source mask: 255.255.255.255
> > > > >  149K   15M DROP_ALL   all  --  *      *       0.0.0.0/0
> > > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=
> >
> > > > > 0.0.0.0/0
> > > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=
> >
> > >           recent: UPDATE seconds: 2 reap hit_count: 150
> > > > > TTL-Match name: limit_all_global side: source mask: 255.255.255.255
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > mod-security-users mailing list
> > > > > mod...@li...
> > > > > https://lists.sourceforge.net/lists/listinfo/mod-security-users
> > > <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_mod-2Dsecurity-2Dusers&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=RUDsPA0iq5WVkW20NWQOl8suSJ4RvNfYZ6TM3FXNtdM&e=
> >
> > > > > Commercial ModSecurity Rules and Support from Trustwave's
> SpiderLabs:
> > > > > http://www.modsecurity.org/projects/commercial/rules/
> > > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_rules_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=rQF299h7fZhdJbhudnhzjEcP4e3Aa8qCG0KvKi4CKiM&e=
> >
> > > > > http://www.modsecurity.org/projects/commercial/support/
> > > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_support_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=ofF4OfFPsr3nKEMOH7j-CQmBqLgK_51fvOnQTavYK3c&e=
> >
> > > > >
> > >
> > >
> > > > _______________________________________________
> > > > mod-security-users mailing list
> > > > mod...@li...
> > > > https://lists.sourceforge.net/lists/listinfo/mod-security-users
> > > <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_mod-2Dsecurity-2Dusers&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=RUDsPA0iq5WVkW20NWQOl8suSJ4RvNfYZ6TM3FXNtdM&e=
> >
> > > > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> > > > http://www.modsecurity.org/projects/commercial/rules/
> > > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_rules_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=rQF299h7fZhdJbhudnhzjEcP4e3Aa8qCG0KvKi4CKiM&e=
> >
> > > > http://www.modsecurity.org/projects/commercial/support/
> > > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_support_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=ofF4OfFPsr3nKEMOH7j-CQmBqLgK_51fvOnQTavYK3c&e=
> >
> > >
> > >
> > >
> > > _______________________________________________
> > > mod-security-users mailing list
> > > mod...@li...
> > > https://lists.sourceforge.net/lists/listinfo/mod-security-users
> > > <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_mod-2Dsecurity-2Dusers&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=RUDsPA0iq5WVkW20NWQOl8suSJ4RvNfYZ6TM3FXNtdM&e=
> >
> > > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> > > http://www.modsecurity.org/projects/commercial/rules/
> > > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_rules_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=rQF299h7fZhdJbhudnhzjEcP4e3Aa8qCG0KvKi4CKiM&e=
> >
> > > http://www.modsecurity.org/projects/commercial/support/
> > > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_support_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=ofF4OfFPsr3nKEMOH7j-CQmBqLgK_51fvOnQTavYK3c&e=
> >
> > >
> > > _______________________________________________
> > > mod-security-users mailing list
> > > mod...@li...
> > > https://lists.sourceforge.net/lists/listinfo/mod-security-users
> > > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> > > http://www.modsecurity.org/projects/commercial/rules/
> > > http://www.modsecurity.org/projects/commercial/support/
> > >
>
>
> > _______________________________________________
> > mod-security-users mailing list
> > mod...@li...
> > https://lists.sourceforge.net/lists/listinfo/mod-security-users
> > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> > http://www.modsecurity.org/projects/commercial/rules/
> > http://www.modsecurity.org/projects/commercial/support/
>
>
>
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/
>

Re: [mod-security-users] How to limit access rate by header?

From: Franziska B. <fra...@gm...> - 2018-12-09 11:14:32

Luciano,

It’s not fair to call someone, who publishes several free tutorials,
invests his free time in open source, and over the years answers tons
of questions on this mailing list, as arrogant, as someone who solves
his problems with money.
And in the end, you tell him he should never write you back again.
I could imagine that your tone probably disappoints and discourages people.

Best regards,
Franziska

Am Fr., 7. Dez. 2018 um 16:24 Uhr schrieb Luciano Guillermo Fantuzzi
<luc...@gm...>:
>
> @Christian First of all, you are completly wrong. Let me explain:
>
> 1. I'm using Cloudflare (free plan) just to save traffic. So no, I'm not a rich guy and I'm looking for a solution server side, because since I'm in the free plan I can't use their WAF solution.
> 2. I'm not a guy looking for someone to do my job. Maybe you didn't see my last email, where I wrote the rule I created and that is not working for some reason. Before spending my time creating an account here and writing emails, I googled the entire day and tried different approaches, but no one worked. I didn't find a single piece of code doing what I need to do, so I tried with different IP rate limit snippets, but after debugging some time I'm facing an error that no one else seem to see. This is why I'm stuck here.
> 3. I'm not paying anyone to do something like this. Maybe you are the kind of people that solve your problems with money and in the easiest way. Sorry, I'm not that kind of people.
>
> And last, I don't want the help of arrogant guys like you. I don't know the reason you think you can talk to people the way you do, specially when they are asking for help. Like if you were born knowing about everything... I can only thank I don't know you and I don't have people like you in my life. Fortunately, you are part of a minority.
>
> Don't write me back again.
>
> El vie., 7 de dic. de 2018 a la(s) 09:31, Christian Folini (chr...@ne...) escribió:
>>
>> Luciano,
>>
>> I understand your troubles. But if you are behind cloudflare and you are
>> giving cloudflare money to solve your problems, why are you coming here
>> asking for free support?
>>
>> You are facing a special problem and you need a special solution. We have
>> given you pointers and hints but it seems it is not enough, so it is
>> likely you need to dig deeper and learn more - or you pay somebody
>> to do that for you.
>>
>> What I am not going to do - and I doubt somebody else is willing to do that -
>> is investing 1-2-3 hours into developing a solution for you in my spare
>> time. And it is very likely this would take 1-2-3 hours for me, so do not
>> expect a quick win.
>>
>> Just my 2 cents. Good luck,
>>
>> Christian
>>
>> On Fri, Dec 07, 2018 at 09:17:45AM -0300, Luciano Guillermo Fantuzzi wrote:
>> > Already tried that, but still the same message in debug log. Anyway, I'm
>> > behind Cloudflare so I need to access that var (that contains the real IP)
>> > from header. Moreover, I tried with global collection and same luck. Am I
>> > missing some initialization step in modsec?
>> >
>> > Thanks.
>> >
>> > El vie., 7 dic. 2018 00:56, Scheblein, Adam <ada...@ma...>
>> > escribió:
>> >
>> > > I had a similar problem.  You need to initialize the collection with
>> > > something like this:
>> > >
>> > >
>> > >
>> > >                 SecAction
>> > > id:'2000000',phase:1,nolog,pass,initcol:IP=%{REMOTE_ADDR}
>> > >
>> > >
>> > >
>> > > *From: *Luciano Guillermo Fantuzzi <luc...@gm...>
>> > > *Reply-To: *"mod...@li..." <
>> > > mod...@li...>
>> > > *Date: *Thursday, December 6, 2018 at 8:51 PM
>> > > *To: *"mod...@li..." <
>> > > mod...@li...>
>> > > *Subject: *Re: [mod-security-users] How to limit access rate by header?
>> > >
>> > >
>> > >
>> > > I've very frustrated... I can't make it work, even for IP control. What am
>> > > I doing wrong here? It always returns:
>> > >
>> > > Could not set variable "IP.access_count" as the collection does not exist.
>> > >
>> > >
>> > >
>> > > 105 <LocationMatch "^/.*">
>> > >
>> > > 109   SecRule REQUEST_HEADERS:CF-Connecting-IP "@unconditionalMatch"
>> > > "phase:2,initcol:IP=%{MATCHED_VAR},pass,nolog,id:35003"
>> > >
>> > > 112   SecRule IP:ACCESS_COUNT "@gt 1"
>> > > "phase:2,pause:300,deny,status:503,setenv:RATELIMITED,skip:1,nolog,id:35004"
>> > >
>> > > 116   SecAction "phase:2,setvar:IP.access_count=+1,pass,nolog,id:35005"
>> > >
>> > >
>> > >
>> > > 119   SecAction
>> > > "phase:5,deprecatevar:IP.access_count=1/10,pass,nolog,id:35006"
>> > >
>> > > 122   Header always set Retry-After "10" env=RATELIMITED
>> > >
>> > > 123 </LocationMatch>
>> > >
>> > > 124
>> > >
>> > > 125 ErrorDocument 503 "Service Unavailable"
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > El jue., 6 de dic. de 2018 a la(s) 20:38, Luciano Guillermo Fantuzzi (
>> > > luc...@gm...) escribió:
>> > >
>> > > Thank you for your answer, Christian. Do you think it's possible for you
>> > > to just build the first part of the rule (in Modsec)? I'm trying but I'm
>> > > not understanding how variables work with the global scope. I was be able
>> > > to build some basic rules like:
>> > >
>> > >
>> > >
>> > > # Banned Bots and Crawlers
>> > >
>> > > SecRule REQUEST_HEADERS:User-Agent "@pmFromFile blacklist-bots.data" \
>> > >
>> > >     "id:350001,phase:1,t:none,deny,log,msg:'BANNED BOT'"
>> > >
>> > >
>> > >
>> > > # Specific IPs
>> > >
>> > > SecRule REMOTE_ADDR "@pmFromFile blacklist-ip.data" \
>> > >
>> > >     "id:350002,phase:1,t:none,deny,log,msg:'BANNED IP'"
>> > >
>> > >
>> > >
>> > > I'm trying to understand examples from stackoverflow and different places,
>> > > but they are all intended to limit by IP and for specific resources (the
>> > > scope of the rule). Eg.:
>> > >
>> > > https://gist.github.com/josnidhin/91d1ea9cd71fde386c27a9228476834e
>> > > <https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_josnidhin_91d1ea9cd71fde386c27a9228476834e&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=L7iKlxwUA3exA-ByaKl7gyvQkoOevQwuEjv4ZKC6hOY&e=>
>> > >
>> > >
>> > >
>> > > I'm not asking for the entire rule, just an example of how var counters
>> > > work in the global scope (directly in /etc/modsecurity/modsecurity.conf)
>> > > and how can I connect them to sum by header instead of IP.
>> > >
>> > >
>> > >
>> > > Thank you!
>> > >
>> > >
>> > >
>> > > El jue., 6 de dic. de 2018 a la(s) 10:30, Christian Folini (
>> > > chr...@ne...) escribió:
>> > >
>> > > Hello Luciano,
>> > >
>> > > You have a peculiar use case, but I see your thinking.
>> > >
>> > > There are examples in the ModSecurity books that are really close to your
>> > > plan. They should be easy to adopt.
>> > >
>> > > Other than that, you may want to look into mod_qos. It has functionality
>> > > that might be useful in your case.
>> > >
>> > > Best,
>> > >
>> > > Christian
>> > >
>> > >
>> > > On Wed, Dec 05, 2018 at 06:26:03PM -0300, Luciano Guillermo Fantuzzi wrote:
>> > > > Thank you for your answer, but maybe I'm not asking it the right way or
>> > > > this is not the right place to ask(?).
>> > > >
>> > > > I need a Modsecurity rule (I'm using it through Apache) to be able to
>> > > > control hits from clients with a specific header, like
>> > > > "facebookexternalhit/1.1".
>> > > > Ie. to stop some agressive bots hitting too often my webservers and
>> > > taking
>> > > > them down eventually. I don't want to block them at all because I need
>> > > some
>> > > > of them (like Facebook bot to parse shared content), but I need a way to
>> > > > tell them "stop, retry in some seconds".
>> > > >
>> > > > Thanks.
>> > > >
>> > > > El mié., 5 de dic. de 2018 a la(s) 16:16, Reindl Harald (
>> > > > h.r...@th...) escribió:
>> > > >
>> > > > >
>> > > > >
>> > > > > Am 05.12.18 um 16:57 schrieb Luciano Guillermo Fantuzzi:
>> > > > > > First of all, I'm new here so I'm not sure this is the right place
>> > > for
>> > > > > > asking for help (free modsec version). If it's not, I'll really
>> > > > > > appreciate it if you can tell me where should I go.
>> > > > > >
>> > > > > > I'm trying to limit hit rate by:
>> > > > > >
>> > > > > > 1. Request's header (like "facebookexternalhit").
>> > > > > > 2. (All hits to non static resources)
>> > > > > >
>> > > > > > And then return a friendly "429 Too Many Requests" and "Retry-After:
>> > > 3"
>> > > > > > (seconds).
>> > > > > > I know I can read a file of headers like:
>> > > > > >
>> > > > > > SecRule REQUEST_HEADERS:User-Agent "@pmFromFile ratelimit-bots.txt"
>> > > > > >
>> > > > > > But I'm getting trouble building the entire rule.
>> > > > > >
>> > > > > > Any help would be really appreciated. Thank you!
>> > > > >
>> > > > > this a non-iusse
>> > > > >
>> > > > > normally you have rate-limits per IP in place and they should not be
>> > > > > within the application layer at all and in the best case not even on
>> > > the
>> > > > > same machine
>> > > > >
>> > > > > that below is from a firewall-vm on a complete /24 network before any
>> > > > > packet reaches a server at all, and for the individual servers are
>> > > > > simimlar rules with lower values per 2 seconds in place
>> > > > >
>> > > > > when the request reachs the webserver damage is long done and if no
>> > > > > damage is done you are wasting expensive ressources with the rules
>> > > > >
>> > > > > Chain INBOUND (2 references)
>> > > > >  pkts bytes target     prot opt in     out     source
>> > > > > destination
>> > > > >  1914  183K IPST_ALL   all  --  *      *       0.0.0.0/0
>> > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=>
>> > > > > 0.0.0.0/0
>> > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=>
>> > >           recent: UPDATE seconds: 2 hit_count: 250 TTL-Match
>> > > > > name: limit_all_global side: source mask: 255.255.255.255
>> > > > >  149K   15M DROP_ALL   all  --  *      *       0.0.0.0/0
>> > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=>
>> > > > > 0.0.0.0/0
>> > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=>
>> > >           recent: UPDATE seconds: 2 reap hit_count: 150
>> > > > > TTL-Match name: limit_all_global side: source mask: 255.255.255.255
>> > > > >
>> > > > >
>> > > > > _______________________________________________
>> > > > > mod-security-users mailing list
>> > > > > mod...@li...
>> > > > > https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> > > <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_mod-2Dsecurity-2Dusers&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=RUDsPA0iq5WVkW20NWQOl8suSJ4RvNfYZ6TM3FXNtdM&e=>
>> > > > > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> > > > > http://www.modsecurity.org/projects/commercial/rules/
>> > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_rules_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=rQF299h7fZhdJbhudnhzjEcP4e3Aa8qCG0KvKi4CKiM&e=>
>> > > > > http://www.modsecurity.org/projects/commercial/support/
>> > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_support_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=ofF4OfFPsr3nKEMOH7j-CQmBqLgK_51fvOnQTavYK3c&e=>
>> > > > >
>> > >
>> > >
>> > > > _______________________________________________
>> > > > mod-security-users mailing list
>> > > > mod...@li...
>> > > > https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> > > <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_mod-2Dsecurity-2Dusers&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=RUDsPA0iq5WVkW20NWQOl8suSJ4RvNfYZ6TM3FXNtdM&e=>
>> > > > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> > > > http://www.modsecurity.org/projects/commercial/rules/
>> > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_rules_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=rQF299h7fZhdJbhudnhzjEcP4e3Aa8qCG0KvKi4CKiM&e=>
>> > > > http://www.modsecurity.org/projects/commercial/support/
>> > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_support_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=ofF4OfFPsr3nKEMOH7j-CQmBqLgK_51fvOnQTavYK3c&e=>
>> > >
>> > >
>> > >
>> > > _______________________________________________
>> > > mod-security-users mailing list
>> > > mod...@li...
>> > > https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> > > <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_mod-2Dsecurity-2Dusers&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=RUDsPA0iq5WVkW20NWQOl8suSJ4RvNfYZ6TM3FXNtdM&e=>
>> > > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> > > http://www.modsecurity.org/projects/commercial/rules/
>> > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_rules_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=rQF299h7fZhdJbhudnhzjEcP4e3Aa8qCG0KvKi4CKiM&e=>
>> > > http://www.modsecurity.org/projects/commercial/support/
>> > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_support_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=ofF4OfFPsr3nKEMOH7j-CQmBqLgK_51fvOnQTavYK3c&e=>
>> > >
>> > > _______________________________________________
>> > > mod-security-users mailing list
>> > > mod...@li...
>> > > https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> > > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> > > http://www.modsecurity.org/projects/commercial/rules/
>> > > http://www.modsecurity.org/projects/commercial/support/
>> > >
>>
>>
>> > _______________________________________________
>> > mod-security-users mailing list
>> > mod...@li...
>> > https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> > http://www.modsecurity.org/projects/commercial/rules/
>> > http://www.modsecurity.org/projects/commercial/support/
>>
>>
>>
>> _______________________________________________
>> mod-security-users mailing list
>> mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> http://www.modsecurity.org/projects/commercial/rules/
>> http://www.modsecurity.org/projects/commercial/support/
>
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/

Re: [mod-security-users] How to limit access rate by header?

From: Luciano G. F. <luc...@gm...> - 2018-12-09 13:52:09

Yes, sorry to try to defend myself. It was not fear to telling the true to
someone suddenly attacking me for no reason. Trying to get help here was
clearly a bad idea. This is not a place for help, at least for newbies with
modsec. Sorry for bothering all of you. Have a nice day.

El dom., 9 dic. 2018 08:16, Franziska Buehler <
fra...@gm...> escribió:

> Luciano,
>
> It’s not fair to call someone, who publishes several free tutorials,
> invests his free time in open source, and over the years answers tons
> of questions on this mailing list, as arrogant, as someone who solves
> his problems with money.
> And in the end, you tell him he should never write you back again.
> I could imagine that your tone probably disappoints and discourages people.
>
> Best regards,
> Franziska
>
> Am Fr., 7. Dez. 2018 um 16:24 Uhr schrieb Luciano Guillermo Fantuzzi
> <luc...@gm...>:
> >
> > @Christian First of all, you are completly wrong. Let me explain:
> >
> > 1. I'm using Cloudflare (free plan) just to save traffic. So no, I'm not
> a rich guy and I'm looking for a solution server side, because since I'm in
> the free plan I can't use their WAF solution.
> > 2. I'm not a guy looking for someone to do my job. Maybe you didn't see
> my last email, where I wrote the rule I created and that is not working for
> some reason. Before spending my time creating an account here and writing
> emails, I googled the entire day and tried different approaches, but no one
> worked. I didn't find a single piece of code doing what I need to do, so I
> tried with different IP rate limit snippets, but after debugging some time
> I'm facing an error that no one else seem to see. This is why I'm stuck
> here.
> > 3. I'm not paying anyone to do something like this. Maybe you are the
> kind of people that solve your problems with money and in the easiest way.
> Sorry, I'm not that kind of people.
> >
> > And last, I don't want the help of arrogant guys like you. I don't know
> the reason you think you can talk to people the way you do, specially when
> they are asking for help. Like if you were born knowing about everything...
> I can only thank I don't know you and I don't have people like you in my
> life. Fortunately, you are part of a minority.
> >
> > Don't write me back again.
> >
> > El vie., 7 de dic. de 2018 a la(s) 09:31, Christian Folini (
> chr...@ne...) escribió:
> >>
> >> Luciano,
> >>
> >> I understand your troubles. But if you are behind cloudflare and you are
> >> giving cloudflare money to solve your problems, why are you coming here
> >> asking for free support?
> >>
> >> You are facing a special problem and you need a special solution. We
> have
> >> given you pointers and hints but it seems it is not enough, so it is
> >> likely you need to dig deeper and learn more - or you pay somebody
> >> to do that for you.
> >>
> >> What I am not going to do - and I doubt somebody else is willing to do
> that -
> >> is investing 1-2-3 hours into developing a solution for you in my spare
> >> time. And it is very likely this would take 1-2-3 hours for me, so do
> not
> >> expect a quick win.
> >>
> >> Just my 2 cents. Good luck,
> >>
> >> Christian
> >>
> >> On Fri, Dec 07, 2018 at 09:17:45AM -0300, Luciano Guillermo Fantuzzi
> wrote:
> >> > Already tried that, but still the same message in debug log. Anyway,
> I'm
> >> > behind Cloudflare so I need to access that var (that contains the
> real IP)
> >> > from header. Moreover, I tried with global collection and same luck.
> Am I
> >> > missing some initialization step in modsec?
> >> >
> >> > Thanks.
> >> >
> >> > El vie., 7 dic. 2018 00:56, Scheblein, Adam <
> ada...@ma...>
> >> > escribió:
> >> >
> >> > > I had a similar problem.  You need to initialize the collection with
> >> > > something like this:
> >> > >
> >> > >
> >> > >
> >> > >                 SecAction
> >> > > id:'2000000',phase:1,nolog,pass,initcol:IP=%{REMOTE_ADDR}
> >> > >
> >> > >
> >> > >
> >> > > *From: *Luciano Guillermo Fantuzzi <luc...@gm...>
> >> > > *Reply-To: *"mod...@li..." <
> >> > > mod...@li...>
> >> > > *Date: *Thursday, December 6, 2018 at 8:51 PM
> >> > > *To: *"mod...@li..." <
> >> > > mod...@li...>
> >> > > *Subject: *Re: [mod-security-users] How to limit access rate by
> header?
> >> > >
> >> > >
> >> > >
> >> > > I've very frustrated... I can't make it work, even for IP control.
> What am
> >> > > I doing wrong here? It always returns:
> >> > >
> >> > > Could not set variable "IP.access_count" as the collection does not
> exist.
> >> > >
> >> > >
> >> > >
> >> > > 105 <LocationMatch "^/.*">
> >> > >
> >> > > 109   SecRule REQUEST_HEADERS:CF-Connecting-IP "@unconditionalMatch"
> >> > > "phase:2,initcol:IP=%{MATCHED_VAR},pass,nolog,id:35003"
> >> > >
> >> > > 112   SecRule IP:ACCESS_COUNT "@gt 1"
> >> > >
> "phase:2,pause:300,deny,status:503,setenv:RATELIMITED,skip:1,nolog,id:35004"
> >> > >
> >> > > 116   SecAction
> "phase:2,setvar:IP.access_count=+1,pass,nolog,id:35005"
> >> > >
> >> > >
> >> > >
> >> > > 119   SecAction
> >> > > "phase:5,deprecatevar:IP.access_count=1/10,pass,nolog,id:35006"
> >> > >
> >> > > 122   Header always set Retry-After "10" env=RATELIMITED
> >> > >
> >> > > 123 </LocationMatch>
> >> > >
> >> > > 124
> >> > >
> >> > > 125 ErrorDocument 503 "Service Unavailable"
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > El jue., 6 de dic. de 2018 a la(s) 20:38, Luciano Guillermo
> Fantuzzi (
> >> > > luc...@gm...) escribió:
> >> > >
> >> > > Thank you for your answer, Christian. Do you think it's possible
> for you
> >> > > to just build the first part of the rule (in Modsec)? I'm trying
> but I'm
> >> > > not understanding how variables work with the global scope. I was
> be able
> >> > > to build some basic rules like:
> >> > >
> >> > >
> >> > >
> >> > > # Banned Bots and Crawlers
> >> > >
> >> > > SecRule REQUEST_HEADERS:User-Agent "@pmFromFile
> blacklist-bots.data" \
> >> > >
> >> > >     "id:350001,phase:1,t:none,deny,log,msg:'BANNED BOT'"
> >> > >
> >> > >
> >> > >
> >> > > # Specific IPs
> >> > >
> >> > > SecRule REMOTE_ADDR "@pmFromFile blacklist-ip.data" \
> >> > >
> >> > >     "id:350002,phase:1,t:none,deny,log,msg:'BANNED IP'"
> >> > >
> >> > >
> >> > >
> >> > > I'm trying to understand examples from stackoverflow and different
> places,
> >> > > but they are all intended to limit by IP and for specific resources
> (the
> >> > > scope of the rule). Eg.:
> >> > >
> >> > > https://gist.github.com/josnidhin/91d1ea9cd71fde386c27a9228476834e
> >> > > <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_josnidhin_91d1ea9cd71fde386c27a9228476834e&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=L7iKlxwUA3exA-ByaKl7gyvQkoOevQwuEjv4ZKC6hOY&e=
> >
> >> > >
> >> > >
> >> > >
> >> > > I'm not asking for the entire rule, just an example of how var
> counters
> >> > > work in the global scope (directly in
> /etc/modsecurity/modsecurity.conf)
> >> > > and how can I connect them to sum by header instead of IP.
> >> > >
> >> > >
> >> > >
> >> > > Thank you!
> >> > >
> >> > >
> >> > >
> >> > > El jue., 6 de dic. de 2018 a la(s) 10:30, Christian Folini (
> >> > > chr...@ne...) escribió:
> >> > >
> >> > > Hello Luciano,
> >> > >
> >> > > You have a peculiar use case, but I see your thinking.
> >> > >
> >> > > There are examples in the ModSecurity books that are really close
> to your
> >> > > plan. They should be easy to adopt.
> >> > >
> >> > > Other than that, you may want to look into mod_qos. It has
> functionality
> >> > > that might be useful in your case.
> >> > >
> >> > > Best,
> >> > >
> >> > > Christian
> >> > >
> >> > >
> >> > > On Wed, Dec 05, 2018 at 06:26:03PM -0300, Luciano Guillermo
> Fantuzzi wrote:
> >> > > > Thank you for your answer, but maybe I'm not asking it the right
> way or
> >> > > > this is not the right place to ask(?).
> >> > > >
> >> > > > I need a Modsecurity rule (I'm using it through Apache) to be
> able to
> >> > > > control hits from clients with a specific header, like
> >> > > > "facebookexternalhit/1.1".
> >> > > > Ie. to stop some agressive bots hitting too often my webservers
> and
> >> > > taking
> >> > > > them down eventually. I don't want to block them at all because I
> need
> >> > > some
> >> > > > of them (like Facebook bot to parse shared content), but I need a
> way to
> >> > > > tell them "stop, retry in some seconds".
> >> > > >
> >> > > > Thanks.
> >> > > >
> >> > > > El mié., 5 de dic. de 2018 a la(s) 16:16, Reindl Harald (
> >> > > > h.r...@th...) escribió:
> >> > > >
> >> > > > >
> >> > > > >
> >> > > > > Am 05.12.18 um 16:57 schrieb Luciano Guillermo Fantuzzi:
> >> > > > > > First of all, I'm new here so I'm not sure this is the right
> place
> >> > > for
> >> > > > > > asking for help (free modsec version). If it's not, I'll
> really
> >> > > > > > appreciate it if you can tell me where should I go.
> >> > > > > >
> >> > > > > > I'm trying to limit hit rate by:
> >> > > > > >
> >> > > > > > 1. Request's header (like "facebookexternalhit").
> >> > > > > > 2. (All hits to non static resources)
> >> > > > > >
> >> > > > > > And then return a friendly "429 Too Many Requests" and
> "Retry-After:
> >> > > 3"
> >> > > > > > (seconds).
> >> > > > > > I know I can read a file of headers like:
> >> > > > > >
> >> > > > > > SecRule REQUEST_HEADERS:User-Agent "@pmFromFile
> ratelimit-bots.txt"
> >> > > > > >
> >> > > > > > But I'm getting trouble building the entire rule.
> >> > > > > >
> >> > > > > > Any help would be really appreciated. Thank you!
> >> > > > >
> >> > > > > this a non-iusse
> >> > > > >
> >> > > > > normally you have rate-limits per IP in place and they should
> not be
> >> > > > > within the application layer at all and in the best case not
> even on
> >> > > the
> >> > > > > same machine
> >> > > > >
> >> > > > > that below is from a firewall-vm on a complete /24 network
> before any
> >> > > > > packet reaches a server at all, and for the individual servers
> are
> >> > > > > simimlar rules with lower values per 2 seconds in place
> >> > > > >
> >> > > > > when the request reachs the webserver damage is long done and
> if no
> >> > > > > damage is done you are wasting expensive ressources with the
> rules
> >> > > > >
> >> > > > > Chain INBOUND (2 references)
> >> > > > >  pkts bytes target     prot opt in     out     source
> >> > > > > destination
> >> > > > >  1914  183K IPST_ALL   all  --  *      *       0.0.0.0/0
> >> > > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=
> >
> >> > > > > 0.0.0.0/0
> >> > > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=
> >
> >> > >           recent: UPDATE seconds: 2 hit_count: 250 TTL-Match
> >> > > > > name: limit_all_global side: source mask: 255.255.255.255
> >> > > > >  149K   15M DROP_ALL   all  --  *      *       0.0.0.0/0
> >> > > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=
> >
> >> > > > > 0.0.0.0/0
> >> > > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=
> >
> >> > >           recent: UPDATE seconds: 2 reap hit_count: 150
> >> > > > > TTL-Match name: limit_all_global side: source mask:
> 255.255.255.255
> >> > > > >
> >> > > > >
> >> > > > > _______________________________________________
> >> > > > > mod-security-users mailing list
> >> > > > > mod...@li...
> >> > > > > https://lists.sourceforge.net/lists/listinfo/mod-security-users
> >> > > <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_mod-2Dsecurity-2Dusers&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=RUDsPA0iq5WVkW20NWQOl8suSJ4RvNfYZ6TM3FXNtdM&e=
> >
> >> > > > > Commercial ModSecurity Rules and Support from Trustwave's
> SpiderLabs:
> >> > > > > http://www.modsecurity.org/projects/commercial/rules/
> >> > > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_rules_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=rQF299h7fZhdJbhudnhzjEcP4e3Aa8qCG0KvKi4CKiM&e=
> >
> >> > > > > http://www.modsecurity.org/projects/commercial/support/
> >> > > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_support_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=ofF4OfFPsr3nKEMOH7j-CQmBqLgK_51fvOnQTavYK3c&e=
> >
> >> > > > >
> >> > >
> >> > >
> >> > > > _______________________________________________
> >> > > > mod-security-users mailing list
> >> > > > mod...@li...
> >> > > > https://lists.sourceforge.net/lists/listinfo/mod-security-users
> >> > > <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_mod-2Dsecurity-2Dusers&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=RUDsPA0iq5WVkW20NWQOl8suSJ4RvNfYZ6TM3FXNtdM&e=
> >
> >> > > > Commercial ModSecurity Rules and Support from Trustwave's
> SpiderLabs:
> >> > > > http://www.modsecurity.org/projects/commercial/rules/
> >> > > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_rules_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=rQF299h7fZhdJbhudnhzjEcP4e3Aa8qCG0KvKi4CKiM&e=
> >
> >> > > > http://www.modsecurity.org/projects/commercial/support/
> >> > > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_support_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=ofF4OfFPsr3nKEMOH7j-CQmBqLgK_51fvOnQTavYK3c&e=
> >
> >> > >
> >> > >
> >> > >
> >> > > _______________________________________________
> >> > > mod-security-users mailing list
> >> > > mod...@li...
> >> > > https://lists.sourceforge.net/lists/listinfo/mod-security-users
> >> > > <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_mod-2Dsecurity-2Dusers&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=RUDsPA0iq5WVkW20NWQOl8suSJ4RvNfYZ6TM3FXNtdM&e=
> >
> >> > > Commercial ModSecurity Rules and Support from Trustwave's
> SpiderLabs:
> >> > > http://www.modsecurity.org/projects/commercial/rules/
> >> > > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_rules_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=rQF299h7fZhdJbhudnhzjEcP4e3Aa8qCG0KvKi4CKiM&e=
> >
> >> > > http://www.modsecurity.org/projects/commercial/support/
> >> > > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_support_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=ofF4OfFPsr3nKEMOH7j-CQmBqLgK_51fvOnQTavYK3c&e=
> >
> >> > >
> >> > > _______________________________________________
> >> > > mod-security-users mailing list
> >> > > mod...@li...
> >> > > https://lists.sourceforge.net/lists/listinfo/mod-security-users
> >> > > Commercial ModSecurity Rules and Support from Trustwave's
> SpiderLabs:
> >> > > http://www.modsecurity.org/projects/commercial/rules/
> >> > > http://www.modsecurity.org/projects/commercial/support/
> >> > >
> >>
> >>
> >> > _______________________________________________
> >> > mod-security-users mailing list
> >> > mod...@li...
> >> > https://lists.sourceforge.net/lists/listinfo/mod-security-users
> >> > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> >> > http://www.modsecurity.org/projects/commercial/rules/
> >> > http://www.modsecurity.org/projects/commercial/support/
> >>
> >>
> >>
> >> _______________________________________________
> >> mod-security-users mailing list
> >> mod...@li...
> >> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> >> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> >> http://www.modsecurity.org/projects/commercial/rules/
> >> http://www.modsecurity.org/projects/commercial/support/
> >
> > _______________________________________________
> > mod-security-users mailing list
> > mod...@li...
> > https://lists.sourceforge.net/lists/listinfo/mod-security-users
> > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> > http://www.modsecurity.org/projects/commercial/rules/
> > http://www.modsecurity.org/projects/commercial/support/
>
>
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/
>