Re: [mod-security-users] How to limit access rate by header?

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

I had a similar problem.  You need to initialize the collection with something like this:

                SecAction id:'2000000',phase:1,nolog,pass,initcol:IP=%{REMOTE_ADDR}

From: Luciano Guillermo Fantuzzi <luc...@gm...>
Reply-To: "mod...@li..." <mod...@li...>
Date: Thursday, December 6, 2018 at 8:51 PM
To: "mod...@li..." <mod...@li...>
Subject: Re: [mod-security-users] How to limit access rate by header?

I've very frustrated... I can't make it work, even for IP control. What am I doing wrong here? It always returns:
Could not set variable "IP.access_count" as the collection does not exist.

105 <LocationMatch "^/.*">
109   SecRule REQUEST_HEADERS:CF-Connecting-IP "@unconditionalMatch" "phase:2,initcol:IP=%{MATCHED_VAR},pass,nolog,id:35003"
112   SecRule IP:ACCESS_COUNT "@gt 1" "phase:2,pause:300,deny,status:503,setenv:RATELIMITED,skip:1,nolog,id:35004"
116   SecAction "phase:2,setvar:IP.access_count=+1,pass,nolog,id:35005"
119   SecAction "phase:5,deprecatevar:IP.access_count=1/10,pass,nolog,id:35006"
122   Header always set Retry-After "10" env=RATELIMITED
123 </LocationMatch>
124
125 ErrorDocument 503 "Service Unavailable"

El jue., 6 de dic. de 2018 a la(s) 20:38, Luciano Guillermo Fantuzzi (luc...@gm...<mailto:luc...@gm...>) escribió:
Thank you for your answer, Christian. Do you think it's possible for you to just build the first part of the rule (in Modsec)? I'm trying but I'm not understanding how variables work with the global scope. I was be able to build some basic rules like:

# Banned Bots and Crawlers
SecRule REQUEST_HEADERS:User-Agent "@pmFromFile blacklist-bots.data" \
    "id:350001,phase:1,t:none,deny,log,msg:'BANNED BOT'"

# Specific IPs
SecRule REMOTE_ADDR "@pmFromFile blacklist-ip.data" \
    "id:350002,phase:1,t:none,deny,log,msg:'BANNED IP'"

I'm trying to understand examples from stackoverflow and different places, but they are all intended to limit by IP and for specific resources (the scope of the rule). Eg.:
https://gist.github.com/josnidhin/91d1ea9cd71fde386c27a9228476834e<https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_josnidhin_91d1ea9cd71fde386c27a9228476834e&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=L7iKlxwUA3exA-ByaKl7gyvQkoOevQwuEjv4ZKC6hOY&e=>

I'm not asking for the entire rule, just an example of how var counters work in the global scope (directly in /etc/modsecurity/modsecurity.conf) and how can I connect them to sum by header instead of IP.

Thank you!

El jue., 6 de dic. de 2018 a la(s) 10:30, Christian Folini (chr...@ne...<mailto:chr...@ne...>) escribió:
Hello Luciano,

You have a peculiar use case, but I see your thinking.

There are examples in the ModSecurity books that are really close to your
plan. They should be easy to adopt.

Other than that, you may want to look into mod_qos. It has functionality
that might be useful in your case.

Best,

Christian

On Wed, Dec 05, 2018 at 06:26:03PM -0300, Luciano Guillermo Fantuzzi wrote:
> Thank you for your answer, but maybe I'm not asking it the right way or
> this is not the right place to ask(?).
>
> I need a Modsecurity rule (I'm using it through Apache) to be able to
> control hits from clients with a specific header, like
> "facebookexternalhit/1.1".
> Ie. to stop some agressive bots hitting too often my webservers and taking
> them down eventually. I don't want to block them at all because I need some
> of them (like Facebook bot to parse shared content), but I need a way to
> tell them "stop, retry in some seconds".
>
> Thanks.
>
> El mié., 5 de dic. de 2018 a la(s) 16:16, Reindl Harald (
> h.r...@th...<mailto:h.r...@th...>) escribió:
>
> >
> >
> > Am 05.12.18 um 16:57 schrieb Luciano Guillermo Fantuzzi:
> > > First of all, I'm new here so I'm not sure this is the right place for
> > > asking for help (free modsec version). If it's not, I'll really
> > > appreciate it if you can tell me where should I go.
> > >
> > > I'm trying to limit hit rate by:
> > >
> > > 1. Request's header (like "facebookexternalhit").
> > > 2. (All hits to non static resources)
> > >
> > > And then return a friendly "429 Too Many Requests" and "Retry-After: 3"
> > > (seconds).
> > > I know I can read a file of headers like:
> > >
> > > SecRule REQUEST_HEADERS:User-Agent "@pmFromFile ratelimit-bots.txt"
> > >
> > > But I'm getting trouble building the entire rule.
> > >
> > > Any help would be really appreciated. Thank you!
> >
> > this a non-iusse
> >
> > normally you have rate-limits per IP in place and they should not be
> > within the application layer at all and in the best case not even on the
> > same machine
> >
> > that below is from a firewall-vm on a complete /24 network before any
> > packet reaches a server at all, and for the individual servers are
> > simimlar rules with lower values per 2 seconds in place
> >
> > when the request reachs the webserver damage is long done and if no
> > damage is done you are wasting expensive ressources with the rules
> >
> > Chain INBOUND (2 references)
> >  pkts bytes target     prot opt in     out     source
> > destination
> >  1914  183K IPST_ALL   all  --  *      *       0.0.0.0/0<https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=>
> > 0.0.0.0/0<https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=>            recent: UPDATE seconds: 2 hit_count: 250 TTL-Match
> > name: limit_all_global side: source mask: 255.255.255.255
> >  149K   15M DROP_ALL   all  --  *      *       0.0.0.0/0<https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=>
> > 0.0.0.0/0<https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=>            recent: UPDATE seconds: 2 reap hit_count: 150
> > TTL-Match name: limit_all_global side: source mask: 255.255.255.255
> >
> >
> > _______________________________________________
> > mod-security-users mailing list
> > mod...@li...<mailto:mod...@li...>
> > https://lists.sourceforge.net/lists/listinfo/mod-security-users<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_mod-2Dsecurity-2Dusers&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=RUDsPA0iq5WVkW20NWQOl8suSJ4RvNfYZ6TM3FXNtdM&e=>
> > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> > http://www.modsecurity.org/projects/commercial/rules/<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_rules_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=rQF299h7fZhdJbhudnhzjEcP4e3Aa8qCG0KvKi4CKiM&e=>
> > http://www.modsecurity.org/projects/commercial/support/<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_support_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=ofF4OfFPsr3nKEMOH7j-CQmBqLgK_51fvOnQTavYK3c&e=>
> >

> _______________________________________________
> mod-security-users mailing list
> mod...@li...<mailto:mod...@li...>
> https://lists.sourceforge.net/lists/listinfo/mod-security-users<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_mod-2Dsecurity-2Dusers&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=RUDsPA0iq5WVkW20NWQOl8suSJ4RvNfYZ6TM3FXNtdM&e=>
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_rules_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=rQF299h7fZhdJbhudnhzjEcP4e3Aa8qCG0KvKi4CKiM&e=>
> http://www.modsecurity.org/projects/commercial/support/<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_support_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=ofF4OfFPsr3nKEMOH7j-CQmBqLgK_51fvOnQTavYK3c&e=>

_______________________________________________
mod-security-users mailing list
mod...@li...<mailto:mod...@li...>
https://lists.sourceforge.net/lists/listinfo/mod-security-users<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_mod-2Dsecurity-2Dusers&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=RUDsPA0iq5WVkW20NWQOl8suSJ4RvNfYZ6TM3FXNtdM&e=>
Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
http://www.modsecurity.org/projects/commercial/rules/<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_rules_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=rQF299h7fZhdJbhudnhzjEcP4e3Aa8qCG0KvKi4CKiM&e=>
http://www.modsecurity.org/projects/commercial/support/<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_support_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=ofF4OfFPsr3nKEMOH7j-CQmBqLgK_51fvOnQTavYK3c&e=>