Re: [mod-security-users] How to limit access rate by header?

[Eero V. <eer...@ik...>]

maybe you need to use this module

https://github.com/gnif/mod_rpaf

Eero

Luciano Guillermo Fantuzzi <luc...@gm...> kirjoitti pe 7.
jouluk. 2018 klo 14.29:

> Apache 2.4.x
>
> El vie., 7 dic. 2018 09:25, Eero Volotinen <eer...@ik...>
> escribió:
>
>> are you using nginx or apache?
>>
>> Luciano Guillermo Fantuzzi <luc...@gm...> kirjoitti pe 7.
>> jouluk. 2018 klo 14.19:
>>
>>> Already tried that, but still the same message in debug log. Anyway, I'm
>>> behind Cloudflare so I need to access that var (that contains the real IP)
>>> from header. Moreover, I tried with global collection and same luck. Am I
>>> missing some initialization step in modsec?
>>>
>>> Thanks.
>>>
>>> El vie., 7 dic. 2018 00:56, Scheblein, Adam <
>>> ada...@ma...> escribió:
>>>
>>>> I had a similar problem.  You need to initialize the collection with
>>>> something like this:
>>>>
>>>>
>>>>
>>>>                 SecAction
>>>> id:'2000000',phase:1,nolog,pass,initcol:IP=%{REMOTE_ADDR}
>>>>
>>>>
>>>>
>>>> *From: *Luciano Guillermo Fantuzzi <luc...@gm...>
>>>> *Reply-To: *"mod...@li..." <
>>>> mod...@li...>
>>>> *Date: *Thursday, December 6, 2018 at 8:51 PM
>>>> *To: *"mod...@li..." <
>>>> mod...@li...>
>>>> *Subject: *Re: [mod-security-users] How to limit access rate by header?
>>>>
>>>>
>>>>
>>>> I've very frustrated... I can't make it work, even for IP control. What
>>>> am I doing wrong here? It always returns:
>>>>
>>>> Could not set variable "IP.access_count" as the collection does not
>>>> exist.
>>>>
>>>>
>>>>
>>>> 105 <LocationMatch "^/.*">
>>>>
>>>> 109   SecRule REQUEST_HEADERS:CF-Connecting-IP "@unconditionalMatch"
>>>> "phase:2,initcol:IP=%{MATCHED_VAR},pass,nolog,id:35003"
>>>>
>>>> 112   SecRule IP:ACCESS_COUNT "@gt 1"
>>>> "phase:2,pause:300,deny,status:503,setenv:RATELIMITED,skip:1,nolog,id:35004"
>>>>
>>>> 116   SecAction
>>>> "phase:2,setvar:IP.access_count=+1,pass,nolog,id:35005"
>>>>
>>>>
>>>>
>>>> 119   SecAction
>>>> "phase:5,deprecatevar:IP.access_count=1/10,pass,nolog,id:35006"
>>>>
>>>> 122   Header always set Retry-After "10" env=RATELIMITED
>>>>
>>>> 123 </LocationMatch>
>>>>
>>>> 124
>>>>
>>>> 125 ErrorDocument 503 "Service Unavailable"
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> El jue., 6 de dic. de 2018 a la(s) 20:38, Luciano Guillermo Fantuzzi (
>>>> luc...@gm...) escribió:
>>>>
>>>> Thank you for your answer, Christian. Do you think it's possible for
>>>> you to just build the first part of the rule (in Modsec)? I'm trying but
>>>> I'm not understanding how variables work with the global scope. I was be
>>>> able to build some basic rules like:
>>>>
>>>>
>>>>
>>>> # Banned Bots and Crawlers
>>>>
>>>> SecRule REQUEST_HEADERS:User-Agent "@pmFromFile blacklist-bots.data" \
>>>>
>>>>     "id:350001,phase:1,t:none,deny,log,msg:'BANNED BOT'"
>>>>
>>>>
>>>>
>>>> # Specific IPs
>>>>
>>>> SecRule REMOTE_ADDR "@pmFromFile blacklist-ip.data" \
>>>>
>>>>     "id:350002,phase:1,t:none,deny,log,msg:'BANNED IP'"
>>>>
>>>>
>>>>
>>>> I'm trying to understand examples from stackoverflow and different
>>>> places, but they are all intended to limit by IP and for specific resources
>>>> (the scope of the rule). Eg.:
>>>>
>>>> https://gist.github.com/josnidhin/91d1ea9cd71fde386c27a9228476834e
>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_josnidhin_91d1ea9cd71fde386c27a9228476834e&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=L7iKlxwUA3exA-ByaKl7gyvQkoOevQwuEjv4ZKC6hOY&e=>
>>>>
>>>>
>>>>
>>>> I'm not asking for the entire rule, just an example of how var counters
>>>> work in the global scope (directly in /etc/modsecurity/modsecurity.conf)
>>>> and how can I connect them to sum by header instead of IP.
>>>>
>>>>
>>>>
>>>> Thank you!
>>>>
>>>>
>>>>
>>>> El jue., 6 de dic. de 2018 a la(s) 10:30, Christian Folini (
>>>> chr...@ne...) escribió:
>>>>
>>>> Hello Luciano,
>>>>
>>>> You have a peculiar use case, but I see your thinking.
>>>>
>>>> There are examples in the ModSecurity books that are really close to
>>>> your
>>>> plan. They should be easy to adopt.
>>>>
>>>> Other than that, you may want to look into mod_qos. It has functionality
>>>> that might be useful in your case.
>>>>
>>>> Best,
>>>>
>>>> Christian
>>>>
>>>>
>>>> On Wed, Dec 05, 2018 at 06:26:03PM -0300, Luciano Guillermo Fantuzzi
>>>> wrote:
>>>> > Thank you for your answer, but maybe I'm not asking it the right way
>>>> or
>>>> > this is not the right place to ask(?).
>>>> >
>>>> > I need a Modsecurity rule (I'm using it through Apache) to be able to
>>>> > control hits from clients with a specific header, like
>>>> > "facebookexternalhit/1.1".
>>>> > Ie. to stop some agressive bots hitting too often my webservers and
>>>> taking
>>>> > them down eventually. I don't want to block them at all because I
>>>> need some
>>>> > of them (like Facebook bot to parse shared content), but I need a way
>>>> to
>>>> > tell them "stop, retry in some seconds".
>>>> >
>>>> > Thanks.
>>>> >
>>>> > El mié., 5 de dic. de 2018 a la(s) 16:16, Reindl Harald (
>>>> > h.r...@th...) escribió:
>>>> >
>>>> > >
>>>> > >
>>>> > > Am 05.12.18 um 16:57 schrieb Luciano Guillermo Fantuzzi:
>>>> > > > First of all, I'm new here so I'm not sure this is the right
>>>> place for
>>>> > > > asking for help (free modsec version). If it's not, I'll really
>>>> > > > appreciate it if you can tell me where should I go.
>>>> > > >
>>>> > > > I'm trying to limit hit rate by:
>>>> > > >
>>>> > > > 1. Request's header (like "facebookexternalhit").
>>>> > > > 2. (All hits to non static resources)
>>>> > > >
>>>> > > > And then return a friendly "429 Too Many Requests" and
>>>> "Retry-After: 3"
>>>> > > > (seconds).
>>>> > > > I know I can read a file of headers like:
>>>> > > >
>>>> > > > SecRule REQUEST_HEADERS:User-Agent "@pmFromFile
>>>> ratelimit-bots.txt"
>>>> > > >
>>>> > > > But I'm getting trouble building the entire rule.
>>>> > > >
>>>> > > > Any help would be really appreciated. Thank you!
>>>> > >
>>>> > > this a non-iusse
>>>> > >
>>>> > > normally you have rate-limits per IP in place and they should not be
>>>> > > within the application layer at all and in the best case not even
>>>> on the
>>>> > > same machine
>>>> > >
>>>> > > that below is from a firewall-vm on a complete /24 network before
>>>> any
>>>> > > packet reaches a server at all, and for the individual servers are
>>>> > > simimlar rules with lower values per 2 seconds in place
>>>> > >
>>>> > > when the request reachs the webserver damage is long done and if no
>>>> > > damage is done you are wasting expensive ressources with the rules
>>>> > >
>>>> > > Chain INBOUND (2 references)
>>>> > >  pkts bytes target     prot opt in     out     source
>>>> > > destination
>>>> > >  1914  183K IPST_ALL   all  --  *      *       0.0.0.0/0
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=>
>>>> > > 0.0.0.0/0
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=>
>>>>           recent: UPDATE seconds: 2 hit_count: 250 TTL-Match
>>>> > > name: limit_all_global side: source mask: 255.255.255.255
>>>> > >  149K   15M DROP_ALL   all  --  *      *       0.0.0.0/0
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=>
>>>> > > 0.0.0.0/0
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=S69x5cd6GIukj5xdZEQNNUnYwCrOIQklblT0zUw7IVM&e=>
>>>>           recent: UPDATE seconds: 2 reap hit_count: 150
>>>> > > TTL-Match name: limit_all_global side: source mask: 255.255.255.255
>>>> > >
>>>> > >
>>>> > > _______________________________________________
>>>> > > mod-security-users mailing list
>>>> > > mod...@li...
>>>> > > https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_mod-2Dsecurity-2Dusers&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=RUDsPA0iq5WVkW20NWQOl8suSJ4RvNfYZ6TM3FXNtdM&e=>
>>>> > > Commercial ModSecurity Rules and Support from Trustwave's
>>>> SpiderLabs:
>>>> > > http://www.modsecurity.org/projects/commercial/rules/
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_rules_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=rQF299h7fZhdJbhudnhzjEcP4e3Aa8qCG0KvKi4CKiM&e=>
>>>> > > http://www.modsecurity.org/projects/commercial/support/
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_support_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=ofF4OfFPsr3nKEMOH7j-CQmBqLgK_51fvOnQTavYK3c&e=>
>>>> > >
>>>>
>>>>
>>>> > _______________________________________________
>>>> > mod-security-users mailing list
>>>> > mod...@li...
>>>> > https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_mod-2Dsecurity-2Dusers&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=RUDsPA0iq5WVkW20NWQOl8suSJ4RvNfYZ6TM3FXNtdM&e=>
>>>> > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>>> > http://www.modsecurity.org/projects/commercial/rules/
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_rules_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=rQF299h7fZhdJbhudnhzjEcP4e3Aa8qCG0KvKi4CKiM&e=>
>>>> > http://www.modsecurity.org/projects/commercial/support/
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_support_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=ofF4OfFPsr3nKEMOH7j-CQmBqLgK_51fvOnQTavYK3c&e=>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> mod-security-users mailing list
>>>> mod...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_mod-2Dsecurity-2Dusers&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=RUDsPA0iq5WVkW20NWQOl8suSJ4RvNfYZ6TM3FXNtdM&e=>
>>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>>> http://www.modsecurity.org/projects/commercial/rules/
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_rules_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=rQF299h7fZhdJbhudnhzjEcP4e3Aa8qCG0KvKi4CKiM&e=>
>>>> http://www.modsecurity.org/projects/commercial/support/
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.modsecurity.org_projects_commercial_support_&d=DwMFaQ&c=S1d2Gs1Y1NQV8Lx35_Qi5FnTH2uYWyh_OhOS94IqYCo&r=E28NzkfUnnOxyipWbMmVvps8QnGe_19SJDNMcPTyffU&m=0DeHAXm5x7u_63IG4vvHEiJ7cWQqPlE3mjexyj6AoOY&s=ofF4OfFPsr3nKEMOH7j-CQmBqLgK_51fvOnQTavYK3c&e=>
>>>>
>>>> _______________________________________________
>>>> mod-security-users mailing list
>>>> mod...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>>> http://www.modsecurity.org/projects/commercial/rules/
>>>> http://www.modsecurity.org/projects/commercial/support/
>>>>
>>> _______________________________________________
>>> mod-security-users mailing list
>>> mod...@li...
>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>> http://www.modsecurity.org/projects/commercial/rules/
>>> http://www.modsecurity.org/projects/commercial/support/
>>>
>> _______________________________________________
>> mod-security-users mailing list
>> mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> http://www.modsecurity.org/projects/commercial/rules/
>> http://www.modsecurity.org/projects/commercial/support/
>>
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/
>