Menu
▾
▴
fail2ban-users
|
From: Marcel B. <mar...@we...> - 2023-10-19 11:50:20
|
<html><head></head><body><div style="font-family: Verdana;font-size: 12.0px;"><div>Hello everyone,</div> <div> </div> <div>i am in the need for some help, as i want to create a new filter.</div> <div> </div> <div>Setup:</div> <div> </div> <div>We are running a nginx-Server in a docker-container and on the system itself a fail2ban-installation.</div> <div> </div> <div>The Docker-Container writes via syslog-module into a file the content of the nginx-Logs and we want to check those logs for repeating 404-error and block those ips, which are creating those entries</div> <div> </div> <div>The Logfile looks like this:</div> <div> </div> <div> <div>Oct 16 15:49:02 localhost cabc0b82e7f9[424]: 192.168.10.10 - - [16/Oct/2023:13:49:02 +0000] "GET /de_DE/infrastruktu?order=website_priority%2Cname+asc HTTP/1.1" 404 3005 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/116.0" "-"</div> <div> <div>Oct 16 15:49:03 localhost cabc0b82e7f9[424]: 192.168.10.10 - - [16/Oct/2023:13:49:02 +0000] "GET /de_DE/infrastruktu?order=website_priority%2Cname+asc HTTP/1.1" 404 3005 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/116.0" "-"</div> <div>Oct 16 15:49:04 localhost cabc0b82e7f9[424]: 192.168.10.10 - - [16/Oct/2023:13:49:02 +0000] "GET /de_DE/infrastruktu?order=website_priority%2Cname+asc HTTP/1.1" 404 3005 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/116.0" "-"</div> <div>Oct 16 15:49:04 localhost cabc0b82e7f9[424]: 192.168.10.10 - - [16/Oct/2023:13:48:56 +0000] "GET /en_UK/theme_clarico/static/src/fileadmin/package/fonts/open-sans/Open_Sans_800.ttf HTTP/1.1" 404 2646 "/web/content/3223-5ddd78d/1/web.assets_frontend.1.css" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/118.0" "-"</div> <div> </div> <div>As you can see, we need to block the IP 192.168.10.10 or any other ip which are found on that position.</div> <div> </div> <div>I tried:</div> <div> </div> <div> <div>failregex = ^.+?(?=: ) <HOST>.*"(GET|POST).*" (403|404) .*$</div> <div> </div> <div> </div> <div>or</div> <div> </div> <div> <div>failregex = ^.+?(?=: ) <HOST> \- \S+ \[\] \"(GET|POST|HEAD) \/<block> \S+\" 404 .+$</div> <div> </div> <div> </div> <div>complete file:</div> <div> </div> <div> <div># Fail2Ban filter to match web requests for selected URLs that don't exist<br/> #</div> <div>[INCLUDES]</div> <div># Load regexes for filtering<br/> before = botsearch-common.conf</div> <div>[Definition]</div> <div>failregex = ^.+?(?=: ) <HOST> \- \S+ \[\] \"(GET|POST|HEAD) \/<block> \S+\" 404 .+$</div> <div>ignoreregex =</div> <div><br/> # DEV Notes:<br/> # Based on apache-botsearch filter<br/> #<br/> # Author: Frantisek Sumsal</div> <div> </div> <div> </div> <div>fail2ban-regex:</div> <div> </div> <div> <div>Running tests<br/> =============</div> <div>Use failregex filter file : nginx-docker, basedir: /etc/fail2ban<br/> Use log file : /root/nginx.log.2<br/> Use encoding : UTF-8</div> <div><br/> Results<br/> =======</div> <div>Failregex: 0 total</div> <div>Ignoreregex: 0 total</div> <div>Date template hits:<br/> |- [# of hits] date format<br/> | [994] (?:DAY )?MON Day 24hour:Minute:Second(?:\.Microseconds)?(?: Year)?<br/> `-</div> <div>Lines: 994 lines, 0 ignored, 0 matched, 994 missed<br/> [processed in 0.06 sec]</div> <div>Missed line(s): too many to print. Use --print-all-missed to print all 994 lines</div> <div> </div> </div> </div> </div> </div> <div> </div> <div> </div> <div>Could someone please point me in the right direction for the failregex?</div> <div> </div> <div>Thanks in advance!</div> <div> </div> <div>Greetings</div> <div> </div> <div>Marcel</div> <div> </div> <div> </div> </div> </div></div></body></html> |
|
From: Peter H. <mai...@ma...> - 2023-10-19 16:53:27
|
I think, you are not aware, what 192.168.10.y means. this is the IP-address seen inside the docker container. This IP is created by NAT on your host. If you block them, you are not blocking access from outside to your host, but blocking the way back from docker container to your host internal. This is output from nginx inside docker, not input. Of course, you can manualy set up a more sophisticated version, but consider this: docker-daemon is changeging the iptables. if you start runing a docker container, usualy iptables is used to add rules to setup NAT. there is a --ip-tables option to dockerd, which prevents the iptables rules from changeing by dockerd, but in most cases i tried, that causes malfunction. if you are runing firewalld there is a zone docker added IIRC, but i not realy know about. My advise would be, not to verify the log of nginx inside the docker. nginx is able to run as a reverse proxy. You probably shhould choose a setup outside --> nginx (reverse proxy) --> NAT --> docker --> nginx (webserver) such a setup is often used for large sites. On them not only 1 nginx(webserver) instances is runing, but a lot of them on different hosts. In most cases, creating a webside by php, perl or other script language need a lot of time. Only to get the answer from a webserver and deliver this to outside is just some kind of copy. however, because caching within reverse proxy, static objects, like .jpg are cached there. So the real webserver has not to serve ( depends on cache-header config), but only once a day or week. However, the logs of the reverse proxy contains the real outside addresses in log and of course the 404 answer generated by real webserver. From this point of view it is just a normal setup runing nginx as webserver, but using "proxy-pass" instead "try-files" within the location rule. Peter Am 19.10.2023 um 13:49 schrieb Marcel Blenkers: > Hello everyone, > i am in the need for some help, as i want to create a new filter. > Setup: > We are running a nginx-Server in a docker-container and on the system > itself a fail2ban-installation. > The Docker-Container writes via syslog-module into a file the content > of the nginx-Logs and we want to check those logs for repeating > 404-error and block those ips, which are creating those entries > The Logfile looks like this: > Oct 16 15:49:02 localhost cabc0b82e7f9[424]: 192.168.10.10 - - > [16/Oct/2023:13:49:02 +0000] "GET > /de_DE/infrastruktu?order=website_priority%2Cname+asc HTTP/1.1" 404 > 3005 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) > Gecko/20100101 Firefox/116.0" "-" > Oct 16 15:49:03 localhost cabc0b82e7f9[424]: 192.168.10.10 - - > [16/Oct/2023:13:49:02 +0000] "GET > /de_DE/infrastruktu?order=website_priority%2Cname+asc HTTP/1.1" 404 > 3005 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) > Gecko/20100101 Firefox/116.0" "-" > Oct 16 15:49:04 localhost cabc0b82e7f9[424]: 192.168.10.10 - - > [16/Oct/2023:13:49:02 +0000] "GET > /de_DE/infrastruktu?order=website_priority%2Cname+asc HTTP/1.1" 404 > 3005 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) > Gecko/20100101 Firefox/116.0" "-" > Oct 16 15:49:04 localhost cabc0b82e7f9[424]: 192.168.10.10 - - > [16/Oct/2023:13:48:56 +0000] "GET > /en_UK/theme_clarico/static/src/fileadmin/package/fonts/open-sans/Open_Sans_800.ttf > HTTP/1.1" 404 2646 > "/web/content/3223-5ddd78d/1/web.assets_frontend.1.css" "Mozilla/5.0 > (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/118.0" "-" > As you can see, we need to block the IP 192.168.10.10 or any other ip > which are found on that position. > I tried: > failregex = ^.+?(?=: ) <HOST>.*"(GET|POST).*" (403|404) .*$ > or > failregex = ^.+?(?=: ) <HOST> \- \S+ \[\] \"(GET|POST|HEAD) \/<block> > \S+\" 404 .+$ > complete file: > # Fail2Ban filter to match web requests for selected URLs that don't exist > # > [INCLUDES] > # Load regexes for filtering > before = botsearch-common.conf > [Definition] > failregex = ^.+?(?=: ) <HOST> \- \S+ \[\] \"(GET|POST|HEAD) \/<block> > \S+\" 404 .+$ > ignoreregex = > > # DEV Notes: > # Based on apache-botsearch filter > # > # Author: Frantisek Sumsal > fail2ban-regex: > Running tests > ============= > Use failregex filter file : nginx-docker, basedir: /etc/fail2ban > Use log file : /root/nginx.log.2 > Use encoding : UTF-8 > > Results > ======= > Failregex: 0 total > Ignoreregex: 0 total > Date template hits: > |- [# of hits] date format > | [994] (?:DAY )?MON Day 24hour:Minute:Second(?:\.Microseconds)?(?: > Year)? > `- > Lines: 994 lines, 0 ignored, 0 matched, 994 missed > [processed in 0.06 sec] > Missed line(s): too many to print. Use --print-all-missed to print > all 994 lines > Could someone please point me in the right direction for the failregex? > Thanks in advance! > Greetings > Marcel > > > _______________________________________________ > Fail2ban-users mailing list > Fai...@li... > https://lists.sourceforge.net/lists/listinfo/fail2ban-users |
|
From: Marcel B. <mar...@we...> - 2023-10-19 16:57:40
|
<!doctype html> <html> <head> <meta name="viewport" content="width=device-width"> <meta http-equiv="Content-Type" content="text/vnd.ui.insecure+html;charset=utf-8"> </head> <body style="overflow-wrap:break-word; word-break: break-word;"><div class="mail_android_message" style="line-height: 1; padding: 0.5em">Hi Peter, <br/><br/>thanks for the reply. <br/><br/>Unfortunatly i forgot something<br/><br/>i changed the ip for datapeotection<br/><br/>the ip <a href="http://192.168.10.10">192.168.10.10</a> is actually the ip which is accessing the webserver.<br/><br/>so it shows the correct ip, just not in my posting as i changed the ip<br/><br/>it is really the ip i need to block<br/><br/>greetings<br/><br/>Marcel<br/><br/>--<br/>Diese Nachricht wurde von meinem Android Mobiltelefon mit <a href="http://WEB.DE">WEB.DE</a> Mail gesendet.</div><div class="mail_android_quote" style="line-height: 1; padding: 0.3em"><html><body>Am 19.10.23, 18:35 schrieb Peter Heirich <mai...@ma...>:</body></html><blockquote class="gmail_quote" style="margin: 0.8ex 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"> I think, you are not aware, what 192.168.10.y means. <br> <br> this is the IP-address seen inside the docker container. This IP is created by NAT on your host. <br> <br> If you block them, you are not blocking access from outside to your host, but blocking the way back from docker container to your host internal. This is output from nginx inside docker, not input. <br> <br> Of course, you can manualy set up a more sophisticated version, but consider this: <br> docker-daemon is changeging the iptables. if you start runing a docker container, usualy iptables is used to add rules to setup NAT. <br> <br> there is a --ip-tables option to dockerd, which prevents the iptables rules from changeing by dockerd, but in most cases i tried, that causes malfunction. <br> <br> if you are runing firewalld there is a zone docker added IIRC, but i not realy know about. <br> <br> My advise would be, not to verify the log of nginx inside the docker. <br> <br> nginx is able to run as a reverse proxy. You probably shhould choose a setup <br> <br> outside --> nginx (reverse proxy) --> NAT --> docker --> nginx (webserver) <br> <br> such a setup is often used for large sites. On them not only 1 nginx(webserver) instances is runing, but a lot of them on different hosts. <br> <br> In most cases, creating a webside by php, perl or other script language need a lot of time. Only to get the answer from a webserver and deliver this to outside is just some kind of copy. however, because caching within reverse proxy, static objects, like .jpg are cached there. So the real webserver has not to serve ( depends on cache-header config), but only once a day or week. <br> <br> However, the logs of the reverse proxy contains the real outside addresses in log and of course the 404 answer generated by real webserver. <br> <br> From this point of view it is just a normal setup runing nginx as webserver, but using "proxy-pass" instead "try-files" within the location rule. <br> <br> Peter <br> <br> <div class="moz-cite-prefix"> Am 19.10.2023 um 13:49 schrieb Marcel Blenkers: <br> </div> <blockquote> <div style="font-family: Verdana;font-size: 12.0px;"> <div> Hello everyone, </div> <div> </div> <div> i am in the need for some help, as i want to create a new filter. </div> <div> </div> <div> Setup: </div> <div> </div> <div> We are running a nginx-Server in a docker-container and on the system itself a fail2ban-installation. </div> <div> </div> <div> The Docker-Container writes via syslog-module into a file the content of the nginx-Logs and we want to check those logs for repeating 404-error and block those ips, which are creating those entries </div> <div> </div> <div> The Logfile looks like this: </div> <div> </div> <div> <div> Oct 16 15:49:02 localhost cabc0b82e7f9[424]: 192.168.10.10 - - [16/Oct/2023:13:49:02 +0000] "GET /de_DE/infrastruktu?order=website_priority%2Cname+asc HTTP/1.1" 404 3005 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/116.0" "-" </div> <div> <div> Oct 16 15:49:03 localhost cabc0b82e7f9[424]: 192.168.10.10 - - [16/Oct/2023:13:49:02 +0000] "GET /de_DE/infrastruktu?order=website_priority%2Cname+asc HTTP/1.1" 404 3005 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/116.0" "-" </div> <div> Oct 16 15:49:04 localhost cabc0b82e7f9[424]: 192.168.10.10 - - [16/Oct/2023:13:49:02 +0000] "GET /de_DE/infrastruktu?order=website_priority%2Cname+asc HTTP/1.1" 404 3005 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/116.0" "-" </div> <div> Oct 16 15:49:04 localhost cabc0b82e7f9[424]: 192.168.10.10 - - [16/Oct/2023:13:48:56 +0000] "GET /en_UK/theme_clarico/static/src/fileadmin/package/fonts/open-sans/Open_Sans_800.ttf HTTP/1.1" 404 2646 "/web/content/3223-5ddd78d/1/web.assets_frontend.1.css" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/118.0" "-" </div> <div> </div> <div> As you can see, we need to block the IP 192.168.10.10 or any other ip which are found on that position. </div> <div> </div> <div> I tried: </div> <div> </div> <div> <div> failregex = ^.+?(?=: ) <HOST>.*"(GET|POST).*" (403|404) .*$ </div> <div> </div> <div> </div> <div> or </div> <div> </div> <div> <div> failregex = ^.+?(?=: ) <HOST> \- \S+ \[\] \"(GET|POST|HEAD) \/<block> \S+\" 404 .+$ </div> <div> </div> <div> </div> <div> complete file: </div> <div> </div> <div> <div> # Fail2Ban filter to match web requests for selected URLs that don't exist <br> # </div> <div> [INCLUDES] </div> <div> # Load regexes for filtering <br> before = botsearch-common.conf </div> <div> [Definition] </div> <div> failregex = ^.+?(?=: ) <HOST> \- \S+ \[\] \"(GET|POST|HEAD) \/<block> \S+\" 404 .+$ </div> <div> ignoreregex = </div> <div> <br> # DEV Notes: <br> # Based on apache-botsearch filter <br> # <br> # Author: Frantisek Sumsal </div> <div> </div> <div> </div> <div> fail2ban-regex: </div> <div> </div> <div> <div> Running tests <br> ============= </div> <div> Use failregex filter file : nginx-docker, basedir: /etc/fail2ban <br> Use log file : /root/nginx.log.2 <br> Use encoding : UTF-8 </div> <div> <br> Results <br> ======= </div> <div> Failregex: 0 total </div> <div> Ignoreregex: 0 total </div> <div> Date template hits: <br> |- [# of hits] date format <br> | [994] (?:DAY )?MON Day 24hour:Minute:Second(?:\.Microseconds)?(?: Year)? <br> `- </div> <div> Lines: 994 lines, 0 ignored, 0 matched, 994 missed <br> [processed in 0.06 sec] </div> <div> Missed line(s): too many to print. Use --print-all-missed to print all 994 lines </div> <div> </div> </div> </div> </div> </div> <div> </div> <div> </div> <div> Could someone please point me in the right direction for the failregex? </div> <div> </div> <div> Thanks in advance! </div> <div> </div> <div> Greetings </div> <div> </div> <div> Marcel </div> <div> </div> <div> </div> </div> </div> </div> <br> <br> <pre class="moz-quote-pre">_______________________________________________ Fail2ban-users mailing list <a class="moz-txt-link-abbreviated" href="mailto:Fai...@li...">Fai...@li...</a> <a class="moz-txt-link-freetext" href="https://lists.sourceforge.net/lists/listinfo/fail2ban-users">https://lists.sourceforge.net/lists/listinfo/fail2ban-users</a> </pre> </blockquote> <br> </blockquote></div></body> </html> |
|
From: Peter H. <mai...@ma...> - 2023-10-19 18:09:58
|
Am 19.10.2023 um 18:52 schrieb Marcel Blenkers:
> Hi Peter,
>
> thanks for the reply.
>
> Unfortunatly i forgot something
>
> i changed the ip for datapeotection
>
> the ip 192.168.10.10 <http://192.168.10.10> is actually the ip which
> is accessing the webserver.
>
> so it shows the correct ip, just not in my posting as i changed the ip
>
> it is really the ip i need to block
>
> greetings
>
> Marcel
OK lets have look:
> complete file:
> # Fail2Ban filter to match web requests for selected URLs that don't
> exist
> #
> [INCLUDES]
> # Load regexes for filtering
> before = botsearch-common.conf
> [Definition]
> failregex = ^.+?(?=: ) <HOST> \- \S+ \[\] \"(GET|POST|HEAD) \/<block>
> \S+\" 404 .+$
> ignoreregex =
>
> # DEV Notes:
> # Based on apache-botsearch filter
> #
> # Author: Frantisek Sumsal
I would use a copy of existing filter nginx-botsearch.conf as
nginx-botsearch.local
on my centos 8 stream:
# Fail2Ban filter to match web requests for selected URLs that don't exist
#
[INCLUDES]
# Load regexes for filtering
before = botsearch-common.conf
[Definition]
failregex = ^<HOST> \- \S+ \[\] \"(GET|POST|HEAD) \/<block> \S+\" 404 .+$
^ \[error\] \d+#\d+: \*\d+ (\S+ )?\"\S+\" (failed|is not
found) \(2\: No such file or directory\), client\: <HOST>\, server\:
\S*\, request: \"(GET|POST|HEAD) \/<block> \S+\"\, .*?$
ignoreregex =
datepattern = {^LN-BEG}%%ExY(?P<_sep>[-/.])%%m(?P=_sep)%%d[T
]%%H:%%M:%%S(?:[.,]%%f)?(?:\s*%%z)?
^[^\[]*\[({DATE})
{^LN-BEG}
journalmatch = _SYSTEMD_UNIT=nginx.service + _COMM=nginx
# DEV Notes:
# Based on apache-botsearch filter
#
# Author: Frantisek Sumsal
just seen: @web.de - 2hours difference to UTC
geändert auf:
failregex = ^<HOST> \- \S+ \[[^\]]*\] \"(GET|POST|HEAD) \/<block> \S+\"
40[34] .+$
Datepattern sollte in der 2. Zeile matchen
^[^\[]*\[({DATE})
^ - Zeilenstart
^[^\[] - alle Zeichen außer "["
* 0- bis beliebig viele davon
\[ - das Zeichen "[" ( ist vor dem 2. Datum / Zeit in UTC )
({DATE}) - das fängt dann Datum und Urzeit.
Zum failregex:
Bei dir \S+ \[\] \"(GET
erfordert [], ist bei dir aber [16/Oct/2023:13:49:02 +0000] ---> kein
match, nie
\[[^\]]*\]
ist
\[ "["
[^\]] - alle Zeichen, die nicht "]" sind
* 0 bis beliebig viele davon.
\] "]"
Es könnte sein, dass du ein Problem bekommst, weil das 2. Datum/Zeit in
UTC ist, das Log selbst aber in MESZ.
IIRC wird +0000 aber beachtet, so dass die Chancen gut sind. Prüfe aber
bitte im log, ob die Zeit korrekt erkannt wird.
40[34] erfasst die Fehler 403 und 404 Ich hatte dein (403|404) so
verstanden, dass das gewünscht ist.
Willst du nur 404 bleibt, es bei 404
Grüße aus Berlin
|
|
From: James M. <moe...@sm...> - 2023-10-19 21:02:31
|
On 10/19/23 4:49 AM, Marcel Blenkers wrote: > The Logfile looks like this: > Oct 16 15:49:02 localhost cabc0b82e7f9[424]: 192.168.10.10 - - > [16/Oct/2023:13:49:02 +0000] "GET > /de_DE/infrastruktu?order=website_priority%2Cname+asc HTTP/1.1" 404 > 3005 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) > Gecko/20100101 Firefox/116.0" "-" Try this: fail2reg = ^.* <HOST> .* HTTP/.* (403|404).* -- |
|
From: John W. <wil...@gm...> - 2023-10-20 15:12:31
|
Here I use, [Definition] failregex = ^<HOST>.*"(GET|POST).*" (404|444|403|400) .*$ ignoreregex = Kind regards, John Willemse LinkedIn: https://www.linkedin.com/in/willemsej/ Twitter: https://twitter.com/willemsej/ Op do 19 okt 2023 om 23:03 schreef James Moe via Fail2ban-users < fai...@li...>: > On 10/19/23 4:49 AM, Marcel Blenkers wrote: > > The Logfile looks like this: > > Oct 16 15:49:02 localhost cabc0b82e7f9[424]: 192.168.10.10 - - > > [16/Oct/2023:13:49:02 +0000] "GET > > /de_DE/infrastruktu?order=website_priority%2Cname+asc HTTP/1.1" 404 > > 3005 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) > > Gecko/20100101 Firefox/116.0" "-" > > Try this: > > fail2reg = ^.* <HOST> .* HTTP/.* (403|404).* > > > -- > > > > > _______________________________________________ > Fail2ban-users mailing list > Fai...@li... > https://lists.sourceforge.net/lists/listinfo/fail2ban-users > |
Thanks for helping keep SourceForge clean.
X