From: Richard C. <ri...@gm...> - 2007-07-29 03:07:08
|
If you run a DNS server on your system you probably have been plagued with external sites trying to forward queries through your DNS server. Even though you probably have told your named.conf to allow-query {"localnets";}; or a list of valid IP's you probably still have a bunch of unnecessary probing that adds to your bandwidth consumption even if you reject the queries and send 'refused' packets back, it ties up your line. I got tired of literally hundreds, sometimes thoussands of such queries which I considered a form of attack and thought that "fail2ban" could be a solution. I know about as much about writing filters as I do about the differance between my posterior and a hole in the ground, but a fellow list member, Yaroslav Halchenko took pity on me and in private E-mail, helped me develop a filter we call 'named-refused'. On 7/24 I installed it into fail2ban and started testing it. The results are in the log summary below. You will notice on the 24th, the filter 'named-refused was innvoked "a lot" and by the next day, it was back to the normal fighting off the sshd worm, and even that has gone way down since fail2ban was installed. I didn't post my entire log, but it is just as impressive to note that as of the 24th, fail2ban as reduced my DNS attack bandwidth to zero because whoever those badguys are have apparantly decided that because I no longer appear to exist that it isn't worth wasting their time trying anymore. As long as I responded to all of their attempts, even though they got 'refused' each time, they kept trying. Yay fail2ban and thank you Cyril for a fine product and Yaroslav for your patience and time. Yaroslav reminds me that because DNS does not normaly use TCP that it would be possible to have a vindictive attacker mount a DDoS attack by sending bogus IP addresses and getting you to ban your entire network. That is true IF he knows you are using fail2ban (which does not identify itself) and by proper unban times, you would restore communication automatically in a short time but the attacker would have to know your defense involves fail2ban and that is unlikely as he has no credible way to detect it remotely. I think it is worth the risk and the log below shows how effective it can be. I run SUSE and I am more than willing to zip up my /etc/fail2ban local files which should work with little or no modification on other distros. Yaroslav said he will submit a patch for Debian to Cyril to consider for distribution. BTW, the report below can be produced by: grep "Ban " /var/log/fail2ban.log | awk '{print $1,$5,$7}' | sort |uniq -c assuming your log file is in that directory with that name. Substitute your log file name if you don't use that name. Richard Nr Date Filter IP of Attacker 1 2007-07-23 [ssh-iptables] 165.230.95.44 2 2007-07-24 [named-refused] 128.110.124.120 1 2007-07-24 [named-refused] 130.57.22.201 1 2007-07-24 [named-refused] 130.57.22.6 1 2007-07-24 [named-refused] 137.65.1.1 1 2007-07-24 [named-refused] 137.65.1.2 6 2007-07-24 [named-refused] 148.160.29.6 2 2007-07-24 [named-refused] 155.101.98.155 2 2007-07-24 [named-refused] 155.101.98.156 1 2007-07-24 [named-refused] 165.230.69.67 1 2007-07-24 [named-refused] 165.230.81.231 1 2007-07-24 [named-refused] 165.230.84.227 1 2007-07-24 [named-refused] 165.230.95.119 3 2007-07-24 [named-refused] 165.230.95.90 2 2007-07-24 [named-refused] 204.127.192.82 2 2007-07-24 [named-refused] 204.127.192.85 2 2007-07-24 [named-refused] 204.127.193.31 1 2007-07-24 [named-refused] 204.127.193.32 1 2007-07-24 [named-refused] 204.127.193.33 1 2007-07-24 [named-refused] 204.127.193.36 1 2007-07-24 [named-refused] 204.127.200.81 1 2007-07-24 [named-refused] 204.127.200.82 1 2007-07-24 [named-refused] 204.127.200.83 2 2007-07-24 [named-refused] 204.127.200.84 1 2007-07-24 [named-refused] 204.127.201.29 1 2007-07-24 [named-refused] 204.127.201.31 1 2007-07-24 [named-refused] 204.127.201.32 1 2007-07-24 [named-refused] 204.127.201.33 1 2007-07-24 [named-refused] 204.127.201.35 1 2007-07-24 [named-refused] 204.127.201.36 1 2007-07-24 [named-refused] 216.148.226.32 1 2007-07-24 [named-refused] 216.148.226.33 1 2007-07-24 [named-refused] 216.148.226.34 1 2007-07-24 [named-refused] 216.148.226.36 1 2007-07-24 [named-refused] 216.148.227.153 3 2007-07-24 [named-refused] 63.240.77.32 1 2007-07-24 [named-refused] 63.240.77.81 1 2007-07-24 [named-refused] 63.240.77.85 1 2007-07-24 [named-refused] 65.110.190.249 1 2007-07-24 [ssh-iptables] 222.107.187.2 1 2007-07-25 [ssh-iptables] 200.61.47.165 1 2007-07-25 [ssh-iptables] 213.176.96.5 1 2007-07-25 [ssh-iptables] 61.185.220.249 1 2007-07-26 [ssh-iptables] 61.200.49.40 1 2007-07-27 [ssh-iptables] 202.172.229.16 1 2007-07-27 [ssh-iptables] 61.172.200.150 |
From: Yaroslav H. <li...@on...> - 2007-07-29 15:44:50
Attachments:
named-refused.conf
|
And please find filter attached (I've sent it to the list few times already but list server for some reason didn't forward my emails to the list). I want to mention once again that there is a possibility of DoS against your server, so use this filter/jail with caution: * Add your local IPs to ignoreip (so your local boxes can't be denied access) * Do not enable permanent banning for the jail (bantime = -1) -- just make it sufficiently long (like an hour: bantime = 3600) * It is better to ban only DNS specific ports (so do not use iptables-all or shorewall actions), so that you do not provoke DoS against the server if it hosts other services * Enable both udp and tcp jails (some small portion of queries from outside can come through TCP) Jail configuration in debian (in stock jail.conf you would need to specify action explicitely) for named-refused would be: # DNS Servers # Mention: by default logging is off with bind installation. # Need smth like # logging { # channel lame-servers_file { file "/var/log/named/lame-servers.log" versions 3 size 30m; severity dynamic; print-time yes; }; # category lame-servers { lame-servers_file; }; # } # in your named.conf to provide proper logging [named-refused-udp] enabled = false port = domain,953 protocol = udp filter = named-refused logpath = /var/log/named/lame-servers.log [named-refused-tcp] enabled = false port = domain,953 protocol = tcp filter = named-refused logpath = /var/log/named/lame-servers.lo On Sat, 28 Jul 2007, Richard Creighton wrote: > If you run a DNS server on your system you probably have been plagued > with external sites trying to forward queries through your DNS server. > Even though you probably have told your named.conf to allow-query -- .-. =------------------------------ /v\ ----------------------------= Keep in touch // \\ (yoh@|www.)onerussian.com Yaroslav Halchenko /( )\ ICQ#: 60653192 Linux User ^^-^^ [175555] |
From: Yaroslav H. <li...@on...> - 2007-08-17 16:46:16
Attachments:
00_named_refused.dpatch
|
Gy gy... we just shut ourselves into the foot ;-) I found a bleeding toe ;-) Or the story on how we got a new buggy feature in fail2ban without my clear understanding on what we are doing. Please correct me if I am wrong anywhere below. ok -- in 2 words, due to my ignorance in bind setup, I misunderstood such entries as 17-Aug-2007 11:56:06.005 unexpected RCODE (REFUSED) resolving 'hermes.nswcl.navy.mil/A/IN': 164.158.250.2#53 to be a 'bad client from 164.158.250.2'. What it is in reality: our DNS server received a query to resolve hermes.nswcl.navy.mil, which is actually forbidden to resolve by the navy for obvious reason. And that is what it was actually said above. If you have query-source address * port 53; in your named.conf.options, then your servers will query from port 53 (not from some arbitrary port which is the default since 8.1) and their replies will be blocked on maxfailures+1 attempt to resolve any forbidden name from them. If all domains served by those servers are such private ones, then such banning can be considered kinda a good feature I guess, since it reduces log polution with those REFUSED logs. But in most cases, I think, it is bug and thus such failregex should be removed since it would provide DoS automatically for some servers which serve restricted and non-restricted domains. The healthy failregex is the 2nd one %(__line_prefix)sclient <HOST>#\S+: query(?: \(cache\))? '.*' denied\s*$ but if you are using the logging schema as I suggested you really need to log category security not the lame-servers (and why didn't I think about that name more?) so use smth like channel security_file { file "/var/log/named/security.log" versions 3 size 30m; severity dynamic; print-time yes; }; category security { security_file; }; and point your jail at that security.log Cyril, please find respective patch with that failregex removed and jail.conf adjusted. I left the name of the chain the same although more logically it should be named-denied. May be we could even create to filters -- named-refused with a "bad" failregex for those who like to have logs clean though being a part of DoS network segment ;-) Thanks everyone -- enjoy ;-) Any comments are welcome. Yarik On Sun, 29 Jul 2007, Yaroslav Halchenko wrote: > And please find filter attached (I've sent it to the list few times > already but list server for some reason didn't forward my emails to the > list). > I want to mention once again that there is a possibility of DoS against > your server, so use this filter/jail with caution: > * Add your local IPs to ignoreip (so your local boxes can't be denied > access) > * Do not enable permanent banning for the jail (bantime = -1) -- just > make it sufficiently long (like an hour: bantime = 3600) > * It is better to ban only DNS specific ports (so do not use iptables-all > or shorewall actions), so that you do not provoke DoS against the > server if it hosts other services > * Enable both udp and tcp jails (some small portion of queries from > outside can come through TCP) > Jail configuration in debian (in stock jail.conf you would need to > specify action explicitely) for named-refused would be: > # DNS Servers > # Mention: by default logging is off with bind installation. > # Need smth like > # logging { > # channel lame-servers_file { file "/var/log/named/lame-servers.log" versions 3 size 30m; severity dynamic; print-time yes; }; > # category lame-servers { lame-servers_file; }; > # } > # in your named.conf to provide proper logging > [named-refused-udp] > enabled = false > port = domain,953 > protocol = udp > filter = named-refused > logpath = /var/log/named/lame-servers.log > [named-refused-tcp] > enabled = false > port = domain,953 > protocol = tcp > filter = named-refused > logpath = /var/log/named/lame-servers.lo > On Sat, 28 Jul 2007, Richard Creighton wrote: > > If you run a DNS server on your system you probably have been plagued > > with external sites trying to forward queries through your DNS server. > > Even though you probably have told your named.conf to allow-query -- .-. =------------------------------ /v\ ----------------------------= Keep in touch // \\ (yoh@|www.)onerussian.com Yaroslav Halchenko /( )\ ICQ#: 60653192 Linux User ^^-^^ [175555] |
From: Justin P. <jp...@lu...> - 2007-08-17 20:30:19
|
I agree and look forward to the patch+changes. On Fri, 17 Aug 2007, Yaroslav Halchenko wrote: > Gy gy... we just shut ourselves into the foot ;-) I found a bleeding toe > ;-) Or the story on how we got a new buggy feature in fail2ban > without my clear understanding on what we are doing. Please correct me > if I am wrong anywhere below. > > ok -- in 2 words, due to my ignorance in bind setup, I misunderstood > such entries as > > 17-Aug-2007 11:56:06.005 unexpected RCODE (REFUSED) resolving 'hermes.nswcl.navy.mil/A/IN': 164.158.250.2#53 > > to be a 'bad client from 164.158.250.2'. > > What it is in reality: our DNS server received a query to resolve > hermes.nswcl.navy.mil, which is actually forbidden to resolve by the > navy for obvious reason. And that is what it was actually said above. > > If you have > > query-source address * port 53; > > in your named.conf.options, then your servers will query from port 53 (not from > some arbitrary port which is the default since 8.1) and their replies will be > blocked on maxfailures+1 attempt to resolve any forbidden name from them. If all > domains served by those servers are such private ones, then such banning can be > considered kinda a good feature I guess, since it reduces log polution with those > REFUSED logs. But in most cases, I think, it is bug and thus such failregex > should be removed since it would provide DoS automatically for some servers > which serve restricted and non-restricted domains. > > The healthy failregex is the 2nd one > > %(__line_prefix)sclient <HOST>#\S+: query(?: \(cache\))? '.*' denied\s*$ > > but if you are using the logging schema as I suggested you really need to > log category security not the lame-servers (and why didn't I think about that > name more?) so use smth like > > channel security_file { file "/var/log/named/security.log" versions 3 size 30m; severity dynamic; print-time yes; }; > category security { security_file; }; > > and point your jail at that security.log > > > Cyril, please find respective patch with that failregex removed and jail.conf > adjusted. I left the name of the chain the same although more logically it > should be named-denied. May be we could even create to filters -- > named-refused with a "bad" failregex for those who like to have logs clean > though being a part of DoS network segment ;-) > > Thanks everyone -- enjoy ;-) Any comments are welcome. > > Yarik > > > On Sun, 29 Jul 2007, Yaroslav Halchenko wrote: > >> And please find filter attached (I've sent it to the list few times >> already but list server for some reason didn't forward my emails to the >> list). > >> I want to mention once again that there is a possibility of DoS against >> your server, so use this filter/jail with caution: > >> * Add your local IPs to ignoreip (so your local boxes can't be denied >> access) >> * Do not enable permanent banning for the jail (bantime = -1) -- just >> make it sufficiently long (like an hour: bantime = 3600) >> * It is better to ban only DNS specific ports (so do not use iptables-all >> or shorewall actions), so that you do not provoke DoS against the >> server if it hosts other services >> * Enable both udp and tcp jails (some small portion of queries from >> outside can come through TCP) > >> Jail configuration in debian (in stock jail.conf you would need to >> specify action explicitely) for named-refused would be: > >> # DNS Servers > >> # Mention: by default logging is off with bind installation. >> # Need smth like >> # logging { >> # channel lame-servers_file { file "/var/log/named/lame-servers.log" versions 3 size 30m; severity dynamic; print-time yes; }; >> # category lame-servers { lame-servers_file; }; >> # } >> # in your named.conf to provide proper logging > >> [named-refused-udp] > >> enabled = false >> port = domain,953 >> protocol = udp >> filter = named-refused >> logpath = /var/log/named/lame-servers.log > >> [named-refused-tcp] > >> enabled = false >> port = domain,953 >> protocol = tcp >> filter = named-refused >> logpath = /var/log/named/lame-servers.lo > > >> On Sat, 28 Jul 2007, Richard Creighton wrote: > >>> If you run a DNS server on your system you probably have been plagued >>> with external sites trying to forward queries through your DNS server. >>> Even though you probably have told your named.conf to allow-query > -- > .-. > =------------------------------ /v\ ----------------------------= > Keep in touch // \\ (yoh@|www.)onerussian.com > Yaroslav Halchenko /( )\ ICQ#: 60653192 > Linux User ^^-^^ [175555] > > > |
From: Yaroslav H. <li...@on...> - 2007-08-18 05:08:17
|
> I agree and look forward to the patch+changes. good -- so my reasoning was correct I guess ;-) FWIW -- patch was supplied with my email. you can patch sources yourself with patch -p1 command while you are in the source of fail2ban. On Fri, 17 Aug 2007, Justin Piszcz wrote: -- .-. =------------------------------ /v\ ----------------------------= Keep in touch // \\ (yoh@|www.)onerussian.com Yaroslav Halchenko /( )\ ICQ#: 60653192 Linux User ^^-^^ [175555] |