From: Charles B. <br...@br...> - 2013-09-14 17:35:18
|
Hi I've been running fail2ban very successfully to block fake registrations for a Yabb forum. I started just blocking offenders for a 1 week period, but this didn't reduced the number of attempts. Next I have started to block offenders permanently, still no reduction in attempts. I presume the script kiddies are moving around faster than I'm blocking them! The problem is I'm seeing something like 5-10 attempts hour. The received wisdom is that iptables will become unstable when the number of rules reaches, in the order of, 25000 and will start to consume processor resource long before that. I will reach 25000 in under 6 months at the current rate. I have been looking for a method of doing mass blocking while relieving iptables of the load. I see ipset here: http://ipset.netfilter.org/ which looks like it might fit the bill. Is there any experience here to guide me in setting up ipset and integrating it with fail2ban? Perhaps there is some other tool to alleviate the inevitable slowdown of iptables when the number of rules reaches multi thousands? Thanks in advance, Charles Bradshaw |
From: Fabian W. <fa...@we...> - 2013-09-14 22:03:30
|
Hello Charles On 14.09.2013 19:35, Charles Bradshaw wrote: > The problem is I'm seeing something like 5-10 attempts hour. The > received wisdom is that iptables will become unstable when the number of > rules reaches, in the order of, 25000 and will start to consume > processor resource long before that. I will reach 25000 in under 6 > months at the current rate. What happens if you use a much shorter bantime, e.g. only a few hours or maybe a few days? Do the same IP address try again? bye Fabian |
From: Daniel B. <dan...@in...> - 2013-09-14 23:00:56
|
On 15/09/13 03:35, Charles Bradshaw wrote: > Hi > > I've been running fail2ban very successfully to block fake registrations > for a Yabb forum. I started just blocking offenders for a 1 week period, > but this didn't reduced the number of attempts. Next I have started to > block offenders permanently, still no reduction in attempts. > > I presume the script kiddies are moving around faster than I'm blocking > them! > > The problem is I'm seeing something like 5-10 attempts hour. The > received wisdom is that iptables will become unstable when the number of > rules reaches, in the order of, 25000 and will start to consume > processor resource long before that. I will reach 25000 in under 6 > months at the current rate. > > I have been looking for a method of doing mass blocking while relieving > iptables of the load. I see ipset here: http://ipset.netfilter.org/ > which looks like it might fit the bill. > > Is there any experience here to guide me in setting up ipset and > integrating it with fail2ban? https://github.com/fail2ban/fail2ban/tree/master/config/action.d has iptables-ipset-proto4.conf and iptables-ipset-proto6.conf depending which version you are running (ipset -V to see the protocol and version). Those configuration files have instructions on them. |
From: Charles B. <br...@br...> - 2013-09-15 09:50:35
|
Hello Fabian, Thanks for the reply. I fail to see how knowing the frequency of repeat offending IPs helps me. I have sometimes determined if the offending IP is still on-line. Some are most aren't. None nmap! The apache logs show the offenders user agent, seem like "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:22.0) Gecko/20100101 Firefox/22.0" is popular. Some of the request strings are huge, having the GET...'?action=register' concatenated many times with different '+garbage' strings. Just looks like a brute force on many different names! Did I miss something? Charles Bradshaw On Sun, 2013-09-15 at 00:03 +0200, Fabian Wenk wrote: > Hello Charles > > On 14.09.2013 19:35, Charles Bradshaw wrote: > > The problem is I'm seeing something like 5-10 attempts hour. The > > received wisdom is that iptables will become unstable when the number of > > rules reaches, in the order of, 25000 and will start to consume > > processor resource long before that. I will reach 25000 in under 6 > > months at the current rate. > > What happens if you use a much shorter bantime, e.g. only a few > hours or maybe a few days? Do the same IP address try again? > > > bye > Fabian > > ------------------------------------------------------------------------------ > LIMITED TIME SALE - Full Year of Microsoft Training For Just $49.99! > 1,500+ hours of tutorials including VisualStudio 2012, Windows 8, SharePoint > 2013, SQL 2012, MVC 4, more. BEST VALUE: New Multi-Library Power Pack includes > Mobile, Cloud, Java, and UX Design. Lowest price ever! Ends 9/22/13. > http://pubads.g.doubleclick.net/gampad/clk?id=64545871&iu=/4140/ostg.clktrk > _______________________________________________ > Fail2ban-users mailing list > Fai...@li... > https://lists.sourceforge.net/lists/listinfo/fail2ban-users |
From: Fabian W. <fa...@we...> - 2013-09-15 12:18:51
|
Hello Charles On 15.09.2013 11:50, Charles Bradshaw wrote: > I fail to see how knowing the frequency of repeat offending IPs helps > me. If the same IP address does not try to auto create new accounts after e.g. a few hours or days, then it is useless to ban it for several months. > I have sometimes determined if the offending IP is still on-line. Some > are most aren't. None nmap! I do not care if an IP is still online or not. I only care if I do still get brute force attacks from it or not. Currently I do not have any services running where anybody can create new accounts / registrations, so I do not have your particular problem on my site. But I do get a lot of password brute force against several services. But most of the scans stop if I ban them with fail2ban. I guess what is helping here is on TCP connections to response with "ICMP port unreachable" instead of just silently dropping the packets. Because so the scanner does get feedback that this port is not open any more and most often then stops hitting the system. bye Fabian |
From: Gregory S. <gr...@sl...> - 2013-09-15 23:43:00
|
CB> Hello Fabian, CB> Thanks for the reply. CB> I fail to see how knowing the frequency of repeat offending IPs helps CB> me. CB> I have sometimes determined if the offending IP is still on-line. Some CB> are most aren't. None nmap! CB> The apache logs show the offenders user agent, seem like "Mozilla/5.0 CB> (Macintosh; Intel Mac OS X 10.7; rv:22.0) Gecko/20100101 Firefox/22.0" CB> is popular. CB> Some of the request strings are huge, having the CB> GET...'?action=register' concatenated many times with different CB> '+garbage' strings. Just looks like a brute force on many different CB> names! CB> Did I miss something? CB> Charles Bradshaw I'm no expert here, but this *looks* to me to be a distributed spam attack. IMO, this does NOT fit an IP block very well. 1) There's a botnet that's being used to attempt these, and are simply "innocent" users machines who have been infected with spyware/malware. You could easily be blocking 50K machines. While this may not bother you (as you never expect to actually see one of these machines as a visitor/customer) - the churn rate on infected machines will likely be very high. So, over time you could be blocking hundreds of thousands of machines - and with no real effect/benefit. i.e. You'll be expending huge levels of resources to block a few annoying machines, but the number of "new" botnet members is essentially limitless for the "attacker" and the benefit from blocking one machine from making new attempts is rather small. If you maintain these forever, it's highly likely you'll start impacting visitors that might legitimately visit your site. [And this doesn't really take into consideration clients on dynamic IP's who will have one IP blocked only to move to another, and leave the "old" IP still blocked with a non-hostile machine on it now.] Essentially, fail2ban works for non-distributed "attacks" - where one attacker at a single IP will make many attempts. It's rather terrible, IMO, for a distributed attack where one attacker will make a few attempts via many thousands of machines. The latter situation is rather harder to address. IMO, captchca's or other methods are a lot more applicable than fail2ban. If I were in your shoes, I'd simply probably try to ignore the attempts [at least via fail2ban] and harden the site so successes would be very hard earned. [And the likelihood they succeed is pretty low anyway.] I'd guess something like manual subscription approval, captchca's etc, are more likely to bring satisfaction, though at increased admin time, and resources. These approaches tend to have better returns on investment than fail2ban for your situation, IMO. [I'm not sure what BB you're running, but wordpress etc, PHPBB etc all have tools that work pretty well for the problems that typically impact such sites and IMO, you'll fare better using those tools. You may *feel* better being able to shake your ban-hammer at the botnet, but I don't think you'll actually *be* in a better place.] But that's all my opinion and you're more than welcome to ignore it. -Greg |
From: Charles B. <br...@br...> - 2013-09-16 00:55:13
|
Danial, Thanks for the headsup. I have installed ipset from my Fedora installation, read the man page and looked at action.d/iptables-ipset-proto4.conf I'm not sure what the .conf file tells me, but anyway I did the following: Modified jail.conf for [forum-noregister] Change the action line from: action = iptables-multiport[name=forum, port="http,https", protocol=tcp] To: action = iptables-ipset-proto4[name=forum, port="http,https", protocol=tcp] # ipset -V ipset v6.14, protocol version: 6 # ipset create fail2ban-forum hash:ip # ipset list Name: fail2ban-forum Type: hash:ip Revision: 0 Header: family inet hashsize 1024 maxelem 65536 Size in memory: 8276 References: 0 Members: And restarted fail2ban # systemctl restart fail2ban Then: # ipset list Name: fail2ban-forum Type: hash:ip Revision: 0 Header: family inet hashsize 1024 maxelem 65536 Size in memory: 8404 References: 0 Members: 91.236.74.7 ... So all working. There is a selinux problem, but that's another story. Thanks, Charles Bradshaw On Sun, 2013-09-15 at 09:00 +1000, Daniel Black wrote: > On 15/09/13 03:35, Charles Bradshaw wrote: > > Hi > > > > I've been running fail2ban very successfully to block fake registrations > > for a Yabb forum. I started just blocking offenders for a 1 week period, > > but this didn't reduced the number of attempts. Next I have started to > > block offenders permanently, still no reduction in attempts. > > > > I presume the script kiddies are moving around faster than I'm blocking > > them! > > > > The problem is I'm seeing something like 5-10 attempts hour. The > > received wisdom is that iptables will become unstable when the number of > > rules reaches, in the order of, 25000 and will start to consume > > processor resource long before that. I will reach 25000 in under 6 > > months at the current rate. > > > > I have been looking for a method of doing mass blocking while relieving > > iptables of the load. I see ipset here: http://ipset.netfilter.org/ > > which looks like it might fit the bill. > > > > Is there any experience here to guide me in setting up ipset and > > integrating it with fail2ban? > > > https://github.com/fail2ban/fail2ban/tree/master/config/action.d has > iptables-ipset-proto4.conf and iptables-ipset-proto6.conf depending > which version you are running (ipset -V to see the protocol and version). > > Those configuration files have instructions on them. > > ------------------------------------------------------------------------------ > LIMITED TIME SALE - Full Year of Microsoft Training For Just $49.99! > 1,500+ hours of tutorials including VisualStudio 2012, Windows 8, SharePoint > 2013, SQL 2012, MVC 4, more. BEST VALUE: New Multi-Library Power Pack includes > Mobile, Cloud, Java, and UX Design. Lowest price ever! Ends 9/22/13. > http://pubads.g.doubleclick.net/gampad/clk?id=64545871&iu=/4140/ostg.clktrk > _______________________________________________ > Fail2ban-users mailing list > Fai...@li... > https://lists.sourceforge.net/lists/listinfo/fail2ban-users |
From: Charles B. <br...@br...> - 2013-09-16 21:39:21
|
Hello list I'm replying to my own post because there's VITAL missing step to get fail2ban and ipset working which I missed. (See VITAL below) The way I read iptables-ipset-proto4.conf ie this line: actionstart = ipset --create fail2ban-<name> iphash iptables -I INPUT -p <protocol> -m multiport --dports <port> -m set --match-set fail2ban-<name> src -j <blocktype> When f2b restarts the necessary above configuration steps should occur for iptables and ipset. This does NOT appear to be the case! I had to do both steps manually. At least not in the event of a restart, as opposed to a clean start. Did I miss something else? I don't think so because the actionban line is clearly working. On Mon, 2013-09-16 at 01:55 +0100, Charles Bradshaw wrote: > Danial, > > Thanks for the headsup. > > I have installed ipset from my Fedora installation, read the man page > and looked at action.d/iptables-ipset-proto4.conf > > I'm not sure what the .conf file tells me, but anyway I did the > following: > Modified jail.conf for > [forum-noregister] > > Change the action line from: > action = iptables-multiport[name=forum, port="http,https", > protocol=tcp] > To: > action = iptables-ipset-proto4[name=forum, port="http,https", > protocol=tcp] > > # ipset -V > ipset v6.14, protocol version: 6 > # ipset create fail2ban-forum hash:ip > # ipset list > Name: fail2ban-forum > Type: hash:ip > Revision: 0 > Header: family inet hashsize 1024 maxelem 65536 > Size in memory: 8276 > References: 0 > Members: The following VITAL step without which fail2ban 'thinks' IPs are banned, but we just see "IP alread banned" when in fact it is not. # iptables -I INPUT 1 -m set --match-set 'fail2ban-forum' src -j REJECT > > And restarted fail2ban > # systemctl restart fail2ban > > Then: > # ipset list > Name: fail2ban-forum > Type: hash:ip > Revision: 0 > Header: family inet hashsize 1024 maxelem 65536 > Size in memory: 8404 > References: 0 > Members: > 91.236.74.7 > ... > > So all working. There is a selinux problem, but that's another story. > > Thanks, Charles Bradshaw > > On Sun, 2013-09-15 at 09:00 +1000, Daniel Black wrote: > > On 15/09/13 03:35, Charles Bradshaw wrote: > > > Hi > > > > > > I've been running fail2ban very successfully to block fake registrations > > > for a Yabb forum. I started just blocking offenders for a 1 week period, > > > but this didn't reduced the number of attempts. Next I have started to > > > block offenders permanently, still no reduction in attempts. > > > > > > I presume the script kiddies are moving around faster than I'm blocking > > > them! > > > > > > The problem is I'm seeing something like 5-10 attempts hour. The > > > received wisdom is that iptables will become unstable when the number of > > > rules reaches, in the order of, 25000 and will start to consume > > > processor resource long before that. I will reach 25000 in under 6 > > > months at the current rate. > > > > > > I have been looking for a method of doing mass blocking while relieving > > > iptables of the load. I see ipset here: http://ipset.netfilter.org/ > > > which looks like it might fit the bill. > > > > > > Is there any experience here to guide me in setting up ipset and > > > integrating it with fail2ban? > > > > > > https://github.com/fail2ban/fail2ban/tree/master/config/action.d has > > iptables-ipset-proto4.conf and iptables-ipset-proto6.conf depending > > which version you are running (ipset -V to see the protocol and version). > > > > Those configuration files have instructions on them. > > > > ------------------------------------------------------------------------------ > > LIMITED TIME SALE - Full Year of Microsoft Training For Just $49.99! > > 1,500+ hours of tutorials including VisualStudio 2012, Windows 8, SharePoint > > 2013, SQL 2012, MVC 4, more. BEST VALUE: New Multi-Library Power Pack includes > > Mobile, Cloud, Java, and UX Design. Lowest price ever! Ends 9/22/13. > > http://pubads.g.doubleclick.net/gampad/clk?id=64545871&iu=/4140/ostg.clktrk > > _______________________________________________ > > Fail2ban-users mailing list > > Fai...@li... > > https://lists.sourceforge.net/lists/listinfo/fail2ban-users > > > > ------------------------------------------------------------------------------ > LIMITED TIME SALE - Full Year of Microsoft Training For Just $49.99! > 1,500+ hours of tutorials including VisualStudio 2012, Windows 8, SharePoint > 2013, SQL 2012, MVC 4, more. BEST VALUE: New Multi-Library Power Pack includes > Mobile, Cloud, Java, and UX Design. Lowest price ever! Ends 9/20/13. > http://pubads.g.doubleclick.net/gampad/clk?id=58041151&iu=/4140/ostg.clktrk > _______________________________________________ > Fail2ban-users mailing list > Fai...@li... > https://lists.sourceforge.net/lists/listinfo/fail2ban-users |
From: Daniel B. <dan...@in...> - 2013-09-16 23:29:54
|
> # ipset -V > ipset v6.14, protocol version: 6 You have protocol 6 so you can use iptables-ipset-proto6.conf On 17/09/13 07:39, Charles Bradshaw wrote: > Hello list I'm replying to my own post because there's VITAL missing > step to get fail2ban and ipset working which I missed. (See VITAL below) > > The way I read iptables-ipset-proto4.conf ie this line: > actionstart = ipset --create fail2ban-<name> iphash > iptables -I INPUT -p <protocol> -m multiport --dports > <port> -m set --match-set fail2ban-<name> src -j <blocktype> > > When f2b restarts the necessary above configuration steps should occur > for iptables and ipset. > > This does NOT appear to be the case! I had to do both steps manually. > > At least not in the event of a restart, as opposed to a clean start. > > Did I miss something else? I don't think so because the actionban line > is clearly working. What does the fail2ban log contain? |
From: Charles B. <br...@br...> - 2013-09-17 01:23:47
|
Oops... I never thought to look. The log contains: > 2013-09-15 18:39:18,625 fail2ban.actions.action: ERROR ipset --create fail2ban-forum iphash That last shoud be "hash:ip" not "iphash" so the distribution iptables-ipset-proto4.conf file is buggy. Thanks On Tue, 2013-09-17 at 09:29 +1000, Daniel Black wrote: > > # ipset -V > > ipset v6.14, protocol version: 6 > > You have protocol 6 so you can use iptables-ipset-proto6.conf > > On 17/09/13 07:39, Charles Bradshaw wrote: > > Hello list I'm replying to my own post because there's VITAL missing > > step to get fail2ban and ipset working which I missed. (See VITAL below) > > > > The way I read iptables-ipset-proto4.conf ie this line: > > actionstart = ipset --create fail2ban-<name> iphash > > iptables -I INPUT -p <protocol> -m multiport --dports > > <port> -m set --match-set fail2ban-<name> src -j <blocktype> > > > > When f2b restarts the necessary above configuration steps should occur > > for iptables and ipset. > > > > This does NOT appear to be the case! I had to do both steps manually. > > > > At least not in the event of a restart, as opposed to a clean start. > > > > Did I miss something else? I don't think so because the actionban line > > is clearly working. > > What does the fail2ban log contain? > > > ------------------------------------------------------------------------------ > LIMITED TIME SALE - Full Year of Microsoft Training For Just $49.99! > 1,500+ hours of tutorials including VisualStudio 2012, Windows 8, SharePoint > 2013, SQL 2012, MVC 4, more. BEST VALUE: New Multi-Library Power Pack includes > Mobile, Cloud, Java, and UX Design. Lowest price ever! Ends 9/20/13. > http://pubads.g.doubleclick.net/gampad/clk?id=58041151&iu=/4140/ostg.clktrk > _______________________________________________ > Fail2ban-users mailing list > Fai...@li... > https://lists.sourceforge.net/lists/listinfo/fail2ban-users |
From: Daniel B. <dan...@in...> - 2013-09-17 13:29:59
|
On 17/09/13 11:23, Charles Bradshaw wrote: > Oops... I never thought to look. > > The log contains: >> 2013-09-15 18:39:18,625 fail2ban.actions.action: ERROR ipset --create > fail2ban-forum iphash > > That last shoud be "hash:ip" not "iphash" so the distribution > iptables-ipset-proto4.conf file is buggy. I'm fairly sure this is correct for protocol version 4. http://manpages.ubuntu.com/manpages/natty/man8/ipset.8.html As mentioned before you have protocol version 6 so the iptables-ipset-proto6 is the action you need. > > Thanks > > On Tue, 2013-09-17 at 09:29 +1000, Daniel Black wrote: >>> # ipset -V >>> ipset v6.14, protocol version: 6 >> >> You have protocol 6 so you can use iptables-ipset-proto6.conf |
From: Bob C. <bo...@mo...> - 2013-09-17 14:43:15
|
Hi Folks, I periodically get these dictionary attacks on my dovecot mail server: dovecot: auth(default): pam(account@MYSERVERNAME.com,200.76.17.206): pam_authenticate() failed: User not known to the underlying authentication module: 2 Time(s) My regex does not seem to catch these pesky intrusions: [Definition] failregex = dovecot.*auth\(default\): pam\(.*,<HOST>\): pam_authenticate\(\) failed: ignoreregex = Suggestions? Bob Cohen Writer, Internet Consultant, Teacher w: bobjcohen.com t: #itsabobworld |
From: Daniel B. <dan...@in...> - 2013-09-18 22:28:55
|
On 18/09/13 00:20, Bob Cohen wrote: > Hi Folks, > > I periodically get these dictionary attacks on my dovecot mail server: > > dovecot: auth(default): pam(account@MYSERVERNAME.com,200.76.17.206): pam_authenticate() failed: User not known to the underlying authentication module: 2 Time(s) > > My regex does not seem to catch these pesky intrusions: > > [Definition] > failregex = dovecot.*auth\(default\): pam\(.*,<HOST>\): pam_authenticate\(\) failed: > ignoreregex = > > Suggestions? I've redone the regex in accordance with recommended practice in in the DEVELOP documentation. https://github.com/grooverdan/fail2ban/commit/89e0520675ad822ab6935ade97554e8fd338e2c4 Does this match all of the entries you need to match? If it doesn't can you provide some more log examples. > > > Bob Cohen > Writer, Internet Consultant, Teacher > w: bobjcohen.com > t: #itsabobworld > > > ------------------------------------------------------------------------------ > LIMITED TIME SALE - Full Year of Microsoft Training For Just $49.99! > 1,500+ hours of tutorials including VisualStudio 2012, Windows 8, SharePoint > 2013, SQL 2012, MVC 4, more. BEST VALUE: New Multi-Library Power Pack includes > Mobile, Cloud, Java, and UX Design. Lowest price ever! Ends 9/20/13. > http://pubads.g.doubleclick.net/gampad/clk?id=58041151&iu=/4140/ostg.clktrk > _______________________________________________ > Fail2ban-users mailing list > Fai...@li... > https://lists.sourceforge.net/lists/listinfo/fail2ban-users > > |
From: Daniel B. <dan...@in...> - 2013-09-25 23:18:35
|
On 19/09/13 08:28, Daniel Black wrote: > On 18/09/13 00:20, Bob Cohen wrote: >> Hi Folks, >> >> I periodically get these dictionary attacks on my dovecot mail server: >> >> dovecot: auth(default): pam(account@MYSERVERNAME.com,200.76.17.206): pam_authenticate() failed: User not known to the underlying authentication module: 2 Time(s) >> >> My regex does not seem to catch these pesky intrusions: >> >> [Definition] >> failregex = dovecot.*auth\(default\): pam\(.*,<HOST>\): pam_authenticate\(\) failed: >> ignoreregex = >> >> Suggestions? > > I've redone the regex in accordance with recommended practice in in the > DEVELOP documentation. > > https://github.com/grooverdan/fail2ban/commit/89e0520675ad822ab6935ade97554e8fd338e2c4 > > Does this match all of the entries you need to match? > > If it doesn't can you provide some more log examples. > Bob, Any feedback on the above? |