From: Neil D. <ne...@da...> - 2013-01-25 13:14:08
|
Hi, I have fail2ban-0.8.8 installed on Archlinux, configured for Shorewall and banning IPs very well. I then installed the sysstat package and fail2ban let through a two day password SSH brute-force attack from a single IP. Removing the sysstat package and restarting fail2ban resulted in an immediate ban of the offending IP. Definitely a fail2ban and sysstat interaction issue. Here is the startup log for fail2ban with sysstat installed. Note the ERROR message relating to /var/log/sa which is the data directory sysstat uses to store its performance data. Under this scenario, once fail2ban is started it fails to Ban IPs. 2013-01-23 22:09:34,632 fail2ban.server : INFO Changed logging target to /var/log/fail2ban.log for Fail2ban v0.8.8 2013-01-23 22:09:34,649 fail2ban.jail : INFO Creating new jail 'sshd' 2013-01-23 22:09:34,875 fail2ban.jail : INFO Jail 'sshd' uses pyinotify 2013-01-23 22:09:35,207 fail2ban.jail : INFO Initiated 'pyinotify' backend 2013-01-23 22:09:35,237 fail2ban.filter : INFO Added logfile = /var/log/auth.log 2013-01-23 22:09:35,240 fail2ban.filter : INFO Set maxRetry = 3 2013-01-23 22:09:35,243 fail2ban.filter : INFO Set findtime = 600 2013-01-23 22:09:35,245 fail2ban.actions: INFO Set banTime = 10800 2013-01-23 22:09:35,389 fail2ban.jail : INFO Creating new jail 'sshd-ddos' 2013-01-23 22:09:35,394 fail2ban.jail : INFO Jail 'sshd-ddos' uses pyinotify 2013-01-23 22:09:35,419 fail2ban.jail : INFO Initiated 'pyinotify' backend 2013-01-23 22:09:35,425 fail2ban.filter : INFO Added logfile = /var/log/auth.log 2013-01-23 22:09:35,428 fail2ban.filter : INFO Set maxRetry = 3 2013-01-23 22:09:35,431 fail2ban.filter : INFO Set findtime = 600 2013-01-23 22:09:35,432 fail2ban.actions: INFO Set banTime = 10800 2013-01-23 22:09:35,452 fail2ban.jail : INFO Jail 'sshd' started 2013-01-23 22:09:35,474 fail2ban.jail : INFO Jail 'sshd-ddos' started 2013-01-24 08:13:20,867 fail2ban.filter : ERROR Unable to get failures in /var/log/sa This is the startup logfile for fail2ban after removing sysstat and restarting fail2ban. Here we see no ERROR message and the offending IP is quickly banned. 2013-01-25 10:50:35,459 fail2ban.server : INFO Changed logging target to /var/log/fail2ban.log for Fail2ban v0.8.8 2013-01-25 10:50:35,460 fail2ban.jail : INFO Creating new jail 'sshd' 2013-01-25 10:50:35,607 fail2ban.jail : INFO Jail 'sshd' uses pyinotify 2013-01-25 10:50:35,696 fail2ban.jail : INFO Initiated 'pyinotify' backend 2013-01-25 10:50:35,700 fail2ban.filter : INFO Added logfile = /var/log/auth.log 2013-01-25 10:50:35,702 fail2ban.filter : INFO Set maxRetry = 3 2013-01-25 10:50:35,704 fail2ban.filter : INFO Set findtime = 600 2013-01-25 10:50:35,705 fail2ban.actions: INFO Set banTime = 10800 2013-01-25 10:50:35,797 fail2ban.jail : INFO Creating new jail 'sshd-ddos' 2013-01-25 10:50:35,797 fail2ban.jail : INFO Jail 'sshd-ddos' uses pyinotify 2013-01-25 10:50:35,816 fail2ban.jail : INFO Initiated 'pyinotify' backend 2013-01-25 10:50:35,819 fail2ban.filter : INFO Added logfile = /var/log/auth.log 2013-01-25 10:50:35,821 fail2ban.filter : INFO Set maxRetry = 3 2013-01-25 10:50:35,823 fail2ban.filter : INFO Set findtime = 600 2013-01-25 10:50:35,824 fail2ban.actions: INFO Set banTime = 10800 2013-01-25 10:50:35,833 fail2ban.jail : INFO Jail 'sshd' started 2013-01-25 10:50:35,838 fail2ban.jail : INFO Jail 'sshd-ddos' started 2013-01-25 10:51:42,113 fail2ban.actions: WARNING [sshd] Ban 211.115.71.171 My question is: Should fail2ban be attempting to examing files in /var/log/sa or have I uncovered a bug? Regards, Neil Darlow |
From: Fabian W. <fa...@we...> - 2013-01-25 14:32:04
|
Hello Neil On 25.01.2013 14:13, Neil Darlow wrote: > 2013-01-23 22:09:35,474 fail2ban.jail : INFO Jail 'sshd-ddos' started > 2013-01-24 08:13:20,867 fail2ban.filter : ERROR Unable to get failures in > /var/log/sa Check the logpath= option for your sshd-ddos jail, something seems to be wrong there. bye Fabian |
From: Neil D. <ne...@da...> - 2013-01-25 14:54:46
|
Hi Fabian, On Friday 25 January 2013 15:31:52 Fabian Wenk wrote: > Check the logpath= option for your sshd-ddos jail, something > seems to be wrong there. Thank you for responding. My jail.conf entries all have "enabled = false" so nothing to see there. My jail.local just contains: [DEFAULT] bantime = 10800 usedns = no [sshd] enabled = true filter = sshd action = shorewall logpath = /var/log/auth.log [sshd-ddos] enabled = true filter = sshd-ddos action = shorewall logpath = /var/log/auth.log Additionally, I did: cd /etc/fail2ban find . -type f -exec grep -Hi 'var/log/sa' '{}' ';' which produced no output. I can't see why fail2ban would want to go into /var/log/sa from my configuration, hence my question. Regards, Neil Darlow |
From: Yaroslav H. <li...@on...> - 2013-01-25 18:45:37
|
interesting! I wonder if things worked just fine with sysstat and some other backend than pyinotify (e.g. polling). NB I would have tried myself but on Debian systems sysstat uses /var/log/sysstat/ directory, not a file under /var/log . Since your problem might be somehow distribution specific, might have been worth first checking with fail2ban maintainer in arch, if there is such a person... also what is the output of fail2ban-client status sshd fail2ban-client status sshd-ddos is that /var/log/sa get's listed in any of those? On Fri, 25 Jan 2013, Neil Darlow wrote: > Hi Fabian, > On Friday 25 January 2013 15:31:52 Fabian Wenk wrote: > > Check the logpath= option for your sshd-ddos jail, something > > seems to be wrong there. > Thank you for responding. > My jail.conf entries all have "enabled = false" so nothing to see there. > My jail.local just contains: > [DEFAULT] > bantime = 10800 > usedns = no > [sshd] > enabled = true > filter = sshd > action = shorewall > logpath = /var/log/auth.log > [sshd-ddos] > enabled = true > filter = sshd-ddos > action = shorewall > logpath = /var/log/auth.log > Additionally, I did: > cd /etc/fail2ban > find . -type f -exec grep -Hi 'var/log/sa' '{}' ';' > which produced no output. > I can't see why fail2ban would want to go into /var/log/sa from my > configuration, hence my question. -- Yaroslav O. Halchenko Postdoctoral Fellow, Department of Psychological and Brain Sciences Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik |
From: Neil D. <ne...@da...> - 2013-01-25 19:26:43
|
Hi Yaroslav, On Friday 25 January 2013 13:45:26 Yaroslav Halchenko wrote: > interesting! > > I wonder if things worked just fine with sysstat and some other backend > than pyinotify (e.g. polling). I could try that but I would rather debug a failing configuration. > NB I would have tried myself but on Debian systems sysstat uses > /var/log/sysstat/ directory, not a file under /var/log . Perhaps I was not clear. sysstat on Arch places its performance data files in /var/log/sa. It is a directory not a file. > Since your problem might be somehow distribution specific, might > have been worth first checking with fail2ban maintainer in arch, if > there is such a person... I might incur the wrath of Arch if I do. The philosophy of bug reporting in Arch is to first determine that it is not an upstream bug before reporting against Arch. Regards, Neil Darlow |
From: Yaroslav H. <li...@on...> - 2013-01-25 20:46:19
|
On Fri, 25 Jan 2013, Neil Darlow wrote: > > interesting! > > I wonder if things worked just fine with sysstat and some other backend > > than pyinotify (e.g. polling). > I could try that but I would rather debug a failing configuration. do not worry -- as soon as you verify that that is working/or not not, we could revert/continue with failing configuration ;-) also, please clone (if you haven't yet) original fail2ban sources and run ./fail2ban-testcases on your box please -- any failures? > > NB I would have tried myself but on Debian systems sysstat uses > > /var/log/sysstat/ directory, not a file under /var/log . > Perhaps I was not clear. sysstat on Arch places its performance data files in > /var/log/sa. It is a directory not a file. aaah... "to /var/log/sa which is the data directory " -- I should have read more carefully indeed > > Since your problem might be somehow distribution specific, might > > have been worth first checking with fail2ban maintainer in arch, if > > there is such a person... > I might incur the wrath of Arch if I do. The philosophy of bug reporting in > Arch is to first determine that it is not an upstream bug before reporting > against Arch. interesting philosophy -- so users must know intricacies of particular tools to drive the judgement... or to buzz upstream developers first to make them troubleshoot the problems on systems they do not have? ;) on my system I have created /var/log/sa, removed all permissions from it, made owned by user... so far fail2ban doesn't complain. Also looking at your log: 2013-01-23 22:09:35,474 fail2ban.jail : INFO Jail 'sshd-ddos' started 2013-01-24 08:13:20,867 fail2ban.filter : ERROR Unable to get failures in /var/log/sa so it took it good 10 hours to get to the ERROR from the start... do you have any other interesting events at that point in time ? (some CRON jobs running, rotating things etc)? -- Yaroslav O. Halchenko Postdoctoral Fellow, Department of Psychological and Brain Sciences Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik |
From: Yaroslav H. <li...@on...> - 2013-01-25 20:52:33
|
aha -- managed to reproduce -- I have removed and recreated that bloody directory and got: 2013-01-25 15:47:05,201 fail2ban.filter : DEBUG Callback for Event: <Event dir=True mask=0x40000100 maskname=IN_CREATE|IN_ISDIR name=sa path=/var/log pathname=/var/log/sa wd=1 > 2013-01-25 15:47:05,201 fail2ban.filter : ERROR Unable to get failures in /var/log/sa we should fix above (eventually) but then fail2ban seems to proceed functioning properly (even banned myself). so for your further troubleshooting (probably no need to try different backend since I confirmed that it does happen with pyinotify and now I really doubt it would happen with anything else) -- follow http://www.fail2ban.org/wiki/index.php/HOWTO_Seek_Help item 3.4: Relevant parts of /var/log/fail2ban.log file, preferably obtained while running fail2ban with loglevel = 4 (set it in [Definition] section of /etc/fail2ban/fail2ban.conf or /etc/fail2ban/fail2ban.local). Send lengthy output (or with long lines) as an attachment (see next one) Cheers On Fri, 25 Jan 2013, Yaroslav Halchenko wrote: > On Fri, 25 Jan 2013, Neil Darlow wrote: > > > interesting! > > > I wonder if things worked just fine with sysstat and some other backend > > > than pyinotify (e.g. polling). > > I could try that but I would rather debug a failing configuration. > do not worry -- as soon as you verify that that is working/or not not, we > could revert/continue with failing configuration ;-) > also, please clone (if you haven't yet) original fail2ban sources and run > ./fail2ban-testcases > on your box please -- any failures? > > > NB I would have tried myself but on Debian systems sysstat uses > > > /var/log/sysstat/ directory, not a file under /var/log . > > Perhaps I was not clear. sysstat on Arch places its performance data files in > > /var/log/sa. It is a directory not a file. > aaah... "to /var/log/sa which is the data directory " -- I should have > read more carefully indeed > > > Since your problem might be somehow distribution specific, might > > > have been worth first checking with fail2ban maintainer in arch, if > > > there is such a person... > > I might incur the wrath of Arch if I do. The philosophy of bug reporting in > > Arch is to first determine that it is not an upstream bug before reporting > > against Arch. > interesting philosophy -- so users must know intricacies of particular > tools to drive the judgement... or to buzz upstream developers first to > make them troubleshoot the problems on systems they do not have? > ;) > on my system I have created /var/log/sa, removed all permissions from > it, made owned by user... so far fail2ban doesn't complain. Also > looking at your log: > 2013-01-23 22:09:35,474 fail2ban.jail : INFO Jail 'sshd-ddos' started > 2013-01-24 08:13:20,867 fail2ban.filter : ERROR Unable to get failures in /var/log/sa > so it took it good 10 hours to get to the ERROR from the start... do you have > any other interesting events at that point in time ? (some CRON jobs running, > rotating things etc)? -- Yaroslav O. Halchenko Postdoctoral Fellow, Department of Psychological and Brain Sciences Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik |
From: Yaroslav H. <li...@on...> - 2013-01-25 21:03:36
|
could you please check if this would fix it altogether for you? http://github.com/yarikoptic/fail2ban/commit/4af83d9dc4ed02b78385bf70b945e6acc7c0ed83 I haven't tested it yet but I guess this could be the spot On Fri, 25 Jan 2013, Yaroslav Halchenko wrote: > aha -- managed to reproduce -- I have removed and recreated that bloody > directory and got: > 2013-01-25 15:47:05,201 fail2ban.filter : DEBUG Callback for Event: <Event dir=True mask=0x40000100 maskname=IN_CREATE|IN_ISDIR name=sa path=/var/log pathname=/var/log/sa wd=1 > > 2013-01-25 15:47:05,201 fail2ban.filter : ERROR Unable to get failures in /var/log/sa > we should fix above (eventually) but then fail2ban seems to proceed > functioning properly (even banned myself). > so for your further troubleshooting (probably no need to try different backend > since I confirmed that it does happen with pyinotify and now I really doubt it > would happen with anything else) -- follow > http://www.fail2ban.org/wiki/index.php/HOWTO_Seek_Help > item 3.4: > Relevant parts of /var/log/fail2ban.log file, preferably obtained while running fail2ban with loglevel = 4 > (set it in [Definition] section of /etc/fail2ban/fail2ban.conf or /etc/fail2ban/fail2ban.local). > Send lengthy output (or with long lines) as an attachment (see next one) > Cheers > On Fri, 25 Jan 2013, Yaroslav Halchenko wrote: > > On Fri, 25 Jan 2013, Neil Darlow wrote: > > > > interesting! > > > > I wonder if things worked just fine with sysstat and some other backend > > > > than pyinotify (e.g. polling). > > > I could try that but I would rather debug a failing configuration. > > do not worry -- as soon as you verify that that is working/or not not, we > > could revert/continue with failing configuration ;-) > > also, please clone (if you haven't yet) original fail2ban sources and run > > ./fail2ban-testcases > > on your box please -- any failures? > > > > NB I would have tried myself but on Debian systems sysstat uses > > > > /var/log/sysstat/ directory, not a file under /var/log . > > > Perhaps I was not clear. sysstat on Arch places its performance data files in > > > /var/log/sa. It is a directory not a file. > > aaah... "to /var/log/sa which is the data directory " -- I should have > > read more carefully indeed > > > > Since your problem might be somehow distribution specific, might > > > > have been worth first checking with fail2ban maintainer in arch, if > > > > there is such a person... > > > I might incur the wrath of Arch if I do. The philosophy of bug reporting in > > > Arch is to first determine that it is not an upstream bug before reporting > > > against Arch. > > interesting philosophy -- so users must know intricacies of particular > > tools to drive the judgement... or to buzz upstream developers first to > > make them troubleshoot the problems on systems they do not have? > > ;) > > on my system I have created /var/log/sa, removed all permissions from > > it, made owned by user... so far fail2ban doesn't complain. Also > > looking at your log: > > 2013-01-23 22:09:35,474 fail2ban.jail : INFO Jail 'sshd-ddos' started > > 2013-01-24 08:13:20,867 fail2ban.filter : ERROR Unable to get failures in /var/log/sa > > so it took it good 10 hours to get to the ERROR from the start... do you have > > any other interesting events at that point in time ? (some CRON jobs running, > > rotating things etc)? -- Yaroslav O. Halchenko Postdoctoral Fellow, Department of Psychological and Brain Sciences Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik |
From: Yaroslav H. <li...@on...> - 2013-01-25 21:11:17
|
nah -- that one is wrong (result of having no unittests for this yet) -- better this one http://github.com/yarikoptic/fail2ban/commit/6b2e76ba7fce1841411183a72a2c75b6826c4b4c ;-) On Fri, 25 Jan 2013, Yaroslav Halchenko wrote: > could you please check if this would fix it altogether for you? > http://github.com/yarikoptic/fail2ban/commit/4af83d9dc4ed02b78385bf70b945e6acc7c0ed83 > I haven't tested it yet but I guess this could be the spot -- Yaroslav O. Halchenko Postdoctoral Fellow, Department of Psychological and Brain Sciences Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik |
From: Neil D. <ne...@da...> - 2013-01-25 21:31:40
|
Hi Yaroslav, On Friday 25 January 2013 16:11:10 Yaroslav Halchenko wrote: > nah -- that one is wrong (result of having no unittests for this yet) -- > better this one > http://github.com/yarikoptic/fail2ban/commit/6b2e76ba7fce1841411183a72a2c75 > b6826c4b4c ;-) I've patched that change into my install and will see how it goes. Thank you for your interest. Regards, Neil Darlow |
From: Neil D. <ne...@da...> - 2013-01-26 08:55:03
|
Hi Yaroslav, On Friday 25 January 2013 21:31:28 Neil Darlow wrote: > I've patched that change into my install and will see how it goes. The patch works fine, I banned/unbanned myself, and one other, overnight. Am I right in thinking that this affects *any* directory in /var/log and not just /var/log/sa? I have other directories in /var/log (journal and old) so I am not sure why sa was hit first (unless inotify traverses items in newest-first order). So, if this problem is more generic than just sysstat and does cause fail2ban to stop banning IPs (which was my experience) then it is potentially quite serious - especially with backend=auto being the default, which will choose pyinotify over other backends if it is available. Not that I want to frighten people but it might be a justification for a 0.8.9 release? Regards, Neil Darlow |
From: Yaroslav H. <li...@on...> - 2013-01-26 17:43:50
|
On Sat, 26 Jan 2013, Neil Darlow wrote: > On Friday 25 January 2013 21:31:28 Neil Darlow wrote: > > I've patched that change into my install and will see how it goes. > The patch works fine, I banned/unbanned myself, and one other, overnight. good... so did you check logs on what was happening around that point before? (e.g. some cron job, logrotation) > Am I right in thinking that this affects *any* directory in /var/log and not > just /var/log/sa? that would be my guess too ;-) > I have other directories in /var/log (journal and old) so I am not sure why sa > was hit first (unless inotify traverses items in newest-first order). since we do not exactly which events lead to this, nor anyone else complained by now, it might be that sysstats do something specific/interesting to it (e.g. during logrotation) which others do not do (could it be possibly removed/recreated - or moved away, new one created, and then old files moved over??) > So, if this problem is more generic than just sysstat and does cause fail2ban > to stop banning IPs (which was my experience) then it is potentially quite > serious - especially with backend=auto being the default, which will choose > pyinotify over other backends if it is available. > Not that I want to frighten people but it might be a justification for a 0.8.9 > release? ;-) yeah -- having 0.8.9 would be good... as for seriousness -- you were the first to report it... I haven't observed it myself although have been using pyinotify backend for a while... so not sure if it is not really Arch-specific in how /sa is handled I would love to address https://github.com/fail2ban/fail2ban/issues?milestone=4&page=1&state=open before kicking 0.8.9 out. Since this patch worked for you nice -- I will just push it to the main repository... hopefully adding some more unittests later on (i.e. before the release) -- your contributions would be welcome as well ;-) -- Yaroslav O. Halchenko Postdoctoral Fellow, Department of Psychological and Brain Sciences Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik |
From: Neil D. <ne...@da...> - 2013-01-26 20:24:24
|
Hi Yaroslav, On Saturday 26 January 2013 12:43:44 Yaroslav Halchenko wrote: > good... so did you check logs on what was happening around that point > before? (e.g. some cron job, logrotation) That is in progress now. So far nothing untoward has happened. I hope this is not a case of Schrodinger's Cat :) > since we do not exactly which events lead to this, nor anyone else > complained by now, it might be that sysstats do something > specific/interesting to it (e.g. during logrotation) which others do not > do (could it be possibly removed/recreated - or moved away, new one > created, and then old files moved over??) I have reverted your patch and started with a clean sysstat install so it will take a little time for sa log file rotation to occur. > Since this patch worked for you nice -- I will just push it to the main > repository... hopefully adding some more unittests later on (i.e. before > the release) -- your contributions would be welcome as well ;-) It is one of those patches that just "Seems logical (TM)" to apply in that you would not want to consider directories for processing as log files. Regards, Neil Darlow |
From: Neil D. <ne...@da...> - 2013-01-28 08:59:51
|
Hi Yaroslav, On Saturday 26 January 2013 12:43:44 Yaroslav Halchenko wrote: > > Am I right in thinking that this affects *any* directory in /var/log and > > not just /var/log/sa? > > that would be my guess too ;-) I have confirmed this on another system, unfortunately without additional logging. This system reported that it could not get failures from /var/log/old which is an empty directory and sysstat is not installed. An additional fact was that there was a ban for an IP in effect. Some breakage in the latest Arch samba package meant that authentication wasn't working on this system and my user was banned by fail2ban. This might explain why I haven't seen any errors on my primary system due to no bans being currently active. Interesting indeed. Regards, Neil Darlow |