Haha I didn't realise the f2b mailing list was configured to send replies
to the message sender and not to the mailing list; sorry! :)
When I talked about updating the ignore list, I meant a script that would
allow me to store false positives for future use, to feed into
fail2ban-client. I've noticed that fail2ban-client (which I already use to
addignoreip) allows multiple entries of the same IP address, which seems to
poor use of resources. So the idea was, if certain IP addresses have been
falsely marked and might be banned, I can store them in a file and run a
script to occasionally compare them against the outout of *fail2ban-client
get [jail] ignoreip* - any that aren't already ignored can then be added
using fail2ban-client. It seemed like a good way to make sure I didn't have
duplicate entries in fail2ban-client.
The issue is partly that there are 3 other people who manage the website,
one of whom has dyspraxia - he does have serious problems mistyping
passwords because he can't see the right keys on the keyboard, even when he
types slowly. Unfortunately, WordPress doesn't seem to allow non password
logins. My SSH logins are done through keys; password access is turned off.
So you can see, there is actually a need to have a fresh ignoreip list -
this person works from a number of different sites and, due to the nature
of his work, cannot stay logged in to websites and thus logs in to the
back-end several times a day. His dyspraxia means he keeps getting it wrong
and getting blocked; my (bad) solution was to use sed to remove any entries
from the log files.
Additionally, the dyspraxia-suffering admin also tends to tap "spam"
instead of "moderate" on troll (but not spam) comments when he's using a
mobile device; thus, his behaviour can create false positives - but this is
not something I can change easily - I am gradually convincing him not to
moderate comments, but our site, which is a political/cultural site, is
very busy and posts frequently get over 500 comments. It needs multiple
admins, and the 4 of us are the people who run the website. But it's
something I'm actively looking at, and is beyond the scope of this mailing
Instead, I'm starting to re-think how I control access. I now have to look
at alternative strategies, which takes us beyond the realm of f2b. I've
tried methods of training the admins, but the combination of dyspraxia,
lazy tapping on mobile devices, and some actual technological illiteracy
means that it's proved almost impossible to change the behaviour of the
You can sort of see why I ended up using sed to change the log files; I'm
so grateful for your post, because you stopped me in my tracks and reminded
me that you don't solve a problem in the way I'd been trying (and failing)
- you have to come at it from different angles.
There's a third problem which you identified, which i'm already aware of:
Lowering security by whitelisting bunches of IP addresses. Unfortunately,
the key admin - the one who can't type properly - uses the BT network, one
of the biggest in the UK. The IP address allocation isn't repeated
frequently, but it *is* repeated sometimes, so he has ended up logging in
from a large range of IP addresses - and logging in incorrectly. He has
frequently been blocked by f2b because, like most users, he's always
convinced he's typed the password correctly so he keeps pressing "submit";
even when he doesn't, he's frequently logged in incorrectly 5-10 times a
day. Thus, if I have this nice big list of ignored IP addresses, I'm
seriously lowering security, because other people are being assigned these
As a result of your advice, I am rethinking how I manage this. I've already
lowered my find/bantime for spam logging, which will help a bit with the
false positives (but not completely - by accidentally marking a legitimate
post as spam, the WordPress spam plugin Akismet can learn to mark others by
that person as spam, and before I know it, they've been blocked because
their next 3 comments have been automatically marked as spam. There's a
clear issue here that I need to resolve - I can't stop the logging of
failed WordPress login attempts; I can't convert WordPress into a different
type of login management; the guys who run the site are really dedicated
writers and contributors, so it's not like I can get rid of them - they're
the heart of the site. Training helps a bit, but I've gone as far as I can
to modify user behaviour.
Again, thank you for your insight into this issue. You can hopefully see
that I'm trying to teach myself as I go along, and your responses have been
really helpful. I know that my response goes beyond f2b support, but I
wanted to clarify why I have done certain things - we've got a fairly
interesting issue in that that guy who started the website has perception
problems that directly cause people to be banned as spammers, and causes
himself to be banned as a hacker :-)
On 17 March 2013 09:14, Tom Hendrikx <tom@...> wrote:
> [ please keep replies on-list ]
> On 16-03-13 18:14, Tony Collins wrote:
> > Thanks for such a thoughtful response Tom.
> > I'm sort of dimly aware that I wasn't doing it right - the truth is, I'm
> > experimenting. I run a dedicated web server for one busy website, so it's
> > kinda amateur.
> > Actually, your response has surprised me :-) - I was hoping someone would
> > accept that there's a bug in f2b, but actually, from your logic, it's a
> > user behaviour (me!) problem, and that's where the change needs to be
> > I need to create a different level of spam logging - there are reasons
> > we have a 3 month window that I can't really go into, but you've
> > me enough to make me realise that I'm the one with the bug. I need to
> > rethink the 3 month issue for sure.
> > I'm going to hunt for a script that allows me to have a text file with a
> > list of ignore IPs that can be updated - for example, I frequently log in
> > from different IP addresses, and I'd like f2b to be updated with a new
> > addignoreip whenever I log in; I think perhaps this is what I need for my
> > false positives - a dynamic add/delignoreip, rather than forcing f2b to
> > keep re-reading log files.
> fail2ban can update ignore lists dynamic, read 'man fail2ban-client',
> and be surprised.
> I don't understand why you are so interested in the ignorelist though:
> you are a valid user and if you behave normally (while logging in as in
> your new example) you'll never give f2b reason to blacklist you. If you
> have a typing disorder (doesn't show in your mails :>) that makes you
> regularly mistype passwords, you should consider using ssh keys or an
> equivalent for your way of logging in, that lowers your error rate. In
> stead you are merely interested in dropping your security by
> white-listing a whole bunch of ip addresses...
> > Thank you for what has turned out to be an interesting lesson!
> > Tony Collins
> > On 16 March 2013 15:29, Tom Hendrikx <tom@...> wrote:
> >> On 15-03-13 19:10, Tony Collins wrote:
> >>> H
> >>> i - I tried sending this but it got held in the queue and "rejected by
> >>> moderator"
> >>> I
> >>> 've massively cut down the log file sample, so it might now be
> >> -
> >>> please let me know if you need more and if you can help. Thank you :-)
> >>> Hi everyone.
> >>> I've been trying to get to the bottom of this for so long. On Centos
> >>> and f2b 0.8.8 from git.
> >>> I have a "spam-log" jail. If my WordPress blog marks someone as spam
> >> twice
> >>> within 3 months, they get banned.
> >>> That all works fine - except if I ever do anything to the spam-log
> >>> As soon as I touch it (let's say, I see a false positive and want to
> >> remove
> >>> it from the spam log), fail2ban starts going through the whole file
> >> again,
> >>> and bans EVERYONE. I get the notification that says ".... after 2
> >> attempts
> >>> against spam-log", but its grep of the log correctly shows just one
> >> attempt.
> >> There you go: it's a log file. You shouldn't edit it. This causes
> >> multiple issues for f2b, as documented in fail2ban.log:
> >> 2013-03-11 10:30:15,849 fail2ban.filter : DEBUG
> >> /PATH/wp-content/plugins/spam-log/spam.log has been modified
> >> 2013-03-11 10:30:15,849 fail2ban.filter : INFO Log rotation detected
> >> for /PATH/wp-content/plugins/spam-log/spam.log
> >> Issue 1: f2b reads file contents as they get added to it and remembers
> >> entries in them by itself. If you remove an entry, you are already too
> >> late: f2b already recorded it.
> >> Issue 2: if the filesize changes because you edited it, f2b needs to
> >> what happened. If the file grew larger, data was appended to the file
> >> and f2b reads the appended data. If the file got smaller, the file must
> >> be rotated, so f2b reads from the start, and counts all occurring
> >> entries as new (but actually duplicates), resulting in unwanted bans.
> >> Generally speaking of logfiles: you removed data that might be
> >> interesting later, for reasons you currently don't know: the number zero
> >> reason to not edit log files. Sysadmin rule of thumb: Don't. Ever Edit.
> >> Log. Files. Period. :)
> >>> This happens with every jail I use. Well, every jail I have made
> >> manually.
> >> Probably not due to the jails you make, but with the log files you
> >>> It's a problem, cos I have a script that I wrote that seds the file and
> >>> removes false positives - if I run the script, that's when everyone
> >>> banned.
> >> You need a different way of handling false positives. I suggest that you
> >> try one of:
> >> - extract false positives, and add them to f2b ignore list for that
> >> specific jail.
> >> - raise your threshold: a real comment spammer will try to find a way
> >> around your antispam stuff (captcha etc), and then load your system with
> >> multiple links to casino/porn/whatever sites to increase pagerank.
> >> Banning on 3 spammy comments within 24 hours would be manageable.
> >> A 3 month findtime is a bad idea any way:
> >> - you need to keep a huge logfile that goes back at least 3 months,
> >> because it needs to be reread when you restart f2b.
> >> - f2b needs to keep all the entries from that log file in memory,
> >> resulting in huge memory use: in fail2ban.log you are seeing the amounts
> >> of ip addresses that f2b is keeping track of.
> >> If the spam is so slow that a 3 month findtime for 2 hits is really
> >> needed to be effective, then you need a different hammer than f2b to hit
> >> this nail.
> >> --
> >> Tom
> >> Everyone hates slow websites. So do we.
> >> Make your web apps faster with AppDynamics
> >> Download AppDynamics Lite for free today:
> >> http://p.sf.net/sfu/appdyn_d2d_mar
> >> _______________________________________________
> >> Fail2ban-users mailing list
> >> Fail2ban-users@...
> >> https://lists.sourceforge.net/lists/listinfo/fail2ban-users
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> Fail2ban-users mailing list