Fluffy isn't doing its job - Part 2 - Blacklists
Status: Beta
Brought to you by:
waynemcdougall
Menu
▾
▴
#115 Fluffy isn't doing its job - Part 2 - Blacklists
Okay, after I saw your file and how to use the DNSBLs.
I managed to get it working with some of the dnsbls
out there. However, you mentioned that its suppose to
change the weights or something, well the weights don't
change. But in any case, I managed to get it to look
at some dnsbls. The next problem is I have blacklisted
some IPs out there, and for the most part, it works.
But there is always this one IP range that I have
blacklisted and it still seems to be able to get
through. I have deleted the DNS.txt file, the
defer.txt file just to get it to reset. But it still
gets through. Any idea why?
I also have a question about the order at which fluffy
does its checks on email. It seems that the order that
it goes through is something to the effect of DEFER,
DNSBL, BLACKLIST, and whatever other checks you have
that I'm not using. Question, shouldn't it have defer
on the end as the last thing it checks. So that if a
site doesn't show up anywhere the last thing it does is
defer, otherwise if the site is blacklisted (either by
file or dns) it gets denied right away and doesn't get
to the deferred state. This is just something I think
I might have seen and it's just a suggestion.
Dan
Logged In: YES
user_id=744036
I've been thinking along the same lines... My pref would be...
check whitelist ... in there? skip everything, allow
check blacklist .... in there? reject immediately
check spam trap... in there? tar pit
check external blacklist servers.. in there? action according to
weighting scores, if not... defer
Logged In: YES
user_id=660239
1. Fluffy isn't supposed to change the weights. You can
change the weights. You can give a weight to each DNSBL
listing report. The total of all the weights (scores) where an
IP address is listed on a DNSBL is used by Fluffy to determine
how to handle the message. So for example, I will block
anything with a total score of 10 or more. Some of my DNSBL
are weighted at 10 - if the IP is listed on the DNSBL it gets
blocked. Some of my DNSBL are weighted at 4. An IP would
have to be listed on 3 different DNSBL weighted at 4 to be
blocked - but I do get a warning that the mail may be spam.
And some of my DNSBL are weighted at 0 - I don't use them.
2. You say you have blacklisted an IP address but it still gets
through. Possible explanations: you have not blacklisted it
correctly or you have also whitelisted it. If you send me the
file list.txt (containing the white/blacklist) and your log file
(extract) for a day when an IP address gets through and
advise me of the IP address in question, I'm sure I can help.
You can email me fluffy@codeworks.gen.nz
3. The order of processing is
a) blacklist/whitelist - any specific local instructions on
handling are followed first
b) defer
c) DNSBL
d) spamtraps/header content
This order is by design. No defer shouldn't be last. Rememebr
that looking up DNSBL takes both time and bandwidth (and
puts a load on the DNSBL server). If a site is listed on a
DNSBL server it is treated almost exactly the same as if it is
blacklisted. The only differwenc eis that instead of sending a
550 response code (permanent failure - go away) we send a
450 response code (temporary failure - try again later).
One big advantage is that a lot of spam doesn't try again. So
on my system, over a 24 hour period I had:
How many emails were deferred and not later received: 557
How many emails were rejected: 3224
That's 557 emails that were never retried and that I didn't
have to wait for a DNSBL lookup or use the bandwidth. I just
got away with deferring them.
If I was to lookup and then defer, I'd have to repeat that
every time they retried. So at a minimum I'd be doing two
DNSBL lookups - and often more as they keep retrying in the
defer timeout period.
But that's just an added bonus. I also defer first because
there's an excellent chance that as part of the defer cycle I'll
find that they are sending to a spam trap address, so then
they get (reliably) blocked without a DNSBL lookup. And in the
future when I allow whitelisting by recipient (or
sender/recipient pairs) I want to do that bfore an DNSBL
lookup.
In summary DNSBL lookup is done at the last possble
opportunity because there are better, faster local choices to
work through first.
4. I think this addresses rontouw's comments except I'd add:
a) remember that DNSBL lists take time to respond - so the
later we query, the more likely fresh spam will be listed. I find
listing chances goes up by about 50% by waiting 15 minutes,
and 90% by waiting 30 minutes.
b) you may misunderstand and misuse star pit
i) when a spam trap is triggered the site is blacklisted
ii) a tar pit is something that slows you down. According to
Internet standards a system should wait 5 minutes for a
response after specifying a mail recipient. Custom spam
software doesn't like to wait that long and will usually give up
after 30 seconds. Fluffy on the other hand needs time to look
up DNSBL servers and domain names and that often takes
longer than 30 seconds. So some spammers give up in that
time. Good. The logging of tar pitting simply notes that has
happened as a by-product of normal Fluffy operations. I am
considering adding deliberate tar pitting as a feature though,
seeing how effective it is at stopping spammers.
I still ahve an open mind on this. Comments are welcome.
Logged In: YES
user_id=744036
I agree Tar pitting / deferral really DOES work. I find that
spammers hardly EVER try again. A very nice feature!
Especially true for those attempting to send THROUGH us. (I.e.
thinking we are an open relay?)
But genuine ones can get stuck in an infinite loop. I have no
idea why. When I see it happen, I add them to my whitelist.
Next time they visit a few minutes later, they come straight in.
I have had to manually traipse through the logs to find them.
Thus my wish for a deferral then if x defers have occurrred,
allow them in but pass to another email account. Perhaps if
loop counter defaults to 0 it means leave it how it is now?
That way existing users who like the way it is, see no change.
Anyone else wish to discuss?
Logged In: NO
Alright, I can see why you have the DNSBL last, however, it
appears that the order is Defer, blacklists, then dnsbl. I
don't know all of what fluffy does but according to what
you've said it does, shouldn't it be whitelists, blacklists,
header blocks (sender and whatever), defer, then dnsbl? It
seems that it whitelists, defers, blacklists, and then
dnsbl. I don't use the header stuff so I don't know where
that is but it seems like there are a few problems with it
following its procedure.
On that note, I have included my log file and my list.txt to
try to find out what is going on. The IP that is getting
through in this case is 216.108.233.48 and earlier today, I
spotted another IP that is doing the same thing. I don't
edit these files directly but use the Fluffy Trainer so if
there is a syntatical error, its in the gui.
Thanks and good luck.
Dan
Logged In: YES
user_id=830967
Alright, I can see why you have the DNSBL last, however, it
appears that the order is Defer, blacklists, then dnsbl. I
don't know all of what fluffy does but according to what
you've said it does, shouldn't it be whitelists, blacklists,
header blocks (sender and whatever), defer, then dnsbl? It
seems that it whitelists, defers, blacklists, and then
dnsbl. I don't use the header stuff so I don't know where
that is but it seems like there are a few problems with it
following its procedure.
On that note, I have included my log file and my list.txt to
try to find out what is going on. The IP that is getting
through in this case is 216.108.233.48 and earlier today, I
spotted another IP that is doing the same thing. I don't
edit these files directly but use the Fluffy Trainer so if
there is a syntatical error, its in the gui.
Thanks and good luck.
Dan
Dan's Log File
Dan's list.txt file
Logged In: YES
user_id=660239
The order is blacklists/whitelists (whatever it can find
matching first), then defer then dnsbl. I can understand your
scepticism (I know what it's like when some programmer says
it can't be foing what yous ee it is doing), but that really is
what happens. So I'm happy to look a the log files to see why
it apparently isn't. :-)
rontouw: if you ahve good senders being deferred, then you
shouldn't be using the DNSBL that is deferring them. I'd
suggest that DNSBL isn't appropriate if it is listing IPs of
people you want to send you email.
I really don't believe you see a message being deferred for
more than 5 days. Can you show me evidence?
Logged In: YES
user_id=660239
Looks like a bug in my code. Perhaps the subnet mask is not
being applied properly. I shall scurry away and work on this. I
may not have an answer for 48 hours.
Sorry.
Logged In: YES
user_id=830967
Thats alright. At least its not something thats program
crippling. I wish you luck. And hey, is there any other
bugs/suggestions being worked on? May be time for another
full release after you fix and test this bug.
Thanks and Good Luck,
Dan
Logged In: YES
user_id=660239
Daniel: I can't replicate this. It seems to work for me. Can
you tell me how long you've had the
216.108.233.0|255.255.255.0 entry in your reject list, versus
the Jan 18 date of the log entry?
Can I ask you to increase the debug log detail to level 4, and
cpature another incident in the logs. I want to see more
detail as to what is happening.
Logged In: YES
user_id=830967
I've had that entry in the list for about 3 or 4 days.
Other entries I have put in seem to be working but this one
is not (seems to means that I haven't received any email in
my inbox from those IPs). I have increased the log level to
4 and I will post when it happens again.
Dan
Logged In: YES
user_id=830967
Well I got another one, I've attached the log file. The
list.txt has not changed.
Dan
Logged In: YES
user_id=660239
Thanks cd83. I really appreciate your patience in helping me
track this down. I will find and fix the problem.
Umm, I don't see the log file attached. Perhaps, like I often
do, you didn't tick the box as well as entering the name and
description?
Logged In: YES
user_id=830967
Yea, I did that. I attached it about a minute later. Its
there now. Its the log20040121.txt file.
Dan
Logged In: YES
user_id=830967
I was just wondering about the progress on this bug. I have
the latest release (1.5.2) that I can find and I was
reminded of this problem when I got it again just this week
(the other sender disappeared). The new IP is
80.100.132.155 and I have blacklisted
80.100.132.0|255.255.255.0 but it still gets through. Any
progress as to the fix on this bug (or any new files I
should have)?
Dan