repeated deferrals
Status: Beta
Brought to you by:
waynemcdougall
Menu
▾
▴
#111 repeated deferrals
I'm seeing the same message from the same server
deferred over and over again, running 1.4.92a (I think:
the title says 1.4.92).
Here's a snipped extract from the status window:
20:47:32 Deferred mail to forbes@hairydog.co.uk from
<Forbes.com@z2c.net> 67.72.99.27
20:58:18 Deferred mail to forbes@hairydog.co.uk from
<Forbes.com@z2c.net> 67.72.99.27
21:24:37 Deferred mail to forbes@hairydog.co.uk from
<Forbes.com@z2c.net> 67.72.99.27
22:01:45 Deferred mail to forbes@hairydog.co.uk from
<Forbes.com@z2c.net> 67.72.99.27
Expiry time is 23 hours, new contact delay is 3 minutes
and old contact expiry is 56 hours.
By the way, how can I stop this stuff appearing every
few minutes?
21:42:35 Total accepted 74
21:42:35 Total rejected 50
21:42:35 Total deferred then not received 30
21:42:35 Total deferred then later processed 12
21:42:35 Total caught in tar pit 0
21:42:35 Total released from tar pit 0
21:42:35 Connection handling since 19/12/2003
00:00:03
21:42:35 Connections Dropped: 9
21:42:35 Connections Blocked: 49
21:42:35 Connections Deferred: 65
21:42:35 Connections Accepted: 75
21:42:35 Connections Unable to accept: 0
21:42:35 Connections Total (including open
connections): 198
Logged In: YES
user_id=660239
Let's start with the easy stuff:
> By the way, how can I stop this stuff appearing every
> few minutes?
The stuff is put in the logs so taht people who are receveing
the logs on a remote system can get an updated count -
same will apply when Fluffy runs as a service as the console
will be a separate user application from the service doing the
processing.
The informtion is transmitted every hour AND when the totals
change by 5%. So it should get slower as the day goes by,
ubnless you get a suddent lfurry of email.
Now you understand the why: if you still object to it, better
give me some guidance as to how you'd like to control it.
Now the difficult problem. What you show should not happen.
I cannot see why it does form the information provided. If you
set the debug detail to 1 you will see in the logs more detail
about why the connection is deferred and more importantly
when it is deferred to.
If you want to send me a full log rather than an extract to
fluffy@codeworks.gen.nz I may be able to puzzle it out more,
but even then I'd still like to see a log at debug detail level 1
when the problem surfaces again.
Logged In: YES
user_id=776625
I thought that a low debug level meant less information, so is
the suggestion to set it to level 1 a typo?
Logged In: YES
user_id=660239
Everything about logging:
1. When configuring Fluffy, the log level setting controls the
detail displayed in the local console window.
2. Regardless of the detail for the local console window, the
log file written to disk is down at full detail (level 7) - the
debug level.
3. When sending out data to a Syslog device, you choose the
log level to send to that device (up to and including that log
level) - this is independent of what is displayed in the console
window.
4. For all three (local console, disk file, syslog) there is a
global setting that controls how much information is
written/displayed IF you choode log level 7 - debugging. At
the highest level of debugging every SMTP command is logged
which is a LOT of information (syslog traffic/confusing
display/large disk log files). So you can control how much
information is recorded at debug level (and increase it only if
you ahve a problem.
5. Because disk log files are always written at debug level,
you want to set the debug detail level appropriately, even if
you choose a lower level for dispaly purposes in the console.
6. So debug detail level 1 was not a typo - just don't confuse
it with the logging level - it's the level of detail in the
devbugging log level - and 1 (or 2) is sufficient to give us
detail about when and why connections are being deferred.
7. And if you happen to have already ahd the debug log detail
set at 1 or higher, the disk log files will already have the
information I'd like to see about the connection with
67.72.99.27
Logged In: YES
user_id=744036
I have seen something similar myself - running 1.4.92a.
I can find that an incoming email will be seen for the first time
by Fluffy, be set a deferred time in the future 15 minutes away
(correct), the incoming email server keeps trying every few
minutes and each time, in the log it states the deferred time
which still has not elapsed, so quite rightly it puts the server
off with another error message, but then when the server
comes back shortly AFTER the set deferral time it still won't be
allowed through!
When you see the logs it can get really silly! Emails can be
still deferred many hours after the elapsed timer has expired.
If you want old logs, let me know.
For the moment I have set the deferral time to 0 minutes as
employees have stated that some emails do not arrive until the
day after they were sent to them.
Since setting deferral time to 0, I haven't scoured the logs to
see if it still doing it.
What logging settings do you suggest best to see all
information required. I thought that both set to 7 would be
maximum reporting. But are you suggesting otherwise? I.e.
that 1 is better?
Ron.
Logged In: YES
user_id=660239
1. The initial complaint was resolved. It turns out that the
sending SMTP server triggered a DNSBL listing which had a
weight which reached the "Defer" threshold for handling. So
the message was always correctly deferred.
My understanding, and implementation was that one keeps on
deferring until it either gets listed on other DNSBL (pushing it
up higher into the block level) or gets unlisted (and falls into
the accept) or something else happens. After repeated
deferrals the sender would just give up (generating a bounce).
However, the original complainant thought deferral should be
a one-off event. I'd like to encourage people to scrap among
themselves on this issue and come back to me explaining how
they want it to work.
Hmm, I now see the original request at
https://sourceforge.net/tracker/index.php?
func=detail&aid=815573&group_id=75935&atid=545512
wanted a "hold for later processing" rather than a defer
indefinitely. I'm not sure what that would mean or how to
implement it.
Please someone let me know. :-)
2. rontouw: My initial guess would be that it is from a large
mail service with mutliple mail servers on different IP
addresses. So one gets deferred, and then it tries again with
another server on an IP address and that gets deferred, and
so on, until eventually it retries with a mail server it has used
before and it gets through.
Fixing that is not impossible, but not trivial either. My take on
this is that often such large mail servers have the whole
domain whitelisted.
More to the point, once the mail server gets recorded by
Fluffy as being used regularly, the defer time should go away,
and drop to nothing because it will be a regular customer.
If this is not happening, I would like to see the logs of a
troublesome sender.
3. Yes log level 7/7 gives the most information - rather a lot.
Makes rather large log files. 7/1 was what was necessary to
troubleshoot this particular problem.
Logged In: YES
user_id=744036
Different IP's / large corps? ... no this was one mail server, one
IP. But you might be right about the DNSBL listing... I need to
re-analyse the logs. Something in the back of mind suggests
they may have been blacklisted with someone! Anyway - I'll
check when I have the time - bit hectic right now and the log
files are big!
On your "new features" point (although I hate to add this
discussion here as really in the wrong place) I have to admit
that I would like the following ...
Use the blocking features but not for "real" SMTP blocking but
allow it in and instead forward to another email account. This
becomes a dumping ground for the initial first few days when
users can contact you and say, I think fluffy swallowed an
incoming email, can you "regurgitate" it for me? OK it will get
all the garbage as well, but at least it gives you a chance to
add them to the whitelist to stop it happening again.
The only prob I can see is that the email account could get full
pretty quickly with all the garbage spam, but that's not fluffy's
problem or yours. It is the sysadmin's!
In the early days I got a lot of flak from some users who had
incoming mails blocked. The incoming server was blacklisted on
a number of DNSBLs. So fluffy did exactly what it was trained
to do! Unfortunately I was not very popular for a while until I
explained that it was not really our fault their server was
blacklisted. I would have preferred to be able to search for
and forward them the offending emails.
Also think you should move the bypass email address to
another tab. The Virus Scanning tab is not really the right
home for it. I appreciate there is not a lot of room elsewhere.
Ron.
Logged In: YES
user_id=744036
In answer to the defer... I think the basic understanding of the
defer logic you use is sound/correct, except emails could get
stuck forever as their IP may never get added to more external
blacklists (as actually it's a "good guy" but has his IP in a
blacklist due to other persons' actions on that ISP). I would
add a timeout that after x hours or x attempts it gets
forwarded into some designated email account for admin
attention.
Then nothing gets lost or stuck for days. You could add a
fluffy-flag to the email header stating why it was spat out the
system into the "admin-check" email account.
I can then read the email, if legit, forward to client account,
then investigate from the logs why that email got stuck and
had to be rescued.
Logged In: YES
user_id=660239
rontouw: It won't get stuck foreever because a sending SMTP
server will give up (usually after 3-5 days). So it will get
bounced back to the original sender.
Given that - do we really need to add more complexity to
Fluffy to handle this case?
Logged In: NO
Since I made the initial request for defer then process later, I
will explain what I was thinking:
Take the message and put it into a hold directory. Wait a
certain period of time specified and then send it back through
the tests. If weight doesn't increase by getting listed on
blacklists, let it either go through or hold for manual review.
If no time is specified to resend through tests, just hold it for
manual review. Manual review would require a way to view
the messages and select delete or send through.
There would need to be a setting to specify how many can
be processed at a time or only do it if there are connections
available to reprocess.
This feature would be good to hold messages that might not
have a high enough score to delete but high enough to be
suspect. One can hold the messages for a few days and wait
until someone complains that they didn't get the email. We
can look and see that we still have it, send it on through and
explain to them that there sender's ISP needs to get off a
blocklist or we need to whitelist them.
Logged In: NO
Another thing we can be done with a defer and process later
feature is that if Fluffy is getting a lot of mail coming in and
can't handle the processing of it, it can accept a message
before the DNSBL tests (and maybe keyword tests) and
process the tests later when it gets where it can do so.
This could help prevent mail delivery delays for legitimate
servers that have to keep trying to find an open connection
to send their message because someone with a cable modem
is sending a ton of viruses to your server.
Logged In: YES
user_id=660239
I'm still seeking clarity on how people would like this to work.
Some comments and observations that may help inform
debate.
1. If Fluffy has more connections that the maximum allowed,
it will refuse new connections. The sending SMTP server will
try again later.
2. It would take longer, as a rule, to accept a message than
to run DNSBL tests. On my limited connection (128 kbits) I
can get flooded with spam (up to 1 every 17 seconds) if I
accept them all. With deferring my PC doesn't even break a
sweat (200 MHz Pentium).
3. Worse is that if we accept a message we have told the
sender we will deliver it. It would be inappropriate to then
block the message later. And if we don't deliver it we should
send a bounce back. All of this is very messy.
4. The issue of a cable modem user swamping your
connection with viruses is really a separate one. I've
experienced that and haven't seen it as a problem. In fact
the deferral usually takes them away. If you have
expereinced this in practice as opposed to a theoretical
concern, we could certainyl devise a method to deal with that
(blacklisting an IP that has sent more than 10 viruses in a day
would do it).
5. Putting the emssage into a hold directory...once we have
accepted itwe are responsible for handling it, delievring ir or
generatign a bounce message. All things to be avoided really.
I would be very concerned if a system had mail addressed to
fluffy@codeworks.gen.nz, and my mail server says yes I
accept it, but in reality it was held in some queue that may
get looked at once a month. What I'd rather do is rewrite the
destination mail address which can be picked up by another
account - so it's all just mail too Fluffy, and how that mailbox
is handled is all external to Fluffy.
But it's hard for me to see an advantage in this, as opposed
to just flagging the message as possibly spam. So we
have "looks clean"..."definitely spam"..."probably spam" and
now a new category of "may be spam - let a human check it
otu before passing it on"....sounds like a step to be avoided
IMO.
6. As things stand now, the mail isn't deferred forever. The
sender will probably get a warning if the message isnm't
delivered within x hours, and should definitely get a boucne
after the sender gives up in x days. They will then know their
message hasn't been delivered and can take action
appropriately. Shift the burden on to the sender. If in that
tiem, the weighting goes up (or down) the message would be
immediately blocked (or passed through).
As I say, I'm still seeking explicit clarity here where senders
don't get false assurance about delivery and Fluffy isn't doing
any sort of tedious manual interference.
Make a case and convince us, and me!