fetchmail-users

[fetchmail-users] Messages not downloaded

[David N. <na...@ma...>]

We've been using fetchmail for a few years, but recently noticed that there was a delay of up to 20minutes in receiving messages sent to us from an external email address.  We're using a POP3 multidrop mailbox, and fetchmail passes the retrieved mail to Trestlemail MDA which then drops messages into the correct user's inbox.  It's all worked fine 'til now.  
 
I logged on to our domain's webmail service, and noticed that our multidrop inbox had a few thousand messages in it.  I don't think this used to happen, but they've started to collect there.  I cleaned them out, and we started to receive external mail instantaneously (obviously fetchmail was taking time to parse the thousands of messages).
 
When new mail hits the multi-drop mailbox, its status is 'unread'.  When fetchmail downloads from the multidrop, the mail is removed from the multidrop mailbox, with the exception of some messages where the status changes to 'read' but it is not downloaded.  These are the ones that are building up.  Some of the recipients are genuine users on our system, but many are not.  From what I've been able to monitor, all these messages are SPAM.  I've looked at the headers, and they seem to be pretty much the same as genuine messages, but they have X-IMAIL-SPAM-xxx lines in their header, eg X-IMAIL-SPAM-VALFROM.  
 
After all that explanation, I just wondered if fetchmail is looking at these header lines and choosing not to download the messages?  
 
I'm probably going to look at the 'flush' parameter to automatically delete them, once I've monitored it for a wee while to make sure that every message affected is genuinely SPAM.  I don't really want to change fetchmail versions if I can help it!  
 
Output of fetchmail -V is:
 
This is fetchmail release 5.8.17+SSL+NLS
Linux scoms1.xxxxx.co.uk 2.4.13 #1 Thu Oct 31 02:32:23 EST 2002 i686 unknown
Taking options from command line and /.fetchmailrc
Poll interval is 300 seconds
Idfile is //.fetchids
Fetchmail will forward misaddressed multidrop messages to postmaster.
Options for retrieving from xx...@xx...@xxx.net:
  True name of server is xxx.net.
  Protocol is POP3.
  All available authentication methods will be tried.
  Server nonresponse timeout is 300 seconds (default).
  Default mailbox selected.
  All messages will be retrieved (--all on).
  Fetched messages will not be kept on the server (--keep off).
  Old messages will not be flushed before message retrieval (--flush off).
  Rewrite of server-local addresses is enabled (--norewrite off).
  Carriage-return stripping is enabled (stripcr on).
  Carriage-return forcing is disabled (forcecr off).
  Interpretation of Content-Transfer-Encoding is enabled (pass8bits off).
  MIME decoding is disabled (mimedecode off).
  Idle after poll is disabled (idle off).
  Nonempty Status lines will be kept (dropstatus off)
  Delivered-To lines will be kept (dropdelivered off)
  Messages will be delivered with "~/trestlemail/bin/trestlemail ~/trestlemail/trestlemail.rc -F %F -- %T".
  Recognized listener spam block responses are: 571 550 501 554
  Multi-drop mode: 1 local name(s) recognized.
  DNS lookup for multidrop addresses is enabled.
  Server aliases will be compared with multidrop addresses by name.
  Envelope header is assumed to be: Received
  Local domains: xxxx.co.uk
  No UIDs saved from this host.
 
Thanks 
 
David

Me

Re: [fetchmail-users] Messages not downloaded

[Frederic M. <fre...@wo...>]

David Nash a écrit :
> We've been using fetchmail for a few years, but recently noticed that there was a delay of up to 20minutes in receiving messages sent to us from an external email address.  We're using a POP3 multidrop mailbox, and fetchmail passes the retrieved mail to Trestlemail MDA which then drops messages into the correct user's inbox.  It's all worked fine 'til now.  
>  
> I logged on to our domain's webmail service, and noticed that our multidrop inbox had a few thousand messages in it.  I don't think this used to happen, but they've started to collect there.  I cleaned them out, and we started to receive external mail instantaneously (obviously fetchmail was taking time to parse the thousands of messages).
>   
I encounter that problem from time to time. In my case, it always occurs 
after a broken connection.

What happens is that fetchmail starts retrieving the mails and marks 
them for deletion as soon as they are delivered to the MDA. Then, at 
some point, the connection break (usually a timeout) and the server 
assumes that it should not delete the mails that were marked so (it is 
the expected behaviour).

When fetchmail starts the next poll, it sees all the old messages and do 
not download nor delete them again. Instead it leaves them on the server.
>  
> When new mail hits the multi-drop mailbox, its status is 'unread'.  When fetchmail downloads from the multidrop, the mail is removed from the multidrop mailbox, with the exception of some messages where the status changes to 'read' but it is not downloaded.  These are the ones that are building up.  Some of the recipients are genuine users on our system, but many are not.  From what I've been able to monitor, all these messages are SPAM.  I've looked at the headers, and they seem to be pretty much the same as genuine messages, but they have X-IMAIL-SPAM-xxx lines in their header, eg X-IMAIL-SPAM-VALFROM.  
>  
> After all that explanation, I just wondered if fetchmail is looking at these header lines and choosing not to download the messages?  
>   
No header filtering that I'm aware of.
>  
> I'm probably going to look at the 'flush' parameter to automatically delete them, once I've monitored it for a wee while to make sure that every message affected is genuinely SPAM.  I don't really want to change fetchmail versions if I can help it!  
>   
You cannot assumes they are all spams but you may assume they were all 
delivered some time in the past. When I notice that a lot of mails are 
left on the server, I just turn the 'flush' option on for one poll and 
then I disable it for normal usage.

To reduce the likelihood of leaving a lot of mails on the server if the 
connection break late in the transaction, I set fetchlimit to 500 mails. 
With that option, fetchmail closes the POP3 connection after retrieving 
500 mails and the server deletes at least those mails.

Frederic

Re: [fetchmail-users] Messages not downloaded

[Matthias A. <mat...@gm...>]

Frederic Marchal schrieb:

> What happens is that fetchmail starts retrieving the mails and marks 
> them for deletion as soon as they are delivered to the MDA. Then, at 
> some point, the connection break (usually a timeout) and the server 
> assumes that it should not delete the mails that were marked so (it is 
> the expected behaviour).
> 
> When fetchmail starts the next poll, it sees all the old messages and do 
> not download nor delete them again. Instead it leaves them on the server.

Frédéric,

your observation is correct. I have discussed with Sunil Shetye how we can
mark these messages "deleted" so fetchmail re-does the deletions at the
next cycle, but there have been open issues we haven't answered at the
time, and finding a solution was shelved for various reasons, not the least
of which that it was rather intrusive and had unresolved borderline issues.

One way to contain this with respect to the message count is --expunge
(which forces the deletion to become effective by logging out and in again
for POP3 or sending an EXPUNGE command for IMAP), another one --fetchlimit,
as you suggest (I'm not quoting that) which proceeds to the next server
after a given number of messages downloaded.

The options aren't quite the same and I'm not guaranteeing anything for
older fetchmail version, but either should help with "mail reappearing
after broken connection" issues to some extent.


-- Matthias

Re: [fetchmail-users] Messages not downloaded

[Matthias A. <mat...@gm...>]

[resending to list]

David Nash schrieb:

> This is fetchmail release 5.8.17+SSL+NLS

David, if you'd like support from this list, update to 6.3.8 and apply the
patch posted to the security advisory, both available via
http://www.fetchmail.info/

This list cannot currently provide support for older versions - if your
distribution is still supported, ask its vendor for help.

Sorry for the inconvenience, but volunteer support resources are very
limited and nonexistent for all but the current version.

Re: [fetchmail-users] Messages not downloaded

[Matthias A. <mat...@gm...>]

So, some more version-agnostic answers now that I've more time at my hands,
note that my answers will still assume you're using version 6.3.8 of fetchmail.

David Nash schrieb:
> I logged on to our domain's webmail service, and noticed that our multidrop inbox had a few thousand messages in it.  I don't think this used to happen, but they've started to collect there.  I cleaned them out, and we started to receive external mail instantaneously (obviously fetchmail was taking time to parse the thousands of messages).

Likely, to be confirmed or denied with "fetchmail -v" or perhaps "fetchmail
-vv".

> When new mail hits the multi-drop mailbox, its status is 'unread'.  When fetchmail downloads from the multidrop, the mail is removed from the multidrop mailbox, with the exception of some messages where the status changes to 'read' but it is not downloaded.  These are the ones that are building up.  Some of the recipients are genuine users on our system, but many are not.  From what I've been able to monitor, all these messages are SPAM.  I've looked at the headers, and they seem to be pretty much the same as genuine messages, but they have X-IMAIL-SPAM-xxx lines in their header, eg X-IMAIL-SPAM-VALFROM.  

> After all that explanation, I just wondered if fetchmail is looking at these header lines and choosing not to download the messages?  

No, it does not. If it did, you'd see that documented as a feature.
However, one of those spam messages may confuse your older version of
fetchmail, or perhaps (though less likely as I see it) your MDA (note I
wrote "may", not "does", I don't know trestlemail and this is not a
statement about trestlemail as such).

> I'm probably going to look at the 'flush' parameter to automatically delete them, once I've monitored it for a wee while to make sure that every message affected is genuinely SPAM.  I don't really want to change fetchmail versions if I can help it!  

It's a dangerous option to use, I'd rather run fetchmail once with that
option on the command line.

You really should update, not only fetchmail, but also your kernel, and
probably HEAPS of your system libraries and basic utilities. Your system is
way outdated and full of security-relevant bugs, and unless your fetchmail
version has been patched against known security holes, it's prone to code
injection.

I'd not be surprised at all if one of the bugs caused the messages to pile
up, by terminating fetchmail prematurely.

In a separate response to Frédéric I describe some workarounds to contain
the damage a bit - consider them regardless of if you're updating.

> Output of fetchmail -V is:
>  
> This is fetchmail release 5.8.17+SSL+NLS
> Linux scoms1.xxxxx.co.uk 2.4.13 #1 Thu Oct 31 02:32:23 EST 2002 i686 unknown
> Taking options from command line and /.fetchmailrc
> Poll interval is 300 seconds
> Idfile is //.fetchids
> Fetchmail will forward misaddressed multidrop messages to postmaster.
> Options for retrieving from xx...@xx...@xxx.net:
>   True name of server is xxx.net.
>   Protocol is POP3.
>   All available authentication methods will be tried.
>   Server nonresponse timeout is 300 seconds (default).
>   Default mailbox selected.
>   All messages will be retrieved (--all on).
>   Fetched messages will not be kept on the server (--keep off).
>   Old messages will not be flushed before message retrieval (--flush off).
>   Rewrite of server-local addresses is enabled (--norewrite off).
>   Carriage-return stripping is enabled (stripcr on).
>   Carriage-return forcing is disabled (forcecr off).
>   Interpretation of Content-Transfer-Encoding is enabled (pass8bits off).
>   MIME decoding is disabled (mimedecode off).
>   Idle after poll is disabled (idle off).
>   Nonempty Status lines will be kept (dropstatus off)
>   Delivered-To lines will be kept (dropdelivered off)
>   Messages will be delivered with "~/trestlemail/bin/trestlemail ~/trestlemail/trestlemail.rc -F %F -- %T".
>   Recognized listener spam block responses are: 571 550 501 554
>   Multi-drop mode: 1 local name(s) recognized.
>   DNS lookup for multidrop addresses is enabled.
>   Server aliases will be compared with multidrop addresses by name.
>   Envelope header is assumed to be: Received
>   Local domains: xxxx.co.uk
>   No UIDs saved from this host.

A general remark: DNS lookup for multidrop is a resource-consuming option
to use and based on the - usually false today - assumption that POP3 and
SMTP MX servers for the given domain are identical. This holds for none but
the smallest sites that run all services off the same computer. If there is
more than the polling domain, list it with "aka", then use the "no dns" option.

Hope that clarifies things a bit.

-- Matthias

Re: [fetchmail-users] Messages not downloaded

[Matthias A. <mat...@gm...>]

Matthias Andree <mat...@gm...> writes:

>> This is fetchmail release 5.8.17+SSL+NLS Linux scoms1.xxxxx.co.uk
>> 2.4.13 #1 Thu Oct 31 02:32:23 EST 2002 i686 unknown Taking options

BTW, sorry for messing up the quote formatting.

-- 
Matthias Andree