Re: [fetchmail-users] Re: fix lost POP3 deletes

[Matthias A. <ma...@dt...>]

On Fri, 19 May 2006, Frederic Marchal wrote:

> >The refinement of the suggestion would be to hash all but a few
> >non-constant headers with some decent hash function. MD5 would be
> >simple, but we shouldn't hardcode anything here.
> >  
> You are right. This is a better.
> 
> To reduce the load, the hash could be used only when both the UID and 
> the size of a previously deleted message are found again on the next 
> poll. That would not occur too often and, therefore, it should not 
> increase significantly the load on a relatively stable connection with a 
> server that uses good UID and reports valid constant mail size.
> 
> The user could also choose to use the hash instead of the size when the 
> size reported by the server is unreliable.

Just to clarify: a hash is not a short-term solution. I don't think
we'll see this in 6.3.X beyond what's already there.

> We have a pop3 proxy (p3scan) through which fetchmail downloads the 
> mails from the external pop3 server. p3scan get the whole mail and hand 
> it to clamd to scan it for a virus. If everything is ok, the mail is 
> passed to fetchmail. Therefore, fetchmail receives the mail after some 
> delay depending on the size of the mail but it can send it successfully 
> to postfix. Then, fetchmail tries to send the dele command and it fails 
> because the scanning of the mail by clamd took too much time and the 
> pop3 server timed out.

...

> If fetchmail can discover that the mail was delivered but it could not 
> be deleted on the previous poll (remember the dele command was never 
> acknowledged by the server) and delete it without downloading it again, 
> then the problem is solved. In my case, I received a 32MB mails and it 
> took 13 minutes to scan it.
> It is far beyond the patience of the most 
> patient server...

Thanks for providing that information. The actual problem is p3scan's
attempt to scan in real-time, which failed miserably. The proper
solution would be to change the setup:

1. fetchmail downloads the message

2. fetchmail injects either into an MTA which has a after-queueing scanner hooked up
   (for instance Postfix with amavisd-new)

3. the MTA queues the message with a "must inspect content".

4. MTA hands the message to the scanner, which can take as long as it
   desires, and then takes action depending on the result. Either
   defang, pass on, bounce, or erase.

5. MTA then hands the message to the local delivery agent (internal,
   maildrop, procmail, deliver, ...) which stuffs the mail into a
   mailbox

6. a local POP3 or IMAP server offers access to the content-scanned
   message to the client.

Step 3 and 4 are important - this makes for fast acceptance to prevent
timeouts and allows the scanner to take all the time it needs, without
respect to network timeouts.

-- 
Matthias Andree