[fetchmail-devel] Re: Incorrect header line and lost mails

[Sunil S. <sh...@bo...>]

Quoting from Matthias Andree's mail on Tue, Mar 14, 2006:
> > Imagine the worst-case scenario: Frederic's mailbox has 2000 mails on
> > this server and it has to be accessed on a slow dialup line which is
> > prone to disconnection. Even if the assumption doesn't work, it will
> > still download a few mails. Going back to linear search will probably
> > fetch even less mails due to the overheads.
> 
> If it confuses the user, it needs to be fixed. If the connection is too
> flakey to regularly provide the full UIDL connection (which will usually
> compress well with V.42bis or MNP5), then the user should re-think if he
> needs "keep messages on server" setups. Usually there are better
> alternatives, such as *copying* new messages from INBOX to a second IMAP
> folder which is then downloaded in nokeep mode, so that there is no need
> to download UID lists.

Apart from the issue of fixing, what is missing is documentation on
this. There is no place where it is written what configuration should
a user use to derive the best mileage from fetchmail. Maybe, some FAQ
entry like:

=========================================================================
C#. What protocol should I choose with my mailserver? My mailserver
supports both IMAP and POP3.

fetchmail, by default, uses first IMAP and then POP3 (protocol auto).
This doesn't work when you are keeping mails on the mailserver or when
there are errors (like socket errors or deliver errors) after a few
mails have been downloaded.

So, as a first step, you should necessarily choose between IMAP and
POP3.

- If your mailserver is delivering to multiple folders, choose IMAP.
  
- If you want to download all mails and not leave a copy on the
  server, choose IMAP.

- If you intend to keep mails on the server, check if your server
  supports UIDL. If it does, choose POP3 and enable the "uidl" option.
  With POP3 and UIDL, it is even possible to access the mailbox
  through multiple e-mail clients without affecting fetchmail.

- If you intend to keep mails on the server, but your server does not
  support UIDL, choose IMAP. However, you should not access the
  mailbox through other e-mail clients as fetchmail will not be
  able to download all new mails.

- If you are having a slow connection to the mailserver and/or expect
  frequent socket errors from the mailserver, choose IMAP and do not
  leave a copy on the mailserver.

C#. Ok, I choose IMAP. What next?

...

C#. Ok, I choose POP3. What next?

...
=========================================================================

> > The question is not just of keeping in memory or database. The
> > question is: What should be done if a mismatch is found? If the only
> > solution is to go back to downloading all UIDs at one go, then I would
> > prefer continuing with the incorrect assumption.
> 
> Well, I'd rather download full UID lists to get rid of bogus
> assumptions. Missing messages because of false UIDL assumptions delays
> messages far more than connection drops (which are usually noticed
> quickly) does, and that causes certainly more reports than long UIDL
> downloads.
> 
> A RANGES extension for POP3 that allowed UIDL n- or UIDL n-m would help
> a lot.

If this extension is being used, using it is the better option.

> My basic idea is: say we have u UIDLs stored, and the server has m
> messages. Then, send "UIDL" u+1 "UIDL" u+2 ... "UIDL" m pipelined
> (non-blocking) (or, on future RANGE capable POP3 servers, "UIDL" u+1 "-"
> m), and if ANY seen message is in that range, do slow UIDL. No binary
> search with many round trips. Of course, this needs to track deletions
> properly and if another client client deletes messages, needs to back
> down and fetch the whole list. Perhaps sanity checking the last known
> UIDL helps detecting third-party deletions on the assumption that the
> likelyhood of the server recycling a UID is low.

This might not work with "no keep" when fetchmail is not sure if the
mails marked by fetchmail for deletion have actually been expunged or
not.

-- 
Sunil Shetye.