Re: [fetchmail-devel] [PATCH] Re: Incorrect header line and lost mails

[Matthias A. <mat...@gm...>]

Sunil Shetye <sh...@bo...> writes:

> Quoting from Matthias Andree's mail on Tue, Mar 14, 2006:
>> > One option is to change the default to a lower value from the current
>> > value of 10. The lowest practical value is 2, which will use linear
>> > search in alternate polls.
>> 
>> The algorithm tries to find the first unused UIDL with a binary search,
>> on the assumption that all new messages were at the end of the
>> list. This doesn't work for Fr?d?ric's server and others.
>
> Imagine the worst-case scenario: Frederic's mailbox has 2000 mails on
> this server and it has to be accessed on a slow dialup line which is
> prone to disconnection. Even if the assumption doesn't work, it will
> still download a few mails. Going back to linear search will probably
> fetch even less mails due to the overheads.

If it confuses the user, it needs to be fixed. If the connection is too
flakey to regularly provide the full UIDL connection (which will usually
compress well with V.42bis or MNP5), then the user should re-think if he
needs "keep messages on server" setups. Usually there are better
alternatives, such as *copying* new messages from INBOX to a second IMAP
folder which is then downloaded in nokeep mode, so that there is no need
to download UID lists.

> The question is not just of keeping in memory or database. The
> question is: What should be done if a mismatch is found? If the only
> solution is to go back to downloading all UIDs at one go, then I would
> prefer continuing with the incorrect assumption.

Well, I'd rather download full UID lists to get rid of bogus
assumptions. Missing messages because of false UIDL assumptions delays
messages far more than connection drops (which are usually noticed
quickly) does, and that causes certainly more reports than long UIDL
downloads.

A RANGES extension for POP3 that allowed UIDL n- or UIDL n-m would help
a lot.

>> Well. Should I lower the default to 4 for the nonce or wait for a patch? :)
>
> Here it is:

I thought about a sanity checking patch that detects when slowuidl is
needed, but I'm taking this for the nonce, too.

My basic idea is: say we have u UIDLs stored, and the server has m
messages. Then, send "UIDL" u+1 "UIDL" u+2 ... "UIDL" m pipelined
(non-blocking) (or, on future RANGE capable POP3 servers, "UIDL" u+1 "-"
m), and if ANY seen message is in that range, do slow UIDL. No binary
search with many round trips. Of course, this needs to track deletions
properly and if another client client deletes messages, needs to back
down and fetch the whole list. Perhaps sanity checking the last known
UIDL helps detecting third-party deletions on the assumption that the
likelyhood of the server recycling a UID is low.

-- 
Matthias Andree