Re: [Fetchmail-devel] IMAP and UIDs

[Matthias A. <mat...@gm...>]

Am 25.08.2016 um 15:34 schrieb Peter Griess:
> Hi,
>
> I'm thinking of taking a stab at the "Let IMAP code use UID and
> UIDVALIDITY rather than relying on flags that everyone can alter" bug
> since relying on the \Seen state is hurting my multi-client workflow.
>
> I have a few questions before I get started
>
> 1. Do we need to tackle UIDVALIDITY storage and validation as part of
> this? Since client-side handling UIDVALIDITY changes is had to do
> correctly, most (all?) IMAP servers don't do this unless they
> absolutely have to (e.g. there has been some sort of corruption in
> their underlying storage layer). If we can punt on UIDVALIDITY, we can
> store IMAP UIDs similarly to how we handle POP UIDs. Any advice here?

Looking at UIDVALIDITY and invalidating all UIDs if the UIDVALIDITY
changes is crucial.
I will not accept a patch that disregards UIDVALIDITY.
Feel free to add a "force-keep-UID-on-changed-UIDVALIDITY" option, too,
if it defaults to off, that's fine.

Note the patch should be against the master branch, as that one has
considerably faster UID database code (O(n log n) rather than O(n^2))
compared to the legacy branches.
If, on the other hand, you mean to write such code in C++ rather than C,
and it's clean enough, then I'll also be fine with that; I'd propose to
not go beyond C++11 at this time (but if there's something in C++14 you
want to use and it's well supported by 2-year old operating systems,
I'll be happy to review that, too.)

> 2. POP UIDs are alphanumeric strings. IMAP UIDs are unsigned 32-bit
> integers. In order to re-use as much of the existing idlist code as
> possible, I was thinking of storing UIDs as strings, e.g. via
> sprintf(id, "%lu", uid). Not my favorite solution but it should be
> less invasive than other ideas (e.g. making 'idlist.id' a union).
Fine with me. It's advantageous if the .fetchids file can be tweaked in
vim, if the need arises, and given the UIDs are decimal strings "on the
wire", it also seems natural so that the .fetchids data and the wire
protocol data match.
> 3. For the configuration impact of this, allowing --uidl to affect
> IMAP sessions would be the natural choice except that it may have
> unexpected consequences for users who were relying on that being a
> no-op. In this case the consequence would be that they would have an
> idfile get populated. I think this is probably okay as long as we
> preserve the existing behavior that marks fetched messages as seen. I
> think we should add a new --leave-unseen flag that allows uses to
> opt-in to marking messages as un-seen after fetching.
Since this is a new feature I am intending to release in fetchmail 7.0
(not 6.x.y) changing behaviour to "not mark seen" will be fine at that time.

I will also tweak TLS options at the time, and there will not be a 100%
compatibility either.
It'll probably better to have a more flexible way of telling fetchmail
what to pull (all being the default, and unread another) and what to
mark after retrieval for IMAP.

Regarding your other message, where you're writing:

> Nevermind about storing lists for UIDs. I had forgotten about UIDNEXT
> and ranged fetches.
>
> So I think the minimum required state for each folder is simply the
> last seen value of UIDNEXT. Given that this is quite a bit different
> than the lists of UIDs that are being tracked in fetchids, we may
> indeed want a separate file for this stuff. I suppose we could live
> alongside the existing fetchids scheme by encoding these values as a
> fake UID. In this case, the fetchids file would contain entries like
> this the following (with a POP account to start things off to show
> contrast):
>
>     pg...@po... some-pop-uid-1
>     pg...@po... some-pop-uid-94
>     [...]
>     pg...@gm... some-folder/1234567890/14
>     pg...@gm... some-folder/1234567890/17
>     [...]
I am open to adding a second file, perhaps even a per-"poll" (account)
file to get some of the login@account redundancy out.
We used to have code (patches) to that end in the late 6.2.5 and early
6.3.X days that never hit mainstream, and the development of this
feature stalled.
They're at the very bottom of
http://krusty.dt.e-technik.uni-dortmund.de/~ma/fetchmail/ in the fine
print, and are also somewhat polluted in that they contain modernization
stuff for the autotools rigging that shouldn't be in.