Re: [Courier-imap] Is Courier IMAP Still "Expensive"?

[Sabahattin G. <lis...@me...>]

On 2 May 2021, at 14:11, Sam Varshavchik <mr...@co...> wrote:
> Sabahattin Gucukoglu via Courier-imap writes:
>>> All of that is somewhat analogous to CONDSTORE's sequence numbers. Except that, from what I can tell, the server must maintain a complete history of all changes to the mailbox, in order to support CONDSTORE+QRESYNC. This seems completely unnecessary itself, too onerous, and it misses the mark entirely. The server only needs to maintain a single per-client snapshot (with the next most recent backup snapshot), identified by a single opaque handle.
>> 
>> OK, so I will be looking at it, but could this infrastructure be reused? If you were to permit the saving of an indefinite number of snapshots (up to some specified time limit), without reference to any particular client, and store the snapshots into easily looked-up files in a directory, then whenever a client references a MODSEQ, it would be an easy matter to reply accordingly with the changes. How expensive is creating a snapshot? Is it done by request of the client, or every time there’s a change? After all, if you are OK with keeping this additional state, then perhaps it’s not such a tough battle.
> 
> No that is not reusable, for a couple of reasons. Because, again, CONDSTORE requires an entire infrastructure that revolves around monotonously increasing sequence numbers. Nothing, with regards to client/server resynchronization, uses or needs any sequence numbers.

I agree. It is an elegant solution, but it won’t help without client support to identify it. It is a pity that this approach was not used or made available in the standards.

One more throw of the dice, then. Do you see yourself computing CONDSTORE’s MODSEQ algorithmically? E.g.: a generation for the uiddb file plus an offset into it for a given message? Of course, you still need an expunge log …

> Anything that involves a monotonous sequence number rubs against the grain of concurrent processes, all using the same mail store. Like what IMAP allows. Using a sequence number to identify messages in a folder, with a strict requirement that it's monotonous, is bad enough. CONDSTORE just adds to the badness. Monotonous, strictly increasing sequence numbers, are not welcome in the age of parallel, concurrent computing. The same mail store could be, for example, accessible by multiple servers, with both of them accessing the same mail folder. And the independent processes on those servers are expected to, somehow, coordinate their ticking of the same sequence number.
> 
> Implementing IMAP, as is, was a small miracle. One miracle is enough.
> 
> Finally, the snapshots in question are snapshots of the entire folder. and are not just some individual, single token. Which is fine, since the resynchronization algorithm only needs the two most recent snapshots, to guarantee resynchronization. CONDSTORE requires everything to be kept forever. Saving the complete list of messages any time something changes is going to get real old, real fast.
> 
> In my previous message I alluded that CONDSTORE "is a perfect IMAP extension, that exhibits the same underlying problems that IMAP itself has". I hope the above clarifies that.

Well, for me, I think it clarifies that this is really a discussion about philosophy, and philosophy is important. I do like Courier’s philosophy (minimalism, state-avoidance, lock-avoidance) a great deal. But, unfortunately, it has casualties that other servers, prepared to keep more per-message state, do not have (as much or to the same degree). Nowt wrong with this, of course, but it’s something to bear in mind when choosing servers for a given task.

BTW, I have since learned that, due to Gmail issues, Thunderbird actually disables CONDSTORE by default.

How do you feel about IMAP5/JMAP? Many of these ideas are explored there (sync-tokens instead of sequence numbers, for instance).

Cheers,
Sabahattin