Re: [Courier-imap] Is Courier IMAP Still "Expensive"?

[Sam V. <mr...@co...>]

Sabahattin Gucukoglu via Courier-imap writes:

> If you were to permit the saving of an indefinite number of snapshots (up to  
> some specified time limit), without reference to any particular client, and  
> store the snapshots into easily looked-up files in a directory, then  
> whenever a client references a MODSEQ, it would be an easy matter to reply  
> accordingly with the changes. How expensive is creating a snapshot? Is it  
> done by request of the client, or every time there’s a change? After all, if  
> you are OK with keeping this additional state, then perhaps it’s not such a  
> tough battle.

Skimming over the specification, the server is supposed to calculate and  
return all changes between the snapshot and the current state of the  
mailbox, but not the complete set of changes, only the changes in the state  
of the flags of any messages that are still in the mailbox. And there's  
still the issue of maintaining a monotonically increasing sequence counter,  
which this does not address at all.

> > No that is not reusable, for a couple of reasons. Because, again,  
> CONDSTORE requires an entire infrastructure that revolves around  
> monotonously increasing sequence numbers. Nothing, with regards to  
> client/server resynchronization, uses or needs any sequence numbers.
>
> I agree. It is an elegant solution, but it won’t help without client support  
> to identify it. It is a pity that this approach was not used or made  
> available in the standards.

Earlier, I wrote:

# The CONDSTORE extension is a perfect example. It's a perfect IMAP extension,
# that exhibits the same underlying problems that IMAP itself has.

There were several nuanced layers in my reply. I chose by words carefully,  
when I wrote that.

The original IMAP was tailored to formalize the behavior of the very first  
IMAP server, and the protocol was designed to accomodate the server, rather  
than the other way around.

Similarly, it looks to me that the CONDSTORE extension is tailored to the  
behavior of a specific IMAP server which was designed in a very specific  
way, and cannot be efficiently implemented generically. It imposes a  
specific, restrictive, overall structure that favors a very specific server  
design.

On the other hand, I believe that an opaque snapshot-based approach gives  
servers wide lattitude and freedom in terms of their implementation details,  
and should be easy to implement in any IMAP server, while the monotonically- 
increasing counter-based approach requires a specific, strict, server design  
that does not directly offer any value-added in of itself.

What could be easier than simply adding a few more commands, to effectively  
tag the server state, without adding any new concepts to the core IMAP, and  
simply letting the existing IMAP features – that already exist –  
resynchronize the server with the client, since the client must implement  
the same IMAP messages, already?

> One more throw of the dice, then. Do you see yourself computing CONDSTORE’s  
> MODSEQ algorithmically? E.g.: a generation for the uiddb file plus an offset  
> into it for a given message? Of course, you still need an expunge log …

Except that the UID file does not grow infinitely. It is pruned when  
messages gets deleted. So a given message's location in the UID file will  
change over time. I don't see how a monotonically-increasing counter value  
can be algorithmically computed, from that.

> Well, for me, I think it clarifies that this is really a discussion about  
> philosophy, and philosophy is important. I do like Courier’s philosophy  
> (minimalism, state-avoidance, lock-avoidance) a great deal. But,  
> unfortunately, it has casualties that other servers, prepared to keep more  
> per-message state, do not have (as much or to the same degree). Nowt wrong  
> with this, of course, but it’s something to bear in mind when choosing  
> servers for a given task.

There's nothing wrong with other servers keeping more detailed per-message  
state. That in itself is completely orthogonal to the requirements of  
representing the state as a "64 bit monotonically increasing integer", or  
however CONDSTORE define that. There is absolutely no reason, whatsoever, to  
impose that restriction, if the goal is to be able to efficiently  
synchronize client/server behavior.

If, on the other hand, a server's state is represented as some opaque  
snapshot handle, then a particular server CAN STILL implement that handle as  
a 64 bit monotonically increasing integer, if it's convenient to do so.

It does NOT restrict servers that are implemented in term of some kind of a  
journal, or a log, making it easy for them to support a version counter. It  
can still use that to implement client/server resynchronization, since a 64- 
bit integer perfectly works as an opaque identifier for the server's  
snapshot.

But FORCING the server's state to be representable as a 64-bit integer  
FORCES an unreasonable burden on other servers that are not designed that  
way.

> BTW, I have since learned that, due to Gmail issues, Thunderbird actually  
> disables CONDSTORE by default.

That does not surprise me.

> How do you feel about IMAP5/JMAP? Many of these ideas are explored there  
> (sync-tokens instead of sequence numbers, for instance).

I have no opinion, I haven't looked at it.