Re: [Courier-imap] Is Courier IMAP Still "Expensive"?

[Sabahattin G. <lis...@me...>]

On 30 Apr 2021, at 13:49, Sam Varshavchik <mr...@co...> wrote:
> The CONDSTORE extension is a perfect example. It's a perfect IMAP extension, that exhibits the same underlying problems that IMAP itself has. If the goal is to provide for efficient client/server resynchronization, this can be done far, far simpler than all the baggage that goes into CONDSTORE+QRESYNC. There's a huge pile of functionality in CONDSTORE+QRESYNC that has nothing to do with resynchronization, but the server has to implement it; and I doubt whether any client actually even uses it. Why? Well, because CONDSTORE+QRESYNC is an optional extension, and if the server doesn't support it, what are the client's choices? Disable whatever extra functionality the client needs the extension is for? That's not going to fly. No, instead the client will simply fallback to IMAP core functionality, and as long as the client has to code that anyway, there's very little reason to then also implement the extension too, in the client, unless the server can do the extension far more efficiently. But in CONDSTORE's case, most of the extension's functionality seems to be rather dubious, and is a solution for an unclear problem.

FWIW, CONDSTORE+QRESYNC are implemented by (at least) Apple Mail (iOS, macOS), Thunderbird, and Sogo (the latter requiring them). It is, of course, an optimisation, just as LITERAL+, SORT/THREAD, UNSELECT, MULTIAPPEND and BINARY are. We can live without them, but having them makes things easier, in this case most especially for mobile clients where efficient bandwidth use / avoiding round-trip latency are totally worth having, so clients implement them. I think CONDSTORE+QRESYNC are a major advantage for a modern IMAP server to have, and personally I would love to see them implemented in Courier, so people have a good choice of server. But, ultimately, you’re the best person to know if it’s worth implementing. I do notice that you’ve been asked this before as well, so I think it’s probably desirable.

No, I don’t think CONDSTORE+QRESYNC are especially beautiful, and it was always my suspicion that a “pure” maildir server would struggle to fit them in neatly, although there is some precedent in IMAP keywords and the uiddb. But, ultimately, it comes down to maintaining the MODSEQ value for every message change, and keeping a log of expunged messages. The hard part would, I think, be deciding on a rational way to compute MODSEQ, and in deciding how long to keep the expunge log for returning clients. You can in fact do it with less state than that—only keep the highest MODSEQ and require clients to fall back to discover expunges, and only keep one MODSEQ per message instead of one MODSEQ per metadata change (see section 5)—but this compromises efficiency to some degree. See RFC 4549 for how disconnected client synchronisation operations are proposed, which RFC 7162 updates.

> As I mentioned, client/server resynchronization is a much simpler task. In your poking of Courier-IMAP's code you must've missed the parallel implementation of an alternative client/server protocol, which I called SMAP, that I did, for some research, a long time ago. It's still there, it just needs to be enabled. And Cone implements it on the client side. And one of its aspects is exactly this: efficient client/server resynchronization.

Well, I *did* notice all the #ifdefs … but thanks, I’ll be taking a closer look.

> Resynchronization is a much, much simpler task than the arcane way CONDSTORE+QRESYNC goes about doing it. All that needs to happen is for the server to save the current state, a snapshot of the mailbox, and send an opaque handle to the client, representing the snapshot. The server only needs to save one snapshot per client, and each snapshot simply discards the previous one (well, two most recent snapshots need to be saved, to make this bulletproof).
> 
> When the client opens a mailbox it provides the snapshot handle it knows about to the server, which then loads the saved snapshot, and then proceeds and resynchronizes its state, the same way it normally does, and then update the client using the responses it already does, i.e. EXPUNGE, FLAGS of any messages' changed flags, the number of EXISTting messages itself. 99% of what needs to be done, to resynchronize the client/server, already exists as part of the core protocol so why invent a completely new way of doing that?
> 
> All of that is somewhat analogous to CONDSTORE's sequence numbers. Except that, from what I can tell, the server must maintain a complete history of all changes to the mailbox, in order to support CONDSTORE+QRESYNC. This seems completely unnecessary itself, too onerous, and it misses the mark entirely. The server only needs to maintain a single per-client snapshot (with the next most recent backup snapshot), identified by a single opaque handle.

OK, so I will be looking at it, but could this infrastructure be reused? If you were to permit the saving of an indefinite number of snapshots (up to some specified time limit), without reference to any particular client, and store the snapshots into easily looked-up files in a directory, then whenever a client references a MODSEQ, it would be an easy matter to reply accordingly with the changes. How expensive is creating a snapshot? Is it done by request of the client, or every time there’s a change? After all, if you are OK with keeping this additional state, then perhaps it’s not such a tough battle.

Cheers,
Sabahattin