#642 DavMail doesn't sanitize illegal characters (RFC 3501 atom-specials) when converting Outlook categories into IMAP keywords

[Brett Keller]

Environment where bug was first noticed:
DavMail 4.7.3 installed from DEB file on Ubuntu 16.04
client1 = Thunderbird 45.7.0 connected to DavMail IMAP on localhost
server = Exchange 2003
client2 = Outlook 2010 on Windows connected to Exchange server

Using client2, an Outlook category named "Neuro (resolved/needs preparation)" had been created and applied to several messages in the INBOX.

DavMail was installed on client1, Thunderbird was configured, and e-mail successfully began downloading through DavMail for all folders except INBOX. Not a single message appeared in the INBOX, and each attempt to sync was abandoned by Thunderbird after about 32 seconds. Adjusting timeout values did not change anything.

After turning on Thunderbird debug logging, I was able to see that Thunderbird was choking on parsing invalid IMAP keywords that corresponded to the aforementioned particular Outlook category. Even though Thunderbird had been able to parse every message up to this point, the first message that it hit that caused the error also caused the entire initial folder sync to abort, so not a single message in the INBOX was visible in the Thunderbird client.

I then logged directly into OWA, created a new folder, and moved all of the messages tagged with this problematic category out of the INBOX and into the new folder. I asked Thunderbird to attempt an initial sync with the INBOX again, and this time it completed without error and all INBOX messages were visible.

Attempting to sync with the newly-created folder with the problem messages failed in the same way that the INBOX had previously been failing.

Attached is a Thunderbird debug log that shows an attempt to sync with the quarantining folder. Notice where Thunderbird aborts the check with "PARSER:Internal Syntax Error on line: %s: * 2 FETCH (UID 30 FLAGS (\Seen \Flagged $Forwarded Neuro (resolved/needs preparation) Neuro))".

Looking at RFC 3501 for IMAP, it seems that the parentheses are illegal characters (a.k.a., "atom-specials") within IMAP keywords, so I'm guessing that this is why Thunderbird chokes on these messages. Incidentally, it also looks like the space character is listed as illegal, but I'm guessing TB doesn't choke on it because it misinterprets it as a keyword delimiter, which is a different problem.

DavMail should escape, substitute, or otherwise sanitize these illegal characters when translating Outlook categories into IMAP keywords. All it takes is a single bug-triggering message to prevent an entire folder of messages from being downloaded.

If necessary, I can also provide a DavMail WIRE debug log, but I cannot post it publicly as it contains sensitive information.

Please let me know if I can provide any further info, and thanks for writing this wonderfully useful piece of software!

[Mickael Guessant]

As a workaround, you can try to map Outlook categories to IMAP flags, see:
https://sourceforge.net/p/davmail/discussion/644057/thread/586e2539/

[Brett Keller]

Thanks! This allowed me to successfully workaround the issue and download the folder with the problematic messages.

In addition to the davmail.imapFlags... settings, I also manually configured matching tags in Thunderbird, and the Outlook categories are now mapped to tags perfectly! The only thing that doesn't seem to work is that it is unidirectional: Outlook categories show up as Thunderbird tags just fine, but when I tried setting the same tags on new messages in Thunderbird, they didn't show up in OWA. I was unable to test on the Outlook 2010 client side of things, so maybe it works there. Considering that version of OWA is from Exchange 2003, and that it's own handling of Outlook 2010 categories is a little odd, I'm not so sure I can say it's a valid test. Anyway, syncing tags/categories in the other direction isn't particularly important to me.

I couldn't find any reference to davmail.imapFlags... in the documentation. Given how useful it is, I'd like to request that you consider adding it to your FAQ, please.

As to a more automatic (for the end user) fix for this sanitizing IMAP keywords bug, I think I've come to understand the problem a little bit better and can appreciate that it's not an easy fix. My original thought was that you could simply replace all atom-specials with underscores, but I realize now that this would be a lossy conversion that would break any chance of tag-to-category mapping being bidirectional.

Perhaps a multi-character substitution for each atom-special would work to prevent the breakage I described, yet still maintain bidirectionality? I'm thinking of something similar to what you're already doing with IMAP folder names. For example, I see the Outlook folder named "Old requests/orders" show up in Thunderbird as "Old requests_xF8FF_orders", which while a bit of an eyesore, at least removes the invalid "/" character while maintaining what appears to be a one-to-one mapping back to the Outlook folder. Does this sound plausible?

[Mickael Guessant]

It it supposed to be bidirectionnal, please check davmail.log to get more info on actual commands (you should see IMAP flags coming from the client, and the translated version in WIRE DEBUG log)

I must admit the documentation is missing for davmail.imapFlags, my only excuse is that it's a user provided patch ;-)
Anyway, can you please provide a simple text that I could include in DavMail documentation, as you just tried it ?

The special character translation looks like an interesting workaround. Just for your information: the xF8FF trick was provided by Exchange, I didn't create it.

Thanks for your feedback.

[Brett Keller]

Sorry I couldn't get around to your request sooner. It looks like you've beaten me to writing up the documentation for imapFlags. It looks good! Thanks for including it.

[Mickael Guessant]

Well, actually I asked the initial patch submitter to write a little bit of doc :-)