#61 malformed PERMANENTFLAGS status when syncing specific folders

[comatory]

Hello. I am newbie when it comes to using command line interfaces with email so forgive my ignorance.
I have set up isync on my macOS machine. My email provider is fastmail, however when attempting to sync mail via imap, I get following error:

IMAP error: malformed PERMANENTFLAGS status
IMAP error: bogus FETCH response

I assume that this might not be related to isync at all and it can be with my email provider. However I was hoping that you could point me at some direction. This is my .mbsyncrc file:

IMAPAccount personal
Host imap.fastmail.com
User username@fastmail.com
Pass password
SSLType IMAPS
SSLVersions TLSv1.2
Port 993
CertificateFile /usr/local/etc/openssl@1.1/cert.pem

IMAPStore personal-remote
Account personal

MaildirStore personal-local
SubFolders Verbatim
Path ~/mail/personal/
Inbox ~/mail/personal/INBOX

Channel personal
CopyArrivalDate yes
Far :personal-remote:
Near :personal-local:
Patterns "*"
Sync All
Create Both
Expunge None
SyncState *

I am using version 1.4.0.

It seems to get stuck at specific folders, some of them get synced and I can see them on my local machine.

[Oswald Buddenhagen]

please re-run with the -Dn option and attach the log. it's sufficient if you do this for a single affected folder.

[comatory]

@ossi hope this helps:

Reading configuration file /Users/username/.mbsyncrc
Channel personal
Opening far side store personal-remote...
Resolving imap.fastmail.com... ok
Connecting to imap.fastmail.com (66.111.4.136:993)...
Opening near side store personal-local...
Connection is now encrypted

* OK IMAP4 ready
>>> 1 CAPABILITY
* CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ENABLE UIDPLUS SASL-IR NAMESPACE CONDSTORE SORT LIST-EXTENDED QRESYNC MOVE SPECIAL-USE CREATE-SPECIAL-USE IDLE AUTH=PLAIN
1 OK completed
Logging in...
Authenticating with SASL mechanism PLAIN...
>>> 2 AUTHENTICATE PLAIN <authdata>
2 OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE ACL ANNOTATE-EXPERIMENT-1 BINARY CATENATE CHILDREN CONDSTORE CREATE-SPECIAL-USE ESEARCH ESORT LIST-EXTENDED LIST-MYRIGHTS LIST-STATUS MAILBOX-REFERRALS METADATA MOVE MULTIAPPEND NAMESPACE OBJECTID PREVIEW QRESYNC QUOTA RIGHTS=kxten SAVEDATE SEARCH=FUZZY SORT SORT=DISPLAY SPECIAL-USE STATUS=SIZE THREAD=ORDEREDSUBJECT THREAD=REFERENCES UIDPLUS UNSELECT WITHIN DIGEST=SHA1 LIST-METADATA NO_ATOMIC_RENAME SCAN SORT=MODSEQ SORT=UID THREAD=REFS X-CREATEDMODSEQ X-REPLICATION XLIST XMOVE XAPPLEPUSHSERVICE LOGINDISABLED XCONVERSATIONS COMPRESS=DEFLATE X-QUOTA=STORAGE X-QUOTA=MESSAGE X-QUOTA=X-ANNOTATION-STORAGE X-QUOTA=X-NUM-FOLDERS IDLE APPENDLIMIT=4294967295] User logged in SESSIONID=<sloti1d3t14-1613332069-3044647-1-13365583335546422966>
>>> 3 COMPRESS DEFLATE
3 OK DEFLATE active
>>> 4 NAMESPACE
* NAMESPACE (("" "/")) (("Other Users/" "/")) (("Other Folders/" "/"))
4 OK Completed
>>> 5 LIST "" "*"
* LIST (\HasNoChildren) "/" INBOX
* LIST (\HasNoChildren \Archive) "/" Archive
* LIST (\HasNoChildren \Drafts) "/" Drafts
* LIST (\HasNoChildren \Sent) "/" Sent
* LIST (\HasNoChildren \Junk) "/" Spam
* LIST (\HasNoChildren \Trash) "/" Trash
* LIST (\HasNoChildren) "/" Zofka
* LIST (\HasNoChildren) "/" apartment
* LIST (\HasNoChildren) "/" doctor
* LIST (\HasNoChildren) "/" finance
* LIST (\HasNoChildren) "/" games
* LIST (\HasNoChildren) "/" personal
* LIST (\HasNoChildren) "/" problems
* LIST (\HasNoChildren) "/" programming
* LIST (\HasNoChildren) "/" registrations
* LIST (\HasNoChildren) "/" school
* LIST (\HasNoChildren) "/" shop
* LIST (\HasNoChildren) "/" skytospeak
* LIST (\HasNoChildren) "/" work
5 OK Completed (0.001 secs 32 calls)
Opening far side box doctor...
>>> 6 SELECT "doctor"
Opening near side box doctor...
* 27 EXISTS
* 0 RECENT
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen $X-ME-Annot-2 $IsMailingList $IsNotification $HasAttachment $HasTD [ARCHIVED] [EXPUNGED])
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen $X-ME-Annot-2 $IsMailingList $IsNotification $HasAttachment $HasTD [ARCHIVED] [EXPUNGED] \*)] Ok
IMAP error: malformed PERMANENTFLAGS status
IMAP error: malformed LIST response

[comatory]

Also not sure if this is related, I actually wanted to file this as a possible bug as well but some of my emails that I have synced seemed to have mangled my name in "Sent" folder. The format of the "FROM" field is specified like this: "FirstName LastName firstnamelastname@email.com".
However my name actually contains accented characters so the downloaded files actually contain something like this: =?utf-8?b?T25kxZllaiBTeW7DocSNZWs=?= <firstnamelastname@email.com>. Which is not great as my email viewer is not able to render this correctly.

As I mentioned, not sure if that can relate to the problem above, some possible encoding issue.

[Oswald Buddenhagen]

from https://tools.ietf.org/html/rfc3501#section-9:

resp-text       = ["[" resp-text-code "]" SP] text
resp-text-code  = [...] /
                  "PERMANENTFLAGS" SP "("
                  [flag-perm *(SP flag-perm)] ")" /
                  [...]
flag-perm       = flag / "\*"
flag            = [...] / flag-keyword / flag-extension
flag-extension  = "\" atom
flag-keyword    = atom
atom            = 1*ATOM-CHAR
ATOM-CHAR       = <any CHAR except atom-specials>
atom-specials   = [...] / resp-specials
resp-specials   = "]"

which means that the unquoted closing brackets in the keywords are illegal.
by extension this means that closing brackets in keywords are illegal altogether, as the above syntax has no provisions for quoting. that's certainly just an omission of the spec, but still.

i think i can work around this somewhat easily, but technically speaking that server is broken and you should report it to them.

[Oswald Buddenhagen]

the encoding problem is unrelated. the protocol doesn't change or even care for the mail contents in any way. check if the outgoing messages are compliant with https://tools.ietf.org/html/rfc5322.

[comatory]

Great! I will be checking new release.

the encoding problem is unrelated. the protocol doesn't change or even care for the mail contents in any way. check if the outgoing messages are compliant with https://tools.ietf.org/html/rfc5322.

I've actually found out it's one of my email clients that does this. Raw emails downloaded via web interface of my email provider have the same weird header value so you're right, not definitely and issue with isync.

Thanks again for help.

[comatory]

@ossi One more thing. I've tried pulling 1.4 branch, read the instructions for installation. I wanted to check out if it works with your changes.

However did something change between 1.4.0 and what is currently in that branch as far as configuration goes?

I've got error saying that SSLType and SSLVersions is not supported within IMAPAccount section.

[Oswald Buddenhagen]

fairly safe bet: openssl was not found. make sure you have the -dev package installed.

[comatory]

By that package do you mean openssl-dev? I'm on macOS so I tried searching for openssl-dev / openssl-devel but no luck. The requirements in README file specify open ssl as optional but do not mention the dev package.

This is the output:

/Users/username/.mbsyncrc:5: keyword 'SSLType' is not recognized in IMAPAccount sections
/Users/username/.mbsyncrc:6: keyword 'SSLVersions' is not recognized in IMAPAccount sections
/Users/username/.mbsyncrc:8: keyword 'CertificateFile' is not recognized in IMAPAccount sections

When running ./configure I get

checking OpenSSL presence with pkg-config... not found

So I guess that is potential culprit?

[comatory]

OK got it and confirm your fix does indeed work. I managed to compile on my macOS system by doing following

1. Have homebrew package installed
2. Replace macOS's openssl version with provided one from hombrew: brew install openssl && brew link openssl --force
3. If you get console warning that you need to add it to your PATH, do that, link openssl --force again and reload terminal session.
4. Then copy the files like this: cp /usr/local/opt/openssl@1.1/lib/pkgconfig/*.pc /usr/local/lib/pkgconfig/. I found it here
5. Follow README to recompile and install again.