Isync / Bugs / #59 IMAP error: unexpected FETCH response

#59 IMAP error: unexpected FETCH response

Milestone: 1.4.0-pre

Status: fixed

Owner: Oswald Buddenhagen

Labels: None

Fixed In: 1.4.1

Priority: 4

Updated: 2021-02-21

Created: 2020-10-27

Creator: Sage

Private: No

VERSIONS:
Using mbsync master commit: 2351356
mu version 1.4.13, emacs 27.1, with mu4e
Mailboxes hosted on GMail. (Gmail is set to Auto-Expunge: Off, and "Move message to Trash")

PROCEDURE:
Open Inbox in mu4e
Mark message as seen, then delete message.
Execute mbsync. An error will be generated:

% mbsync personal-all
C: 0/1  B: 0/1  F: +0/0 *0/1 #0/0  N: +0/0 *0/0 #0/0
IMAP error: unexpected FETCH response (UID 472018)
C: 1/1  B: 1/1  F: +0/0 *0/1 #0/0  N: +0/0 *0/0 #0/0

If multiple messages are deleted, mbsync will continue to give errors until an "unexpected FETCH response" is given for each message that was marked seen, then deleted. Sometimes, I also get a "Notice: conflicting changes in (x, y)" message.

% mbsync -Dm work-all work-trash
Reading configuration file /Users/user/.mbsyncrc
Channel work-all
Opening far side store work-remote...
Resolving imap.gmail.com... ok
Connecting to imap.gmail.com (142.250.112.108:993)...
Opening near side store work-local...
Connection is now encrypted
Logging in...
Authenticating with SASL mechanism PLAIN...
Opening far side box [Gmail]/All Mail...
Opening near side box [Gmail].All Mail...
Loading far side box...
Loading near side box...
near side: 152915 messages, 209 recent
Warning: lost track of 5 pulled message(s)
far side: 76511 messages, 0 recent
Synchronizing...
Notice: conflicting changes in (190312,153805)
Notice: conflicting changes in (190314,153806)
Channel work-trash
Opening far side store work-remote...
Opening near side store work-local...
Opening far side box [Gmail]/Trash...
Opening near side box [Gmail].Trash...
Loading far side box...
Loading near side box...
near side: 169 messages, 0 recent
far side: 165 messages, 0 recent
Warning: lost track of 10 pushed message(s)
Synchronizing...
Notice: conflicting changes in (78116,8283)
Notice: conflicting changes in (78117,8284)
IMAP error: unexpected FETCH response (UID 78143)

Relevant Configuration files, below.

.mbsyncrc:

#Relatively safe optimization
FSync no
CopyArrivalDate yes
Sync All
Create Both
Expunge Both

# PERSONAL ACCOUNT

IMAPAccount personal
# Address to connect to
Host imap.gmail.com
User user@personal.ca
UseKeychain yes
# To avoid going over-bandwidth limit on GMail
PipelineDepth 30
SSLType IMAPS
CertificateFile /opt/local/share/curl/curl-ca-bundle.crt

# THEN WE SPECIFY THE LOCAL AND REMOTE STORAGE
# - THE REMOTE STORAGE IS WHERE WE GET THE MAIL FROM (E.G., THE
#   SPECIFICATION OF AN IMAP ACCOUNT)
# - THE LOCAL STORAGE IS WHERE WE STORE THE EMAIL ON OUR COMPUTER

# REMOTE STORAGE (USE THE IMAP ACCOUNT SPECIFIED ABOVE)
IMAPStore personal-remote
Account personal

# LOCAL STORAGE (CREATE DIRECTORIES with mkdir -p Maildir/gmail)
MaildirStore personal-local
Path ~/.mail/personal/
Inbox ~/.mail/personal/Inbox

# CONNECTIONS SPECIFY LINKS BETWEEN REMOTE AND LOCAL FOLDERS
#
# CONNECTIONS ARE SPECIFIED USING PATTERNS, WHICH MATCH REMOTE MAIl
# FOLDERS. SOME COMMONLY USED PATTERS INCLUDE:
#
# 1 "*" TO MATCH EVERYTHING
# 2 "!DIR" TO EXCLUDE "DIR"
# 3 "DIR" TO MATCH DIR

Channel personal-inbox
Far :personal-remote:"INBOX"
Near :personal-local:"Inbox"
SyncState *

Channel personal-trash
Far :personal-remote:"[Gmail]/Trash"
Near :personal-local:"[Gmail].Trash"
SyncState *

Channel personal-sent
Far :personal-remote:"[Gmail]/Sent Mail"
Near :personal-local:"[Gmail].Sent Mail"
SyncState *

Channel personal-all
Far :personal-remote:"[Gmail]/All Mail"
Near :personal-local:"[Gmail].All Mail"
SyncState *

Channel personal-drafts
Far :personal-remote:"[Gmail]/Drafts"
Near :personal-local:"[Gmail].Drafts"
SyncState *

Group personal-gmail
Channel personal-inbox
Channel personal-trash
Channel personal-sent
Channel personal-all
Channel personal-drafts

mu4e config (these are relevant snippets from a larger config):

(setq +mu4e-backend 'mbsync
       mu4e-change-filenames-when-moving t
       mu4e-context-policy `ask
       mu4e-headers-include-related nil
       mu4e-index-cleanup t
       mu4e-index-lazy-check nil
       mu4e-headers-visible-lines 15
       max-specpdl-size 5000
       message-kill-buffer-on-exit t
       mu4e-sent-messages-behavior 'delete)

  (setq mu4e-contexts
           `(,(make-mu4e-context
             :name "Personal"
             :enter-func (lambda () (mu4e-message "Entering Personal Context"))
             :leave-func (lambda () (mu4e-message "Leaving Personal Context"))
             :match-func (lambda (msg)
                       (when msg
                         (or (mu4e-message-contact-field-matches msg :to "user@personal.ca")
                             (mu4e-message-contact-field-matches msg :from "user@personal.ca")
                             (mu4e-message-contact-field-matches msg :cc "user@personal.ca")
                             (mu4e-message-contact-field-matches msg :bcc "user@personal.ca")
                             (string-match-p "/personal" (mu4e-message-field msg :maildir)))))
             :vars '(
                     (user-mail-address . "user@personal.ca")
                     (mu4e-sent-folder      . "/personal/[Gmail].Sent Mail")
                     (mu4e-drafts-folder . "/personal/[Gmail].Drafts")
                     (mu4e-trash-folder     . "/personal/[Gmail].Trash")
                     (mu4e-refile-folder    . "/personal/[Gmail].All Mail")
                     (mu4e-get-mail-command   . "mbsync personal-gmail")
                     (mu4e-maildir-shortcuts .
                           ( ("/personal/Inbox"               . ?i)
                              ("/personal[Gmail].All Mail"   . ?a)
                              ("/personal[Gmail].Sent Mail"   . ?s)
                              ("/personal/[Gmail].Trash"       . ?t)
                              ("/personal[Gmail].Drafts"    . ?d)))
                     (smtpmail-smtp-user     . "user@personal.ca")
                     (query-prefix              . "maildir:/personal/.*/ AND ")
                     (mu4e-compose-signature .
                                             (concat
                                              "Patrick Toal\n"
                                              "user@personal.ca"
                                              ))
                     (mu4e-bookmarks    .       (( :query "maildir:/personal/Inbox" :name "Inbox" :key ?i)
                                                  ( :query (concat query-prefix "flag:flagged") :name "Flagged messages" :key ?f)
                                                  ( :query (concat query-prefix "flag:unread AND"
                                                                    "(NOT maildir:/[Gmail].Spam) AND "
                                                                    "(NOT maildir:/[Gmail].Trash")
                                                    :name "Unread messages" :key ?u))))

Discussion

Oswald Buddenhagen - 2020-10-27

i think you meant to use -Dn instead of -Dm (plain -D would also work, and is generally preferable unless you know what you're doing).

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Sage - 2020-10-28

Ah, sorry. I already downgraded back to 1.3.

My mailbox is enormous, and the logs have hundreds of thousands of lines, so I didn't think you wanted to see the full log. I do have one log that I captured with full debugging. It's 285,307 lines long. I have attached it, in case it helps.

mbsync.log.gz

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Oswald Buddenhagen - 2020-10-28

well, to debug imap errors i need to see the corresponding network traffic. ;)
the log isn't too helpful, because it doesn't contain the error in question. i'm afraid this won't go anywhere unless you re-upgrade (just keep a copy of the old config and folders, to make downgrading trivial).

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Sage - 2020-10-28

Understood. What I'll do is create a VM in my lab, a new Gmail account for testing, and reproduce. Do you want wireshark traces as well?

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Oswald Buddenhagen - 2020-10-28

sounds excessively labor-intensive to me, given that the client side is trivial to sandbox (by just using a copy) and the server side didn't appear negatively impacted anyway. but on the upside, if you create a test case with a minimal number of messages and reproducible instructions from scratch, that might be worth something in itself (depending on how tricky the problem turns out to be).

a wireshark trace won't help in this case, as you'll see only encrypted ssl traffic, while the problem is clearly in the application layer.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Sage - 2020-10-28

Ah. Fair enough. I misunderstood your "see the network traffic" comment. Of course it would be useless with the TLS.

I'll set up a simple test case with a fresh mailbox, and see if I can reproduce. Since I downgraded to 1.3, I haven't had any errors, but I still get the occasional warning that messages have changed on both sides. My guess is because I'm changing the flags (unread to seen) at the same time I'm refiling or trashing messages.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Oswald Buddenhagen - 2020-10-28

that guess is certainly correct, though usually that would happen only if you did one of these things on the server and the other on the client. but you have server-side overlap between All Mail and the other mailboxes, which mbsync doesn't know, so it thinks someone else is doing the things it's actually doing itself. that's in fact quite possibly the trigger for the real problem, even if it really shouldn't.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Oswald Buddenhagen - 2021-02-03

status: reported --> need-more-info
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Hi! I may also be affected by this.

My mbsync configuration:

IMAPAccount gmail
CertificateFile /etc/ssl/certs/ca-certificates.crt
Host imap.gmail.com
PassCmd "/nix/store/0lf7wcl0spladqlcypn8zvaysld13ba8-gnupg-2.2.27/bin/gpg -q --for-your-eyes-only --no-tty --exit-on-status-write-error -d /home/dominik/.config/nixpkgs/config/gmail.password.gpg"
SSLType IMAPS
User dominik.schrempf@gmail.com

IMAPStore gmail-remote
Account gmail

MaildirStore gmail-local
Inbox /home/dominik/Maildir/gmail/Inbox
Path /home/dominik/Maildir/gmail/
SubFolders Verbatim

Channel gmail
Create Both
Expunge Both
Master :gmail-remote:
Patterns INBOX Linux University "[Gmail]/Sent Mail"
Remove None
Slave :gmail-local:
SyncState *

I attach a log file that stops at IMAP error: unexpected FETCH response (UID 93148)

Let me know if I can provide more information!

Thank you!

EDIT: My isync version is:

+ mbsync --version
isync 1.4.0

Last edit: Dominik Schrempf 2021-02-08

error.log

Jan Palus - 2021-02-14

For me this is 100% reproducible with FastMail server:

Ensure new email is received not tracked by isync

Invoke mbsync to fetch new message

Open mail client to read it

Invoke mbsync again to mark mail as read on remote side

Last step always results in "unexpected FETCH response". Next invocation always succeeds.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Oswald Buddenhagen - 2021-02-14

status: need-more-info --> open

assigned_to: Oswald Buddenhagen
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Oswald Buddenhagen - 2021-02-14

i see. the servers are being a bit funny, but within the limits of what's permitted.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Oswald Buddenhagen - 2021-02-14

status: open --> fixed

Fixed In: unknown --> 1.4.1
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Dominik Schrempf - 2021-02-16

Thank you! I can confirm that 1.4.1 works in my case!

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Jan Palus - 2021-02-17

Any chance for 1.4.1 release? With multiple new messages in multiple folders (mailing lists, build logs etc) it takes isync quite a while to synchronize fully when invoked with interval of 10min.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Evan Farrar - 2021-02-19

+1, built using origin/1.4 and I am no longer seeing this issue.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Emin Martinian - 2021-02-21

I had this same problem, clone and built from source and the problem went away.

Thanks for this great software and having clear instructions on the README for a clean build.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

IMAP error: unexpected FETCH response

mailbox synchronizer

Affected

Fixed In

Searches

Help

#59 IMAP error: unexpected FETCH response

Discussion