isync-devel

"lost track of <n> pulled message(s)" and other warnings

[Sean H. <isy...@sn...>]

Hi, I'm using mbsync to pull a lot of email (several GB) from my gmail
account to a local maildir. It seems to be working fine, but it has
printed a number of warnings and I wanted to check that these are safe
to ignore? mbsync has stopped mid-pull with these messages:

"IMAP error: unexpected BYE response: [OVERQUOTA] Account exceeded
command or bandwidth limits." After this one I tried using the program
`trickle` to throttle mbsync's download speed, but that caused mbsync to
freeze (using no cpu or bandwidth, not giving any output) for a few
hours until I killed it. I've now put `PipelineDepth 1` in my .mbsyncrc
instead, and that seems to work just fine.

"IMAP error: unexpected BYE response: System Error" mbsync sometimes
exits with this error, it seems to continue working again fine if I just
start the command again, and I'm happy to run it in a while loop.

"Warning: lost track of 57473 pulled message(s)" is printed whenever I
restart mbsync after one of the above messages stops it, or after a
severed network connection or a ctrl-c from me stops it  (the number is
different every time).

I wanted to check whether it's ok for mbsync to be ctrl-c'd and to stop
with the above errors and just be restarted again? It won't miss any
emails, or leave any in a partial state, etc?

Re: "lost track of <n> pulled message(s)" and other warnings

[Oswald B. <osw...@gm...>]

On Sat, Oct 29, 2016 at 05:10:49PM +0100, Sean Hammond wrote:
> Hi, I'm using mbsync to pull a lot of email (several GB) from my gmail
> account to a local maildir. It seems to be working fine, but it has
> printed a number of warnings and I wanted to check that these are safe
> to ignore? mbsync has stopped mid-pull with these messages:
> 
> "IMAP error: unexpected BYE response: [OVERQUOTA] Account exceeded
> command or bandwidth limits."
>
that seems quite expected given your usage.

> After this one I tried using the program `trickle` to throttle
> mbsync's download speed, but that caused mbsync to freeze (using no
> cpu or bandwidth, not giving any output) for a few hours until I
> killed it.
>
try the master branch - it has proper timeout handling.

> I've now put `PipelineDepth 1` in my .mbsyncrc instead, and that seems
> to work just fine.
> 
a less drastical limit like 50 may still work, and would be considerably
faster.

> "IMAP error: unexpected BYE response: System Error" mbsync sometimes
> exits with this error, it seems to continue working again fine if I just
> start the command again, and I'm happy to run it in a while loop.
> 
i'm fairly sure google finds a nice stack trace in their logs every
time this happens. ^^

> "Warning: lost track of 57473 pulled message(s)" is printed whenever I
> restart mbsync after one of the above messages stops it, or after a
> severed network connection or a ctrl-c from me stops it  (the number
> is different every time).
> 
the worst that can happen is that you get some duplicate mails if the
loss of tracking has profound reasons (we actually found a few mbsync
and server bugs this way). usually, it just means as much as "the
connection was interrupted. you wasted some bandwidth."

> I wanted to check whether it's ok for mbsync to be ctrl-c'd and to stop
> with the above errors and just be restarted again? It won't miss any
> emails, or leave any in a partial state, etc?
> 
it's just fine.

Re: "lost track of <n> pulled message(s)" and other warnings

[Sean H. <isy...@sn...>]

On Mon, 31 Oct 2016, at 08:26 AM, Oswald Buddenhagen wrote:
> On Sat, Oct 29, 2016 at 05:10:49PM +0100, Sean Hammond wrote:
> > I've now put `PipelineDepth 1` in my .mbsyncrc instead, and that seems
> > to work just fine.
> > 
> a less drastical limit like 50 may still work, and would be considerably
> faster.
> 
> > I wanted to check whether it's ok for mbsync to be ctrl-c'd and to stop
> > with the above errors and just be restarted again? It won't miss any
> > emails, or leave any in a partial state, etc?
> > 
> it's just fine.

Thanks, good to know. I've put my PipelineDepth up to 50, it's going
much faster (and seems like it must surely be exceeding Gmail's stated
IMAP bandwidth limit of 2500MB per day, but so far Gmail hasn't
complained that I've exceeded my quota again).

Seeing lots of this error message now:

Warning: message <n> from master has incomplete header.

And I've also seen this one:

Error decompressing data from imap.gmail.com (64.233.166.108:993):
(null)

I assume there's aren't anything to worry about either?

Re: "lost track of <n> pulled message(s)" and other warnings

[Oswald B. <osw...@gm...>]

On Mon, Oct 31, 2016 at 07:21:05PM +0000, Sean Hammond wrote:
> Seeing lots of this error message now:
> 
> Warning: message <n> from master has incomplete header.
> 
that indicates an invalid message that was cut before the end of the
headers (the body can be empty, but should be preceded by an empty line).
you better find out where these come from.

> And I've also seen this one:
> 
> Error decompressing data from imap.gmail.com (64.233.166.108:993):
> (null)
> 
hmpf. that possibly indicates a bug in mbsync. otoh, given that google
was also throwing internal server errors, i wouldn't be surprised if
their tls compression was botched, either.

this could be worked around by using DisableExtension COMPRESS=DEFLATE,
but that would come with a rather huge bandwidth cost, so it's probably
better to just let the connection break down from time to time.

> I assume there's aren't anything to worry about either?
> 
well, data loss is unlikely, but it's still better to find out the root
causes.