fetchmail-users

[Fetchmail-users] mail expunge mismatch

[Hans C. <fo...@gm...>]

I'm using fetchmail-6.4.4.

For about a year now I've been seeing the following in my log file for 
mail coming from yahoo (imap.mail.yahoo.com).  This happens regardless of 
the user.

   fetchmail: 1 message for USER at SERVERNAME.
   fetchmail: reading message US...@in...:1 of 1 (3733 header octets) (29948 body octets) flushed
   fetchmail: mail expunge mismatch (0 actual != 1 expected)
   fetchmail: client/server synchronization error while fetching from USER@SERVERNAME
   fetchmail: Query status=7 (ERROR)

The main problem with this, is the connection is dropped each time it 
happens and it happens for every message.  That means if there are 10 
messages waiting, fetchmail needs to poll this server 10 times before all 
the messages are retrieved.

Here's my config entry for this user:

   poll SERVERNAME via imap.mail.yahoo.com protocol IMAP service 993
     user 'REMOTE_USER' there password 'PASSWORD' is 'LOCAL_USER' here;

I did a debug session and from what I can tell the problem appears to be 
the response from the imap EXPUNGE command.  It seems to be missing some 
information?  (at least based on the response from another server that 
does work - see below).

   # This is imap.mail.yahoo.com
   $ fetchmail -vvv -d0 --nodetach --nosyslog SERVERNAME
   ...
   fetchmail: IMAP> A0009 STORE 1 +FLAGS (\Seen \Deleted)
   fetchmail: IMAP< * 1 FETCH (FLAGS (\Deleted \Seen $NotJunk))
   fetchmail: IMAP< * 1 EXPUNGE
   fetchmail: IMAP< * 0 EXISTS
   fetchmail: IMAP< A0009 OK STORE completed
   fetchmail: IMAP> A0010 EXPUNGE
   fetchmail: IMAP< A0010 OK EXPUNGE completed
   fetchmail: mail expunge mismatch (0 actual != 1 expected)
   fetchmail: IMAP> A0011 LOGOUT
   fetchmail: IMAP< * BYE IMAP4rev1 Server logging out
   fetchmail: IMAP< A0011 OK LOGOUT completed

If I try an account on a different server that functions correctly 
(imap.gmx.com), the response from EXPUNGE shows what I assume fetchmail is 
expecting to see.  BUT comparing the yahoo response to the gmx response it 
appears yahoo may be sending that information as part of the STORE 
response?

   # This is imap.gmx.com
   $ fetchmail -vvv -d0 --nodetach --nosyslog SERVERNAME
   ...
   fetchmail: IMAP> A0008 STORE 1 +FLAGS (\Seen \Deleted)
   fetchmail: IMAP< * 1 FETCH (FLAGS (\Recent \Deleted \Seen))
   fetchmail: IMAP< A0008 OK STORE completed
   fetchmail: IMAP> A0009 EXPUNGE
   fetchmail: IMAP< * 1 EXPUNGE
   fetchmail: IMAP< * 0 EXISTS
   fetchmail: IMAP< * 0 RECENT
   fetchmail: IMAP< A0009 OK EXPUNGE completed
   fetchmail: selecting or re-polling default folder
   fetchmail: 0 messages waiting after re-poll
   fetchmail: IMAP> A0010 LOGOUT
   fetchmail: IMAP< * BYE Server logging out
   fetchmail: IMAP< A0010 OK LOGOUT completed

So... is there any way around this issue?

Re: [Fetchmail-users] mail expunge mismatch

[Matthias A. <mat...@gm...>]

Am 06.05.20 um 20:14 schrieb Hans Carlson:
> I'm using fetchmail-6.4.4.
>
> For about a year now I've been seeing the following in my log file for
> mail coming from yahoo (imap.mail.yahoo.com).  This happens regardless
> of the user.

Hans,

Yahoo's mail server is broken, and your trace proves it. Please report
this to Yahoo (but don't ask my how, I don't know their contacts).
Details below.


I did a debug session and from what I can tell the problem appears to be
the response from the imap EXPUNGE command.  It seems to be missing some
information?  (at least based on the response from another server that
does work - see below).


The Yahoo IMAP server is expunging prematurely, at a time when it is not
allowed to. This is a server behaviour (malfunction) that is EXPLICITLY
FORBIDDEN by the IMAP4r1 standard (RFC3501) page 73 section 7.4.1
EXPUNGE Response:

      An EXPUNGE response MUST NOT be sent when no command is in
      progress, nor while responding to a FETCH, STORE, or SEARCH
      command.  This rule is necessary to prevent a loss of
      synchronization of message sequence numbers between client and
      server. 


>   # This is imap.mail.yahoo.com
>   $ fetchmail -vvv -d0 --nodetach --nosyslog SERVERNAME
>   ...
>   fetchmail: IMAP> A0009 STORE 1 +FLAGS (\Seen \Deleted)
>   fetchmail: IMAP< * 1 FETCH (FLAGS (\Deleted \Seen $NotJunk))
>   fetchmail: IMAP< * 1 EXPUNGE
>   fetchmail: IMAP< * 0 EXISTS
>   fetchmail: IMAP< A0009 OK STORE completed
>   fetchmail: IMAP> A0010 EXPUNGE
>   fetchmail: IMAP< A0010 OK EXPUNGE completed
>   fetchmail: mail expunge mismatch (0 actual != 1 expected)
>   fetchmail: IMAP> A0011 LOGOUT
>   fetchmail: IMAP< * BYE IMAP4rev1 Server logging out
>   fetchmail: IMAP< A0011 OK LOGOUT completed
>
> If I try an account on a different server that functions correctly
> (imap.gmx.com), the response from EXPUNGE shows what I assume
> fetchmail is expecting to see.  BUT comparing the yahoo response to
> the gmx response it appears yahoo may be sending that information as
> part of the STORE response?
>
>   # This is imap.gmx.com
>   $ fetchmail -vvv -d0 --nodetach --nosyslog SERVERNAME
>   ...
>   fetchmail: IMAP> A0008 STORE 1 +FLAGS (\Seen \Deleted)
>   fetchmail: IMAP< * 1 FETCH (FLAGS (\Recent \Deleted \Seen))
>   fetchmail: IMAP< A0008 OK STORE completed
>   fetchmail: IMAP> A0009 EXPUNGE
>   fetchmail: IMAP< * 1 EXPUNGE
>   fetchmail: IMAP< * 0 EXISTS
>   fetchmail: IMAP< * 0 RECENT
>   fetchmail: IMAP< A0009 OK EXPUNGE completed
>   fetchmail: selecting or re-polling default folder
>   fetchmail: 0 messages waiting after re-poll
>   fetchmail: IMAP> A0010 LOGOUT
>   fetchmail: IMAP< * BYE Server logging out
>   fetchmail: IMAP< A0010 OK LOGOUT completed
>
> So... is there any way around this issue? 

I will not add new code that works around MUST-NOT violations of a server.

You may get away with avoiding to mark messages deleted (--keep), but
then you cannot use --all.

HTH

Regards,
Matthias

Re: [Fetchmail-users] mail expunge mismatch

[Matthias A. <mat...@gm...>]

Am 07.05.20 um 19:54 schrieb Matthias Andree:
> Am 06.05.20 um 20:14 schrieb Hans Carlson:
Sorry - this message has botched citation levels. I'll resend.

Re: [Fetchmail-users] mail expunge mismatch

[Matthias A. <mat...@gm...>]

[re-send with fixed quotation markers]

Am 06.05.20 um 20:14 schrieb Hans Carlson:

> I'm using fetchmail-6.4.4.
>
> For about a year now I've been seeing the following in my log file for
> mail coming from yahoo (imap.mail.yahoo.com).  This happens regardless
> of the user.

Hans,

Yahoo's mail server is broken, and your trace proves it. Please report
this to Yahoo (but don't ask my how, I don't know their contacts).
Details below.


> I did a debug session and from what I can tell the problem appears to be
> the response from the imap EXPUNGE command.  It seems to be missing some
> information?  (at least based on the response from another server that
> does work - see below).


The Yahoo IMAP server is expunging prematurely, at a time when it is not
allowed to. This is a server behaviour (malfunction) that is EXPLICITLY
FORBIDDEN by the IMAP4r1 standard (RFC3501) page 73 section 7.4.1
EXPUNGE Response:

An EXPUNGE response MUST NOT be sent when no command is in
progress, nor while responding to a FETCH, STORE, or SEARCH
command. This rule is necessary to prevent a loss of
synchronization of message sequence numbers between client and
server.

>   # This is imap.mail.yahoo.com
>   $ fetchmail -vvv -d0 --nodetach --nosyslog SERVERNAME
>   ...
>   fetchmail: IMAP> A0009 STORE 1 +FLAGS (\Seen \Deleted)
>   fetchmail: IMAP< * 1 FETCH (FLAGS (\Deleted \Seen $NotJunk))
>   fetchmail: IMAP< * 1 EXPUNGE
>   fetchmail: IMAP< * 0 EXISTS
>   fetchmail: IMAP< A0009 OK STORE completed
>   fetchmail: IMAP> A0010 EXPUNGE
>   fetchmail: IMAP< A0010 OK EXPUNGE completed
>   fetchmail: mail expunge mismatch (0 actual != 1 expected)
>   fetchmail: IMAP> A0011 LOGOUT
>   fetchmail: IMAP< * BYE IMAP4rev1 Server logging out
>   fetchmail: IMAP< A0011 OK LOGOUT completed
>
> If I try an account on a different server that functions correctly
> (imap.gmx.com), the response from EXPUNGE shows what I assume
> fetchmail is expecting to see.  BUT comparing the yahoo response to
> the gmx response it appears yahoo may be sending that information as
> part of the STORE response?
>
>   # This is imap.gmx.com
>   $ fetchmail -vvv -d0 --nodetach --nosyslog SERVERNAME
>   ...
>   fetchmail: IMAP> A0008 STORE 1 +FLAGS (\Seen \Deleted)
>   fetchmail: IMAP< * 1 FETCH (FLAGS (\Recent \Deleted \Seen))
>   fetchmail: IMAP< A0008 OK STORE completed
>   fetchmail: IMAP> A0009 EXPUNGE
>   fetchmail: IMAP< * 1 EXPUNGE
>   fetchmail: IMAP< * 0 EXISTS
>   fetchmail: IMAP< * 0 RECENT
>   fetchmail: IMAP< A0009 OK EXPUNGE completed
>   fetchmail: selecting or re-polling default folder
>   fetchmail: 0 messages waiting after re-poll
>   fetchmail: IMAP> A0010 LOGOUT
>   fetchmail: IMAP< * BYE Server logging out
>   fetchmail: IMAP< A0010 OK LOGOUT completed
>
> So... is there any way around this issue?

I will not add new code that works around MUST-NOT violations of a server.

You may get away with avoiding to mark messages deleted (--keep), but
then you cannot use --all.

HTH

Regards,
Matthias