Re: [fetchmail-users] Problem with Demon's new Microsoft Exchange Server 2010 service

[John C. <jr...@sk...>]

On Mon, 07 May 2012 19:26:27 +0100, Matthias Andree  
<mat...@gm...> wrote:
> Am 07.05.2012 12:10, schrieb John Connett:
>> Operating system:
>>     $ cat /etc/SuSE-release
>>     openSUSE 12.1 (x86_64)
>>     VERSION = 12.1
>>     CODENAME = Asparagus
>>
>> Name and origin of the RPM:
>>     $ rpm -q fetchmail
>>     fetchmail-6.3.21-4.1.3.x86_64
>>
>> The name and version of the SMTP listener:
>>     $ rpm -q exim
>>     exim-4.75-5.1.3.x86_64
>>
>> Command-line options you used:
>>     openSUSE 12.1 distributed /etc/inid.d/fetchmail with added "-vvv".
>>
>> /etc/fetchmailrc (with names changed to protect the innocent):
>>     $ cat /etc/fetchmailrc
>>     # Configuration created Thu Apr 26 22:58:13 2012
>>     set bouncemail
>>     set properties ""
>>     set daemon 300
>>     poll mail.demon.co.uk proto IMAP
>>            localdomains yyy.demon.co.uk
>>            user 'adm...@yy...' there password 'secret'
>> is xxx here
>>            sslproto TLS1
>>            sslcertck
>>     poll mail.demon.co.uk proto IMAP
>>            localdomains yyy.demon.co.uk
>>            user 'xx...@yy...' there password 'secret' is xxx  
>> here
>>            sslproto TLS1
>>            sslcertck
>>
>> Problem:
>> I have recently been migrated to Demon's new service which appears to be
>> based on Microsoft Exchange Server 2010.  The migration appeared to have
>> been successful with messages being fetched.  Then fetching of messages
>> ceased.  I added "-vvv" to /etc/init.d/fetchmail and restarted
>> fetchmail.  Here are edited highlights from /var/log/fetchmail:
>>
>> fetchmail: IMAP< A0004 NO AUTHENTICATE failed.
>> fetchmail: IMAP> A0005 LOGIN "xx...@yy..." *
>> fetchmail: IMAP< A0005 OK LOGIN completed.
>> fetchmail: selecting or re-polling default folder
>> fetchmail: IMAP> A0006 SELECT "INBOX"
>> fetchmail: IMAP< * 264 EXISTS
>> fetchmail: IMAP< * 1 RECENT
>> fetchmail: IMAP< * FLAGS (\Seen \Answered \Flagged \Deleted \Draft
>> $MDNSent)
>> fetchmail: IMAP< * OK [PERMANENTFLAGS (\Seen \Answered \Flagged \Deleted
>> \Draft $MDNSent)] Permanent flags
>> fetchmail: IMAP< * OK [UNSEEN 1] Is the first unseen message
>> fetchmail: IMAP< * OK [UIDVALIDITY 3203] UIDVALIDITY value
>> fetchmail: IMAP< * OK [UIDNEXT 2124] The next unique identifier value
>> fetchmail: IMAP< A0006 OK [READ-WRITE] SELECT completed.
>> fetchmail: 264 messages waiting after first poll
>> fetchmail: IMAP> A0007 EXPUNGE
>> fetchmail: IMAP< * 264 EXISTS
>> fetchmail: IMAP< A0007 OK EXPUNGE completed.
>> fetchmail: 264 messages waiting after expunge
>> fetchmail: 264 messages for xx...@yy... at mail.demon.co.uk.
>> fetchmail: IMAP> A0008 FETCH 1:100 RFC822.SIZE
>> fetchmail: IMAP< * 1 FETCH (RFC822.SIZE 8512)
>> fetchmail: IMAP< * 2 FETCH (RFC822.SIZE 6404)
>> [...]
>> fetchmail: IMAP< * 99 FETCH (RFC822.SIZE 2942)
>> fetchmail: IMAP< * 100 FETCH (RFC822.SIZE 3914)
>> fetchmail: IMAP< A0008 OK FETCH completed.
>> fetchmail: IMAP> A0009 FETCH 1 RFC822.HEADER
>> fetchmail: IMAP< A0009 OK FETCH completed.
>> fetchmail: Incorrect FETCH response: OK FETCH COMPLETED..
>> fetchmail: IMAP> A0010 LOGOUT
>> fetchmail: IMAP< * BYE Microsoft Exchange Server 2010 IMAP4 server
>> signing off.
>> fetchmail: IMAP< A0010 OK LOGOUT completed.
>> fetchmail: client/server synchronization error while fetching from
>> xx...@yy...@mail.demon.co.uk
>> fetchmail: 6.3.21 querying mail.demon.co.uk (protocol IMAP) at Mon May
>> 7 10:07:45 2012: poll completed
>> fetchmail: Merged UID list from mail.demon.co.uk: <empty>
>> fetchmail: Query status=7 (ERROR)
>> fetchmail: sleeping at Mon May  7 10:07:45 2012 for 600 seconds
>>
>> The problem seems to be the "A0009 OK FETCH completed." response to
>> "A0009 FETCH 1 RFC822.HEADER".  There are no patches touching "imap.c"
>> in the openSUSE source RPM and I have confirmed that it is identical to
>> the 6.3.21 original.
>>
>> The "Incorrect FETCH response: OK FETCH COMPLETED.." appears to be
>> generated from within imap_fetch_headers() by "an unexpected tagged
>> response".  Is this an exciting new innovation in Microsoft Exchange
>> Server 2010?
>
> John,
>
> at least it's the first report of its kind that I've become aware of.
>
> I sat down with RFC-3501, the IMAP 4rev1 standard, and my current
> understanding is that the server sent an invalid response.

I came to a similar conclusion but probably from a shallower depth of  
understanding.

> The "unexpected tagged response" means that the server reply lacked
> mandatory untagged data.  A valid response to the A0009 FETCH 1
> RFC822.HEADER would have been either:
>
> * FETCH (RFC822.HEADER NIL)
> A0009 OK FETCH completed.
>
> Or:
>
> * FETCH (RFC822.HEADER "")
> A0009 OK FETCH completed.
>
> Or:
>
> * FETCH (RFC822.HEADER {0}
> )
> A0009 OK FETCH completed.
>
>
> Having said that, I'll recommend that you report the problem to Demon;
> chances are that the message is corrupted on the server and can be
> repaired, or that the Exchange 2010 behaviour can be tweaked by their
> admins; otherwise and/or on top of that, it would be Demon's task to
> talk to Microsoft and have the Redmond guys make Exchange 2010 compliant
> to the IMAP protocol, even in the face of incomplete and/or corrupted
> messages.

The message appears to be well-formed when viewed via the Outlook WebApp.   
The message is from the aus...@op... list, probably has  
the Subject "Event [Reminder: Austin Group teleconference] updated" and is  
generated by The Open Group's web service.  I have successfully received a  
number of similar messages via the older Demon mail service.

It may be that calendar events are subject to extra processing within  
Exchange 2010 and it doesn't feel the need to satisfy a fetch headers  
request.  Maybe the behaviour can be tweaked to treat calendar events in  
the same way as other messages?  Unfortunately, I am pressed for time this  
week and can't really chase this.  I've copied this message to  
hel...@de... and hope they can investigate.

> It is likely that the message has no formal Internet header; earlier
> Exchange versions used to reassemble messages on-the-fly; and chances
> are that configuration options might avoid the server bug.
>
> Note that per RFC-5322, the From: (sender) and Date: (originator's date)
> headers are mandatory, so the message couldn't reliably be shipped
> forward anyways if it doesn't have these headers, and
> garbage-in-garbage-out attempts often fail these days on modern SMTP
> servers.

My best guess is that it is a server problem rather than a corrupt  
message.  The Outlook WebApp view of the message suggests that it probably  
has both From: (sender) and Date: (originator's date) headers available.

> Chances are also that later fetchmail releases may be able to skip over
> these messages and leave them on the server unfetched, or delete them
> should the user opt to delete corrupted messages.  Given that fetchmail
> received the only pending tagged reply, it shouldn't assume it has gone
> out of synch though.

I also looked at the code in fetchmail-7.0.0-alpha2+MAPI.  The imap.c code  
behaves in a similar manner to fetchmail-6.3.21 when handling headers.   
Not keen on any mail software silently deleting what it interprets as a  
corrupt message!

> Sunil, for resilience towards massive faults on the server side, we'll
> probably have to add an error counter so that fetchmail won't attempt to
> fetch hundreds of messages' headers and skip over all of them when the
> server exhibits systematic faults (and not just some individual
> messages, as your report suggests).
>
> Hope that helps for now,

Many thanks.  I have a work around and if the flow of e-mail stops I  
should be able to apply it again.

> Matthias