fetchmail-users

[fetchmail-users] Problem with Demon's new Microsoft Exchange Server 2010 service

[John C. <jr...@sk...>]

Operating system:
     $ cat /etc/SuSE-release
     openSUSE 12.1 (x86_64)
     VERSION = 12.1
     CODENAME = Asparagus

Name and origin of the RPM:
     $ rpm -q fetchmail
     fetchmail-6.3.21-4.1.3.x86_64

The name and version of the SMTP listener:
     $ rpm -q exim
     exim-4.75-5.1.3.x86_64

Command-line options you used:
     openSUSE 12.1 distributed /etc/inid.d/fetchmail with added "-vvv".

/etc/fetchmailrc (with names changed to protect the innocent):
     $ cat /etc/fetchmailrc
     # Configuration created Thu Apr 26 22:58:13 2012
     set bouncemail
     set properties ""
     set daemon 300
     poll mail.demon.co.uk proto IMAP
            localdomains yyy.demon.co.uk
            user 'adm...@yy...' there password 'secret' is  
xxx here
            sslproto TLS1
            sslcertck
     poll mail.demon.co.uk proto IMAP
            localdomains yyy.demon.co.uk
            user 'xx...@yy...' there password 'secret' is xxx here
            sslproto TLS1
            sslcertck

Problem:
I have recently been migrated to Demon's new service which appears to be  
based on Microsoft Exchange Server 2010.  The migration appeared to have  
been successful with messages being fetched.  Then fetching of messages  
ceased.  I added "-vvv" to /etc/init.d/fetchmail and restarted fetchmail.   
Here are edited highlights from /var/log/fetchmail:

fetchmail: IMAP< A0004 NO AUTHENTICATE failed.
fetchmail: IMAP> A0005 LOGIN "xx...@yy..." *
fetchmail: IMAP< A0005 OK LOGIN completed.
fetchmail: selecting or re-polling default folder
fetchmail: IMAP> A0006 SELECT "INBOX"
fetchmail: IMAP< * 264 EXISTS
fetchmail: IMAP< * 1 RECENT
fetchmail: IMAP< * FLAGS (\Seen \Answered \Flagged \Deleted \Draft  
$MDNSent)
fetchmail: IMAP< * OK [PERMANENTFLAGS (\Seen \Answered \Flagged \Deleted  
\Draft $MDNSent)] Permanent flags
fetchmail: IMAP< * OK [UNSEEN 1] Is the first unseen message
fetchmail: IMAP< * OK [UIDVALIDITY 3203] UIDVALIDITY value
fetchmail: IMAP< * OK [UIDNEXT 2124] The next unique identifier value
fetchmail: IMAP< A0006 OK [READ-WRITE] SELECT completed.
fetchmail: 264 messages waiting after first poll
fetchmail: IMAP> A0007 EXPUNGE
fetchmail: IMAP< * 264 EXISTS
fetchmail: IMAP< A0007 OK EXPUNGE completed.
fetchmail: 264 messages waiting after expunge
fetchmail: 264 messages for xx...@yy... at mail.demon.co.uk.
fetchmail: IMAP> A0008 FETCH 1:100 RFC822.SIZE
fetchmail: IMAP< * 1 FETCH (RFC822.SIZE 8512)
fetchmail: IMAP< * 2 FETCH (RFC822.SIZE 6404)
[...]
fetchmail: IMAP< * 99 FETCH (RFC822.SIZE 2942)
fetchmail: IMAP< * 100 FETCH (RFC822.SIZE 3914)
fetchmail: IMAP< A0008 OK FETCH completed.
fetchmail: IMAP> A0009 FETCH 1 RFC822.HEADER
fetchmail: IMAP< A0009 OK FETCH completed.
fetchmail: Incorrect FETCH response: OK FETCH COMPLETED..
fetchmail: IMAP> A0010 LOGOUT
fetchmail: IMAP< * BYE Microsoft Exchange Server 2010 IMAP4 server signing  
off.
fetchmail: IMAP< A0010 OK LOGOUT completed.
fetchmail: client/server synchronization error while fetching from  
xx...@yy...@mail.demon.co.uk
fetchmail: 6.3.21 querying mail.demon.co.uk (protocol IMAP) at Mon May  7  
10:07:45 2012: poll completed
fetchmail: Merged UID list from mail.demon.co.uk: <empty>
fetchmail: Query status=7 (ERROR)
fetchmail: sleeping at Mon May  7 10:07:45 2012 for 600 seconds

The problem seems to be the "A0009 OK FETCH completed." response to "A0009  
FETCH 1 RFC822.HEADER".  There are no patches touching "imap.c" in the  
openSUSE source RPM and I have confirmed that it is identical to the  
6.3.21 original.

The "Incorrect FETCH response: OK FETCH COMPLETED.." appears to be  
generated from within imap_fetch_headers() by "an unexpected tagged  
response".  Is this an exciting new innovation in Microsoft Exchange  
Server 2010?

Please let me know if you need any further information.
--
John Connett

Re: [fetchmail-users] Problem with Demon's new Microsoft Exchange Server 2010 service

[Sunil S. <sun...@ro...>]

Dear John Connett,




________________________________
 > From: John Connett <jr...@sk...>
> 
> I have recently been migrated to Demon's new service which appears to be based on Microsoft Exchange Server 2010.  The migration appeared to have been successful with messages being fetched.  Then fetching of messages ceased.  I added "-vvv" to /etc/init.d/fetchmail and restarted fetchmail.  Here are edited highlights from /var/log/fetchmail:
> 
> fetchmail: IMAP< A0004 NO AUTHENTICATE failed.
> fetchmail: IMAP> A0005 LOGIN "xx...@yy..." *
> fetchmail: IMAP< A0005 OK LOGIN completed.

Please include the signature line / output of CAPABILITY command of your imap server in your report. This is to show the imap version supported.

> fetchmail: IMAP> A0009 FETCH 1 RFC822.HEADER
> fetchmail: IMAP< A0009 OK FETCH completed.
> fetchmail: Incorrect FETCH response: OK FETCH COMPLETED..

...

> The problem seems to be the "A0009 OK FETCH completed." response to "A0009 FETCH 1 RFC822.HEADER".  There are no patches touching "imap.c" in the openSUSE source RPM and I have confirmed that it is identical to the 6.3.21 original.
> 
> The "Incorrect FETCH response: OK FETCH COMPLETED.." appears to be generated from within imap_fetch_headers() by "an unexpected tagged response".  Is this an exciting new innovation in Microsoft Exchange Server 2010?

As is documented in imap.c, fetchmail expects response in the following format:

IMAP> A0006 FETCH 1 RFC822.HEADER                                     
IMAP< * 1 FETCH (RFC822.HEADER {1360}                                 
.... mail headers here ...
IMAP< A0006 OK FETCH completed.
                                                         

In your case, the middle part of the response is missing. Can you check by using telnet to the server if there are any headers in the mail and response to the variouse commands listed below:

A0010 FETCH 1 RFC822.TEXT

A0011 FETCH 1 BODY[TEXT]

for the mail causing the problem? Or check the mail through some other client and see if it is visible properly.

-- 
Sunil Shetye.

Re: [fetchmail-users] Problem with Demon's new Microsoft Exchange Server 2010 service

[John C. <jr...@sk...>]

On Mon, 07 May 2012 15:32:20 +0100, Sunil Shetye  
<sun...@ro...> wrote:
> Please include the signature line / output of CAPABILITY command of your  
> imap server in your report. This is to show the imap version supported.

fetchmail: awakened at Mon May  7 13:48:18 2012
fetchmail: 6.3.21 querying mail.demon.co.uk (protocol IMAP) at Mon May  7  
13:48:18 2012: poll started
fetchmail: Trying to connect to 91.221.168.28/143...connected.
fetchmail: IMAP< * OK The Microsoft Exchange IMAP4 service is ready.
fetchmail: IMAP> A0001 CAPABILITY
fetchmail: IMAP< * CAPABILITY IMAP4 IMAP4rev1 AUTH=NTLM AUTH=GSSAPI  
AUTH=PLAIN STARTTLS UIDPLUS CHILDREN IDLE NAMESPACE LITERAL+
fetchmail: IMAP< A0001 OK CAPABILITY completed.
fetchmail: Protocol identified as IMAP4 rev 1
fetchmail: IMAP> A0002 STARTTLS
fetchmail: IMAP< A0002 OK Begin TLS negotiation now.
fetchmail: Certificate chain, from root to peer, starting at depth 2:
fetchmail: Issuer Organization: GeoTrust Inc.
fetchmail: Issuer CommonName: GeoTrust Global CA
fetchmail: Subject CommonName: GeoTrust Global CA
fetchmail: Certificate at depth 1:
fetchmail: Issuer Organization: GeoTrust Inc.
fetchmail: Issuer CommonName: GeoTrust Global CA
fetchmail: Subject CommonName: GeoTrust DV SSL CA
fetchmail: Server certificate:
fetchmail: Issuer Organization: GeoTrust Inc.
fetchmail: Issuer CommonName: GeoTrust DV SSL CA
fetchmail: Subject CommonName: mail.demon.co.uk
fetchmail: Subject Alternative Name: mail.demon.co.uk
fetchmail: mail.demon.co.uk key fingerprint:  
6F:AB:CD:95:D8:AE:FE:33:83:33:4C:8A:93:60:FC:7C
fetchmail: IMAP> A0003 CAPABILITY
fetchmail: IMAP< * CAPABILITY IMAP4 IMAP4rev1 AUTH=NTLM AUTH=GSSAPI  
AUTH=PLAIN UIDPLUS CHILDREN IDLE NAMESPACE LITERAL+
fetchmail: IMAP< A0003 OK CAPABILITY completed.
fetchmail: Protocol identified as IMAP4 rev 1
fetchmail: mail.demon.co.uk: upgrade to TLS succeeded.
fetchmail: GSSAPI error gss_inquire_cred: Unspecified GSS failure.  Minor  
code may provide more information
fetchmail: GSSAPI error gss_inquire_cred: Credentials cache file  
'/tmp/krb5cc_109' not found
fetchmail: No suitable GSSAPI credentials found. Skipping GSSAPI  
authentication.
fetchmail: If you want to use GSSAPI, you need credentials first, possibly  
 from kinit.

Hope that is sufficient.

> As is documented in imap.c, fetchmail expects response in the following  
> format:
>
> IMAP> A0006 FETCH 1 RFC822.HEADER                                    
> IMAP< * 1 FETCH (RFC822.HEADER {1360}                               ....  
> mail headers here ...
> IMAP< A0006 OK FETCH completed.
>                                                         
> In your case, the middle part of the response is missing. Can you check  
> by using telnet to the server if there are any headers in the mail and  
> response to the variouse commands listed below:
>
> A0010 FETCH 1 RFC822.TEXT
>
> A0011 FETCH 1 BODY[TEXT]
>
> for the mail causing the problem? Or check the mail through some other  
> client and see if it is visible properly.

I will see if I can find time to experiment.  Unfortunately, this is my  
main e-mail account so I am keen not to break it!  As noted in my earlier  
message to the fetchmail-user list the problem message is a meeting  
reminder with a "meeting.ics" attachment.

Many thanks
--
John Connett

Re: [fetchmail-users] Problem with Demon's new Microsoft Exchange Server 2010 service

[Sunil S. <sun...@ro...>]

> From: John Connett <jr...@sk...>

> 
> > Please include the signature line / output of CAPABILITY command of your imap server in your report. This is to show the imap version supported.
> 
> fetchmail: IMAP< * OK The Microsoft Exchange IMAP4 service is ready.
> fetchmail: IMAP> A0003 CAPABILITY
> fetchmail: IMAP< * CAPABILITY IMAP4 IMAP4rev1 AUTH=NTLM AUTH=GSSAPI AUTH=PLAIN UIDPLUS CHILDREN IDLE NAMESPACE LITERAL+
> fetchmail: IMAP< A0003 OK CAPABILITY completed.

Yes, that is what I was looking for.

> I will see if I can find time to experiment.  Unfortunately, this is my main e-mail account so I am keen not to break it!  As noted in my earlier message to the fetchmail-user list the problem message is a meeting reminder with a "meeting.ics" attachment.

If you can view the source of the mail through another email client and check what it looks like, that would help a lot. Otherwise, the only option fetchmail has is to skip over this mail.



fetchmail does have a problem as it is quitting straightaway on finding an error. Instead, it should have continued to the next mail. Though this would not have completely fixed the problem, it would have atleast not stopped mail traffic completely.

This part can be fixed. However, in that case, the problematic mail may remain permanently unnoticed in the mailbox.

-- 
Sunil Shetye.

Re: [fetchmail-users] Problem with Demon's new Microsoft Exchange Server 2010 service

[John C. <jr...@sk...>]

On Mon, 07 May 2012 11:10:47 +0100, John Connett <jr...@sk...>  
wrote:
> The "Incorrect FETCH response: OK FETCH COMPLETED.." appears to be  
> generated from within imap_fetch_headers() by "an unexpected tagged  
> response".  Is this an exciting new innovation in Microsoft Exchange  
> Server 2010?

I took a look via Demon's Microsoft Outlook WebApp to see if I could  
identify the problem message.  The oldest message was about the right size  
and was not flagged with a little yellow closed envelope but with a little  
blue calendar and an attachment paperclip.  If I click on the message it  
is displayed via the WebApp calendar page.  Naturally, the WebApp is  
clever enough not to allow me to forward this message to another address  
to inspect the contents!  It has a single attachment, "meeting.ics".  I  
created a new "Bad Items" folder and moved the message into it.  Fetching  
of e-mail messages resumed.

Looks like Exchange Server 2010 may be doing something special for  
calendar e-mails.  A side effect of sending someone a meeting reminder is  
that they get no reminder and no more messages are fetched!
--
John Connett

Re: [fetchmail-users] Problem with Demon's new Microsoft Exchange Server 2010 service

[Matthias A. <mat...@gm...>]

Am 07.05.2012 12:10, schrieb John Connett:
> Operating system:
>     $ cat /etc/SuSE-release
>     openSUSE 12.1 (x86_64)
>     VERSION = 12.1
>     CODENAME = Asparagus
> 
> Name and origin of the RPM:
>     $ rpm -q fetchmail
>     fetchmail-6.3.21-4.1.3.x86_64
> 
> The name and version of the SMTP listener:
>     $ rpm -q exim
>     exim-4.75-5.1.3.x86_64
> 
> Command-line options you used:
>     openSUSE 12.1 distributed /etc/inid.d/fetchmail with added "-vvv".
> 
> /etc/fetchmailrc (with names changed to protect the innocent):
>     $ cat /etc/fetchmailrc
>     # Configuration created Thu Apr 26 22:58:13 2012
>     set bouncemail
>     set properties ""
>     set daemon 300
>     poll mail.demon.co.uk proto IMAP
>            localdomains yyy.demon.co.uk
>            user 'adm...@yy...' there password 'secret'
> is xxx here
>            sslproto TLS1
>            sslcertck
>     poll mail.demon.co.uk proto IMAP
>            localdomains yyy.demon.co.uk
>            user 'xx...@yy...' there password 'secret' is xxx here
>            sslproto TLS1
>            sslcertck
> 
> Problem:
> I have recently been migrated to Demon's new service which appears to be
> based on Microsoft Exchange Server 2010.  The migration appeared to have
> been successful with messages being fetched.  Then fetching of messages
> ceased.  I added "-vvv" to /etc/init.d/fetchmail and restarted
> fetchmail.  Here are edited highlights from /var/log/fetchmail:
> 
> fetchmail: IMAP< A0004 NO AUTHENTICATE failed.
> fetchmail: IMAP> A0005 LOGIN "xx...@yy..." *
> fetchmail: IMAP< A0005 OK LOGIN completed.
> fetchmail: selecting or re-polling default folder
> fetchmail: IMAP> A0006 SELECT "INBOX"
> fetchmail: IMAP< * 264 EXISTS
> fetchmail: IMAP< * 1 RECENT
> fetchmail: IMAP< * FLAGS (\Seen \Answered \Flagged \Deleted \Draft
> $MDNSent)
> fetchmail: IMAP< * OK [PERMANENTFLAGS (\Seen \Answered \Flagged \Deleted
> \Draft $MDNSent)] Permanent flags
> fetchmail: IMAP< * OK [UNSEEN 1] Is the first unseen message
> fetchmail: IMAP< * OK [UIDVALIDITY 3203] UIDVALIDITY value
> fetchmail: IMAP< * OK [UIDNEXT 2124] The next unique identifier value
> fetchmail: IMAP< A0006 OK [READ-WRITE] SELECT completed.
> fetchmail: 264 messages waiting after first poll
> fetchmail: IMAP> A0007 EXPUNGE
> fetchmail: IMAP< * 264 EXISTS
> fetchmail: IMAP< A0007 OK EXPUNGE completed.
> fetchmail: 264 messages waiting after expunge
> fetchmail: 264 messages for xx...@yy... at mail.demon.co.uk.
> fetchmail: IMAP> A0008 FETCH 1:100 RFC822.SIZE
> fetchmail: IMAP< * 1 FETCH (RFC822.SIZE 8512)
> fetchmail: IMAP< * 2 FETCH (RFC822.SIZE 6404)
> [...]
> fetchmail: IMAP< * 99 FETCH (RFC822.SIZE 2942)
> fetchmail: IMAP< * 100 FETCH (RFC822.SIZE 3914)
> fetchmail: IMAP< A0008 OK FETCH completed.
> fetchmail: IMAP> A0009 FETCH 1 RFC822.HEADER
> fetchmail: IMAP< A0009 OK FETCH completed.
> fetchmail: Incorrect FETCH response: OK FETCH COMPLETED..
> fetchmail: IMAP> A0010 LOGOUT
> fetchmail: IMAP< * BYE Microsoft Exchange Server 2010 IMAP4 server
> signing off.
> fetchmail: IMAP< A0010 OK LOGOUT completed.
> fetchmail: client/server synchronization error while fetching from
> xx...@yy...@mail.demon.co.uk
> fetchmail: 6.3.21 querying mail.demon.co.uk (protocol IMAP) at Mon May 
> 7 10:07:45 2012: poll completed
> fetchmail: Merged UID list from mail.demon.co.uk: <empty>
> fetchmail: Query status=7 (ERROR)
> fetchmail: sleeping at Mon May  7 10:07:45 2012 for 600 seconds
> 
> The problem seems to be the "A0009 OK FETCH completed." response to
> "A0009 FETCH 1 RFC822.HEADER".  There are no patches touching "imap.c"
> in the openSUSE source RPM and I have confirmed that it is identical to
> the 6.3.21 original.
> 
> The "Incorrect FETCH response: OK FETCH COMPLETED.." appears to be
> generated from within imap_fetch_headers() by "an unexpected tagged
> response".  Is this an exciting new innovation in Microsoft Exchange
> Server 2010?

John,

at least it's the first report of its kind that I've become aware of.

I sat down with RFC-3501, the IMAP 4rev1 standard, and my current
understanding is that the server sent an invalid response.

The "unexpected tagged response" means that the server reply lacked
mandatory untagged data.  A valid response to the A0009 FETCH 1
RFC822.HEADER would have been either:

* FETCH (RFC822.HEADER NIL)
A0009 OK FETCH completed.

Or:

* FETCH (RFC822.HEADER "")
A0009 OK FETCH completed.

Or:

* FETCH (RFC822.HEADER {0}
)
A0009 OK FETCH completed.


Having said that, I'll recommend that you report the problem to Demon;
chances are that the message is corrupted on the server and can be
repaired, or that the Exchange 2010 behaviour can be tweaked by their
admins; otherwise and/or on top of that, it would be Demon's task to
talk to Microsoft and have the Redmond guys make Exchange 2010 compliant
to the IMAP protocol, even in the face of incomplete and/or corrupted
messages.

It is likely that the message has no formal Internet header; earlier
Exchange versions used to reassemble messages on-the-fly; and chances
are that configuration options might avoid the server bug.

Note that per RFC-5322, the From: (sender) and Date: (originator's date)
headers are mandatory, so the message couldn't reliably be shipped
forward anyways if it doesn't have these headers, and
garbage-in-garbage-out attempts often fail these days on modern SMTP
servers.


Chances are also that later fetchmail releases may be able to skip over
these messages and leave them on the server unfetched, or delete them
should the user opt to delete corrupted messages.  Given that fetchmail
received the only pending tagged reply, it shouldn't assume it has gone
out of synch though.

Sunil, for resilience towards massive faults on the server side, we'll
probably have to add an error counter so that fetchmail won't attempt to
fetch hundreds of messages' headers and skip over all of them when the
server exhibits systematic faults (and not just some individual
messages, as your report suggests).

Hope that helps for now,
Matthias

Re: [fetchmail-users] Problem with Demon's new Microsoft Exchange Server 2010 service

[John C. <jr...@sk...>]

On Mon, 07 May 2012 19:26:27 +0100, Matthias Andree  
<mat...@gm...> wrote:
> Am 07.05.2012 12:10, schrieb John Connett:
>> Operating system:
>>     $ cat /etc/SuSE-release
>>     openSUSE 12.1 (x86_64)
>>     VERSION = 12.1
>>     CODENAME = Asparagus
>>
>> Name and origin of the RPM:
>>     $ rpm -q fetchmail
>>     fetchmail-6.3.21-4.1.3.x86_64
>>
>> The name and version of the SMTP listener:
>>     $ rpm -q exim
>>     exim-4.75-5.1.3.x86_64
>>
>> Command-line options you used:
>>     openSUSE 12.1 distributed /etc/inid.d/fetchmail with added "-vvv".
>>
>> /etc/fetchmailrc (with names changed to protect the innocent):
>>     $ cat /etc/fetchmailrc
>>     # Configuration created Thu Apr 26 22:58:13 2012
>>     set bouncemail
>>     set properties ""
>>     set daemon 300
>>     poll mail.demon.co.uk proto IMAP
>>            localdomains yyy.demon.co.uk
>>            user 'adm...@yy...' there password 'secret'
>> is xxx here
>>            sslproto TLS1
>>            sslcertck
>>     poll mail.demon.co.uk proto IMAP
>>            localdomains yyy.demon.co.uk
>>            user 'xx...@yy...' there password 'secret' is xxx  
>> here
>>            sslproto TLS1
>>            sslcertck
>>
>> Problem:
>> I have recently been migrated to Demon's new service which appears to be
>> based on Microsoft Exchange Server 2010.  The migration appeared to have
>> been successful with messages being fetched.  Then fetching of messages
>> ceased.  I added "-vvv" to /etc/init.d/fetchmail and restarted
>> fetchmail.  Here are edited highlights from /var/log/fetchmail:
>>
>> fetchmail: IMAP< A0004 NO AUTHENTICATE failed.
>> fetchmail: IMAP> A0005 LOGIN "xx...@yy..." *
>> fetchmail: IMAP< A0005 OK LOGIN completed.
>> fetchmail: selecting or re-polling default folder
>> fetchmail: IMAP> A0006 SELECT "INBOX"
>> fetchmail: IMAP< * 264 EXISTS
>> fetchmail: IMAP< * 1 RECENT
>> fetchmail: IMAP< * FLAGS (\Seen \Answered \Flagged \Deleted \Draft
>> $MDNSent)
>> fetchmail: IMAP< * OK [PERMANENTFLAGS (\Seen \Answered \Flagged \Deleted
>> \Draft $MDNSent)] Permanent flags
>> fetchmail: IMAP< * OK [UNSEEN 1] Is the first unseen message
>> fetchmail: IMAP< * OK [UIDVALIDITY 3203] UIDVALIDITY value
>> fetchmail: IMAP< * OK [UIDNEXT 2124] The next unique identifier value
>> fetchmail: IMAP< A0006 OK [READ-WRITE] SELECT completed.
>> fetchmail: 264 messages waiting after first poll
>> fetchmail: IMAP> A0007 EXPUNGE
>> fetchmail: IMAP< * 264 EXISTS
>> fetchmail: IMAP< A0007 OK EXPUNGE completed.
>> fetchmail: 264 messages waiting after expunge
>> fetchmail: 264 messages for xx...@yy... at mail.demon.co.uk.
>> fetchmail: IMAP> A0008 FETCH 1:100 RFC822.SIZE
>> fetchmail: IMAP< * 1 FETCH (RFC822.SIZE 8512)
>> fetchmail: IMAP< * 2 FETCH (RFC822.SIZE 6404)
>> [...]
>> fetchmail: IMAP< * 99 FETCH (RFC822.SIZE 2942)
>> fetchmail: IMAP< * 100 FETCH (RFC822.SIZE 3914)
>> fetchmail: IMAP< A0008 OK FETCH completed.
>> fetchmail: IMAP> A0009 FETCH 1 RFC822.HEADER
>> fetchmail: IMAP< A0009 OK FETCH completed.
>> fetchmail: Incorrect FETCH response: OK FETCH COMPLETED..
>> fetchmail: IMAP> A0010 LOGOUT
>> fetchmail: IMAP< * BYE Microsoft Exchange Server 2010 IMAP4 server
>> signing off.
>> fetchmail: IMAP< A0010 OK LOGOUT completed.
>> fetchmail: client/server synchronization error while fetching from
>> xx...@yy...@mail.demon.co.uk
>> fetchmail: 6.3.21 querying mail.demon.co.uk (protocol IMAP) at Mon May
>> 7 10:07:45 2012: poll completed
>> fetchmail: Merged UID list from mail.demon.co.uk: <empty>
>> fetchmail: Query status=7 (ERROR)
>> fetchmail: sleeping at Mon May  7 10:07:45 2012 for 600 seconds
>>
>> The problem seems to be the "A0009 OK FETCH completed." response to
>> "A0009 FETCH 1 RFC822.HEADER".  There are no patches touching "imap.c"
>> in the openSUSE source RPM and I have confirmed that it is identical to
>> the 6.3.21 original.
>>
>> The "Incorrect FETCH response: OK FETCH COMPLETED.." appears to be
>> generated from within imap_fetch_headers() by "an unexpected tagged
>> response".  Is this an exciting new innovation in Microsoft Exchange
>> Server 2010?
>
> John,
>
> at least it's the first report of its kind that I've become aware of.
>
> I sat down with RFC-3501, the IMAP 4rev1 standard, and my current
> understanding is that the server sent an invalid response.

I came to a similar conclusion but probably from a shallower depth of  
understanding.

> The "unexpected tagged response" means that the server reply lacked
> mandatory untagged data.  A valid response to the A0009 FETCH 1
> RFC822.HEADER would have been either:
>
> * FETCH (RFC822.HEADER NIL)
> A0009 OK FETCH completed.
>
> Or:
>
> * FETCH (RFC822.HEADER "")
> A0009 OK FETCH completed.
>
> Or:
>
> * FETCH (RFC822.HEADER {0}
> )
> A0009 OK FETCH completed.
>
>
> Having said that, I'll recommend that you report the problem to Demon;
> chances are that the message is corrupted on the server and can be
> repaired, or that the Exchange 2010 behaviour can be tweaked by their
> admins; otherwise and/or on top of that, it would be Demon's task to
> talk to Microsoft and have the Redmond guys make Exchange 2010 compliant
> to the IMAP protocol, even in the face of incomplete and/or corrupted
> messages.

The message appears to be well-formed when viewed via the Outlook WebApp.   
The message is from the aus...@op... list, probably has  
the Subject "Event [Reminder: Austin Group teleconference] updated" and is  
generated by The Open Group's web service.  I have successfully received a  
number of similar messages via the older Demon mail service.

It may be that calendar events are subject to extra processing within  
Exchange 2010 and it doesn't feel the need to satisfy a fetch headers  
request.  Maybe the behaviour can be tweaked to treat calendar events in  
the same way as other messages?  Unfortunately, I am pressed for time this  
week and can't really chase this.  I've copied this message to  
hel...@de... and hope they can investigate.

> It is likely that the message has no formal Internet header; earlier
> Exchange versions used to reassemble messages on-the-fly; and chances
> are that configuration options might avoid the server bug.
>
> Note that per RFC-5322, the From: (sender) and Date: (originator's date)
> headers are mandatory, so the message couldn't reliably be shipped
> forward anyways if it doesn't have these headers, and
> garbage-in-garbage-out attempts often fail these days on modern SMTP
> servers.

My best guess is that it is a server problem rather than a corrupt  
message.  The Outlook WebApp view of the message suggests that it probably  
has both From: (sender) and Date: (originator's date) headers available.

> Chances are also that later fetchmail releases may be able to skip over
> these messages and leave them on the server unfetched, or delete them
> should the user opt to delete corrupted messages.  Given that fetchmail
> received the only pending tagged reply, it shouldn't assume it has gone
> out of synch though.

I also looked at the code in fetchmail-7.0.0-alpha2+MAPI.  The imap.c code  
behaves in a similar manner to fetchmail-6.3.21 when handling headers.   
Not keen on any mail software silently deleting what it interprets as a  
corrupt message!

> Sunil, for resilience towards massive faults on the server side, we'll
> probably have to add an error counter so that fetchmail won't attempt to
> fetch hundreds of messages' headers and skip over all of them when the
> server exhibits systematic faults (and not just some individual
> messages, as your report suggests).
>
> Hope that helps for now,

Many thanks.  I have a work around and if the flow of e-mail stops I  
should be able to apply it again.

> Matthias

Re: [fetchmail-users] Problem with Demon's new Microsoft Exchange Server 2010 service

[Matthias A. <mat...@gm...>]

Greetings,

please be sure do not carbon copy multiple lists (like your ISP's
helpdesk and the fetchmail list), it can confuse recipients.


Just to ease concerns of "delete corrupted messages", that will continue
to require explicit and clear configuration, the default will be the
safer, less astonishing "leave corrupt mail on server".


John,

I have some austin group reminders lying around, I'll see what they look
like on my computer later and see if those sell any clue as to what's
wrong.  It wouldn't be a first to see Exchange choke an rarely-used but
still compliant features.

(For instance, you want to avoid Content-Transfer-Encoding: binary with
certain versions of Exchange and configuration options because that
leads to inconsistent boundary=... lines between message header and body.)


Best regards
Matthias

[fetchmail-users] [PATCH] Re: Problem with Demon's new Microsoft Exchange Server 2010 service

[Sunil S. <sun...@ro...>]

Hi,

> > Problem:

> > I have recently been migrated to Demon's new service which appears to be
> > based on Microsoft Exchange Server 2010.  The migration appeared to have
> > been successful with messages being fetched.  Then fetching of messages
> > ceased.  I added "-vvv" to /etc/init.d/fetchmail and restarted
> > fetchmail.  Here are edited highlights from /var/log/fetchmail:

...

> > fetchmail: IMAP> A0009 FETCH 1 RFC822.HEADER
> > fetchmail: IMAP< A0009 OK FETCH completed.
> > fetchmail: Incorrect FETCH response: OK FETCH COMPLETED..

Attached is a patch to fix this problem. Please note that this patch will not change the behaviour for mails with such responses. It will ensure that mails after the current mail will be fetched as is expected.

> Sunil, for resilience towards massive faults on the server side, we'll
> probably have to add an error counter so that fetchmail won't attempt to
> fetch hundreds of messages' headers and skip over all of them when the
> server exhibits systematic faults (and not just some individual
> messages, as your report suggests).

I have added a counter to this patch to log the number of transient errors. I am tempted to add a open_warning_by_mail() instead of just logging it for too many transient errors. However, if such mails are accumulating, there will be a warning mail per poll leading to excessive warning mails.

Also note that the bug is that fetchmail currently stops fetching the remaining mails from the folder the moment it gets an unexpected response. As this problematic mail is not removed, fetchmail stops fetching mails from the folder completely across polls. This patch ensures that fetching of the remaining mails continues.

-- 
Sunil Shetye.

Re: [fetchmail-users] [PATCH] Re: Problem with Demon's new Microsoft Exchange Server 2010 service

[Matthias A. <mat...@gm...>]

Am 09.05.2012 10:35, schrieb Sunil Shetye:
> Hi,
> 
>>> Problem:
> 
>>> I have recently been migrated to Demon's new service which appears to be
>>> based on Microsoft Exchange Server 2010.  The migration appeared to have
>>> been successful with messages being fetched.  Then fetching of messages
>>> ceased.  I added "-vvv" to /etc/init.d/fetchmail and restarted
>>> fetchmail.  Here are edited highlights from /var/log/fetchmail:
> 
> ...
> 
>>> fetchmail: IMAP> A0009 FETCH 1 RFC822.HEADER
>>> fetchmail: IMAP< A0009 OK FETCH completed.
>>> fetchmail: Incorrect FETCH response: OK FETCH COMPLETED..
> 
> Attached is a patch to fix this problem. Please note that this patch will not change the behaviour for mails with such responses. It will ensure that mails after the current mail will be fetched as is expected.
> 
>> Sunil, for resilience towards massive faults on the server side, we'll
>> probably have to add an error counter so that fetchmail won't attempt to
>> fetch hundreds of messages' headers and skip over all of them when the
>> server exhibits systematic faults (and not just some individual
>> messages, as your report suggests).
> 
> I have added a counter to this patch to log the number of transient errors. I am tempted to add a open_warning_by_mail() instead of just logging it for too many transient errors. However, if such mails are accumulating, there will be a warning mail per poll leading to excessive warning mails.
> 
> Also note that the bug is that fetchmail currently stops fetching the remaining mails from the folder the moment it gets an unexpected response. As this problematic mail is not removed, fetchmail stops fetching mails from the folder completely across polls. This patch ensures that fetching of the remaining mails continues.

Sunil,

thanks a lot, the patch looks good at first inspection.

I think if the count of transient errors goes too high, we should skip
to the next server (else counting is pointless).  I'll add that, too.  I
understand your itch to add that "send warning mail", but I think
fetchmail needs a generic framework for reporting trouble by mail first
- or at least such a feature would have to be structured in a way to
support such a framework later on.  Not 6.3.X stuff though, but for
future development version (although fine for 7.0.X).
The underlying issue that has always been nagging (but not being
annoying enough) is that fetchmail makes a distinction between daemon
and regular mode...

I don't think a tool like fetchmail can survive without any operator
attention though.  We certainly don't want a message on every poll, but
a message after 4 hours and then every 24 hours should be good.  Only we
don't have code to support that outside daemon mode.

I think that part is stuff for the -devel list.

John,

can you try Sunil's patch, i. e. unpack the fetchmail sources, apply the
patch, ./configure --with-ssl --with-gssapi and install, then move the
offending message back in your inbox, send yourself another test
message, and see if the patched fetchmail version skips over the
offending message and retrieves the test message?

Thanks a bunch.

Best regards
Matthias

Re: [fetchmail-users] [PATCH] Re: Problem with Demon's new Microsoft Exchange Server 2010 service

[John C. <jr...@sk...>]

On Thu, 10 May 2012 16:35:13 +0100, Matthias Andree  
<mat...@gm...> wrote:
> John,
>
> can you try Sunil's patch, i. e. unpack the fetchmail sources, apply the
> patch, ./configure --with-ssl --with-gssapi and install, then move the
> offending message back in your inbox, send yourself another test
> message, and see if the patched fetchmail version skips over the
> offending message and retrieves the test message?

Sorry I haven't had chance to try the patch yet.  However, on Friday I had  
another meeting reminder and a correction from the same source.  Both of  
them stopped the unmodified fetchmail from fetching any further messages  
as before.

I did find the following articles about Exchange 2010 Calendar Repair  
Assistant:

http://www.msexchange.org/articles_tutorials/exchange-server-2010/management-administration/exchange-2010-calendar-repair-part1.html
http://www.msexchange.org/articles_tutorials/exchange-server-2010/management-administration/exchange-2010-calendar-repair-part2.html

which might be a hint at the source of the problem.  Maybe it is  
appropriate for Exchange 2010 to provide special calendar handling if it  
is the central IMAP server for a single organization.  However, for an ISP  
using it as a Mail Transfer Agent I would expect Exchange 2010 to pass on  
all (non-spam) e-mail messages without inspecting, modifying or making  
unavailable the body of those messages!  At the very least it indicates a  
possible misconfiguration at Demon and probably a bug in the IMAP part of  
Exchange 2010.

No response from Demon yet.  I will forward a copy of this message to them  
tagged with their Case reference.

Thanks again
--
John Connett