fetchmail-devel

[fetchmail-devel] Continuation of "Domino IMAP and missing Content-Transfer-Encoding"

[Jason O. <jo...@ra...>]

I am experiencing similar problems reported in March by Anthony Kim. See 
old thread here:
http://lists.ccil.org/pipermail/fetchmail-friends/2006-March/010015.html

In this case, I'm using fetchmail 6.4.3 with and without Debian patches, 
and am subjected to Lotus 6.5.4.

Anthony's fix (changing the Lotus mail format) does not work in our 
configuration. In fact, the "Keep in Sender's format" is still the 
default here for all users. We see mutilated emails regularly. They are 
generally from the Lotus webmail system or Lotus notes. Attachments from 
those are almost always mangled. Internet emails are seldom broken.

We have seen the Content-Transfer-Encoding and Content-Type headers not 
reported by the server in response to the FETCH RFC822.HEADER command. 
The server also tends to report bogus header sizes, although that didn't 
seem to phase fetchmail.

I've done a few hours hacking away at the server and fetchmail and can 
only conclude that separate header and body retrieval must be abandoned 
for the very broken actions of Lotus. In digging through fetchmail, it 
appears that the only way to "cleanly" (I use that loosely) accomplish 
this is to make a new protocol. "LotusIMAP" maybe?

I did manage a very quick and very very ugly hack to accomplish this, 
and it now works! I can finally abandon my bastardization MDA that 
attempts to figure out what those headers should be and add them. As you 
can guess, that approach didn't work very well.

Some mention was made in the old thread to adding the single retrieval 
as an option. Is there an easy way to do that? I do have time to provide 
small modifications.

Thanks!
Jason

Re: [fetchmail-devel] Continuation of "Domino IMAP and missing Content-Transfer-Encoding"

[Matthias A. <mat...@gm...>]

Jason Ostermann <jo...@ra...> writes:

> Anthony's fix (changing the Lotus mail format) does not work in our 
> configuration. In fact, the "Keep in Sender's format" is still the 
> default here for all users. We see mutilated emails regularly. They are 
> generally from the Lotus webmail system or Lotus notes. Attachments from 
> those are almost always mangled.

Heck, can't vendors be bothered to repair the known defects in their
software or at least sponsor workarounds in third-party software they
are breaking?

> I've done a few hours hacking away at the server and fetchmail and can 
> only conclude that separate header and body retrieval must be abandoned 
> for the very broken actions of Lotus. In digging through fetchmail, it 
> appears that the only way to "cleanly" (I use that loosely) accomplish 
> this is to make a new protocol. "LotusIMAP" maybe?

Jason,

I wonder if a separate protocol is needed - and I'd like to avoid that
if possible since that will sooner or later lead to bit rot.

Some alternatives to FETCH 1234 RFC822.HEADER (where 1234 is the message
number) I could think of:

  BODY.PEEK[HEADER]
  BODY.PEEK[MIME]   - nonstandard, may cause BAD reply
  BODY.PEEK[0.MIME] - nonstandard, may cause BAD reply
  BODY.PEEK[HEADER.FIELDS (MIME-Version Content-Type Content-Transfer-Encoding Content-Disposition)]
  BODY.PEEK[HEADER.FIELDS.NOT (foo)]

Does any of these provide you with the full header when RFC822.HEADER is
incomplete?

What does

  FETCH 1 BODYSTRUCTURE

give you (replace 1 by the number of a 'broken' message)?

A full IMAP4r1 session might look like:

A LOGIN USER SECRETPASSWORDGOESHERE
B SELECT INBOX
C FETCH 1 BODY.PEEK[HEADER]
D LOGOUT


Generally, fetchmail has support for protocols that can only retrieve
full messages (POP3, for instance, uses that), so some hacking away at
imap.c might be sufficient to make the rest of fetchmail believe that
IMAP could only support full messages. The protocol interface is OOP
style (yes it's clumsy in C, but it works), in that driver.c/transact.c
call protocol (imap.c/pop3.c) methods^Wfunctions.

Best would be a workaround that could automatically kick in, since
that's easiest for users and distributors.

Such code might require minor internal rearrangements, such as moving
what used to be hard-wired protocol attributes into methods that query
the capabilities of the current connection.

-- 
Matthias Andree

Re: [fetchmail-devel] Continuation of "Domino IMAP and missing Content-Transfer-Encoding"

[Matthias A. <mat...@gm...>]

Jason Ostermann <jo...@ra...> writes:

>> Heck, can't vendors be bothered to repair the known defects in their
>> software or at least sponsor workarounds in third-party software they
>> are breaking?
>
> Agreed. Of course, if I had a say (or if any of my fellow 80,000 minions 
> had a say) we'd not be using Lotus.
>
>>   BODY.PEEK[HEADER.FIELDS (MIME-Version Content-Type Content-Transfer-Encoding Content-Disposition)]

> This returned the fields, although only the Content-Transfer-Encoding 
> was missing from the RFC822 header output.

OK, so we'll have to re-request that header.

>> What does
>> 
>>   FETCH 1 BODYSTRUCTURE
>> 
>> give you (replace 1 by the number of a 'broken' message)?
>
> * 574 FETCH (BODYSTRUCTURE ("text" "html"  ("charset" "ISO-8859-1") NIL 
> NIL "quoted-printable" 1287 22 NIL NIL NIL))
> I don't know how to parse this output, although the strings presented 
> are correct. Don't know what the numbers are supposed to reflect.

These are the contents of some headers, see section 7.4.2 in RFC-3501.

> Sadly, I didn't manage to do an imap isolated patch yesterday. To get my 
> hack to work, I mutilated imap.c to make the getsizes, getpartialsizes 
> and fetch body NULL and changed fetch_headers to submit FETCH %d 
> (BODY[]). I also had to change driver.c by adding len-=msgblk.msglen 
> before the readbody. Otherwise, it'd try to read the whole message size 
> off of the socket. I was very confused by the last bit, as it seemed 
> that changing the fetch body to NULL should've induced driver.c to do 
> the same to handle the POP protocols. I expect I missed something somewhere.
>
> Is there a way to identify an imap server? The banner on login only states:
> * OK IMAP Server Ready Tue, 8 Aug 2006 08:37:04 -0500

Perhaps the reply to a CAPABILITY request can help with identifying the
server.

What does Lotus respond to such a command?

Another workaround idea: what if we check the received headers from the
RFC822.HEADERS request if it contains "MIME-Header: 1.0", and if it
does, but is missing Content-Transfer-Encoding, request the latter
explicitly and attach it to the headers if we get one?

Such a workaround would reorder headers, but better that
than a missing quoted-printable or base64 tag.

What do you think?

-- 
Matthias Andree

Re: [fetchmail-devel] Continuation of "Domino IMAP and missing Content-Transfer-Encoding"

[Jason O. <jo...@ra...>]

> Heck, can't vendors be bothered to repair the known defects in their
> software or at least sponsor workarounds in third-party software they
> are breaking?

Agreed. Of course, if I had a say (or if any of my fellow 80,000 minions 
had a say) we'd not be using Lotus.

> Some alternatives to FETCH 1234 RFC822.HEADER (where 1234 is the message
> number) I could think of:
> 
>   BODY.PEEK[HEADER]
Still missing fields.

>   BODY.PEEK[MIME]   - nonstandard, may cause BAD reply
BAD Invalid Fetch argument body.peek[mime]

>   BODY.PEEK[0.MIME] - nonstandard, may cause BAD reply
BAD Invalid Fetch argument body.peek[o.mime]

>   BODY.PEEK[HEADER.FIELDS (MIME-Version Content-Type Content-Transfer-Encoding Content-Disposition)]
This returned the fields, although only the Content-Transfer-Encoding 
was missing from the RFC822 header output.

>   BODY.PEEK[HEADER.FIELDS.NOT (foo)]
> 
> Does any of these provide you with the full header when RFC822.HEADER is
> incomplete?
> 
> What does
> 
>   FETCH 1 BODYSTRUCTURE
> 
> give you (replace 1 by the number of a 'broken' message)?

* 574 FETCH (BODYSTRUCTURE ("text" "html"  ("charset" "ISO-8859-1") NIL 
NIL "quoted-printable" 1287 22 NIL NIL NIL))
I don't know how to parse this output, although the strings presented 
are correct. Don't know what the numbers are supposed to reflect.


> A full IMAP4r1 session might look like:
> 
> A LOGIN USER SECRETPASSWORDGOESHERE
> B SELECT INBOX
> C FETCH 1 BODY.PEEK[HEADER]
> D LOGOUT
> 
> 
> Generally, fetchmail has support for protocols that can only retrieve
> full messages (POP3, for instance, uses that), so some hacking away at
> imap.c might be sufficient to make the rest of fetchmail believe that
> IMAP could only support full messages. The protocol interface is OOP
> style (yes it's clumsy in C, but it works), in that driver.c/transact.c
> call protocol (imap.c/pop3.c) methods^Wfunctions.
> 
> Best would be a workaround that could automatically kick in, since
> that's easiest for users and distributors.
> 
> Such code might require minor internal rearrangements, such as moving
> what used to be hard-wired protocol attributes into methods that query
> the capabilities of the current connection.
> 

I've used structs of function pointers to do similar things in the past. 
Few surprises in that area. =)
Sadly, I didn't manage to do an imap isolated patch yesterday. To get my 
hack to work, I mutilated imap.c to make the getsizes, getpartialsizes 
and fetch body NULL and changed fetch_headers to submit FETCH %d 
(BODY[]). I also had to change driver.c by adding len-=msgblk.msglen 
before the readbody. Otherwise, it'd try to read the whole message size 
off of the socket. I was very confused by the last bit, as it seemed 
that changing the fetch body to NULL should've induced driver.c to do 
the same to handle the POP protocols. I expect I missed something somewhere.

Is there a way to identify an imap server? The banner on login only states:
* OK IMAP Server Ready Tue, 8 Aug 2006 08:37:04 -0500


Thanks!
Jason

Re: [fetchmail-devel] Continuation of "Domino IMAP and missing Content-Transfer-Encoding"

[Jason O. <jo...@ra...>]

Matthias Andree wrote:
> Jason Ostermann <jo...@ra...> writes:
> 
> 
> Perhaps the reply to a CAPABILITY request can help with identifying the
> server.
> 
> What does Lotus respond to such a command?

* OK IMAP Server Ready Tue, 8 Aug 2006 17:11:19 -0500
a capability
* CAPABILITY IMAP4rev1 AUTH=PLAIN LITERAL+ NAMESPACE QUOTA UIDPLUS
a OK CAPABILITY completed


> 
> Another workaround idea: what if we check the received headers from the
> RFC822.HEADERS request if it contains "MIME-Header: 1.0", and if it
> does, but is missing Content-Transfer-Encoding, request the latter
> explicitly and attach it to the headers if we get one?
> 
> Such a workaround would reorder headers, but better that
> than a missing quoted-printable or base64 tag.
> 
> What do you think?
> 

Sounds like a PITA to code. =( Would that fit into the current execution 
scheme of
1) request headers
2) read headers
3) request body
4) ready body??

Or, at least, would it fit any better than making a single header/body 
read version of imap?

Jason

Re: [fetchmail-devel] Continuation of "Domino IMAP and missing Content-Transfer-Encoding"

[Matthias A. <mat...@gm...>]

Jason Ostermann <jo...@ra...> writes:

> Matthias Andree wrote:
>> Jason Ostermann <jo...@ra...> writes:
>> 
>> 
>> Perhaps the reply to a CAPABILITY request can help with identifying the
>> server.
>> 
>> What does Lotus respond to such a command?
>
> * OK IMAP Server Ready Tue, 8 Aug 2006 17:11:19 -0500
> a capability
> * CAPABILITY IMAP4rev1 AUTH=PLAIN LITERAL+ NAMESPACE QUOTA UIDPLUS
> a OK CAPABILITY completed

Not very distinctive.

>> Another workaround idea: what if we check the received headers from the
>> RFC822.HEADERS request if it contains "MIME-Header: 1.0", and if it
>> does, but is missing Content-Transfer-Encoding, request the latter
>> explicitly and attach it to the headers if we get one?
>> 
>> Such a workaround would reorder headers, but better that
>> than a missing quoted-printable or base64 tag.
>> 
>> What do you think?
>
> Sounds like a PITA to code. =( Would that fit into the current execution 
> scheme of
> 1) request headers
> 2) read headers
> 3) request body
> 4) ready body??

Sort of. We could move writing of the delimiter out of
transact.c/readheaders() into driver.c/fetch_messages(), in order to
make room for a new header fixup method, and make that method re-request
the missing header.

> Or, at least, would it fit any better than making a single header/body 
> read version of imap?

Whatever, I wonder if I should defer such a workaround for 6.4.0 since
it involves major code changes I'm not comfortable with for the stable
version, the bug isn't a new regression and it's really IBM's turn to
fix it.

-- 
Matthias Andree