Re: [Fetchmail-users] UTF-8 support?

[Matthias A. <mat...@gm...>]

Am 06.03.20 um 14:19 schrieb Sergey Organov:
> Thanks, that's what I've expected.
>
> To be honest, I don't even understand what actual problem those 2 RFCs
> are to solve. I mean I don't understand why server can't just send what
> it has "as is".

Traditionally, from "way back then"; meaning short after the first human
landing on the moon, we used parity bits in US-ASCII systems and nobody
thought of umlauts, Cyrillic or whatnot. The early transmission
protocols were 7-bit protocols, no 8th bit permitted, hence crutches
such as the UTF-7 stuff in IMAP, MIME content transfer encodings
(base64, quoted-printable) for headers and whatnot.

Some transmission protocols actively prohibit sending 8 bit data, and
have been extended that clients and servers can negotiate 8-bit clean
transmissions.

The UTF8 RFC set strives to pull the plug from that "you must only send
7-bit data" requirement, but requires negotiation between client and server.

> Do you mean: https://tools.ietf.org/html/rfc6858 ?
> Dunno, but to me it looks like server rather does not create surrogate
> messages.
You just showed us one, and I'll leave this snippet in.
>>> Please use an E-mail reader that supports POP3 with UTF-8
>>> (see https://tools.ietf.org/html/rfc6856.html).
>>>
>>> This can also happen when the sender's E-mail program does not
>>> correctly format the sent message.
>> What is your mail server type and address?
> It's corporate server mail.javad.com and I dunno its type. Probably I
> can figure through our IT team if needed.
No need, it identifies on the greeting line as Courier-IMAP. It appears
that Courier-IMAP 5.0.0 introduced UTF8 support in the way you're
currently seeing, quoting its ChangeLog:
> 5.0.0
>
> 2018-07-21  Sam Varshavchik  <mr...@co...>
>
>         * pop3dserver.c: update Courier-IMAP to support UTF8 POP3. Update
>         version of the courierpop3dsizelist cache file.
>
> 2018-07-17  Sam Varshavchik  <mr...@co...>
>
>         * courier-imap, sqwebmail: update Courier-IMAP to support UTF8
> IMAP.
>         Convert maildir folders to use UTF-8 for folder names. Add
>         --checkutf8 and --convutf8 options to maildirmake to convert
>         pre-UTF8 maildirs to UTF8 maildirs, a mandatory upgrade procedure.
>
> ...

> If I get a patch, I think I can test it against the server. I guess
> it should be something rather simple like sending additional
> command claiming UTF-8 support? 

Nah, there's a bit more to it because at least for IMAP I've seen this
incurs protocol changes from UTF-7 to UTF-8 after sending it, and I
haven't fully read the RFCs yet to know what's on the table to do with
them, so we'll at least need to:

* implement CAPABILITY detection

* implement sending UTF8 willingness commands where applicable

* assess where protocol is changed in a way that affects fetchmail's
command generators and response parsers

* assess where fetchmail documentation needs to be changed

* TBD


And it's not just that we can assume UTF8 a given. It's been around for
years, and thank you for pointing me to the RFCs, but I queried all my
servers that I regularly poll and none talks UTF8 today except my
Courier IMAP 5.0.8 test installation.

So nothing's simple...

>
> I didn't try to include attachment and I'd rather not, as it might be
> confidential.
Oh, of course. I only now understand that the mail server was wrapping
the original message with this surrogate banner.
> Why? Suppose the server got proper e-mail with UTF-8 encoded headers.
> Now fetchmail tries to fetch the e-mail, and server in turn tries to
> ensure that it will send content that will be understood by the
> requester. As fetchmail doesn't support relevant RFC indeed, server
> decides support is absent and issues warning message.
Actually it's more of a wrapper message, on second reading of your
original post.
> Anyway, there is no other mail client by fetchmail on my side that
> connects to the server, and my googling about the problem revealed that
> some recent mail servers start to issue such warning e-mail indeed when
> they can't detect proper UTF-8 support when server needs to present a
> message with UTF-8 encoded headers to client.
Yup, and client support doesn't seem abundant. Python's poplib appears
to support it in recent versions.

So, bottom line: I'll put this on the TODO for 6.5, but I will probably
not implement downgrading or conversion code, fetchmail will need to
assume the destination SMTP/LMTP server or MDA is 8bit clean
(8BITMIME/SMTPUTF8).

This will take a while. https://gitlab.com/fetchmail/fetchmail/-/issues/14