#68 quoted-printable not encoding umlaut / DKIM problems

[Benjamin Kaineder]

Hello!

I have the issue that DKIM fails on messages sent with Blat. Our ERP system uses Blat to send e-mails to a MTA, which sends them to Microsoft365 cloud servers where they are signed with DKIM and passed on to the recipient servers.

The problem is that DKIM fails with "message has been altered". Foremost, it's a problem with the upstream server, but in my attempt to fix it, I found Blat behaviour that I think is a bug in the encoding process.

(TEST A)
If I were to sent an e-mail with content "Hallo", the mail in the inbox contains these headers:
Content-Transfer-Encoding: 7BIT
Content-Type: text/plain; charset="ISO-8859-1"
The content is simply "Hallo" and the DKIM result is PASS

(TEST B)
With content "Hallö", the headers will be
Content-Transfer-Encoding: 8BIT
Content-Type: text/plain; charset="utf-8"
The DKIM result is FAIL (message altered)
Now, Blat sends these headers as
Content-Transfer-Encoding: 8BIT
Content-Type: text/plain; charset="ISO-8859-1"
That means some server upstream changes the charset which alters the body hash and results in a FAIL.
So far, it's a server problem.

(TEST C)
Since 8bit is problematic anyway, I tried using quoted-printable using the -mime switch.
Sending "Hallo=Hello", the headers sent and received look like this:
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset="ISO-8859-1"
The body is simply "Hallo=3DHello" and DKIM result is PASS.

(TEST D)
Great, now back to umlaut, sending "Hallö=Hello", the headers received are:
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset="utf-8"
The body is "Hallö=3DHello" and DKIM result of course is FAIL again.
Blat again specifies the charset as "ISO-8859-1" in its outgoing e-mail and an upstream mail server changes that.

My question is, why does the -mime switch not encode the "ö"?
In ISO-8859-1 the body should be "Hall=F6=3DHello" with the "ö" encoded as =F6. I can't input already encoded text since of course it becomes encoded again (e.g. "Hall=3DF6=3D3DHello").

Best regards,
Ben

[Benjamin Kaineder]

Doing some more testing, I tried to used -bodyF with a file saved with iso-8859-1 charset, same thing happened, the upstream server changes to utf-8 because of the "ö".

Consequentially, if I specify the charset as
-charset "UTF-8"
on the command line, the DKIM result is PASS
This is a fix at least for my test setup. Mails with umlauts are passing now.
But there are two caveats with this fix, one, mail headers are reading

Content-Transfer-Encoding: 8BIT
Content-Type: text/plain; charset="UTF-8"

So we are at 8bit encoding, which may break depending on mail servers in the chain.
Two, I do not know yet if I can use this in the ERP (which is still on cp1252 charset).

So my question still stands, shouldn't -mime encode 8bit characters?

BR, Ben

[Chip]

When using -mime and using 8-bit encoding, your umlaut character can be send as-is. When using -mime, but using 7-bit encoding, your umlaut should be encoded. If your SMTP server allows 8-bit mime encoding, then Blat will use that. Changing the mime function to always encode values greater than 0x7F is easy, I just need to disable one line in an "if" condition. As a quick test, if you use -charset utf-7, this should give you want you want. Please give this a try.

Chip

[Benjamin Kaineder]

Hello Chip,

Thanks for your response. I did extensive testing to isolate the reason. The summarize, I can send an e-mail in three ways:

1. do not specify charset, e-mail contains only ASCII => Blat specifies ISO-8859-1 / 7BIT => DKIM=PASS
2. do not specify charset, e-mail contains e.g. one umlaut => Blat specifies ISO-8859-1 / 8BIT => DKIM=FAIL
3. specify charset=UTF-8 => Blat specifies UTF-8 / 8BIT => DKIM=PASS

Option 2 fails because the "final" upstream server (Microsoft O365) does not like the ISO encoding and calculates a wrong hash. It's not the 8BIT per se because Option 3 is working. I did not find much on this issue but I strongly suspect it is a bug in O365.
The characters are rendered correctly in the receiving e-mail client in all three cases.


About UTF-7, you're correct:

Content-Transfer-Encoding: 7BIT
Content-Type: text/plain; charset="UTF-7"

Body is "Hell+APY-" and DKIM result is PASS

The issue I have with UTF-7 is that I think it's deprecated? I don't know how wide it's supported and how long this support will last. Especially with O365 since MS nixed it from .NET in 2022.


You wrote

When using -mime, but using 7-bit encoding, your umlaut should be encoded

But how do I force 7-bit encoding (besides UTF-7)?
If I use e.g. "-charset ASCII -mime" the umlaut is completely missing (and DKIM fails). So no conversion is happening.

I'm not sure, but shouldn't there be some option to enforce quoted-printable for all characters. Even if my SMTP proxy supports 8BITMIME I can't be sure for the complete chain and I'd want to maximize compatibility with all SMTP servers.

I can't speak to all use cases, but 8BIT plus quoted-printable does not make sense to me? If I used 8BIT, why would I also want quoted-printable? Either I encode all characters to stay in 7BIT mode, or I'm already in 8BIT territory and there is no need to encode anything.


JFYI, my pressing issue is resolved in this way: Our ERP sends two types of e-mails, one containing text/html and the other only text/plain.

1. With text/html, I now use ASCII encoding and escape all umlauts and other special characters using html entities (e.g. ö). The e-mail is then sent as 7BIT to maximize compatibility. I do not use quoted-printable since all characters are 7BIT.
2. For text/plain, there is no encoding like in html, so I output the body into a text file, convert it from CP1252 to UTF-8 and then send it as UTF-8 / 8BIT.

This way both types now have a working signature. The text/plain type could run into some issues due to 8BIT if there really is a system that does not support it.

Thanks,
Ben