#30 Subject/From Headers Are Lost When Replying in Japanese

[tetsuo13]

When replying to received messages from other mail
services that are in Japanese (iso-2022-jp),
SquirrelMail seems to mis-encode the first few
characters of the Subject and From name headers. It
appears to be mis-encoded because if the Subject and
From name headers are in Japanese, the first few
characters will be missing and instead some random
roman letters will be in their place.

I've tried replying to Hotmail Japan, Gmail with
Japanese enabled, and Yahoo! Japan. Every message
received from any of these other mail service providers
falls under this bahavior.

It's unclear to me what the cause of this is but it
appears, in the case of Subject, that SM decodes the
header when adding "Re: " and something happens when
re-encoding.

The From name is generally separated with a half-width
space between the first and last name. I belive this is
the cause for everything after the space in the name to
be incorrectly encoded when replying.

Example from Hotmail Japan. (This is from an actual
email, the last name has been replaced with "〜") The
original From name and Subject headers:

From: 〜 圭子
Subject: ひさしぶりのメールでごめんね(´∩｀.)

When replying to this message, the generated To and
Subject headers:

To: 〜 -P
Subject: Re: rUWvりのメールでごめんね(=
?ISO-2022-JP?B?GyRCIS0iQSEuGyhC?=.)

[Tomas Kuliavas]

Logged In: YES
user_id=225877

SquirrelMail version?

Selected SquirrelMail translation?

[tetsuo13]

Logged In: YES
user_id=1015520

SquirrelMail 1.4.5 with Japanese selected.

[Tomas Kuliavas]

Logged In: YES
user_id=225877

Could you provide information about OS, webserver, used php
version, mbstring extension settings, used browser and
headers of affected email?

[tetsuo13]

Logged In: YES
user_id=1015520

OS started out as RedHat 9.
Apache HTTP 2.0.55.
PHP 4.4.1 compiled with enable-mbregex, enable-mbstring=all,
enable-mbstr-enc-trans.
I can't remember how long this has been happening, I would
guess since Firefox 1.0.3 (?). Using Firefox 1.5 now but
also happening with IE 6.

Headers of the email as received (Email address and
originating IP address has been removed by me):

Return-Path: <xxxxxxxxx@hotmail.co.jp>
Received: from mail.neo-anime.org (localhost.localdomain
[127.0.0.1])
by neo-anime.org (Cyrus v2.2.12) with LMTPA;
Thu, 01 Dec 2005 05:04:07 -0500
X-Sieve: CMU Sieve 2.2
Received: from bay111-f23.bay111.hotmail.com ([64.4.17.33]
helo=hotmail.com)
by mail.neo-anime.org with esmtp (Exim 4.60)
(envelope-from <xxxxxxxxx@hotmail.co.jp>)
id 1EhlIK-0005j4-GD
for andre@neo-anime.org; Thu, 01 Dec 2005 05:04:07 -0500
Received: from mail pickup service by hotmail.com with
Microsoft SMTPSVC;
Thu, 1 Dec 2005 02:03:52 -0800
Received-SPF: none ()
Message-ID: <BAY111-F2378BBA7C3BBCA99B9640C994D0@phx.gbl>
Received: from 64.4.17.200 by by111fd.bay111.hotmail.msn.com
with HTTP;
Thu, 01 Dec 2005 10:03:52 GMT
X-Originating-IP: [x.x.x.x]
X-Originating-Email: [xxxxxxxxx@hotmail.co.jp]
X-Sender: xxxxxxxxx@hotmail.co.jp
From: =?iso-2022-jp?B?GyRCPi5AbhsoQiAbJEI3PTtSGyhC?=
<xxxxxxxxx@hotmail.co.jp>
To: andre@neo-anime.org
Bcc:
Subject:
=?iso-2022-jp?B?GyRCJFIkNSQ3JFYkaiROJWEhPCVrJEckNCRhJHMkTRsoQig=?=
=?iso-2022-jp?B?GyRCIS0iQSEuGyhCLik=?=
Date: Thu, 01 Dec 2005 19:03:52 +0900
Mime-Version: 1.0
Content-Type: text/plain; charset=iso-2022-jp; format=flowed
X-OriginalArrivalTime: 01 Dec 2005 10:03:52.0595 (UTC)
FILETIME=[886A3E30:01C5F65E]

Headers of the reply email generated by SM (IPs and email
removed again):

Received: from xxxxxxxxxx.xxxx.xxx ([x.x.x.x])
(SquirrelMail authenticated user andre@neo-anime.org)
by www.neo-anime.org with HTTP;
Thu, 1 Dec 2005 17:56:01 -0500 (EST)
Message-ID:
<52847.x.x.x.x.1133477761.squirrel@www.neo-anime.org>
In-Reply-To: <BAY111-F2378BBA7C3BBCA99B9640C994D0@phx.gbl>
References: <BAY111-F2378BBA7C3BBCA99B9640C994D0@phx.gbl>
Date: Thu, 1 Dec 2005 17:56:01 -0500 (EST)
Subject: Re:

=?ISO-2022-JP?B?clVXdhskQiRqJE4lYSE8JWskRyQ0JGEkcyRNGyhC?=(=
?ISO-2022-JP?B?GyRCIS0iQSEuGyhC?=.)
From: "Andre Nicholson" <andre@neo-anime.org>
To: =?ISO-2022-JP?B?GyRCPi5AbhsoQg==?= -P
<xxxxxxxxx@hotmail.co.jp>
Reply-To: andre@neo-anime.org
User-Agent: SquirrelMail/1.4.5
MIME-Version: 1.0
Content-Type: text/plain;charset=iso-2022-jp
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
Importance: Normal

[Tomas Kuliavas]

Logged In: YES
user_id=225877

Find 'encodeheader' part in japanese_charset_xtra() function
(functions/i18n.php file or similar function in
locale/ja_JP/setup.php file for 1.5.1cvs) and replace whole
encodeheader code with

----
$ret = mb_encode_mimeheader($ret,'iso-2022-jp','B','');
break;
----

Break call is not used in 1.5.1cvs.

Current implementation tries to make some smart decisions
and incorrectly combines multiline headers when first symbol
in second line is standard us-ascii character.

Fix disables any smart header parsing and should encode
header correctly. I think it should work correctly. Header
decoding code still has problems with \r\n\s+ folding and
might add extra spaces between Re: and encoded header part.

[tetsuo13]

Logged In: YES
user_id=1015520

I made the change to i18n.php as directed.

When I brought up the same original email message and hit
the reply button, this is what I noticed. The Subject form
input correctly displayed the original subject along with
the prefixed "Re: " as expected, but the To input got the
name completely wrong, displaying "$B&gt;.@n(B $B7=;R(B
<xxxxxxxxx@hotmail.co.jp>" (email address replaced by me).
So I clicked on 'Send' anyway.

I received an error message, "Server replied: 501
<$B>.@n>: missing or malformed local part," and was shown
the message reply screen once again. The interesting thing
is that now the To input correctly displayed the name: first
and last along with the space inbetween. I click on 'Send'
again and it goes through.

Viewing the Sent folder, the recipient of this email is
correctly displayed in the messages list but the subject
displays as "Re: rUWvりのメールでごめんね(´∩｀.)". It
appears that the added "Re: " prefix of the subject once
again caused incorrect encoding, although it did display
properly in the form input when I replied to the message.
Here are the headers of the message which went out:

Received: from xx.xxxxxxxxx.xxx ([x.x.x.x])
(SquirrelMail authenticated user andre@neo-anime.org)
by mail.neo-anime.org with HTTP;
Thu, 15 Dec 2005 06:45:43 -0500 (EST)
Message-ID:
<46616.x.x.x.x.1134647143.squirrel@mail.neo-anime.org>
In-Reply-To: <BAY111-F2378BBA7C3BBCA99B9640C994D0@phx.gbl>
References: <BAY111-F2378BBA7C3BBCA99B9640C994D0@phx.gbl>
Date: Thu, 15 Dec 2005 06:45:43 -0500 (EST)
Subject: Re:

=?ISO-2022-JP?B?clVXdhskQiRqJE4lYSE8JWskRyQ0JGEkcyRNGyhCKBskQiEtGyhC?=
=?ISO-2022-JP?B?GyRCIkEhLhsoQi4p?=
From: "Andre Nicholson" <andre@neo-anime.org>
To: =?ISO-2022-JP?B?GyRCPi5AbhsoQiAbJEI3PTtSGyhC?=
<xxxxxxxxx@hotmail.co.jp>
User-Agent: SquirrelMail/1.4.5
MIME-Version: 1.0
Content-Type: text/plain;charset=iso-2022-jp
Content-Transfer-Encoding: 8bit
X-Priority: 3 (Normal)
Importance: Normal

[Tomas Kuliavas]

Logged In: YES
user_id=225877

To: issue is reproducable only in 4.4.1 or depends on php
mbstring extension configuration. Haven't noticed it in php
4.3.10 system used for testing.

[Tomas Kuliavas]

Logged In: YES
user_id=225877

will have to recheck encoding function during weekend.
confused fix with broken modification in other part of code.

[Tomas Kuliavas]

Logged In: YES
user_id=225877

if (sq_is8bit($ret)) {
$ret = mb_encode_mimeheader($ret,'iso-2022-jp','b','');
}
break;

In some cases SquirrelMail compose.php tries to encode
already encoded header.