#30 Subject/From Headers Are Lost When Replying in Japanese

open
nobody
None
5
2006-10-09
2005-12-09
tetsuo13
No

When replying to received messages from other mail
services that are in Japanese (iso-2022-jp),
SquirrelMail seems to mis-encode the first few
characters of the Subject and From name headers. It
appears to be mis-encoded because if the Subject and
From name headers are in Japanese, the first few
characters will be missing and instead some random
roman letters will be in their place.

I've tried replying to Hotmail Japan, Gmail with
Japanese enabled, and Yahoo! Japan. Every message
received from any of these other mail service providers
falls under this bahavior.

It's unclear to me what the cause of this is but it
appears, in the case of Subject, that SM decodes the
header when adding "Re: " and something happens when
re-encoding.

The From name is generally separated with a half-width
space between the first and last name. I belive this is
the cause for everything after the space in the name to
be incorrectly encoded when replying.

Example from Hotmail Japan. (This is from an actual
email, the last name has been replaced with "〜") The
original From name and Subject headers:

From: 〜 圭子
Subject: ひさしぶりのメールでごめんね(´∩`.)

When replying to this message, the generated To and
Subject headers:

To: 〜 -P
Subject: Re: rUWvりのメールでごめんね(=
?ISO-2022-JP?B?GyRCIS0iQSEuGyhC?=.)

Discussion

  • Tomas Kuliavas

    Tomas Kuliavas - 2005-12-13

    Logged In: YES
    user_id=225877

    SquirrelMail version?

    Selected SquirrelMail translation?

     
  • tetsuo13

    tetsuo13 - 2005-12-13

    Logged In: YES
    user_id=1015520

    SquirrelMail 1.4.5 with Japanese selected.

     
  • Tomas Kuliavas

    Tomas Kuliavas - 2005-12-14

    Logged In: YES
    user_id=225877

    Could you provide information about OS, webserver, used php
    version, mbstring extension settings, used browser and
    headers of affected email?

     
  • tetsuo13

    tetsuo13 - 2005-12-14

    Logged In: YES
    user_id=1015520

    OS started out as RedHat 9.
    Apache HTTP 2.0.55.
    PHP 4.4.1 compiled with enable-mbregex, enable-mbstring=all,
    enable-mbstr-enc-trans.
    I can't remember how long this has been happening, I would
    guess since Firefox 1.0.3 (?). Using Firefox 1.5 now but
    also happening with IE 6.

    Headers of the email as received (Email address and
    originating IP address has been removed by me):

    Return-Path: <xxxxxxxxx@hotmail.co.jp>
    Received: from mail.neo-anime.org (localhost.localdomain
    [127.0.0.1])
    by neo-anime.org (Cyrus v2.2.12) with LMTPA;
    Thu, 01 Dec 2005 05:04:07 -0500
    X-Sieve: CMU Sieve 2.2
    Received: from bay111-f23.bay111.hotmail.com ([64.4.17.33]
    helo=hotmail.com)
    by mail.neo-anime.org with esmtp (Exim 4.60)
    (envelope-from <xxxxxxxxx@hotmail.co.jp>)
    id 1EhlIK-0005j4-GD
    for andre@neo-anime.org; Thu, 01 Dec 2005 05:04:07 -0500
    Received: from mail pickup service by hotmail.com with
    Microsoft SMTPSVC;
    Thu, 1 Dec 2005 02:03:52 -0800
    Received-SPF: none ()
    Message-ID: <BAY111-F2378BBA7C3BBCA99B9640C994D0@phx.gbl>
    Received: from 64.4.17.200 by by111fd.bay111.hotmail.msn.com
    with HTTP;
    Thu, 01 Dec 2005 10:03:52 GMT
    X-Originating-IP: [x.x.x.x]
    X-Originating-Email: [xxxxxxxxx@hotmail.co.jp]
    X-Sender: xxxxxxxxx@hotmail.co.jp
    From: =?iso-2022-jp?B?GyRCPi5AbhsoQiAbJEI3PTtSGyhC?=
    <xxxxxxxxx@hotmail.co.jp>
    To: andre@neo-anime.org
    Bcc:
    Subject:
    =?iso-2022-jp?B?GyRCJFIkNSQ3JFYkaiROJWEhPCVrJEckNCRhJHMkTRsoQig=?=
    =?iso-2022-jp?B?GyRCIS0iQSEuGyhCLik=?=
    Date: Thu, 01 Dec 2005 19:03:52 +0900
    Mime-Version: 1.0
    Content-Type: text/plain; charset=iso-2022-jp; format=flowed
    X-OriginalArrivalTime: 01 Dec 2005 10:03:52.0595 (UTC)
    FILETIME=[886A3E30:01C5F65E]

    Headers of the reply email generated by SM (IPs and email
    removed again):

    Received: from xxxxxxxxxx.xxxx.xxx ([x.x.x.x])
    (SquirrelMail authenticated user andre@neo-anime.org)
    by www.neo-anime.org with HTTP;
    Thu, 1 Dec 2005 17:56:01 -0500 (EST)
    Message-ID:
    <52847.x.x.x.x.1133477761.squirrel@www.neo-anime.org>
    In-Reply-To: <BAY111-F2378BBA7C3BBCA99B9640C994D0@phx.gbl>
    References: <BAY111-F2378BBA7C3BBCA99B9640C994D0@phx.gbl>
    Date: Thu, 1 Dec 2005 17:56:01 -0500 (EST)
    Subject: Re:

    =?ISO-2022-JP?B?clVXdhskQiRqJE4lYSE8JWskRyQ0JGEkcyRNGyhC?=(=
    ?ISO-2022-JP?B?GyRCIS0iQSEuGyhC?=.)
    From: "Andre Nicholson" <andre@neo-anime.org>
    To: =?ISO-2022-JP?B?GyRCPi5AbhsoQg==?= -P
    <xxxxxxxxx@hotmail.co.jp>
    Reply-To: andre@neo-anime.org
    User-Agent: SquirrelMail/1.4.5
    MIME-Version: 1.0
    Content-Type: text/plain;charset=iso-2022-jp
    Content-Transfer-Encoding: 7bit
    X-Priority: 3 (Normal)
    Importance: Normal

     
  • Tomas Kuliavas

    Tomas Kuliavas - 2005-12-15

    Logged In: YES
    user_id=225877

    Find 'encodeheader' part in japanese_charset_xtra() function
    (functions/i18n.php file or similar function in
    locale/ja_JP/setup.php file for 1.5.1cvs) and replace whole
    encodeheader code with

    ----
    $ret = mb_encode_mimeheader($ret,'iso-2022-jp','B','');
    break;
    ----

    Break call is not used in 1.5.1cvs.

    Current implementation tries to make some smart decisions
    and incorrectly combines multiline headers when first symbol
    in second line is standard us-ascii character.

    Fix disables any smart header parsing and should encode
    header correctly. I think it should work correctly. Header
    decoding code still has problems with \r\n\s+ folding and
    might add extra spaces between Re: and encoded header part.

     
  • tetsuo13

    tetsuo13 - 2005-12-15

    Logged In: YES
    user_id=1015520

    I made the change to i18n.php as directed.

    When I brought up the same original email message and hit
    the reply button, this is what I noticed. The Subject form
    input correctly displayed the original subject along with
    the prefixed "Re: " as expected, but the To input got the
    name completely wrong, displaying "$B&gt;.@n(B $B7=;R(B
    <xxxxxxxxx@hotmail.co.jp>" (email address replaced by me).
    So I clicked on 'Send' anyway.

    I received an error message, "Server replied: 501
    <$B>.@n>: missing or malformed local part," and was shown
    the message reply screen once again. The interesting thing
    is that now the To input correctly displayed the name: first
    and last along with the space inbetween. I click on 'Send'
    again and it goes through.

    Viewing the Sent folder, the recipient of this email is
    correctly displayed in the messages list but the subject
    displays as "Re: rUWvりのメールでごめんね(´∩`.)". It
    appears that the added "Re: " prefix of the subject once
    again caused incorrect encoding, although it did display
    properly in the form input when I replied to the message.
    Here are the headers of the message which went out:

    Received: from xx.xxxxxxxxx.xxx ([x.x.x.x])
    (SquirrelMail authenticated user andre@neo-anime.org)
    by mail.neo-anime.org with HTTP;
    Thu, 15 Dec 2005 06:45:43 -0500 (EST)
    Message-ID:
    <46616.x.x.x.x.1134647143.squirrel@mail.neo-anime.org>
    In-Reply-To: <BAY111-F2378BBA7C3BBCA99B9640C994D0@phx.gbl>
    References: <BAY111-F2378BBA7C3BBCA99B9640C994D0@phx.gbl>
    Date: Thu, 15 Dec 2005 06:45:43 -0500 (EST)
    Subject: Re:

    =?ISO-2022-JP?B?clVXdhskQiRqJE4lYSE8JWskRyQ0JGEkcyRNGyhCKBskQiEtGyhC?=
    =?ISO-2022-JP?B?GyRCIkEhLhsoQi4p?=
    From: "Andre Nicholson" <andre@neo-anime.org>
    To: =?ISO-2022-JP?B?GyRCPi5AbhsoQiAbJEI3PTtSGyhC?=
    <xxxxxxxxx@hotmail.co.jp>
    User-Agent: SquirrelMail/1.4.5
    MIME-Version: 1.0
    Content-Type: text/plain;charset=iso-2022-jp
    Content-Transfer-Encoding: 8bit
    X-Priority: 3 (Normal)
    Importance: Normal

     
  • Tomas Kuliavas

    Tomas Kuliavas - 2005-12-15

    Logged In: YES
    user_id=225877

    To: issue is reproducable only in 4.4.1 or depends on php
    mbstring extension configuration. Haven't noticed it in php
    4.3.10 system used for testing.

     
  • Tomas Kuliavas

    Tomas Kuliavas - 2005-12-15
    • assigned_to: nobody --> tokul
     
  • Tomas Kuliavas

    Tomas Kuliavas - 2005-12-15

    Logged In: YES
    user_id=225877

    will have to recheck encoding function during weekend.
    confused fix with broken modification in other part of code.

     
  • Tomas Kuliavas

    Tomas Kuliavas - 2005-12-19

    Logged In: YES
    user_id=225877

    if (sq_is8bit($ret)) {
    $ret = mb_encode_mimeheader($ret,'iso-2022-jp','b','');
    }
    break;

    In some cases SquirrelMail compose.php tries to encode
    already encoded header.

     
  • Tomas Kuliavas

    Tomas Kuliavas - 2006-10-09
    • assigned_to: tokul --> nobody
     

Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:

JavaScript is required for this form.





No, thanks