#18 Bad handling of UTF-8 mail

open
nobody
None
5
2005-08-04
2005-07-09
Anonymous
No

It seems that when one replies to an UTF-8 mail
squirrelmail does not mark the result's encoding as
UTF-8 too

Unfortunately the quoted text is UTF-8 even if the
reply is pure iso-8859-1

Discussion

  • Tomas Kuliavas
    Tomas Kuliavas
    2005-07-10

    Logged In: YES
    user_id=225877

    enable loosy_encoding in 10. Language Options.

    SquirrelMail compose uses same charset that is set by
    selected translation and does not convert different charset
    to selected charset, if selected charset does not cover all
    characters from used charset. Conversion is enabled by
    default, only when utf-8 translation is used or email and
    translation uses matching Cyrillic and Baltic charsets
    (koi8-r, koi8-u, iso-8859-4, iso-8859-5, iso-8859-13,
    cp1251, cp1257).

    For example:
    a) email is utf-8, you use German translation in iso-8859-1.
    if loosy_encoding is not enabled, 8bit text is broken
    because iso-8859-1 covers only ~220 characters and utf-8 can
    contain millions of different chars.

    b) email is iso-8859-15, you use German translation in
    iso-8859-1. squirrelmail does not convert charsets by
    default, but you don't see problems, because there is only 6
    character difference between these charsets. You might see
    problem only when Euro symbol is used.

    c) email is in some charset, you use translation in utf-8 -
    squirrelmail will convert it to utf-8 by default, if charset
    decoding functions are present in functions/decode/. Unicode
    should cover all characters used in other charsets or these
    characters sets use symbols unlisted in unicode (i think, it
    can happen only in rarely used Eastern chars).

    d) email is written in koi-8r, you use German in iso-8859-1.
    if loosy_encoding is enabled and you reply to email - all
    Russian texts are converted to question marks.

    I think, this is not a bug. SquirrelMail is only using
    safer, but broken approach by default. 1.4.5/1.5.1 provide
    features that fix this issue.

    Maybe rewriting compose code to use Unicode can solve all
    encoding issues, but it needs cross browser testing.

     
  • Mailhot
    Mailhot
    2005-07-10

    Logged In: YES
    user_id=420924

    Well I strongly disagree it's not a bug. I use the french
    translation - how can I force UTF-8 everywhere ?

    (and no question marks is not the "right" thing either)

     
  • Tomas Kuliavas
    Tomas Kuliavas
    2005-07-10

    Logged In: YES
    user_id=225877

    enable loosy_encoding or change charset used in French
    translation (functions/i18n.php and all translation files).

    If loosy_encoding is enabled by default, it can strip
    unsupported characters. If encoding is not enabled by
    default, characters might look garbled. Both ways are
    broken. SquirrelMail uses older broken way.

    Only SquirrelMail translators can change character set used
    with packaged SquirrelMail translations. SquirrelMail
    developers are not going to enforce use of Unicode in every
    translation. Character set used in every translation can't
    be made configurable option, because gettext charset
    conversion is not supported in some setups and it would
    allow misconfiguration of interface.

    I think currently $loosy_encoding = true; is best thing that
    you can do for 1.4.x versions. Any other way requires more
    experimental or currently unimplemented/untested changes in
    SquirrelMail code.

    If you want to see how everything works in utf-8
    translation, test Russian or Lithuanian translations without
    locale/ru_RU/ and locale/lt_LT/ files installed. Interface
    will remain in English, system will use utf-8. Or set utf-8
    in custom_charset plugin. Or set default_charset to utf-8 in
    1.4.5rc1/1.5.1cvs and check English translation.

    SquirrelMail interface is implemented in html with minimal
    php requirements. We can't provide charset selection
    features that are supported by standalone email clients,
    because it is harder to implement those features in pure
    html/php or they might cause more problems than existing
    situation.

    We might solve it by using http-accept attribute in compose
    form and rewriting some parts of compose.php code, but these
    changes are radical and can cause more problems.

    If you know any better way to implement it without
    increasing SquirrelMail php requirements, patches are welcome.

     
  • Tomas Kuliavas
    Tomas Kuliavas
    2005-08-04

    • labels: 102905 -->