[Tcl-bugs] [ tcl-Bugs-1513659 ] Unicode-related tests fail for non-C LANG

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 454-5900

Bugs item #1513659, was opened at 2006-06-27 23:43
Message generated for change (Comment added) made by dkf
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=110894&aid=1513659&group_id=10894

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: 34. tcltest Package
Group: obsolete: 8.4.13
>Status: Closed
>Resolution: Fixed
Priority: 5
Private: No
Submitted By: Mikhail Teterin (kot)
Assigned to: Donal K. Fellows (dkf)
Summary: Unicode-related tests fail for non-C LANG

Initial Comment:
When running tests with LANG set to uk_UA.KOI8-U, for 
example, some of the Unicode-using tests fail.

These are in exec and env, but not only.

The other annoying part, is that, when tcltest 
outputs the failures to the screen, it does not set 
stderr's encoding to -binary, so both -- the expected 
and the actual -- results appear as identical 
question marks.

----------------------------------------------------------------------

>Comment By: Donal K. Fellows (dkf)
Date: 2009-05-07 11:37

Message:
Found a way to recheck and fixed the problems that remained. (There were
tests with unwarranted assumptions about the system encoding...)

Suggest no changes to tcltest; better to write tests to be robust and only
use ASCII scripts and comparisons (possible with suitable quoting and
careful test design).

----------------------------------------------------------------------

Comment By: Donal K. Fellows (dkf)
Date: 2009-03-24 16:14

Message:
What's the current state of this issue now that at least some of it has
been fixed? I can't recheck because I'm on OSX where everything is UTF-8
anyway for all locales.

----------------------------------------------------------------------

Comment By: Donal K. Fellows (dkf)
Date: 2008-07-19 11:03

Message:
Logged In: YES 
user_id=79902
Originator: NO

env.test now fixed on HEAD; it didn't need to produce non-ASCII characters
to test what it was testing

----------------------------------------------------------------------

Comment By: Donal K. Fellows (dkf)
Date: 2006-08-09 15:24

Message:
Logged In: YES 
user_id=79902

Perhaps the best way forward is to provide an option to
tcltest which tells it to \u encode any non-ascii chars in
its output? That would at least mean that if things are
going wrong, you can set something so that you can figure
out what's failed in a way that lets you start bug
hunting... :-)

----------------------------------------------------------------------

Comment By: Mikhail Teterin (kot)
Date: 2006-07-03 18:25

Message:
Logged In: YES 
user_id=173641

As I mentioned from the beginning, uk_UA.KOI8-U is just an 
example. ru_RU.ISO8859-5 is causing similar breakage. As do 
la_LN.ISO8859-4, hu_HU.ISO8859-2, while those matching 
*ISO8859-1* seem to work.

==== env-4.3 setting international environment variables 
FAILED
==== Contents of test case:

    set env(\ua7) \ub6
    getenv

---- Result was:
?=?
---- Result should have been (exact matching):
?=?
==== env-4.3 FAILED

==== env-4.4 changing international environment variables 
FAILED
==== Contents of test case:

    set env(\ua7) \ua7
    getenv

---- Result was:
?=?
---- Result should have been (exact matching):
?=?
==== env-4.4 FAILED

==== env-4.5 unsetting international environment variables 
FAILED
==== Contents of test case:

    set env(\ub6) \ua7
    unset env(\ua7)
    set result [getenv]
    unset env(\ub6)
    set result

---- Result was:
?=?
---- Result should have been (exact matching):
?=?
==== env-4.5 FAILED

error.test
eval.test
event.test
exec.test

==== exec-2.6 redirecting input from immediate source, with 
UTF FAILED
==== Contents of test case:

    # If this fails, it may give back:
    # "\uC3\uA9\uC3\uA0\uC3\uBC\uC3\uB1"
    # If it does, this means that the UTF -> external 
conversion did not 
    # occur before writing out the temp file.
    exec [interpreter] $path(cat) << "\uE9\uE0\uFC\uF1"

---- Result was:
????
---- Result should have been (exact matching):
????
==== exec-2.6 FAILED

execute.test
expr-old.test
expr.test

==== expr-8.13 CompileBitAndExpr: equality expr FAILED
==== Contents of test case:
expr {"\374" eq "Э"}
---- Result was:
0
---- Result should have been (exact matching):
1
==== expr-8.13 FAILED
[...]
unixInit.test

==== unixInit-2.4 TclpInitLibraryPath: TCL_LIBRARY: INTL 
FAILED
==== Contents of test case:

    # Child process translates env variable from native 
encoding.

    set env(TCL_LIBRARY) "\xa7"
    set x [lindex [getlibpath] 0]
    unset env(TCL_LIBRARY)
    unset env(LANG)

    set x

---- Result was:
?
---- Result should have been (exact matching):
?
==== unixInit-2.4 FAILED

----------------------------------------------------------------------

Comment By: Donal K. Fellows (dkf)
Date: 2006-07-03 14:54

Message:
Logged In: YES 
user_id=79902

<sigh>

I've not got a machine that has the uk_UA.KOI8-U locale
installed anyway, and libc's fallback that I have got (the C
locale!) works for me, but that's obviously not useful given
the report. :-)

As such, we *need* the submitter to cut-n-paste the entire
log of what is failing inside the test suite (or inside Tcl
itself) in here so that we can chase down what is going
wrong. Attach it as a patch if it is too big to paste. Thanks.

I'm not sure how to proceed with the stdout/stderr encoding
issue, or at least not in a way that won't cause trouble
elsewhere. Just configuring the output to be done in
-translation binary mode is not enough, as that works by
chopping high bytes from characters, and so loses
information in a different way. Perhaps the right approach
is to use -translation binary after an [encoding convertto
utf-8]? Or maybe to provide some kind of option to turn on
this behaviour, even if it is not default?

----------------------------------------------------------------------

Comment By: Don Porter (dgp)
Date: 2006-07-03 14:20

Message:
Logged In: YES 
user_id=80530

I'd rather not work on something
without a partnership with the
reporter.  You take it, dkf.

----------------------------------------------------------------------

Comment By: Mikhail Teterin (kot)
Date: 2006-07-03 14:15

Message:
Logged In: YES 
user_id=173641

Dgp, just set the LANG to anything (other than "C") and run
the tests -- I tried the 8.4.13, but it should be
reproducible with any.

You'll see, and be able to enumerate the failing tests
yourself. 

----------------------------------------------------------------------

Comment By: Donal K. Fellows (dkf)
Date: 2006-07-03 09:11

Message:
Logged In: YES 
user_id=79902

I think there is (independently of everything else) a
tcltest issue here relating to the loss of information when
tests fail without providing enough information to hunt the
error. Because of this, I'm re-marking this bug as a tcltest
issue.

But on the other issues (test numbers, version number) I
quite agree with dgp.

----------------------------------------------------------------------

Comment By: Don Porter (dgp)
Date: 2006-06-30 20:15

Message:
Logged In: YES 
user_id=80530

Please report the names of the
failing tests so the issue can be
assigned to the right maintainer.

Please also report what version
of Tcl you see the failures in.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=110894&aid=1513659&group_id=10894

[Tcl-bugs] [ tcl-Bugs-1513659 ] Unicode-related tests fail for non-C LANG

The Tool Command Language implementation

[Tcl-bugs] [ tcl-Bugs-1513659 ] Unicode-related tests fail for non-C LANG