[stlport-bugs] [ stlport-Bugs-1670439 ] some locale tests fail in 5.1.1/2 under MSVC8 with Boost

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 454-5900

Bugs item #1670439, was opened at 2007-02-27 19:48
Message generated for change (Comment added) made by tartak
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=766244&aid=1670439&group_id=146814

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Alex Tartakovsky (tartak)
Assigned to: Francois Dumont (dums)
Summary: some locale tests fail in 5.1.1/2 under MSVC8 with Boost

Initial Comment:
8 tests failed (out of 383). All 8 are in LocaleTest.
Build under VStudio 2005 (SP1) with Boost 1.33.1
Output:
===============
LocaleTest::locale_by_name
LocaleTest::loc_has_facet
LocaleTest::num_put_get
	fr_FR
	en_GB
	en_US
	C
LocaleTest::money_put_get
	fr_FR
	en_GB

../../../test/unit\locale_test.cpp(316) : CPPUNIT_ASSERT(str_res.substr(index, p) == rl.money_prefix);
	en_US
	C
LocaleTest::money_put_X_bug
	fr_FR
	en_GB

../../../test/unit\locale_test.cpp(419) : CPPUNIT_ASSERT(str_res.substr(index, p) == rl.money_prefix);
	en_US
	C
LocaleTest::time_put_get
	fr_FR
	en_GB
	en_US
	C
LocaleTest::collate_facet
LocaleTest::ctype_facet
	fr_FR
	en_GB
	en_US
	C
LocaleTest::locale_init_problem
	fr_FR
	en_GB
	en_US
	C
LocaleTest::default_locale
LocaleTest::facet_id
LocaleTest::combine
	combining 'en_GB' money facets with 'fr_FR'

../../../test/unit\locale_test.cpp(316) : CPPUNIT_ASSERT(str_res.substr(index, p) == rl.money_prefix);

../../../test/unit\locale_test.cpp(316) : CPPUNIT_ASSERT(str_res.substr(index, p) == rl.money_prefix);

../../../test/unit\locale_test.cpp(316) : CPPUNIT_ASSERT(str_res.substr(index, p) == rl.money_prefix);

../../../test/unit\locale_test.cpp(316) : CPPUNIT_ASSERT(str_res.substr(index, p) == rl.money_prefix);
	combining 'en_US' money facets with 'en_GB'

../../../test/unit\locale_test.cpp(316) : CPPUNIT_ASSERT(str_res.substr(index, p) == rl.money_prefix);

../../../test/unit\locale_test.cpp(316) : CPPUNIT_ASSERT(str_res.substr(index, p) == rl.money_prefix);
	combining 'C' money facets with 'en_US'

----------------------------------------------------------------------

>Comment By: Alex Tartakovsky (tartak)
Date: 2007-03-09 14:00

Message:
Logged In: YES 
user_id=1331880
Originator: YES

Petr, what you describe is not what I do.  I use locale settings expressly
supported by OS, and documented by MS as such.  

Here is what I do, so you can reproduce it. I don't actually change the
codepage directly. I set up my locale settings (which indirectly changes
the codepage) in Control Panel/Regional and Language Options. I go to
Advanced tab there and select the "Language for non-Unicode Programs". I
change it from the default English(US) to Russian. This should not affect
Unicode programs. I need it simply because I have a few non-Unicode
programs where I want to be able to type in some text in Russian - they
would not show the text otherwise. By the way, on the same tab there is a
list of code page conversion tables installed on my computer: the ones you
mentioned, 1250-1252 are installed among many others, and I don't make any
changes there.

Now, when I change the language as described above, and then issue "chcp"
(without a parameter) command in console, I get 866. If I switch the
language back (using the same procedure) to English(US), chcp in console
gives me 437. 

According to MS Windows documentation of chcp, it shows the active console
code page. That code page is obviously what's the tests are run under.  The
documentation lists the supported code pages, here is the complete list
copied from there:

Code page Country/region or language 
437 United States 
850 Multilingual (Latin I) 
852 Slavic (Latin II) 
855 Cyrillic (Russian) 
857 Turkish 
860 Portuguese 
861 Icelandic 
863 Canadian-French 
865 Nordic 
866 Russian 
869 Modern Greek 

As you see, there is nothing outdated about 437 or 866. 437 is the
default, and 866 is supported.  All those 125x code pages are not on the
list for console.

It may very well be that there is absolutely nothing wrong with STLport,
and the problem is with the test itself. From the above, it appears clear
that, on a standard US Win XP installation, the console tests are run under
code page 437.  But the code page may be switched (even without user's
knowledge) to one of the listed above. I think the test should check the
code page or, at least, warn the user to switch to the proper code page -
the one needed for the test to succeed.

Regards, 
Lesha

----------------------------------------------------------------------

Comment By: Petr Ovtchenkov (complement)
Date: 2007-03-09 07:27

Message:
Logged In: YES 
user_id=615813
Originator: NO

In POSIX world locale assume both locale settings and charset. You can see
explicit specification of charset: "ru_RU.koi8r". Every locale has
'default' charset, that used when charset not mentioned explicitly. en_GB
assume ISO-8859-1. The pound sterling has code 0xa3 in ISO-8859-1, so we
hardcoded this value and don't worry.

But you use old (DOS) codepages (hehe, under system with native UTF16):
866 and 437 (hmmm, why not CP1251 and CP1250/CP1252?). Or why not give
chance to wins track all encodings via native UTF16?

Really, looks like you made such OE settings that lead to poor locales
support. And tests inform you about this indeed. STLport locales based on
system support for locale, so if system don't support [may be some]
locales, we haven't chances to do work correctly.

----------------------------------------------------------------------

Comment By: Francois Dumont (dums)
Date: 2007-03-09 04:54

Message:
Logged In: YES 
user_id=1096600
Originator: NO

No it is not something expected, it would be great to make this tests more
portable. I will try to play with my system settings to see if I can break
the tests like that and then I will see if forcing code page in
localization string can safe our life.

Bests

----------------------------------------------------------------------

Comment By: Alex Tartakovsky (tartak)
Date: 2007-03-06 14:32

Message:
Logged In: YES 
user_id=1331880
Originator: YES

I tested on Win XP, US version.  But the code page was switched to 866 -
Russian.  It's under 866 that I had those errors in LocalTest.  When I
switch back to the default 437 (US), all tests succeed.  Is it something to
be expected? I mean, the tests do not warn that they must be run under a
specific code page...

----------------------------------------------------------------------

Comment By: Francois Dumont (dums)
Date: 2007-03-04 14:56

Message:
Logged In: YES 
user_id=1096600
Originator: NO

I think the fact that you are in chinese explain why tests are not working
as expected. Could you try to specify code page in locale name and tell us
if you still have problem. You can do so replacing "fr_FR" with
"fr_FR.1252", idem for "en_EN" and "en_US".

There is a problem in your script, all builds are done configured with
msvc71 (configure -c msvc71). For VC6 and VC8 it should be different,
'configure -c msvc6' and 'configure -c msvc8' respectively.

As you are a chinese user I would like to take the opportunity of asking
you if you are happy with STLport and especially with its localization
support under Windows and Linux. Is it working fine with your language ?
Thanks for help.

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2007-03-03 09:39

Message:
Logged In: NO 

some UnitTest also failed in STLport SVN(2945) under MSVC71(.Net 2003)
with SP1(KB918007) on Window XP sp2 (Traditional Chinese Version),
and Microsoft (R) Program Maintenance Utility Version 7.10.3077

Output:
=======================
LocaleTest::num_put_get
        fr_FR
        en_GB
        en_US
        C
LocaleTest::money_put_get
        fr_FR

../../../test/unit\locale_test.cpp(243) : CPPUNIT_ASSERT(str_res[index++]
== *rl.money_thousands_sep);
        en_GB

../../../test/unit\locale_test.cpp(316) :
CPPUNIT_ASSERT(str_res.substr(index, p) == rl.money_prefix);
        en_US
        C
LocaleTest::money_put_X_bug
        fr_FR
        en_GB

../../../test/unit\locale_test.cpp(419) :
CPPUNIT_ASSERT(str_res.substr(index, p) == rl.money_prefix);
        en_US
        C
LocaleTest::time_put_get
        fr_FR
        en_GB
        en_US
        C
LocaleTest::collate_facet
LocaleTest::ctype_facet
        fr_FR
        en_GB
        en_US
        C
LocaleTest::locale_init_problem
        fr_FR
        en_GB
        en_US
        C
LocaleTest::default_locale
LocaleTest::facet_id
LocaleTest::combine
        combining 'en_GB' money facets with 'fr_FR'

../../../test/unit\locale_test.cpp(316) :
CPPUNIT_ASSERT(str_res.substr(index, p) == rl.money_prefix);

../../../test/unit\locale_test.cpp(243) : CPPUNIT_ASSERT(str_res[index++]
== *rl.money_thousands_sep);

../../../test/unit\locale_test.cpp(316) :
CPPUNIT_ASSERT(str_res.substr(index, p) == rl.money_prefix);

../../../test/unit\locale_test.cpp(316) :
CPPUNIT_ASSERT(str_res.substr(index, p) == rl.money_prefix);

../../../test/unit\locale_test.cpp(243) : CPPUNIT_ASSERT(str_res[index++]
== *rl.money_thousands_sep);

../../../test/unit\locale_test.cpp(316) :
CPPUNIT_ASSERT(str_res.substr(index, p) == rl.money_prefix);
        combining 'en_US' money facets with 'en_GB'

../../../test/unit\locale_test.cpp(316) :
CPPUNIT_ASSERT(str_res.substr(index, p) == rl.money_prefix);

../../../test/unit\locale_test.cpp(316) :
CPPUNIT_ASSERT(str_res.substr(index, p) == rl.money_prefix);
        combining 'C' money facets with 'en_US'

========================
My build using almost the same batch scripts as followings except some
echo strings and comments are in big5 encoding. The batch scripts was
placed under the directory the same as INSTALL, doc, etc, and lib (i.e. the
root of Stlport package).

=======batch_start=======
@ECHO OFF
:: Was nmake available?
nmake /?
IF %ERRORLEVEL% EQU 0 GOTO START_COMPILE

IF EXIST "%VS80COMNTOOLS%vsvars32.bat" GOTO VC80
IF EXIST "%VS71COMNTOOLS%vsvars32.bat" GOTO VC71
IF EXIST "C:\Program Files\Microsoft Visual Studio\VC98\Bin\VCVARS32.bat"
GOTO VC6

:VC6
CALL "C:\Program Files\Microsoft Visual Studio\VC98\Bin\VCVARS32.bat"
GOTO START_COMPILE

:VC71
CALL "%VS71COMNTOOLS%vsvars32.bat"
GOTO START_COMPILE

:VC80
CALL "%VS80COMNTOOLS%vsvars32.bat"
GOTO START_COMPILE

:START_COMPILE
CD build\lib
CALL configure.bat --clean
CALL configure.bat -c msvc71 --extra-cxxflag /G7
nmake /f msvc.mak clean all install
cd ..\..\
GOTO END

:ERROR_MSG
ECHO Could not find vcvars32.bat!
GOTO END

:END
@ECHO on

=======End of batch=======

And to build unit-test
=======batch start=======
CD build\test\unit
nmake /f msvc.mak clean install
CD ..\..\..\

CALL bin\stl_unit_test.exe
PAUSE
CALL bin\stl_unit_testd.exe
PAUSE
CALL bin\stl_unit_testd_static.exe
PAUSE
CALL bin\stl_unit_teststld.exe
PAUSE
CALL bin\stl_unit_teststld_static.exe
PAUSE
CALL bin\stl_unit_test_static.exe
PAUSE
=======end of batch=======

Thanks to all authors of Stlport :)

----------------------------------------------------------------------

Comment By: Francois Dumont (dums)
Date: 2007-03-02 03:16

Message:
Logged In: YES 
user_id=1096600
Originator: NO

The localization unit tests highly depends on your platform settings.
Haven't you touch to your english locale money symbol setting for instance
the euro (€) sign for english people :-) Expected values are hard coded
in test so if you done so it might explain failure.

Have you check that boost support has any impact on this failure ?

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=766244&aid=1670439&group_id=146814