Thread: [Cppcms-users] Boost Locale: Undefined behavior in POSIX backend
Brought to you by:
artyom-beilis
Menu
▾
▴
cppcms-users
|
From: Kevin A. <kev...@ne...> - 2010-09-16 19:37:38
|
There is an issue in the POSIX backend that causes a crash on Mac OS X, and undefined behavior elsewhere. In file libs/locale/src/posix/codecvt.cpp: The declaration of iconv_t d at line 41 shadows the declaration of d at line 38. This causes undefined behavior at line 85 in the call to iconv_close. On Mac OS X this causes a crash, though I would expect there to be at least a memory leak on every platform. |
|
From: Artyom <art...@ya...> - 2010-09-16 19:55:03
|
Thank you very much! I've got reports of faults on Mac OS X however I hadn't this platform to debug. My brother usually helps me with Mac platform, but now he is busy. Thank you for this good point. I commit the changes to both CppCMS and Boost.Locale svn (https://cppcms.svn.sourceforge.net/svnroot/cppcms/boost_locale/branches/rework) Would you be so kind to check if tests now pass and what fails? Thank you, Artyom ----- Original Message ---- > From: Kevin Arunski <kev...@ne...> > To: cpp...@li... > Sent: Thu, September 16, 2010 9:25:13 PM > Subject: [Cppcms-users] Boost Locale: Undefined behavior in POSIX backend > > There is an issue in the POSIX backend that causes a crash on Mac OS X, > and undefined behavior elsewhere. > > In file libs/locale/src/posix/codecvt.cpp: > > The declaration of iconv_t d at line 41 shadows the declaration of d at > line 38. This causes undefined behavior at line 85 in the call to > iconv_close. On Mac OS X this causes a crash, though I would expect > there to be at least a memory leak on every platform. > > > ------------------------------------------------------------------------------ > Start uncovering the many advantages of virtual appliances > and start using them to simplify application deployment and > accelerate your shift to cloud computing. > http://p.sf.net/sfu/novell-sfdev2dev > _______________________________________________ > Cppcms-users mailing list > Cpp...@li... > https://lists.sourceforge.net/lists/listinfo/cppcms-users > |
|
From: Kevin A. <kev...@ne...> - 2010-09-16 20:08:57
|
Here are the results from your SVN head. Note I disabled the ICU backend.
make test
Running tests...
Test project $HOME/boost_locale_rework_build
Start 1: test_codepage
1/10 Test #1: test_codepage .................... Passed 0.01 sec
Start 2: test_message
2/10 Test #2: test_message ..................... Passed 0.02 sec
Start 3: test_ios_prop
3/10 Test #3: test_ios_prop .................... Passed 0.00 sec
Start 4: test_codepage_converter
4/10 Test #4: test_codepage_converter .......... Passed 0.00 sec
Start 5: test_std_convert
5/10 Test #5: test_std_convert ................. Passed 0.00 sec
Start 6: test_std_formatting
6/10 Test #6: test_std_formatting .............. Passed 0.00 sec
Start 7: test_std_collate
7/10 Test #7: test_std_collate ................. Passed 0.00 sec
Start 8: test_posix_collate
8/10 Test #8: test_posix_collate ............... Passed 0.00 sec
Start 9: test_posix_convert
9/10 Test #9: test_posix_convert ...............***Failed 0.01 sec
Start 10: test_posix_formatting
10/10 Test #10: test_posix_formatting ............ Passed 0.01 sec
90% tests passed, 1 tests failed out of 10
Total Test time (real) = 0.08 sec
The following tests FAILED:
9 - test_posix_convert (Failed)
Errors while running CTest
make: *** [test] Error 8
Here is the output of the failed test:
$ ./test_posix_convert
Testing char
- Testing at least C
- Testing en_US.UTF-8
Testing en_US.ISO8859-1
Error in line:25
boost::locale::to_upper(to_correct_string<CharType>(src,l),l) ==
to_correct_string<CharType>(tgtu,l)
Testing tr_TR.UTF-8
Turkish locale is not supported well
Testing wchar_t
- Testing at least C
- Testing en_US.UTF-8
Testing en_US.ISO8859-1
Testing tr_TR.UTF-8
Turkish locale is not supported well
Passed 25 tests
Failed 1 tests
96.2% of tests completed sucsessefully
On 09/16/2010 03:54 PM, Artyom wrote:
> Thank you very much!
>
> I've got reports of faults on Mac OS X however I hadn't
> this platform to debug. My brother usually helps
> me with Mac platform, but now he is busy.
>
> Thank you for this good point.
>
> I commit the changes to both CppCMS and Boost.Locale
> svn
> (https://cppcms.svn.sourceforge.net/svnroot/cppcms/boost_locale/branches/rework)
>
> Would you be so kind to check if tests now pass and what fails?
>
> Thank you,
> Artyom
>
>
>
>
>
> ----- Original Message ----
>> From: Kevin Arunski<kev...@ne...>
>> To: cpp...@li...
>> Sent: Thu, September 16, 2010 9:25:13 PM
>> Subject: [Cppcms-users] Boost Locale: Undefined behavior in POSIX backend
>>
>> There is an issue in the POSIX backend that causes a crash on Mac OS X,
>> and undefined behavior elsewhere.
>>
>> In file libs/locale/src/posix/codecvt.cpp:
>>
>> The declaration of iconv_t d at line 41 shadows the declaration of d at
>> line 38. This causes undefined behavior at line 85 in the call to
>> iconv_close. On Mac OS X this causes a crash, though I would expect
>> there to be at least a memory leak on every platform.
>>
>>
>> ------------------------------------------------------------------------------
>> Start uncovering the many advantages of virtual appliances
>> and start using them to simplify application deployment and
>> accelerate your shift to cloud computing.
>> http://p.sf.net/sfu/novell-sfdev2dev
>> _______________________________________________
>> Cppcms-users mailing list
>> Cpp...@li...
>> https://lists.sourceforge.net/lists/listinfo/cppcms-users
>>
>
>
>
>
> ------------------------------------------------------------------------------
> Start uncovering the many advantages of virtual appliances
> and start using them to simplify application deployment and
> accelerate your shift to cloud computing.
> http://p.sf.net/sfu/novell-sfdev2dev
> _______________________________________________
> Cppcms-users mailing list
> Cpp...@li...
> https://lists.sourceforge.net/lists/listinfo/cppcms-users
|
|
From: Artyom <art...@ya...> - 2010-09-16 20:18:37
|
Thanks, The test that fails in actually Mac libc issue as it does not handle ISO-8859-1 encoding well in standard library. (Same of FreeBSD, but BSD does not have _l API at all) I've marked for Apple not to do the test this specific one. Actually locale support on Mac is quite weak, for example collation is quite useless and case handling is far from being good. Basically what you have on Mac with POSIX backend is messages, and localized data formatting like dates, times, money etc. Thanks again, Artyom ----- Original Message ---- > From: Kevin Arunski <kev...@ne...> > To: cpp...@li... > Sent: Thu, September 16, 2010 10:08:49 PM > Subject: Re: [Cppcms-users] Boost Locale: Undefined behavior in POSIX backend > > Here are the results from your SVN head. Note I disabled the ICU backend. > > make test > Running tests... > Test project $HOME/boost_locale_rework_build > Start 1: test_codepage > 1/10 Test #1: test_codepage .................... Passed 0.01 sec > Start 2: test_message > 2/10 Test #2: test_message ..................... Passed 0.02 sec > Start 3: test_ios_prop > 3/10 Test #3: test_ios_prop .................... Passed 0.00 sec > Start 4: test_codepage_converter > 4/10 Test #4: test_codepage_converter .......... Passed 0.00 sec > Start 5: test_std_convert > 5/10 Test #5: test_std_convert ................. Passed 0.00 sec > Start 6: test_std_formatting > 6/10 Test #6: test_std_formatting .............. Passed 0.00 sec > Start 7: test_std_collate > 7/10 Test #7: test_std_collate ................. Passed 0.00 sec > Start 8: test_posix_collate > 8/10 Test #8: test_posix_collate ............... Passed 0.00 sec > Start 9: test_posix_convert > 9/10 Test #9: test_posix_convert ...............***Failed 0.01 sec > Start 10: test_posix_formatting > 10/10 Test #10: test_posix_formatting ............ Passed 0.01 sec > > 90% tests passed, 1 tests failed out of 10 > > Total Test time (real) = 0.08 sec > > The following tests FAILED: > 9 - test_posix_convert (Failed) > Errors while running CTest > make: *** [test] Error 8 > > > Here is the output of the failed test: > $ ./test_posix_convert > Testing char > - Testing at least C > - Testing en_US.UTF-8 > Testing en_US.ISO8859-1 > Error in line:25 > boost::locale::to_upper(to_correct_string<CharType>(src,l),l) == > to_correct_string<CharType>(tgtu,l) > Testing tr_TR.UTF-8 > Turkish locale is not supported well > Testing wchar_t > - Testing at least C > - Testing en_US.UTF-8 > Testing en_US.ISO8859-1 > Testing tr_TR.UTF-8 > Turkish locale is not supported well > > Passed 25 tests > Failed 1 tests > 96.2% of tests completed sucsessefully > > > On 09/16/2010 03:54 PM, Artyom wrote: > > Thank you very much! > > > > I've got reports of faults on Mac OS X however I hadn't > > this platform to debug. My brother usually helps > > me with Mac platform, but now he is busy. > > > > Thank you for this good point. > > > > I commit the changes to both CppCMS and Boost.Locale > > svn > > >(https://cppcms.svn.sourceforge.net/svnroot/cppcms/boost_locale/branches/rework) > > > > Would you be so kind to check if tests now pass and what fails? > > > > Thank you, > > Artyom > > > > > > > > > > > > ----- Original Message ---- > >> From: Kevin Arunski<kev...@ne...> > >> To: cpp...@li... > >> Sent: Thu, September 16, 2010 9:25:13 PM > >> Subject: [Cppcms-users] Boost Locale: Undefined behavior in POSIX backend > >> > >> There is an issue in the POSIX backend that causes a crash on Mac OS X, > >> and undefined behavior elsewhere. > >> > >> In file libs/locale/src/posix/codecvt.cpp: > >> > >> The declaration of iconv_t d at line 41 shadows the declaration of d at > >> line 38. This causes undefined behavior at line 85 in the call to > >> iconv_close. On Mac OS X this causes a crash, though I would expect > >> there to be at least a memory leak on every platform. > >> > >> > >> >------------------------------------------------------------------------------ > >> Start uncovering the many advantages of virtual appliances > >> and start using them to simplify application deployment and > >> accelerate your shift to cloud computing. > >> http://p.sf.net/sfu/novell-sfdev2dev > >> _______________________________________________ > >> Cppcms-users mailing list > >> Cpp...@li... > >> https://lists.sourceforge.net/lists/listinfo/cppcms-users > >> > > > > > > > > > > >------------------------------------------------------------------------------ > > ; Start uncovering the many advantages of virtual appliances > > and start using them to simplify application deployment and > > accelerate your shift to cloud computing. > > http://p.sf.net/sfu/novell-sfdev2dev > > _______________________________________________ > > Cppcms-users mailing list > > Cpp...@li... > > https://lists.sourceforge.net/lists/listinfo/cppcms-users > > ------------------------------------------------------------------------------ > Start uncovering the many advantages of virtual appliances > and start using them to simplify application deployment and > accelerate your shift to cloud computing. > http://p.sf.net/sfu/novell-sfdev2dev > _______________________________________________ > Cppcms-users mailing list > Cpp...@li... > https://lists.sourceforge.net/lists/listinfo/cppcms-users > |
|
From: Kevin A. <kev...@ne...> - 2010-09-16 20:42:07
|
Yes, the POSIX locale support even under C appears to be weak for those two BSD derived systems. However, this particular failure does not disuade me from using the new POSIX backend, since I intend to do any conversions or collation after normalizing input to UTF-8. This seemed like an acceptable tradeoff versus linking with ICU. Fortunately it appears the POSIX locale now passes all the tests in the UTF-8 encoding, so thank you! Kevin On 09/16/2010 04:18 PM, Artyom wrote: > Thanks, > > The test that fails in actually Mac libc issue > as it does not handle ISO-8859-1 encoding well in standard > library. (Same of FreeBSD, but BSD does not have _l API at all) > > I've marked for Apple not to do the test this specific one. > > Actually locale support on Mac is quite weak, for example > collation is quite useless and case handling is far from being > good. Basically what you have on Mac with POSIX backend is > messages, and localized data formatting like dates, times, > money etc. > > Thanks again, > Artyom > > > > ----- Original Message ---- >> From: Kevin Arunski<kev...@ne...> >> To: cpp...@li... >> Sent: Thu, September 16, 2010 10:08:49 PM >> Subject: Re: [Cppcms-users] Boost Locale: Undefined behavior in POSIX backend >> >> Here are the results from your SVN head. Note I disabled the ICU backend. >> >> make test >> Running tests... >> Test project $HOME/boost_locale_rework_build >> Start 1: test_codepage >> 1/10 Test #1: test_codepage .................... Passed 0.01 sec >> Start 2: test_message >> 2/10 Test #2: test_message ..................... Passed 0.02 sec >> Start 3: test_ios_prop >> 3/10 Test #3: test_ios_prop .................... Passed 0.00 sec >> Start 4: test_codepage_converter >> 4/10 Test #4: test_codepage_converter .......... Passed 0.00 sec >> Start 5: test_std_convert >> 5/10 Test #5: test_std_convert ................. Passed 0.00 sec >> Start 6: test_std_formatting >> 6/10 Test #6: test_std_formatting .............. Passed 0.00 sec >> Start 7: test_std_collate >> 7/10 Test #7: test_std_collate ................. Passed 0.00 sec >> Start 8: test_posix_collate >> 8/10 Test #8: test_posix_collate ............... Passed 0.00 sec >> Start 9: test_posix_convert >> 9/10 Test #9: test_posix_convert ...............***Failed 0.01 sec >> Start 10: test_posix_formatting >> 10/10 Test #10: test_posix_formatting ............ Passed 0.01 sec >> >> 90% tests passed, 1 tests failed out of 10 >> >> Total Test time (real) = 0.08 sec >> >> The following tests FAILED: >> 9 - test_posix_convert (Failed) >> Errors while running CTest >> make: *** [test] Error 8 >> >> >> Here is the output of the failed test: >> $ ./test_posix_convert >> Testing char >> - Testing at least C >> - Testing en_US.UTF-8 >> Testing en_US.ISO8859-1 >> Error in line:25 >> boost::locale::to_upper(to_correct_string<CharType>(src,l),l) == >> to_correct_string<CharType>(tgtu,l) >> Testing tr_TR.UTF-8 >> Turkish locale is not supported well >> Testing wchar_t >> - Testing at least C >> - Testing en_US.UTF-8 >> Testing en_US.ISO8859-1 >> Testing tr_TR.UTF-8 >> Turkish locale is not supported well >> >> Passed 25 tests >> Failed 1 tests >> 96.2% of tests completed sucsessefully >> >> >> On 09/16/2010 03:54 PM, Artyom wrote: >>> Thank you very much! >>> >>> I've got reports of faults on Mac OS X however I hadn't >>> this platform to debug. My brother usually helps >>> me with Mac platform, but now he is busy. >>> >>> Thank you for this good point. >>> >>> I commit the changes to both CppCMS and Boost.Locale >>> svn >>> >> (https://cppcms.svn.sourceforge.net/svnroot/cppcms/boost_locale/branches/rework) >>> >>> Would you be so kind to check if tests now pass and what fails? >>> >>> Thank you, >>> Artyom >>> >>> >>> >>> >>> >>> ----- Original Message ---- >>>> From: Kevin Arunski<kev...@ne...> >>>> To: cpp...@li... >>>> Sent: Thu, September 16, 2010 9:25:13 PM >>>> Subject: [Cppcms-users] Boost Locale: Undefined behavior in POSIX backend >>>> >>>> There is an issue in the POSIX backend that causes a crash on Mac OS X, >>>> and undefined behavior elsewhere. >>>> >>>> In file libs/locale/src/posix/codecvt.cpp: >>>> >>>> The declaration of iconv_t d at line 41 shadows the declaration of d at >>>> line 38. This causes undefined behavior at line 85 in the call to >>>> iconv_close. On Mac OS X this causes a crash, though I would expect >>>> there to be at least a memory leak on every platform. >>>> >>>> >>>> >> ------------------------------------------------------------------------------ >>>> Start uncovering the many advantages of virtual appliances >>>> and start using them to simplify application deployment and >>>> accelerate your shift to cloud computing. >>>> http://p.sf.net/sfu/novell-sfdev2dev >>>> _______________________________________________ >>>> Cppcms-users mailing list >>>> Cpp...@li... >>>> https://lists.sourceforge.net/lists/listinfo/cppcms-users >>>> >>> >>> >>> >>> >>> >> ------------------------------------------------------------------------------ >>> ; Start uncovering the many advantages of virtual appliances >>> and start using them to simplify application deployment and >>> accelerate your shift to cloud computing. >>> http://p.sf.net/sfu/novell-sfdev2dev >>> _______________________________________________ >>> Cppcms-users mailing list >>> Cpp...@li... >>> https://lists.sourceforge.net/lists/listinfo/cppcms-users >> >> ------------------------------------------------------------------------------ >> Start uncovering the many advantages of virtual appliances >> and start using them to simplify application deployment and >> accelerate your shift to cloud computing. >> http://p.sf.net/sfu/novell-sfdev2dev >> _______________________________________________ >> Cppcms-users mailing list >> Cpp...@li... >> https://lists.sourceforge.net/lists/listinfo/cppcms-users >> > > > > > ------------------------------------------------------------------------------ > Start uncovering the many advantages of virtual appliances > and start using them to simplify application deployment and > accelerate your shift to cloud computing. > http://p.sf.net/sfu/novell-sfdev2dev > _______________________________________________ > Cppcms-users mailing list > Cpp...@li... > https://lists.sourceforge.net/lists/listinfo/cppcms-users |
|
From: Artyom <art...@ya...> - 2010-09-16 21:10:57
|
Yes, indeed, if you don't use any advanced
locale features then these small backends
are the way to go.
That is why I created them.
Aryom
> Yes, the POSIX locale support even under C appears to be weak for those
> two BSD derived systems. However, this particular failure does not
> disuade me from using the new POSIX backend, since I intend to do any
> conversions or collation after normalizing input to UTF-8. This seemed
> like an acceptable tradeoff versus linking with ICU. Fortunately it
> appears the POSIX locale now passes all the tests in the UTF-8 encoding,
> so thank you!
>
> Kevin
|
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X