Re: [Log4cplus-devel] Unicode Text Logging

[Václav Z. <vha...@gm...>]

On 05/31/2012 05:46 PM, vvv jjj wrote:
> Hi log4cplus Devel Team,
>
> Thanks Václav Zeman.
>
> I gave Unicode text directly in the cpp file. (Ex: CString str =
> _T("Ω⅓∂₤◊");)
> The file is saved with unicode support (the default yes button when
> Unicode file is saved in Visual Studio 2008). The configuration used
> is Debug Unicode dynamic library.
>
> Please let me know if you need any other details.
>
> Regards
> J.Viswanath
>
> ------------------------------------------------------------------------
> *From:* Václav Zeman <vha...@gm...>
> *To:* vvv jjj <vv...@ya...>
> *Cc:* "log...@li..."
> <log...@li...>
> *Sent:* Thursday, 31 May 2012 3:11 PM
> *Subject:* Re: [Log4cplus-devel] Unicode Text Logging
>
> On 30 May 2012 19:08, vvv jjj wrote:
> > Hi log4cplus Devel Team,
> >
> > I got some bug when trying to log Unicode text. The logger could log
> ASCII
> > text correctly but when I give Unicode text the Logger is not
> logging any
> > other text also.
> >
> > Is this the same problem as Not able to log unicode character on Linux
> > Platform. - ID: 3446131.
> >
> > I'm using log4cplus in Windows XP with Visual Studio 2008. Working with
> > dynamic library with Debug Unicode support.
> >
> > Please let me know if it is the same problem and still existing.
> Also let me
> > know if there any option to support Unicode text logging without
> changing
> > the log4cplus library (1.0.4.1-RC2).
> How are you giving Unicode text to log4cplus? Is it part of your C++
> source? Then how is your C++ source encoded? What configuration of
> log4cplus are you using (Release or Release_Unicode)?
>
> -- 
> VZ
>
>
I have commented on the bug #3446131. But because your problems are on
Windows and not Linux, the solution might be different:

Ok, I think I have reproduced your problem with log4cplus 1.0.4.1
release.

The problem has two layers:

1. There is the std::locale::global (LOCALE); line that is or is not
   in your source, where LOCALE is either std::locale ("") or
   std::locale ().

2. ./configure script flags: NONE, --with-working-locale (and
   --with-working-c-locale or --with-iconv)

This gives us 6 combinations of these two parameters. To test this I
have used ftests/fileappender_test with "Entering loop číslo/番："
string to log. Log4cplus was compiled with CPPFLAGS="-DUNICODE=1".


(1)
no parameters to ./configure
std::locale::global: missing

The test breaks exactly as you have reported:

log4cplus: Just opened file: Test.log
log4cplus:WARN RollingFileAppender: MaxFileSize property value is too
small. Resetting to 204800.
log4cplus:ERROR file is not open: Test.log
log4cplus: Destroying appender named [First].
log4cplus: Destroying appender named [First].


(2)
no parameters to ./configure
std::locale::global(std::locale());

Same result as for (1).


(3)
no parameters to ./configure
std::locale::global(std::locale(""));

The logger keeps on logging. ASCII text goes through unmodified,
numbers get formatted according to my locale (my environment variable
LANG is set to en_US.UTF-8). The testing Unicode characters pass
through unmodified as well. I see 4 bytes per character
(sizeof(wchar_t) == 4 on Linux platforms).


(4)
std::locale::global: missing
./configure with --with-working-locale

The logging works but the non-ASCII characters are replaced by
question marks:

600 [139678217910080] DEBUG test.subtest <loop> - Entering loop
????slo/??????19759


(5)
std::locale::global(std::locale());
./configure with --with-working-locale

Same result as for (4).


(6)
std::locale::global(std::locale(""));
./configure with --with-working-locale

The logging works, numbers are formatted according to locale
(en_US.UTF-8) and the non-ASCII characters show up as well:

165 [139,711,023,560,512] DEBUG test.subtest <loop> - Entering loop
číslo/番：18,467


CONCLUSION

The whole matter is unfortunately very complicated. The result depends
on your locale settings, on capabilities of your compiler and standard
C++ library. It also depends on compile time settings of log4cplus
itself.

Unless you really (really) have to handle wchar_t strings, then I
strongly suggest that you do not compile log4cplus with -DUNICODE
switch. Most of Linux distributions these days use UTF-8 as their
native encoding and that passes through to the Test.log file unharmed
in all cases (that I have tested) whether the --with-working-locale
switch is present or not, whether any std::locale::global (LOCALE)
line is present or not.

Conclusion of the conclusion, I suggest configuring log4cplus on
_Linux_ platforms with GCC with the --with-working-locale switch and
_without_ -DUNICODE.

-- 
VZ