Re: [Cppcms-users] Questions about decode UTF-8 to Unicode code point
Brought to you by:
artyom-beilis
Menu
▾
▴
Re: [Cppcms-users] Questions about decode UTF-8 to Unicode code point
|
From: Artyom B. <art...@ya...> - 2013-06-09 06:30:02
|
There is a set of functions: booster::locale::conv::utf_to_utf http://www.boost.org/doc/libs/1_53_0/libs/locale/doc/html/group__codepage.html#gaf0ad39959911b000706e0538ec059d44 All you need it to do is to convert UTF-32 to UTF-16 or UTF-8 to UTF-16. Also cast to uint16_t is INCORRECT is it would fail on characters outside of BMP - i.e. characters that are encoded as two units sequence in UTF-16 (so called surrogate pairs) Artyom Beilis -------------- CppCMS - C++ Web Framework: http://cppcms.com/ CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/ >________________________________ > From: 陈抒 <csf...@gm...> >To: Artyom Beilis <art...@ya...>; "cpp...@li..." <cpp...@li...> >Sent: Saturday, June 8, 2013 6:52 PM >Subject: Re: [Cppcms-users] Questions about decode UTF-8 to Unicode code point > > > >Hello Artyom: > I know the code_point type is uint32_t. In my app, I need to convert it into uint16_t. Which one of the following ways is portable and don't need to worry about the endianness(big-endian or little-endian), also can be used in both 32bit and 64bit platform. I am using GCC4.x above. >Assume x is a code_point value >1. static_cast<uint16_t>(x) >2. x & 0xffff > > >Correct me if I am wrong, thanks. > > > > >Dean Chen >Best regards >http://blog.csdn.net/sheismylife > > >On Wed, Feb 6, 2013 at 12:20 AM, Artyom Beilis <art...@ya...> wrote: > >First of all there is utf_to_utf functions and if sizeof(wchar_t) == 4 (which is on Linux/Unix) >>than all you need is to use them. >> >> >>About the error >> >> >>code_point c = utf_traits<char, sizeof(char)>::decode(str.begin(), str.end()); >> >> >> >>the first parameter is reference that is moved forward and std.begin() can't be changed >>so what you need to do is: >> >> >> >>std::string::const_iterator pos = std.begin(),end =str.end(); >>code_point c = utf_traits<char, sizeof(char)>::decode(pos,end ); >> >> >> >>See: http://www.boost.org/doc/libs/1_51_0/libs/locale/doc/html/structboost_1_1locale_1_1utf_1_1utf__traits.html#a65f0b0e1075dd000d2c2c15af30be372 >> >>Artyom Beilis >>-------------- >>CppCMS - C++ Web Framework: http://cppcms.com/ >>CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/ >> >> >> >>>________________________________ >>> From: 陈抒 <csf...@gm...> >>>To: cpp...@li... >>>Sent: Tuesday, February 5, 2013 5:32 PM >>>Subject: [Cppcms-users] Questions about decode UTF-8 to Unicode code point >>> >>> >>> >>>Hello Artyom: >>> I am using the boost.locale library contributed by you to decode UTF-8. >>>The following codes works fine: >>>char const * p = "一"; // one Chinese character here >>> >>>code_point c = utf_traits<char, sizeof(char)>::decode(p, p + 3); >>>cout << "code point: 0x" << std::hex << c << " binary format:B" << PrintIntAsBinaryString(c) << endl; >>> >>> >>>My Print.. function prints it as what I expect. >>>code point: 0x4e00 binary format:B00000000000000000100111000000000 >>> >>> >>> >>>But don't know how to apply the decode function to std::string. when using iterator of string object, got some compilation error: >>> >>> >>>string str = "一"; >>> >>>code_point c = utf_traits<char, sizeof(char)>::decode(str.begin(), str.end()); >>> >>> >>> >>>/home/chenshu/work/asio_echo/codes/main/test/main.cc:18:65: error: no matching function for call to ‘boost::locale::utf::utf_traits<char>::decode(std::basic_string<char>::iterator, std::basic_string<char>::iterator)’ >>>/home/chenshu/work/asio_echo/codes/main/test/main.cc:18:65: note: candidate is: >>>/usr/include/boost/locale/utf.hpp:193:27: note: static boost::locale::utf::code_point boost::locale::utf::utf_traits<CharType, 1>::decode(Iterator&, Iterator) [with Iterator = __gnu_cxx::__normal_iterator<char*, std::basic_string<char> >, CharType = char, boost::locale::utf::code_point = unsigned int] >>>/usr/include/boost/locale/utf.hpp:193:27: note: no known conversion for argument 1 from ‘std::basic_string<char>::iterator {aka __gnu_cxx::__normal_iterator<char*, std::basic_string<char> >}’ to ‘__gnu_cxx::__normal_iterator<char*, std::basic_string<char> >&’ >>>make[2]: *** [test_bin/CMakeFiles/util_test.dir/main.cc.o] Error 1 >>>make[1]: *** [test_bin/CMakeFiles/util_test.dir/all] Error 2 >>>make: *** [all] Error 2 >>> >>> >>>The above is my first question, that's about converting one single Unicode character. >>>Another question is how to decode all Unicode code points from string object? >>> >>> >>> >>>陈抒 >>>Best regards >>>http://blog.csdn.net/sheismylife >>>------------------------------------------------------------------------------ >>>Free Next-Gen Firewall Hardware Offer >>>Buy your Sophos next-gen firewall before the end March 2013 >>>and get the hardware for free! Learn more. >>>http://p.sf.net/sfu/sophos-d2d-feb >>>_______________________________________________ >>>Cppcms-users mailing list >>>Cpp...@li... >>>https://lists.sourceforge.net/lists/listinfo/cppcms-users >>> >>> >>> >>------------------------------------------------------------------------------ >>Free Next-Gen Firewall Hardware Offer >>Buy your Sophos next-gen firewall before the end March 2013 >>and get the hardware for free! Learn more. >>http://p.sf.net/sfu/sophos-d2d-feb >>_______________________________________________ >>Cppcms-users mailing list >>Cpp...@li... >>https://lists.sourceforge.net/lists/listinfo/cppcms-users >> >> > > > |
