Recent changes to strcvt

strcvt modified by Ruediger Helsch

Ruediger Helsch — Wed, 02 Jul 2014 09:50:54 -0000

--- v7
+++ v8
@@ -1,734 +0,0 @@
-Strcvt - Character type converter for C++11
-============================================
-
-
-
-Contents
---------
-- [Introduction](#Introduction)
-- [Installation](#Installation)
-- [Usage](#Usage)
-- [String conversion](#string_conversion)
-- [UTF-8 string type `u8string`](#U8String)
-- [UTF string conversion](#utf_string_conversion)
-- [Locale dependent stream conversion](#locale_dependent)
-- [UTF stream conversion](#utf_stream_conversion)
-- [Character type converting iterator](#converting_iterator)
-- [Operators](#operators)
-- [Creating a Library](#Library)
-- [Compatibility](#Compatibility)
-- [Rationale](#Rationale)
-- [License](#License)
-
-
-
-
-Introduction 
-------------
-
-C++11 has the new character types `char16_t` and `char32_t`, which use
-the UTF-16 and UTF-32 Unicode encoding. The question is how to convert
-between the four character types of C++.
-
-This library converts between the different character types of C++.
-
-- Conversion between the types `char`, `wchar_t`, `char16_t` and
-  `char32_t` in all combinations.
-
-- Implements the API of the C++ `codecvt` member functions in() and
-  out() for stream conversion, and convenience functions for string
-  conversion.
-
-- Locale-dependent conversion and UTF conversion.
-
-
-
-Installation 
-------------
-
-Unpack the distribution to a directory on your local machine.  You can
-include the headers in subdirectory `include/xprintf` from your
-program.  To make inclusion of the headers easier, it is recommended to
-add the subdirectory `include` of the distribution to the include file
-search path of the compiler.  This is commonly achieved with the
-option `-I/path/to/strcvt/include` (assuming that the xprintf
-distribution has been upacked to directory `/path/to/xprintf`).  Then
-you can include the code converter headers through their standard names
-like "strcvt/strcvt.h".
-
-
-
-Usage 
------
-
-In order to use the code converter, its headers have to be included.
-They are:
-- [UTF-8 string type `u8string`](#U8String)
-~~~cpp
-
-    // The main header defining the strcvt() string conversion functions
-    #include "/path/to/strcvt/include/strcvt/strcvt.h"
-    // The main header defining the u8string type
-    #include "/path/to/strcvt/include/strcvt/u8string.h"
-    // The header defining the strcvt_utf() UTF string conversion functions
-    #include "/path/to/strcvt/include/strcvt/strcvt_utf.h"
-    // The header defining the charcvt() stream conversion functions
-    #include "/path/to/strcvt/include/strcvt/charcvt.h"
-    // The header defining the charcvt_utf() UTF stream conversion functions
-    #include "/path/to/strcvt/include/strcvt/charcvt_utf.h"
-    // The header defining the strcvt_iterator
-    #include "/path/to/strcvt/include/strcvt/strcvt_iterator.h"
-    // The header defining codecvt_utf
-    #include "/path/to/strcvt/include/strcvt/codecvt_utf.h"
-
-    // Optional header defining output operators
-    #include "/path/to/strcvt/include/strcvt/strcvt_operators.h"
-
-    // Import symbols into users namespace
-    using namespace StrCvt;
-~~~
-
-If the xprintf include directory `/path/to/strcvt/include` has been
-added to the include file search path of the compiler, e.g. using the
-compiler option `-I/path/to/strcvt/include`, this reduces to:
-
-~~~{.cpp}
-
-    // The main header defining the strcvt() string conversion functions
-    #include "strcvt/strcvt.h"
-    // The main header defining the u8string type
-    #include "strcvt/u8string.h"
-    // The header defining the strcvt_utf() UTF string conversion functions
-    #include "strcvt/strcvt_utf.h"
-    // The header defining the charcvt() stream conversion functions
-    #include "strcvt/charcvt.h"
-    // The header defining the charcvt_utf() UTF stream conversion functions
-    #include "strcvt/charcvt_utf.h"
-    // The header defining the strcvt_iterator
-    #include "strcvt/strcvt_iterator.h"
-    // The header defining codecvt_utf
-    #include "strcvt/codecvt_utf.h"
-
-    // Optional header defining output operators
-    #include "strcvt/strcvt_operators.h"
-
-    // Import symbols into users namespace
-    using namespace StrCvt;
-~~~
-
-The strcvt() functions are exported through namespace `StrCvt`.  It
-is recommented to make them available via a `using namespace`
-directive like in the example above.  Alternatively it is possible to
-import the functions strcvt() and strcvt_utf() etc. separately
-through `using` declarations:
-
-~~~{.cpp}
-
-    using StrCvt::strcvt;
-    using StrCvt::strcvt_utf;
-    using StrCvt::strcvt_utf_strict;
-    using StrCvt::make_strcvt_iterator;
-~~~
-
-The character type converter is pre-configured for header-only use.
-This means: Just include the header and you are done.  In order to
-reduce space overhead and compilation time, a precompiled library can
-be used.  See section [Creating a Library](#library).
-
-
-
-String conversion 
------------------
-
-Function strcvt\(source) converts strings from the
-source character type to the destination string type.  The
-`dest_string_type` can be `std::string`, `std::wstring`,
-`std::u16string` and `std::u32string`. An additional string type
-`StrCvt::u8string` with UTF-8 encoding is also provided; see section
-[UTF-8 string type `u8string`](#U8String). Example usage:
-
-~~~{.cpp}
-
-    #include "strcvt/strcvt.h"
-
-    // ...
-
-    using namespace StrCvt;
-
-    // Convert const char* C string to wide wchar_t string
-    std::wstring w = strcvt("Hello, world!");
-    // Convert wide string to string of different character type
-    std::string s = strcvt(w);
-    // Convert string to char16_t string
-    std::u16string s16 = strcvt(s);
-    // The source buffer can also be specified as (pointer, size):
-    std::cout << strcvt(&s16[0], 5) << std::endl;
-~~~
-
-The source string can be specified as:
-- a character pointer of any character type (C-style null-terminated string),
-- a C++ string of arbitrary character type, or
-- a character pointer and a size.
-
-The destination string type is specified as a template argument to
-strcvt().  The function is specialized on all combinations of source
-and destination character types. So the full signatures are:
-
-~~~{.cpp}
-
-    template
-    std::basic_string
-    strcvt>(const std::basic_string& source);
-
-    template
-    std::basic_string
-    strcvt>(const source_charT* source);
-
-    template
-    std::basic_string
-    strcvt>(const source_charT* source, std::size_t size);
-~~~
-
-The function with string argument `dest_string
-strcvt(source_string source)` is overloaded for the case
-`dest_string` == `source_string`: it moves its argument to the result
-(for rvalue arguments) or returns a reference to its argument (for
-lvalue arguments).  This is useful in generic code where the concrete
-types are not known.
-
-It is also possible to omit the template argument determining the
-destination string type.  In this case, strcvt() does no conversion at
-all, but returns an object which can be converted to all string types.
-
-~~~{.cpp}
-
-    // strcvt() returns an object which can be assigned to any string type
-    std::wstring w = strcvt("Hello, world!");
-    std::u16string = strcvt(w);
-~~~
-
-For string arguments, the return value of strcvt() is a subclass of
-the class of its argument.  The subclass adds conversion operators to
-all string types.  For character pointer arguments, the return value
-of strcvt() is an object which can be converted to a character
-pointer (of the character type of the argument) or to any string type.
-In each case, the return value of strcvt() can e.g. be assigned to
-any string type, or can be passed as argument to a function expecting a
-string.
-
-
-
-UTF-8 string type `u8string`
-----------------------------
-
-In addition to the four standard string types `std::string`,
-`std::wstring`, std::u16string` and `std::u32string` provided by the
-C++ standard library, header strcvt/u8string.h defines an additional
-string type `StrCvt::u8string`. Like the locale dependent string type
-`std::string` it has character type `char`, but it always uses a
-locale-independent UTF-8 encoding.  Example usage:
-
-~~~{.cpp}
-
-    #include "strcvt/u8string.h"
-    #include "strcvt/strcvt.h"
-
-    // ...
-
-    using namespace StrCvt;
-
-    u8string us(u8"Hello, World");
-    std::cout << strcvt(us) << std::endl;
-~~~
-
-In the example above, the `u8string` is initialized using the C++11
-UTF-8 character literal `u8"string"`.  For output to the
-locale-dependent standard output stream, it is converted from UTF-8 to
-the locale dependent `char` endoding.
-
-
-
-UTF string conversion 
----------------------
-
-In the examples above, strcvt() can be replaced with strcvt_utf(),
-which ignores the locale and always performs a UTF transformation.
-The header is "strcvt/strcvt_utf.h".
-
-It is also possible to configure strcvt() to only use UTF conversions
-instead of the locale dependent character conversions.  Open the header
-`strcvt/strcvt_config.h` with an editor and change the preprocessor
-symbol `STRCVT_IMPL_UTF8_ONLY` from 0 to 1.
-
-
-
-Locale-dependent stream conversion 
-----------------------------------
-
-The character type stream converter function charcvt(state, from,
-from_end, from_next, to, to_end, to_next) is defined in header
-strcvt/charcvt.h and uses the API of the member functions in()
-and out() of the standard code conversion interface `std::codecvt`:
-
-~~~{.cpp}
-
-    #include "strcvt/charcvt.h"
-
-    // ...
-
-    using namespace StrCvt;
-
-    std::mbstate_t state = std::mbstate_t(); // Zero-initialize
-    // Convert between character types:
-    result r = charcvt(state, from, from_end, from_next, to, to_end, to_next);
-
-    // ... More calls of charcvt()
-
-    // Return output to initial shift state
-    r = charcvt_unshift(state, to, to_end, to_next);
-~~~
-
-The `state` must be of type `std::mbstate_t`.  It holds the conversion
-state between successive calls of charcvt() and must be explicitly
-zero-initialized before the first use, like shown above.  The
-arguments `from` and `from_end` delimit the source character buffer,
-`to` and `to_end` delimit the destination character buffer, and on
-exit from the function `from_next` and `to_next` point past the last
-converted character.  The function returns `std::codecvt_base::ok` on
-success and `std::codecvt_base::partial` if the output buffer was too
-small to convert the entire input buffer, or if the input buffer ended
-in a part of a multibyte sequence.  After the input has been
-completely converted, possibly by multiple calls to charcvt(),
-charcvt_unshift() must be called to move the output to the initial
-shift state.  The source character type must be specified as a
-template argument to charcvt_unshift(), since different converters are
-used depending on the source character type.  Charcvt_unshift() should
-even be used if the character encoding is known not to be state
-dependent, like UTF-8.  Charcvt_unshift() checks whether trailing
-incomplete Unicode character input sequences are pending, and appends
-a replacement character to the output buffer if necessary to signal
-the presence of trailing garbage.
-
-Charcvt() is specialized for all combinations of source and
-destination character types `char`, `wchar_t`, `char16_t` and
-`char32_t` and converts between the locale-dependent character type
-`char`, the Unicode UTF-16 and UTF-32 character types `char16_t` and
-`char32_t`, and the implementation defined wide character type
-`wchar_t` which is assumed to be equivalent to either `char16_t` or
-`char32_t`.  It is able to consume or deliver single characters from
-multi-character Unicode characters.
-
-The full signature of charcvt() is:
-
-~~~{.cpp}
-
-    template
-    std::codecvt_base::result
-    charcvt(std::mbstate_t& state,
-            const src_charT* from, const src_charT* from_end,
-            const src_charT*& from_next,
-            dst_charT* to, dst_charT* to_end, dst_charT*& dst_next,
-            int flags = 0);
-~~~
-
-The `flags` can be omitted, or they can be set to the constant
-`strcvt_flags_no_partial_conversions` to prevent partial conversions.
-If the flag is omitted or zero, charcvt() is eager and consumes even
-single partial multibyte characters.
-
-
-
-UTF stream conversion 
----------------------
-
-The function charcvt_utf(state, from, from_end, from_next, to, to_end,
-to_next) is defined in header strcvt/charcvt_utf.h and works like
-charcvt() but treats the type `char` as having UTF-8 encoding instead
-of a locale dependent implementation defined encoding:
-
-~~~{.cpp}
-
-    #include "strcvt/charcvt_utf.h"
-
-    // ...
-
-    using namespace StrCvt;
-
-    std::mbstate_t state = std::mbstate_t(); // Zero-initialize
-    // Convert between character types:
-    charcvt_utf(state, from, from_end, from_next, to, to_end, to_next);
-
-    // ... More calls of charcvt_utf()
-
-    // Return output to initial shift state
-    charcvt_utf_unshift(state, to, to_end, to_next);
-~~~
-
-The `state` must be of type `std::mbstate_t`.  It holds the conversion
-state between successive calls of charcvt_utf() and must be explicitly
-zero-initialized before the first use, like shown above.  The
-arguments `from` and `from_end` delimit the source character buffer,
-`to` and `to_end` delimit the destination character buffer, and on
-exit from the function `from_next` and `to_next` point past the last
-converted character.  The function returns `std::codecvt_base::ok` on
-success and `std::codecvt_base::partial` if the output buffer was too
-small to convert the entire input buffer, or if the input buffer ended
-in a part of a multibyte sequence.  After the input has been
-completely converted, possibly by multiple calls to charcvt_utf(),
-charcvt_utf_unshift() checks whether trailing incomplete Unicode
-character input sequences are pending, and appends a replacement
-character to the output buffer if necessary to signal the presence of
-trailing garbage.
-
-Charcvt_utf() is specialized for all combinations of source and
-destination character types `char`, `wchar_t`, `char16_t` and
-`char32_t` and converts between UTF-coded `char`, the Unicode UTF-16
-and UTF-32 character types `char16_t` and `char32_t`, and the
-implementation defined wide character type `wchar_t` which is assumed
-to be equivalent to either `char16_t` or `char32_t`.  Contrary to
-member functions in() and out() of `codecvt`, charcvt_utf() always
-converts (even if the input character type is the same as the output
-character type, in which case it checks the validity of the encoding).
-It always generates valid UTF-8, UTF-16 or UTF-32 output sequences.
-
-For each invalid encoding in the input buffer, charcvt_utf() inserts
-the replacement character U+0xFFFD into the output buffer.  If this is
-not wanted, function charcvt_utf_strict() can be used, which has the
-same interface as charcvt_utf() but returns `std::codecvt_base::error`
-on encoding errors, with `from_next` pointing to the first element of
-the invalid sequence.
-
-There is also a UTF code conversion facet defined in header
-strcvt/codecvt_utf.h
-
-~~~{.cpp}
-
-    class codecvt_utf;
-~~~
-
-This code conversion facet uses charcvt_utf() to convert between UTF
-coded characters `intern_charT` and `extern_charT`, and is specialized
-for all character type combinations.
-
-
-
-Character type converting character iterator 
---------------------------------------------
-
-Instead of converting the entire buffer, it can be accessed trough a
-converting character iterator.  The iterator is created by function
-make_strcvt_iterator():
-
-~~~{.cpp}
-
-    #include "strcvt/strcvt_iterator.h"
-
-    // ...
-
-    using namespace StrCvt;
-
-    // Create char32_t iterator for access to source C string
-    auto it = make_strcvt_iterator("Hello, world!");
-~~~
-
-The end iterator is returned by make_strcvt_iterator() without
-arguments, or it can be obtained from member function end() of the
-iterator.  Since member function begin() is also implemented, and
-returns the iterator itself, the iterator can be used just like a
-container, for example with the range-based for statement:
-
-~~~{.cpp}
-
-    std::u32string w;
-    // Use range-based for statement: process all Unicode characters
-    for (char32_t c: it)
-        w.push_back(c);
-
-    // Alternatively, use simple iterator interface
-    auto e = it.end();
-    for (auto j = it; j != e; ++j)
-        w.push_back(*j);
-~~~
-
-The converting iterator is (indirectly) derived from base class
-`strcvt_iterator_base` which does not depend on the source
-iterator type or the source character type. It has a virtual
-destructor and can be used polymorphically.  The inheritance hierarchy
-looks like this:
-
-~~~{.cpp}
-
-    template
-    class strcvt_iterator_base {
-        // Operators *(), ++(), ==()
-        // Member functions begin(), end()
-    };
-
-    template
-    class strcvt_iterator_impl : public strcvt_iterator_base {
-        // Implements operator ++() which does the conversion using
-        // virtual member function get_next_source_char()
-    };
-
-    template
-    class strcvt_iterator
-        : public strcvt_iterator_impl::value_type> {
-        // Implements get_next_source_char()
-    };
-~~~
-
-Class template `strcvt_iterator_base` does not know the source
-character type and can be used to handle a `strcvt_iterator`
-polymorphically.  Class template `strcvt_iterator_impl` knows the
-source character type but not the type of the iterator.  If the
-precompiled library is used, it is precompiled for all combinations of
-source and destination character type.  Class template
-`strcvt_iterator` is the return value of function
-make_strcvt_iterator().  It implements the source iterator handling.
-
-
-
-Operators 
----------
-
-Some operators for strings are defined in header
-"strcvt/strcvt_operators.h".  These are the output operators `<<` for
-strings and character pointers, and the appending `+=` operators for
-strings.  To use them, include header "strcvt/strcvt_operators.h".
-The operators reside in namespace StrCvt and should be imported into
-the user's namespace through a `using namespace` directive:
-
-~~~{.cpp}
-
-    #include "strcvt/strcvt_operators.h"
-    using namespace StrCvt;
-
-    std::cout << U"Hello, world!\n";
-~~~
-
-
-It would be nice if we could define conversion operators to enable
-assignment between different string types.  But in C++, conversion and
-assignment operators can only be defined as member functions.
-
-
-
-Creating a Library 
-------------------
-
-The character type converter is preconfigured for header-only use.
-This means: Just include the header and you are done.  In order to
-reduce space overhead and compilation time, a precompiled library can
-be used.
-
-The main advantage in using a library is that each time the strcvt
-headers are included, the compiler does not need to look at the
-implementation details.  This can speed up compilation significantly.
-
-To create the library, the C++ source files libstrcvt.cpp,
-libstrcvt_iterator.cpp, libstrcvt_utf.cpp and libcodecvt_utf.cpp in
-directory `lib` of the distribution must be compiled.  Under Linux,
-just run `make`.  Before compiling, you may want to select the
-compiler to use: Uncomment to proper *CXX=* - line in the toplevel
-`Makefile.template`.  Running `make` should create a library
-`lib/libxprintf.a`, which has to be linked to the programs.
-
-In Visual C++, instead of building a library, you may just add the
-library source files to your project.
-
-In order to make the headers use the library, you must open the header
-`strcvt/strcvt_config.h` with an editor and change the preprocessor
-symbol `STRCVT_IMPL_USE_LIBRARY` from 0 to 1.  The next time the
-header is included, the library will be used.  You can check that the
-library is used as intended by omitting the library when linking.
-Linking should fail with missing externals.
-
-In order to run the tests, the headers for the *boost* test framework
-are required.
-
-
-
-Compatibility
--------------------------------------------
-
-The code converter has been tested with:
-
-- GCC 4.7, 4.8 and 4.9 on Linux
-- Visual Studio Express 2013 for Windows Desktop with November 2013 CTP
-- Intel C++ 14.0.2 on Linux
-- Clang 3.5.0 on Linux
-
-
-
-Rationale 
----------
-
-C++11 has the new character types `char16_t` and `char32_t`, which use
-the UTF-16 and UTF-32 Unicode encoding. The question is how to convert
-between the four character types of C++.
-
-The `codecvt` part of the C++11 library looks like some ruins left
-over at the front line between warring factions.
-
-The `codecvt` class template is part of the header \ and
-described in section 22.4.1.4 of the standard.  The first thing to
-note is that according to this specification `codecvt` transforms
-between an *internal* and an *external* character encoding, so it is
-not intended to transform between internal character encodings like
-`char16_t` and `wchar_t`.
-
-According to the standard, each locale shall have specializations of
-`codecvt` for tranformation between the internal character types
-`char`, `wchar_t`, `char16_t` and `char32_t` and the external
-character type `char`.  The standard first says that these
-specializations "convert the implementation-defined native character
-set", only to continue specifying that
-
-- the specialization with external and internal character type both
-  `char` must not convert at all (effectively saying that the internal
-  `char` must have the same encoding as the external `char`),
-
-- the specializations with internal character types `char16_t` and
-  `char32_t` must treat the external character code as UTF-8, so they
-  are explicitly not allowed to treat it as an "implementation-defined
-  native character set".
-
-This leaves the transformation between internal character type
-`wchar_t` and external character type `char` as the only
-locale-dependent transformation, and it is specified to convert
-"between the native character sets for narrow and wide characters".
-So no luck using `codecvt` to transform between native character sets
-and `char16_t` or `char32_t`.
-
-Section 22.5 of the standard specifies the "standard code conversion
-facets".  It contains:
-
-- Facet `codecvt_utf8`, which converts between a UTF-8 coded `char`
-  buffer and UCS2 or UCS4, so it can be used to transform between
-  UTF-8 and `char32_t`. It is not usable for transformation between
-  UTF-8 and `char16_t` because `char16_t` strings are coded as
-  UTF-16, not UCS2.
-
-- Facet `codecvt_utf8_utf16` converts between UTF-8 and UTF-16, so it
-  can be used to transform between UTF-8-coded `char` and `char16_t`.
-
-- Facet `codecvt_utf16` looks like a hack from the 20th century to
-  adapt UTF-16 wide characters to character byte streams.  The
-  UTF-16-coded buffer is addressed through a `char*`.
-
-So we can use these standard code conversion facets to convert between
-UTF-8-coded `char` and `char16_t` or `char32_t`.  We could even
-transform between `char16_t` and `char32_t` by going through an
-intermediate UTF-8-coded `char` buffer.  But again, no
-transformation between native character sets and `char16_t` or
-`char32_t`.
-
-The standard does provide an interface for transformation between
-native `char` and `char16_t` or `char32_t`.  It is hidden at the very
-end of section 21.7 in the string library and consists of the four
-functions mbrtoc16(), c16rtomb(), mbrtoc32() and c32rtomb().
-Unfortunately, a locale can not be specified for these functions.
-They always work with the currently active global locale.
-
-For the following things I know of no standard-conforming procedure:
-
-- Conversion between `wchar_t` and `char16_t` or `char32_t`.  Clearly,
-  going through native `char` is not an option, unless it happens to
-  use UTF-8 encoding.  I worked around this by assuming that the native
-  `wchar_t` type uses Unicode coding and is equivalent to either
-  `char16_t` or `char32_t`, depending on `sizeof(wchar_t)`.  This
-  assumption holds for the systems I have access to, but there may be
-  other `wchar_t` encodings in use.  Older Windows systems used UCS2,
-  where this assumption would not hold, but modern versions of Windows
-  use UTF-16.
-
-- Locale-dependent conversion: especially for output streams where a
-  locale is known that may not be the global locale, it would be
-  useful if the locale could be passed as an argument to the character
-  transformation.  Unfortunately the standard library does not provide
-  a portable way to do this, so I left it out.
-
-First I thought I would reuse the standard-specified UTF coders, but
-first they are incomplete (going from UTF-16 to UTF-32 via an
-intermediate representation as UTF-8 is not very attractive), second
-they are distributed over several different interfaces, and third GCC
-does not have them.  So as a first step I implemented a transformation
-between UTF-coded `char`, `char16_t`, `char32_t` and `wchar_t`.  The
-function charcvt_utf(state, from, from_end, from_next, to, to_end,
-to_next) uses the API of the member functions in() and out() of
-`codecvt`.  The `state` holds the conversion state, `from` and
-`from_end` delimit the source character buffer, `to` and `to_end`
-delimit the destination character buffer, and on exit from the
-function `from_next` and `to_next` point past the last converted
-character.  Like in() and out(), charcvt_utf() returns
-`std::codecvt_base::ok` on success and `std::codecvt_base::partial` if
-the output buffer was too small to convert the entire input buffer, or
-if the input buffer ended in a part of a multibyte sequence.  For each
-invalid encoding in the input buffer, the replacement character
-U+0xFFFD is inserted into the output stream.  If this is not wanted,
-function charcvt_utf_strict() can be used, which has the same
-interface as charcvt_utf() but returns `std::codecvt_base::error` on
-encoding errors, with `from_next` pointing to the first element of the
-invalid sequence.  Both functions are specialized on all combinations
-of `char`, `wchar_t`, `char16_t` and `char32_t`.  They always convert
-(even if input and output type is the same, in which case they check
-the validity of the encoding).  They are able to consume and produce
-single characters forming part of multibyte Unicode characters.
-
-Based on the UTF coder, a locale-dependent converter has been
-implemented. The function charcvt(state, from, from_end, from_next,
-to, to_end, to_next) also uses the API of the member functions in()
-and out() of `codecvt`:  The `state` holds the conversion state,
-`from` and `from_end` delimit the source character buffer, `to` and
-`to_end` delimit the destination character buffer, and on exit from
-the function `from_next` and `to_next` point past the last converted
-character.  This implementation uses the assumption that `wchar_t` is
-encoded as either UTF-16 or UTF-32.  Transformations between wide
-characters `wchar_t`, `char16_t` and `char32_t` use UTF
-transformations, transformations between `char` and wide characters
-treat `char` as a native locale-defined character code.
-
-Convenience functions for the code transformation are:
-
-- std::basic_string strcvt>(const C*
-  from, std::size_t size) converts the `size` characters at `from`
-  into a string of character type `charT`, which is specified as a
-  template parameter.
-
-- std::basic_string strcvt>(const C*
-  from) converts the characters of the null-terminated string `from`
-  into a string of character type `charT`, which is specified as a
-  template parameter.
-
-- std::basic_string strcvt>(const
-  std::basic_string& from) converts the characters of the string
-  `from` into a string of character type `charT`, which is specified
-  as a template parameter.  Specializations for `charT == C` move the
-  string argument to the return value (if the argument is an rvalue)
-  or return the reference to the argument (if the argument is an
-  lvalue).  This is useful in generic code where the concrete types
-  are not known.
-
-- strcvt(const C* from) (strcvt() without template parameter
-  specifying the destination string type) creates an object which
-  can be converted back to a `const C*` or to any string type.  The
-  result can e.g. be assigned to any string type, or passed to a
-  function expecting any string.
-
-- strcvt(const std::basic_string& from) (strcvt() without template
-  parameter specifying the destination string type) casts the type of
-  the argument to a reference to a class derived from
-  std::basic_string, which adds conversion operators to all string
-  types.  The result can e.g. be assigned to any string type, or
-  passed to a function expecting any string.
-
-
-
-License 
--------
-
-Copyright (c) 2014 Ruediger Helsch; All rights reserved
-
-Permission to use, copy, modify, and distribute this software for any
-purpose and without fee is hereby granted.  The author disclaims all
-warranties with regard to this software.

strcvt modified by Ruediger Helsch

Ruediger Helsch — Tue, 01 Jul 2014 23:35:02 -0000

--- v6
+++ v7
@@ -9,6 +9,7 @@
 - [Installation](#Installation)
 - [Usage](#Usage)
 - [String conversion](#string_conversion)
+- [UTF-8 string type `u8string`](#U8String)
 - [UTF string conversion](#utf_string_conversion)
 - [Locale dependent stream conversion](#locale_dependent)
 - [UTF stream conversion](#utf_stream_conversion)
@@ -21,6 +22,7 @@

+
 Introduction 
 ------------

@@ -61,11 +63,13 @@

 In order to use the code converter, its headers have to be included.
 They are:
-
+- [UTF-8 string type `u8string`](#U8String)
 ~~~cpp

     // The main header defining the strcvt() string conversion functions
     #include "/path/to/strcvt/include/strcvt/strcvt.h"
+    // The main header defining the u8string type
+    #include "/path/to/strcvt/include/strcvt/u8string.h"
     // The header defining the strcvt_utf() UTF string conversion functions
     #include "/path/to/strcvt/include/strcvt/strcvt_utf.h"
     // The header defining the charcvt() stream conversion functions
@@ -77,14 +81,11 @@
     // The header defining codecvt_utf
     #include "/path/to/strcvt/include/strcvt/codecvt_utf.h"

+    // Optional header defining output operators
+    #include "/path/to/strcvt/include/strcvt/strcvt_operators.h"
+
     // Import symbols into users namespace
     using namespace StrCvt;
-
-    // Optional header defining output operators
-    #include "/path/to/strcvt/include/strcvt/strcvt_operators.h"
-
-    // Import operators into users namespace
-    using namespace StrCvtOperators;
 ~~~

 If the xprintf include directory `/path/to/strcvt/include` has been
@@ -95,6 +96,8 @@

     // The main header defining the strcvt() string conversion functions
     #include "strcvt/strcvt.h"
+    // The main header defining the u8string type
+    #include "strcvt/u8string.h"
     // The header defining the strcvt_utf() UTF string conversion functions
     #include "strcvt/strcvt_utf.h"
     // The header defining the charcvt() stream conversion functions
@@ -106,14 +109,11 @@
     // The header defining codecvt_utf
     #include "strcvt/codecvt_utf.h"

+    // Optional header defining output operators
+    #include "strcvt/strcvt_operators.h"
+
     // Import symbols into users namespace
     using namespace StrCvt;
-
-    // Optional header defining output operators
-    #include "strcvt/strcvt_operators.h"
-
-    // Import operators into users namespace
-    using namespace StrCvtOperators;
 ~~~

 The strcvt() functions are exported through namespace `StrCvt`.  It
@@ -141,7 +141,11 @@
 -----------------

 Function strcvt\(source) converts strings from the
-source character type to the destination string type.  Example usage:
+source character type to the destination string type.  The
+`dest_string_type` can be `std::string`, `std::wstring`,
+`std::u16string` and `std::u32string`. An additional string type
+`StrCvt::u8string` with UTF-8 encoding is also provided; see section
+[UTF-8 string type `u8string`](#U8String). Example usage:

 ~~~{.cpp}

@@ -185,9 +189,12 @@
     strcvt>(const source_charT* source, std::size_t size);
 ~~~

-Instead of the destination string type, it is also possible to give
-the destination character type as template argument.  It is
-automatically translated to the string type.
+The function with string argument `dest_string
+strcvt(source_string source)` is overloaded for the case
+`dest_string` == `source_string`: it moves its argument to the result
+(for rvalue arguments) or returns a reference to its argument (for
+lvalue arguments).  This is useful in generic code where the concrete
+types are not known.

 It is also possible to omit the template argument determining the
 destination string type.  In this case, strcvt() does no conversion at
@@ -197,7 +204,7 @@

     // strcvt() returns an object which can be assigned to any string type
     std::wstring w = strcvt("Hello, world!");
-    std::basic_string = strcvt(w);
+    std::u16string = strcvt(w);
 ~~~

 For string arguments, the return value of strcvt() is a subclass of
@@ -208,6 +215,36 @@
 In each case, the return value of strcvt() can e.g. be assigned to
 any string type, or can be passed as argument to a function expecting a
 string.
+
+
+
+UTF-8 string type `u8string`
+----------------------------
+
+In addition to the four standard string types `std::string`,
+`std::wstring`, std::u16string` and `std::u32string` provided by the
+C++ standard library, header strcvt/u8string.h defines an additional
+string type `StrCvt::u8string`. Like the locale dependent string type
+`std::string` it has character type `char`, but it always uses a
+locale-independent UTF-8 encoding.  Example usage:
+
+~~~{.cpp}
+
+    #include "strcvt/u8string.h"
+    #include "strcvt/strcvt.h"
+
+    // ...
+
+    using namespace StrCvt;
+
+    u8string us(u8"Hello, World");
+    std::cout << strcvt(us) << std::endl;
+~~~
+
+In the example above, the `u8string` is initialized using the C++11
+UTF-8 character literal `u8"string"`.  For output to the
+locale-dependent standard output stream, it is converted from UTF-8 to
+the locale dependent `char` endoding.

@@ -402,7 +439,7 @@

 ~~~{.cpp}

-    std::basic_string w;
+    std::u32string w;
     // Use range-based for statement: process all Unicode characters
     for (char32_t c: it)
         w.push_back(c);
@@ -455,17 +492,16 @@
 ---------

 Some operators for strings are defined in header
-"strcvt/strcvt_operators.h".  These are the output operators `<<`
-for strings and character pointers, and the appending `+=` operators
-for strings.  To use them, include header
-"strcvt/strcvt_operators.h".  The operators reside in namespace
-StrCvtOperators and should be imported into the user's namespace
-through a `using namespace` directive:
+"strcvt/strcvt_operators.h".  These are the output operators `<<` for
+strings and character pointers, and the appending `+=` operators for
+strings.  To use them, include header "strcvt/strcvt_operators.h".
+The operators reside in namespace StrCvt and should be imported into
+the user's namespace through a `using namespace` directive:

 ~~~{.cpp}

     #include "strcvt/strcvt_operators.h"
-    using namespace StrCvtOperators;
+    using namespace StrCvt;

     std::cout << U"Hello, world!\n";
 ~~~
@@ -654,23 +690,24 @@

 Convenience functions for the code transformation are:

-- basic_string strcvt>(const C* from,
-  std::size_t size) converts the `size` characters at `from` into a
-  string of character type `charT`, which is specified as a template
-  parameter.
-
-- basic_string strcvt>(const C* from)
-  converts the characters of the null-terminated string `from` into a
-  string of character type `charT`, which is specified as a template
-  parameter.
-
-- basic_string strcvt>(const
+- std::basic_string strcvt>(const C*
+  from, std::size_t size) converts the `size` characters at `from`
+  into a string of character type `charT`, which is specified as a
+  template parameter.
+
+- std::basic_string strcvt>(const C*
+  from) converts the characters of the null-terminated string `from`
+  into a string of character type `charT`, which is specified as a
+  template parameter.
+
+- std::basic_string strcvt>(const
   std::basic_string& from) converts the characters of the string
   `from` into a string of character type `charT`, which is specified
-  as a template parameter.  A specialization for `charT == C` and
-  rvalue argument is provided, which moves the input string to the
-  return value.  Note that this is the only conversion which does not
-  check the validity of the encoding of the string.
+  as a template parameter.  Specializations for `charT == C` move the
+  string argument to the return value (if the argument is an rvalue)
+  or return the reference to the argument (if the argument is an
+  lvalue).  This is useful in generic code where the concrete types
+  are not known.

 - strcvt(const C* from) (strcvt() without template parameter
   specifying the destination string type) creates an object which
@@ -695,4 +732,3 @@
 Permission to use, copy, modify, and distribute this software for any
 purpose and without fee is hereby granted.  The author disclaims all
 warranties with regard to this software.
-

strcvt modified by Ruediger Helsch

Ruediger Helsch — Mon, 30 Jun 2014 15:40:42 -0000

--- v5
+++ v6
@@ -655,7 +655,7 @@
 Convenience functions for the code transformation are:

 - basic_string strcvt>(const C* from,
-  std::size_t size) converts the `size* characters at `from` into a
+  std::size_t size) converts the `size` characters at `from` into a
   string of character type `charT`, which is specified as a template
   parameter.

strcvt modified by Ruediger Helsch

Ruediger Helsch — Mon, 30 Jun 2014 15:37:27 -0000

--- v4
+++ v5
@@ -21,7 +21,6 @@

-
 Introduction 
 ------------

@@ -35,7 +34,8 @@
   `char32_t` in all combinations.

 - Implements the API of the C++ `codecvt` member functions in() and
-  out() for stream conversion, and convenience functions for string conversion.
+  out() for stream conversion, and convenience functions for string
+  conversion.

 - Locale-dependent conversion and UTF conversion.

@@ -68,9 +68,9 @@
     #include "/path/to/strcvt/include/strcvt/strcvt.h"
     // The header defining the strcvt_utf() UTF string conversion functions
     #include "/path/to/strcvt/include/strcvt/strcvt_utf.h"
-    // The header defining the strcvt() stream conversion functions
+    // The header defining the charcvt() stream conversion functions
     #include "/path/to/strcvt/include/strcvt/charcvt.h"
-    // The header defining the strcvt_utf() UTF stream conversion functions
+    // The header defining the charcvt_utf() UTF stream conversion functions
     #include "/path/to/strcvt/include/strcvt/charcvt_utf.h"
     // The header defining the strcvt_iterator
     #include "/path/to/strcvt/include/strcvt/strcvt_iterator.h"
@@ -97,9 +97,9 @@
     #include "strcvt/strcvt.h"
     // The header defining the strcvt_utf() UTF string conversion functions
     #include "strcvt/strcvt_utf.h"
-    // The header defining the strcvt() stream conversion functions
+    // The header defining the charcvt() stream conversion functions
     #include "strcvt/charcvt.h"
-    // The header defining the strcvt_utf() UTF stream conversion functions
+    // The header defining the charcvt_utf() UTF stream conversion functions
     #include "strcvt/charcvt_utf.h"
     // The header defining the strcvt_iterator
     #include "strcvt/strcvt_iterator.h"
@@ -174,15 +174,15 @@

     template
     std::basic_string
-    strcvt >(const std::basic_string& source);
+    strcvt>(const std::basic_string& source);

     template
     std::basic_string
-    strcvt >(const source_charT* source);
+    strcvt>(const source_charT* source);

     template
     std::basic_string
-    strcvt >(const source_charT* source, std::size_t size);
+    strcvt>(const source_charT* source, std::size_t size);
 ~~~

 Instead of the destination string type, it is also possible to give
@@ -306,9 +306,8 @@

 The function charcvt_utf(state, from, from_end, from_next, to, to_end,
 to_next) is defined in header strcvt/charcvt_utf.h and works like
-charcvt(state, from, from_end, from_next, to, to_end, to_next) but
-treats the type `char` as having UTF-8 encoding instead of a locale
-dependent implementation defined encoding:
+charcvt() but treats the type `char` as having UTF-8 encoding instead
+of a locale dependent implementation defined encoding:

 ~~~{.cpp}

@@ -521,6 +520,7 @@
 - GCC 4.7, 4.8 and 4.9 on Linux
 - Visual Studio Express 2013 for Windows Desktop with November 2013 CTP
 - Intel C++ 14.0.2 on Linux
+- Clang 3.5.0 on Linux

@@ -618,20 +618,20 @@
 they are distributed over several different interfaces, and third GCC
 does not have them.  So as a first step I implemented a transformation
 between UTF-coded `char`, `char16_t`, `char32_t` and `wchar_t`.  The
-function strcvt_utf(state, from, from_end, from_next, to, to_end,
+function charcvt_utf(state, from, from_end, from_next, to, to_end,
 to_next) uses the API of the member functions in() and out() of
 `codecvt`.  The `state` holds the conversion state, `from` and
 `from_end` delimit the source character buffer, `to` and `to_end`
 delimit the destination character buffer, and on exit from the
 function `from_next` and `to_next` point past the last converted
-character.  Like in() and out(), strcvt_utf() returns
+character.  Like in() and out(), charcvt_utf() returns
 `std::codecvt_base::ok` on success and `std::codecvt_base::partial` if
 the output buffer was too small to convert the entire input buffer, or
 if the input buffer ended in a part of a multibyte sequence.  For each
 invalid encoding in the input buffer, the replacement character
 U+0xFFFD is inserted into the output stream.  If this is not wanted,
-function strcvt_utf_strict() can be used, which has the same
-interface as strcvt_utf() but returns `std::codecvt_base::error` on
+function charcvt_utf_strict() can be used, which has the same
+interface as charcvt_utf() but returns `std::codecvt_base::error` on
 encoding errors, with `from_next` pointing to the first element of the
 invalid sequence.  Both functions are specialized on all combinations
 of `char`, `wchar_t`, `char16_t` and `char32_t`.  They always convert
@@ -640,7 +640,7 @@
 single characters forming part of multibyte Unicode characters.

 Based on the UTF coder, a locale-dependent converter has been
-implemented. The function strcvt(state, from, from_end, from_next,
+implemented. The function charcvt(state, from, from_end, from_next,
 to, to_end, to_next) also uses the API of the member functions in()
 and out() of `codecvt`:  The `state` holds the conversion state,
 `from` and `from_end` delimit the source character buffer, `to` and
@@ -654,36 +654,37 @@

 Convenience functions for the code transformation are:

-- basic_string strcvt(const C* from, const C*
-  from_end) converts the characters between `from` and `from_end` into
-  a string of character type `charT`, which is specified as a template
+- basic_string strcvt>(const C* from,
+  std::size_t size) converts the `size* characters at `from` into a
+  string of character type `charT`, which is specified as a template
   parameter.

-- basic_string strcvt(const C* from) converts the
-  characters of the null-terminated string `from` into a string of
-  character type `charT`, which is specified as a template parameter.
-
-- basic_string strcvt(const std::basic_string&
-  from) converts the characters of the string `from` into a string of
-  character type `charT`, which is specified as a template parameter.
-  A specialization for `charT = C` and rvalue `from` is provided, which
-  moves the input string to the return value.  Note that this is the
-  only conversion which does not check the validity of the encoding of
-  the string.
-
-- strcvt(const std::basic_string& from) (strcvt() without
-  template parameter `charT` specifying the destination character
-  type) casts the type of the argument to a reference to a class
-  derived from std::basic_string, which adds conversion operators
-  to all string types.  The result can e.g. be assigned to any string
-  type, or passed to a function expecting any string.
-
-- strcvt(const C* from) (strcvt() without template parameter `charT`
-  specifying the destination character type) creates an object which
+- basic_string strcvt>(const C* from)
+  converts the characters of the null-terminated string `from` into a
+  string of character type `charT`, which is specified as a template
+  parameter.
+
+- basic_string strcvt>(const
+  std::basic_string& from) converts the characters of the string
+  `from` into a string of character type `charT`, which is specified
+  as a template parameter.  A specialization for `charT == C` and
+  rvalue argument is provided, which moves the input string to the
+  return value.  Note that this is the only conversion which does not
+  check the validity of the encoding of the string.
+
+- strcvt(const C* from) (strcvt() without template parameter
+  specifying the destination string type) creates an object which
   can be converted back to a `const C*` or to any string type.  The
   result can e.g. be assigned to any string type, or passed to a
   function expecting any string.

+- strcvt(const std::basic_string& from) (strcvt() without template
+  parameter specifying the destination string type) casts the type of
+  the argument to a reference to a class derived from
+  std::basic_string, which adds conversion operators to all string
+  types.  The result can e.g. be assigned to any string type, or
+  passed to a function expecting any string.
+

 License 
@@ -694,3 +695,4 @@
 Permission to use, copy, modify, and distribute this software for any
 purpose and without fee is hereby granted.  The author disclaims all
 warranties with regard to this software.
+

strcvt modified by Ruediger Helsch

Ruediger Helsch — Sun, 29 Jun 2014 22:15:19 -0000

--- v3
+++ v4
@@ -214,7 +214,7 @@
 UTF string conversion 
 ---------------------

-In the examples above, strcvt() can be replaces with strcvt_utf(),
+In the examples above, strcvt() can be replaced with strcvt_utf(),
 which ignores the locale and always performs a UTF transformation.
 The header is "strcvt/strcvt_utf.h".

strcvt modified by Ruediger Helsch

Ruediger Helsch — Sun, 29 Jun 2014 22:13:17 -0000

--- v2
+++ v3
@@ -140,8 +140,8 @@
 String conversion 
 -----------------

-Function strcvt\(source) converts strings from one
-character type to the other.  Example usage:
+Function strcvt\(source) converts strings from the
+source character type to the destination string type.  Example usage:

 ~~~{.cpp}

strcvt modified by Ruediger Helsch

Ruediger Helsch — Sun, 29 Jun 2014 22:06:03 -0000

--- v1
+++ v2
@@ -35,7 +35,7 @@
   `char32_t` in all combinations.

 - Implements the API of the C++ `codecvt` member functions in() and
-  out(), and convenience functions for string conversion.
+  out() for stream conversion, and convenience functions for string conversion.

 - Locale-dependent conversion and UTF conversion.

@@ -51,7 +51,7 @@
 search path of the compiler.  This is commonly achieved with the
 option `-I/path/to/strcvt/include` (assuming that the xprintf
 distribution has been upacked to directory `/path/to/xprintf`).  Then
-you can include the code converter header through their standard names
+you can include the code converter headers through their standard names
 like "strcvt/strcvt.h".

strcvt modified by Ruediger Helsch

Ruediger Helsch — Sun, 29 Jun 2014 21:14:10 -0000

Strcvt - Character type converter for C++11

Introduction
Installation
Usage
String conversion
UTF string conversion
Locale dependent stream conversion
UTF stream conversion
Character type converting iterator
Operators
Creating a Library
Compatibility
Rationale
License

Introduction

C++11 has the new character types char16_t and char32_t, which use
the UTF-16 and UTF-32 Unicode encoding. The question is how to convert
between the four character types of C++.

This library converts between the different character types of C++.

Conversion between the types char, wchar_t, char16_t and
char32_t in all combinations.
Implements the API of the C++ codecvt member functions in() and
out(), and convenience functions for string conversion.
Locale-dependent conversion and UTF conversion.

Installation

Unpack the distribution to a directory on your local machine. You can
include the headers in subdirectory include/xprintf from your
program. To make inclusion of the headers easier, it is recommended to
add the subdirectory include of the distribution to the include file
search path of the compiler. This is commonly achieved with the
option -I/path/to/strcvt/include (assuming that the xprintf
distribution has been upacked to directory /path/to/xprintf). Then
you can include the code converter header through their standard names
like "strcvt/strcvt.h".

Usage

In order to use the code converter, its headers have to be included.
They are:

~~~cpp

// The main header defining the strcvt() string conversion functions
#include "/path/to/strcvt/include/strcvt/strcvt.h"
// The header defining the strcvt_utf() UTF string conversion functions
#include "/path/to/strcvt/include/strcvt/strcvt_utf.h"
// The header defining the strcvt() stream conversion functions
#include "/path/to/strcvt/include/strcvt/charcvt.h"
// The header defining the strcvt_utf() UTF stream conversion functions
#include "/path/to/strcvt/include/strcvt/charcvt_utf.h"
// The header defining the strcvt_iterator
#include "/path/to/strcvt/include/strcvt/strcvt_iterator.h"
// The header defining codecvt_utf
#include "/path/to/strcvt/include/strcvt/codecvt_utf.h"

// Import symbols into users namespace
using namespace StrCvt;

// Optional header defining output operators
#include "/path/to/strcvt/include/strcvt/strcvt_operators.h"

// Import operators into users namespace
using namespace StrCvtOperators;

~~~

If the xprintf include directory /path/to/strcvt/include has been
added to the include file search path of the compiler, e.g. using the
compiler option -I/path/to/strcvt/include, this reduces to:

~~~{.cpp}

// The main header defining the strcvt() string conversion functions
#include "strcvt/strcvt.h"
// The header defining the strcvt_utf() UTF string conversion functions
#include "strcvt/strcvt_utf.h"
// The header defining the strcvt() stream conversion functions
#include "strcvt/charcvt.h"
// The header defining the strcvt_utf() UTF stream conversion functions
#include "strcvt/charcvt_utf.h"
// The header defining the strcvt_iterator
#include "strcvt/strcvt_iterator.h"
// The header defining codecvt_utf
#include "strcvt/codecvt_utf.h"

// Import symbols into users namespace
using namespace StrCvt;

// Optional header defining output operators
#include "strcvt/strcvt_operators.h"

// Import operators into users namespace
using namespace StrCvtOperators;

~~~

The strcvt() functions are exported through namespace StrCvt. It
is recommented to make them available via a using namespace
directive like in the example above. Alternatively it is possible to
import the functions strcvt() and strcvt_utf() etc. separately
through using declarations:

~~~{.cpp}

using StrCvt::strcvt;
using StrCvt::strcvt_utf;
using StrCvt::strcvt_utf_strict;
using StrCvt::make_strcvt_iterator;

~~~

The character type converter is pre-configured for header-only use.
This means: Just include the header and you are done. In order to
reduce space overhead and compilation time, a precompiled library can
be used. See section Creating a Library.

String conversion

Function strcvt\<dest_charT>(source) converts strings from one
character type to the other. Example usage:

~~~{.cpp}

#include "strcvt/strcvt.h"

// ...

using namespace StrCvt;

// Convert const char* C string to wide wchar_t string
std::wstring w = strcvt<std::wstring>("Hello, world!");
// Convert wide string to string of different character type
std::string s = strcvt<std::string>(w);
// Convert string to char16_t string
std::u16string s16 = strcvt<std::u16string>(s);
// The source buffer can also be specified as (pointer, size):
std::cout << strcvt<std::string>(&s16[0], 5) << std::endl;

~~~

The source string can be specified as:
- a character pointer of any character type (C-style null-terminated string),
- a C++ string of arbitrary character type, or
- a character pointer and a size.

The destination string type is specified as a template argument to
strcvt(). The function is specialized on all combinations of source
and destination character types. So the full signatures are:

~~~{.cpp}

template<class charT, class source_charT>
std::basic_string<charT>
strcvt<std::basic_string<charT> >(const std::basic_string<source_charT>& source);

template<class charT, class source_charT>
std::basic_string<charT>
strcvt<std::basic_string<charT> >(const source_charT* source);

template<class charT, class source_charT>
std::basic_string<charT>
strcvt<std::basic_string<charT> >(const source_charT* source, std::size_t size);

~~~

Instead of the destination string type, it is also possible to give
the destination character type as template argument. It is
automatically translated to the string type.

It is also possible to omit the template argument determining the
destination string type. In this case, strcvt() does no conversion at
all, but returns an object which can be converted to all string types.

~~~{.cpp}

// strcvt() returns an object which can be assigned to any string type
std::wstring w = strcvt("Hello, world!");
std::basic_string<char16_t> = strcvt(w);

~~~

For string arguments, the return value of strcvt() is a subclass of
the class of its argument. The subclass adds conversion operators to
all string types. For character pointer arguments, the return value
of strcvt() is an object which can be converted to a character
pointer (of the character type of the argument) or to any string type.
In each case, the return value of strcvt() can e.g. be assigned to
any string type, or can be passed as argument to a function expecting a
string.

UTF string conversion

In the examples above, strcvt() can be replaces with strcvt_utf(),
which ignores the locale and always performs a UTF transformation.
The header is "strcvt/strcvt_utf.h".

It is also possible to configure strcvt() to only use UTF conversions
instead of the locale dependent character conversions. Open the header
strcvt/strcvt_config.h with an editor and change the preprocessor
symbol STRCVT_IMPL_UTF8_ONLY from 0 to 1.

Locale-dependent stream conversion

The character type stream converter function charcvt(state, from,
from_end, from_next, to, to_end, to_next) is defined in header
strcvt/charcvt.h and uses the API of the member functions in()
and out() of the standard code conversion interface std::codecvt:

~~~{.cpp}

#include "strcvt/charcvt.h"

// ...

using namespace StrCvt;

std::mbstate_t state = std::mbstate_t(); // Zero-initialize
// Convert between character types:
result r = charcvt(state, from, from_end, from_next, to, to_end, to_next);

// ... More calls of charcvt()

// Return output to initial shift state
r = charcvt_unshift<source_charT>(state, to, to_end, to_next);

~~~

The state must be of type std::mbstate_t. It holds the conversion
state between successive calls of charcvt() and must be explicitly
zero-initialized before the first use, like shown above. The
arguments from and from_end delimit the source character buffer,
to and to_end delimit the destination character buffer, and on
exit from the function from_next and to_next point past the last
converted character. The function returns std::codecvt_base::ok on
success and std::codecvt_base::partial if the output buffer was too
small to convert the entire input buffer, or if the input buffer ended
in a part of a multibyte sequence. After the input has been
completely converted, possibly by multiple calls to charcvt(),
charcvt_unshift() must be called to move the output to the initial
shift state. The source character type must be specified as a
template argument to charcvt_unshift(), since different converters are
used depending on the source character type. Charcvt_unshift() should
even be used if the character encoding is known not to be state
dependent, like UTF-8. Charcvt_unshift() checks whether trailing
incomplete Unicode character input sequences are pending, and appends
a replacement character to the output buffer if necessary to signal
the presence of trailing garbage.

Charcvt() is specialized for all combinations of source and
destination character types char, wchar_t, char16_t and
char32_t and converts between the locale-dependent character type
char, the Unicode UTF-16 and UTF-32 character types char16_t and
char32_t, and the implementation defined wide character type
wchar_t which is assumed to be equivalent to either char16_t or
char32_t. It is able to consume or deliver single characters from
multi-character Unicode characters.

The full signature of charcvt() is:

~~~{.cpp}

template<class src_charT, class dst_charT>
std::codecvt_base::result
charcvt(std::mbstate_t& state,
        const src_charT* from, const src_charT* from_end,
        const src_charT*& from_next,
        dst_charT* to, dst_charT* to_end, dst_charT*& dst_next,
        int flags = 0);

~~~

The flags can be omitted, or they can be set to the constant
strcvt_flags_no_partial_conversions to prevent partial conversions.
If the flag is omitted or zero, charcvt() is eager and consumes even
single partial multibyte characters.

UTF stream conversion

The function charcvt_utf(state, from, from_end, from_next, to, to_end,
to_next) is defined in header strcvt/charcvt_utf.h and works like
charcvt(state, from, from_end, from_next, to, to_end, to_next) but
treats the type char as having UTF-8 encoding instead of a locale
dependent implementation defined encoding:

~~~{.cpp}

#include "strcvt/charcvt_utf.h"

// ...

using namespace StrCvt;

std::mbstate_t state = std::mbstate_t(); // Zero-initialize
// Convert between character types:
charcvt_utf(state, from, from_end, from_next, to, to_end, to_next);

// ... More calls of charcvt_utf()

// Return output to initial shift state
charcvt_utf_unshift(state, to, to_end, to_next);

~~~

The state must be of type std::mbstate_t. It holds the conversion
state between successive calls of charcvt_utf() and must be explicitly
zero-initialized before the first use, like shown above. The
arguments from and from_end delimit the source character buffer,
to and to_end delimit the destination character buffer, and on
exit from the function from_next and to_next point past the last
converted character. The function returns std::codecvt_base::ok on
success and std::codecvt_base::partial if the output buffer was too
small to convert the entire input buffer, or if the input buffer ended
in a part of a multibyte sequence. After the input has been
completely converted, possibly by multiple calls to charcvt_utf(),
charcvt_utf_unshift() checks whether trailing incomplete Unicode
character input sequences are pending, and appends a replacement
character to the output buffer if necessary to signal the presence of
trailing garbage.

Charcvt_utf() is specialized for all combinations of source and
destination character types char, wchar_t, char16_t and
char32_t and converts between UTF-coded char, the Unicode UTF-16
and UTF-32 character types char16_t and char32_t, and the
implementation defined wide character type wchar_t which is assumed
to be equivalent to either char16_t or char32_t. Contrary to
member functions in() and out() of codecvt, charcvt_utf() always
converts (even if the input character type is the same as the output
character type, in which case it checks the validity of the encoding).
It always generates valid UTF-8, UTF-16 or UTF-32 output sequences.

For each invalid encoding in the input buffer, charcvt_utf() inserts
the replacement character U+0xFFFD into the output buffer. If this is
not wanted, function charcvt_utf_strict() can be used, which has the
same interface as charcvt_utf() but returns std::codecvt_base::error
on encoding errors, with from_next pointing to the first element of
the invalid sequence.

There is also a UTF code conversion facet defined in header
strcvt/codecvt_utf.h

~~~{.cpp}

class codecvt_utf<intern_charT, extern_charT>;

~~~

This code conversion facet uses charcvt_utf() to convert between UTF
coded characters intern_charT and extern_charT, and is specialized
for all character type combinations.

Character type converting character iterator

Instead of converting the entire buffer, it can be accessed trough a
converting character iterator. The iterator is created by function
make_strcvt_iterator():

~~~{.cpp}

#include "strcvt/strcvt_iterator.h"

// ...

using namespace StrCvt;

// Create char32_t iterator for access to source C string
auto it = make_strcvt_iterator<char32_t>("Hello, world!");

~~~

The end iterator is returned by make_strcvt_iterator() without
arguments, or it can be obtained from member function end() of the
iterator. Since member function begin() is also implemented, and
returns the iterator itself, the iterator can be used just like a
container, for example with the range-based for statement:

~~~{.cpp}

std::basic_string<char32_t> w;
// Use range-based for statement: process all Unicode characters
for (char32_t c: it)
    w.push_back(c);

// Alternatively, use simple iterator interface
auto e = it.end();
for (auto j = it; j != e; ++j)
    w.push_back(*j);

~~~

The converting iterator is (indirectly) derived from base class
strcvt_iterator_base<charT> which does not depend on the source
iterator type or the source character type. It has a virtual
destructor and can be used polymorphically. The inheritance hierarchy
looks like this:

~~~{.cpp}

template<class charT>
class strcvt_iterator_base {
    // Operators *(), ++(), ==()
    // Member functions begin(), end()
};

template<class charT, class source_charT>
class strcvt_iterator_impl : public strcvt_iterator_base<charT> {
    // Implements operator ++() which does the conversion using
    // virtual member function get_next_source_char()
};

template<class charT, class SourceIterator>
class strcvt_iterator
    : public strcvt_iterator_impl<charT, iterator_traits<SourceIterator>::value_type> {
    // Implements get_next_source_char()
};

~~~

Class template strcvt_iterator_base does not know the source
character type and can be used to handle a strcvt_iterator
polymorphically. Class template strcvt_iterator_impl knows the
source character type but not the type of the iterator. If the
precompiled library is used, it is precompiled for all combinations of
source and destination character type. Class template
strcvt_iterator is the return value of function
make_strcvt_iterator(). It implements the source iterator handling.

Operators

Some operators for strings are defined in header
"strcvt/strcvt_operators.h". These are the output operators <<
for strings and character pointers, and the appending += operators
for strings. To use them, include header
"strcvt/strcvt_operators.h". The operators reside in namespace
StrCvtOperators and should be imported into the user's namespace
through a using namespace directive:

~~~{.cpp}

#include "strcvt/strcvt_operators.h"
using namespace StrCvtOperators;

std::cout << U"Hello, world!\n";

~~~

It would be nice if we could define conversion operators to enable
assignment between different string types. But in C++, conversion and
assignment operators can only be defined as member functions.

Creating a Library

The character type converter is preconfigured for header-only use.
This means: Just include the header and you are done. In order to
reduce space overhead and compilation time, a precompiled library can
be used.

The main advantage in using a library is that each time the strcvt
headers are included, the compiler does not need to look at the
implementation details. This can speed up compilation significantly.

To create the library, the C++ source files libstrcvt.cpp,
libstrcvt_iterator.cpp, libstrcvt_utf.cpp and libcodecvt_utf.cpp in
directory lib of the distribution must be compiled. Under Linux,
just run make. Before compiling, you may want to select the
compiler to use: Uncomment to proper CXX= - line in the toplevel
Makefile.template. Running make should create a library
lib/libxprintf.a, which has to be linked to the programs.

In Visual C++, instead of building a library, you may just add the
library source files to your project.

In order to make the headers use the library, you must open the header
strcvt/strcvt_config.h with an editor and change the preprocessor
symbol STRCVT_IMPL_USE_LIBRARY from 0 to 1. The next time the
header is included, the library will be used. You can check that the
library is used as intended by omitting the library when linking.
Linking should fail with missing externals.

In order to run the tests, the headers for the boost test framework
are required.

Compatibility

The code converter has been tested with:

GCC 4.7, 4.8 and 4.9 on Linux
Visual Studio Express 2013 for Windows Desktop with November 2013 CTP
Intel C++ 14.0.2 on Linux

Rationale

C++11 has the new character types char16_t and char32_t, which use
the UTF-16 and UTF-32 Unicode encoding. The question is how to convert
between the four character types of C++.

The codecvt part of the C++11 library looks like some ruins left
over at the front line between warring factions.

The codecvt class template is part of the header \<locale> and
described in section 22.4.1.4 of the standard. The first thing to
note is that according to this specification codecvt transforms
between an internal and an external character encoding, so it is
not intended to transform between internal character encodings like
char16_t and wchar_t.

According to the standard, each locale shall have specializations of
codecvt for tranformation between the internal character types
char, wchar_t, char16_t and char32_t and the external
character type char. The standard first says that these
specializations "convert the implementation-defined native character
set", only to continue specifying that

the specialization with external and internal character type both
char must not convert at all (effectively saying that the internal
char must have the same encoding as the external char),
the specializations with internal character types char16_t and
char32_t must treat the external character code as UTF-8, so they
are explicitly not allowed to treat it as an "implementation-defined
native character set".

This leaves the transformation between internal character type
wchar_t and external character type char as the only
locale-dependent transformation, and it is specified to convert
"between the native character sets for narrow and wide characters".
So no luck using codecvt to transform between native character sets
and char16_t or char32_t.

Section 22.5 of the standard specifies the "standard code conversion
facets". It contains:

Facet codecvt_utf8, which converts between a UTF-8 coded char
buffer and UCS2 or UCS4, so it can be used to transform between
UTF-8 and char32_t. It is not usable for transformation between
UTF-8 and char16_t because char16_t strings are coded as
UTF-16, not UCS2.
Facet codecvt_utf8_utf16 converts between UTF-8 and UTF-16, so it
can be used to transform between UTF-8-coded char and char16_t.
Facet codecvt_utf16 looks like a hack from the 20th century to
adapt UTF-16 wide characters to character byte streams. The
UTF-16-coded buffer is addressed through a char*.

So we can use these standard code conversion facets to convert between
UTF-8-coded char and char16_t or char32_t. We could even
transform between char16_t and char32_t by going through an
intermediate UTF-8-coded char buffer. But again, no
transformation between native character sets and char16_t or
char32_t.

The standard does provide an interface for transformation between
native char and char16_t or char32_t. It is hidden at the very
end of section 21.7 in the string library and consists of the four
functions mbrtoc16(), c16rtomb(), mbrtoc32() and c32rtomb().
Unfortunately, a locale can not be specified for these functions.
They always work with the currently active global locale.

For the following things I know of no standard-conforming procedure:

Conversion between wchar_t and char16_t or char32_t. Clearly,
going through native char is not an option, unless it happens to
use UTF-8 encoding. I worked around this by assuming that the native
wchar_t type uses Unicode coding and is equivalent to either
char16_t or char32_t, depending on sizeof(wchar_t). This
assumption holds for the systems I have access to, but there may be
other wchar_t encodings in use. Older Windows systems used UCS2,
where this assumption would not hold, but modern versions of Windows
use UTF-16.
Locale-dependent conversion: especially for output streams where a
locale is known that may not be the global locale, it would be
useful if the locale could be passed as an argument to the character
transformation. Unfortunately the standard library does not provide
a portable way to do this, so I left it out.

First I thought I would reuse the standard-specified UTF coders, but
first they are incomplete (going from UTF-16 to UTF-32 via an
intermediate representation as UTF-8 is not very attractive), second
they are distributed over several different interfaces, and third GCC
does not have them. So as a first step I implemented a transformation
between UTF-coded char, char16_t, char32_t and wchar_t. The
function strcvt_utf(state, from, from_end, from_next, to, to_end,
to_next) uses the API of the member functions in() and out() of
codecvt. The state holds the conversion state, from and
from_end delimit the source character buffer, to and to_end
delimit the destination character buffer, and on exit from the
function from_next and to_next point past the last converted
character. Like in() and out(), strcvt_utf() returns
std::codecvt_base::ok on success and std::codecvt_base::partial if
the output buffer was too small to convert the entire input buffer, or
if the input buffer ended in a part of a multibyte sequence. For each
invalid encoding in the input buffer, the replacement character
U+0xFFFD is inserted into the output stream. If this is not wanted,
function strcvt_utf_strict() can be used, which has the same
interface as strcvt_utf() but returns std::codecvt_base::error on
encoding errors, with from_next pointing to the first element of the
invalid sequence. Both functions are specialized on all combinations
of char, wchar_t, char16_t and char32_t. They always convert
(even if input and output type is the same, in which case they check
the validity of the encoding). They are able to consume and produce
single characters forming part of multibyte Unicode characters.

Based on the UTF coder, a locale-dependent converter has been
implemented. The function strcvt(state, from, from_end, from_next,
to, to_end, to_next) also uses the API of the member functions in()
and out() of codecvt: The state holds the conversion state,
from and from_end delimit the source character buffer, to and
to_end delimit the destination character buffer, and on exit from
the function from_next and to_next point past the last converted
character. This implementation uses the assumption that wchar_t is
encoded as either UTF-16 or UTF-32. Transformations between wide
characters wchar_t, char16_t and char32_t use UTF
transformations, transformations between char and wide characters
treat char as a native locale-defined character code.

Convenience functions for the code transformation are:

basic_string strcvt(const C from, const C
from_end) converts the characters between from and from_end into
a string of character type charT, which is specified as a template
parameter.
basic_string strcvt(const C* from) converts the
characters of the null-terminated string from into a string of
character type charT, which is specified as a template parameter.
basic_string strcvt(const std::basic_string&
from) converts the characters of the string from into a string of
character type charT, which is specified as a template parameter.
A specialization for charT = C and rvalue from is provided, which
moves the input string to the return value. Note that this is the
only conversion which does not check the validity of the encoding of
the string.
strcvt(const std::basic_string& from) (strcvt() without
template parameter charT specifying the destination character
type) casts the type of the argument to a reference to a class
derived from std::basic_string, which adds conversion operators
to all string types. The result can e.g. be assigned to any string
type, or passed to a function expecting any string.
strcvt(const C* from) (strcvt() without template parameter charT
specifying the destination character type) creates an object which
can be converted back to a const C* or to any string type. The
result can e.g. be assigned to any string type, or passed to a
function expecting any string.

License

Permission to use, copy, modify, and distribute this software for any
purpose and without fee is hereby granted. The author disclaims all
warranties with regard to this software.