[Flex-devel] %option utf16 utf16le & utf16be

[Paul <pa...@pr...>]

The flex unicode version has now been changed so that:
%option utf16 generates a scanner that accepts the native utf of the 
machine.
%option utf16le generates a scanner that accepts UTF-16LE regardless of 
the machine byte order.
%option utf16be generates a scanner that accepts UTF-16BE regardless of 
the machine byte order.

This means that when utf16le or utf16be is an option the scanner tests 
the byte order of the machine. This code is not generated otherwise or 
for utf16. The test is done at startup, but checked at each read to 
decide to swap bytes. The byte swap is also only generated with utf16le 
or utf16be. Thus a Flex scanner.c may be moved between machines with 
hopefully least astonishment.

There are separate tests for C, C++, reentrant and non-reentrant 
scanners, with options utf16le & utf16be.
There are now 105 tests all of which pass.

The flag -U or --utf is the same as %option utf16.
The flag --utf16le is the same as %option utf16le.
The flag --utf16be is the same as %option utf16be.

Suggestions?

Paul Neelands