Iconv has major problems with UTF-16 file on the
Apparently the files are opened as text instead of binary.
With UTF-16, this means special characters will be seen
where they do not exist (the value was only part of an
extended character), and modifications in the
input/output made by the OS will generate invalid UTF-16.
Conversion of a valid UTF-16-LE file to the same format :
iconv -f utf-16LE -t utf-16LE index.txt > ind.txt
shows that :
0D 00 - 0A 00 - 32 00 in input changes to :
0D 00 - 0D 0A 00 - 32 00 in output.
The erronous 0D desynchonizes all the data after that
and it becomes invalid.
When in the input is present :
3A 00 - 1A 90 - 3A 00
the outputs stop after
without any error message, nothing saying that invalid
input is seen. It seems to think 1A is an end of file.
When converting to utf-8 (there should not be a problem
iconv -f utf-16LE -t utf-8 index.txt > ind_u8.txt
the input sequence :
0D 00 - 0A 00
0D - 0D - 0A