Re: [Scrollkeeper-devel] omf encoding

[Dan M. <mu...@al...>]

Does anybody happen to know if there is a command-line tool to convert
files from one encoding to another, and one character set to another?  I
see you can do it with vim, but I figure there must be a command-line tool
somewhere.

Similarly, does anybody know of a tool for detecting the encoding and 
character set of a file?

Dan


On 9 May 2002, John Fleck wrote:

> On Thu, 2002-05-09 at 20:50, Malcolm Tredinnick wrote:
> > On Thu, May 09, 2002 at 06:45:33PM -0600, John Fleck wrote:
> > > Unless the document's encoding is declared otherwise, libxml (and
> > > therefore ScrollKeeper) assumes UTF-8:
> > > 
> > > http://xmlsoft.org/encoding.html
> > > 
> > > "If there is no encoding declaration, then the input has to be in either
> > > UTF-8 or UTF-16, if it is not then at some point when processing the
> > > input, the converter/checker of UTF-8 form will raise an encoding error.
> > > You may end-up with a garbled document, or no document at all !"
> > > 
> > > This means that in the case of OMF files in other encodings (like the
> > > one for ggv that started this discussion) we need to specify the
> > > encoding in the xml declaration.
> > > 
> > > The encodings that seem to be of importance to us:
> > > 
> > >    1. UTF-8 is supported by default (null handlers)
> > >    2. UTF-16, both little and big endian
> > 
> > Where is UTF-16 being used commonly? Asian languages I know about (so
> > they tend to prefer local encodings at the moment), but are there any
> > others.
> > 
> > >    3. ISO-Latin-1 (ISO-8859-1) covering most western languages
> > 
> > We'll need to be aware of all iso-8859, not just iso-8869-1, since many
> > European languages need them. For example, Czechs and the Slovaks use
> > 8859-2, a lot of the Scandanavian countries use 8859-6, etc. Unless you
> > add the requirement that everything must be UTF-8, then we just need to
> > be able to do this generally (the Japanese and Chinese translators might
> > like us more, then, too).
> > 
> > And that's all assuming you don't want the Euro symbol, and ... oh ...
> > it's all getting too hard ... I need to go lie down now. :-(
> > 
> 
> Well, unless we're particularly industrious, we're restricted to the
> encodings supported by libxml2, which are the ones I listed above plus a
> couple that didn't seem particularly relevant. The full list*:
> 
>    1. UTF-8 is supported by default (null handlers)
>    2. UTF-16, both little and big endian
>    3. ISO-Latin-1 (ISO-8859-1) covering most western languages
>    4. ASCII, useful mostly for saving
>    5. HTML, a specific handler for the conversion of UTF-8 to ASCII with
> HTML predefined entities like © for the Copyright sign.
> 
> * http://www.xmlsoft.org/encoding.html
> 
> Beyond that, we'll have to write our own encoding converters.
> 
> Cheers,
> John
>