Re: Please test UTF-8 support in revised SVG driver

[Ethan A M. <merritt@u.washington.edu>]

On Sunday 15 October 2006 01:09 pm, Petr Mikulik wrote:
> > This weekend I decided to sit down and try to understand how utf-8
> > encoded works.  Afterwards I modified the svg terminal driver
> > to re-encode utf-8 strings as escaped Unicode hex constants. This
> > makes it work properly on my machine using either firefox or
> > konqueror, whereas without re-encoding it worked only partially.
>=20
> I've tried it with a Czech text like
>  	P=C5=99=C3=ADli=C5=A1 =C5=BElu=C5=A5ou=C4=8Dk=C3=BD k=C5=AF=C5=88 =C3=
=BAp=C4=9Bl =C4=8F=C3=A1belsk=C3=A9 =C3=B3dy.
> It works with "set encoding default" and the text written in UTF-8.=20

Good. If you look in the *.svg output file, you will see that it now
starts with the line
<?xml version=3D"1.0" encoding=3D"utf-8"  standalone=3D"no"?>
The UTF-8 encoding part is working as intended in this case.

> For "set encoding iso8859_2" and the text in that encoding, it gets
> displayed wrong. =20

OK. That's exactly the sort of feedback I needed.
I can't test this easily because I don't have a 8859-2 locale installed.

> If I do so, and having Czech text in those encodings, the text appears
> correctly in both Firefox 1.5.0.7 and Konqueror 3.5.2.=20

I have now made the re-encoding dependent on the current setting of
gnuplot's "set encoding". It will re-encode into Unicode only for settings
"default" and "iso_8859_1".

Please give it another test.=20

> And, how is it supposed to work on Windows with cp1250 encoding?

Why is that special?

> Well, I wonder why it is not enough to have
>    <?xml version=3D"1.0" encoding=3D"utf-8"  standalone=3D"no"?>
> as the first line of the output file?=20

Two problems:

=2D Some older versions of svg.trm sort of worked, but there were bug repor=
ts
  about poor handling of characters with octal values >127, and we
  changed svg.trm to escape all of these byte-by-byte into the form
  &#xXX;  Since the individual bytes of a multi-byte UTF-8 sequence
  are not legal characters on their own, this comes out total garbage.

=2D Even if I revert to passing the bytes through without escaping, the
  viewers I have access to do not correctly handle multi-byte utf-8 sequenc=
es.
  I haven't figured out exactly what the limitations are, but mostly it
  doesn't come out right.  Perhaps they can handle 2-byte sequences, but
  not 3- and 4-byte sequences?  I don't know.

> Thus, I don't see any advantage of the reencoding.=20
> I prefer to keep the text as-written.

Well, up until now it has not worked for UTF-8.  Now it does.
Let's try and fix it so that the UTF-8 support doesn't break your
other encodings.

=2D-=20
Ethan A Merritt
Biomolecular Structure Center
University of Washington, Seattle 98195-7742