Menu
▾
▴
gnuplot-beta
|
From: Ethan A M. <merritt@u.washington.edu> - 2006-10-15 06:02:05
|
Hi all, This weekend I decided to sit down and try to understand how utf-8 encoded works. Afterwards I modified the svg terminal driver to re-encode utf-8 strings as escaped Unicode hex constants. This makes it work properly on my machine using either firefox or konqueror, whereas without re-encoding it worked only partially. Please let me know if you find problems viewing the new svg output in other svg viewers. Known limitations: text centering in svg was not good to begin with, and if the text includes utf-8 characters the centering gets even worse. You need to select a utf-8 font, of course. I tested with set term svg font "ArialUnicode MS" -- Ethan A Merritt Biomolecular Structure Center University of Washington, Seattle 98195-7742 |
|
From: <br...@ph...> - 2006-10-15 11:49:59
|
Ethan A Merritt wrote: > Known limitations: > text centering in svg was not good to begin with, and if the > text includes utf-8 characters the centering gets even worse. If so, that's a bug in the enhanced mode implementation. Non-enhanced SVG_put_text passes this job to the only instance that can handle it: the SVG file itself, and its interpreter(s). |
|
From: Ethan A M. <merritt@u.washington.edu> - 2006-10-15 14:38:32
|
On Sunday 15 October 2006 04:52 am, Hans-Bernhard Br=F6ker wrote: > Ethan A Merritt wrote: >=20 > > Known limitations: > > text centering in svg was not good to begin with, and if the > > text includes utf-8 characters the centering gets even worse. >=20 > If so, that's a bug in the enhanced mode implementation. Non-enhanced=20 > SVG_put_text passes this job to the only instance that can handle it:=20 > the SVG file itself, and its interpreter(s). I agree. It's a failure of the enhanced mode implementation. The current code is reduced to counting characters and approximating their net width so that it can place them properly. A better implementation would somehow tell the svg interpreter to save/restore a particular position so that we can return to it later.=20 The corresponding terminal entry points at ENHsvg_open() are marked =46IXME, waiting for someone who knows svg better than I do. =2D-=20 Ethan A Merritt Biomolecular Structure Center University of Washington, Seattle 98195-7742 |
|
From: Petr M. <mi...@ph...> - 2006-10-15 20:09:37
|
> This weekend I decided to sit down and try to understand how utf-8 > encoded works. Afterwards I modified the svg terminal driver > to re-encode utf-8 strings as escaped Unicode hex constants. This > makes it work properly on my machine using either firefox or > konqueror, whereas without re-encoding it worked only partially. I've tried it with a Czech text like =09P=F8=EDli=B9 =BElu=BBou=E8k=FD k=F9=F2 =FAp=ECl =EF=E1belsk=E9 =F3dy. It works with "set encoding default" and the text written in UTF-8. For "se= t=20 encoding iso8859_2" and the text in that encoding, it gets displayed wrong.= =20 So you reencoding is wrong. And, how is it supposed to work on Windows=20 with cp1250 encoding? I think your patch should work only for the default encoding or "set=20 encoding utf8" (to be added), otherwise it should not do any &#xxxx;=20 changes. Well, I wonder why it is not enough to have <?xml version=3D"1.0" encoding=3D"utf-8" standalone=3D"no"?> <?xml version=3D"1.0" encoding=3D"iso-8859-2" standalone=3D"no"?> as the first line of the output file? Why it does not work for you and=20 UTF-8? If I do so, and having Czech text in those encodings, the text=20 appears correctly in both Firefox 1.5.0.7 and Konqueror 3.5.2. Thus, I don't see any advantage of the reencoding. I prefer to keep the tex= t=20 as-written. If you need it, then I propose to force it by an option to "set term svg". --- PM |
|
From: Ethan A M. <merritt@u.washington.edu> - 2006-10-15 21:59:06
|
On Sunday 15 October 2006 01:09 pm, Petr Mikulik wrote: > > This weekend I decided to sit down and try to understand how utf-8 > > encoded works. Afterwards I modified the svg terminal driver > > to re-encode utf-8 strings as escaped Unicode hex constants. This > > makes it work properly on my machine using either firefox or > > konqueror, whereas without re-encoding it worked only partially. >=20 > I've tried it with a Czech text like > P=C5=99=C3=ADli=C5=A1 =C5=BElu=C5=A5ou=C4=8Dk=C3=BD k=C5=AF=C5=88 =C3= =BAp=C4=9Bl =C4=8F=C3=A1belsk=C3=A9 =C3=B3dy. > It works with "set encoding default" and the text written in UTF-8.=20 Good. If you look in the *.svg output file, you will see that it now starts with the line <?xml version=3D"1.0" encoding=3D"utf-8" standalone=3D"no"?> The UTF-8 encoding part is working as intended in this case. > For "set encoding iso8859_2" and the text in that encoding, it gets > displayed wrong. =20 OK. That's exactly the sort of feedback I needed. I can't test this easily because I don't have a 8859-2 locale installed. > If I do so, and having Czech text in those encodings, the text appears > correctly in both Firefox 1.5.0.7 and Konqueror 3.5.2.=20 I have now made the re-encoding dependent on the current setting of gnuplot's "set encoding". It will re-encode into Unicode only for settings "default" and "iso_8859_1". Please give it another test.=20 > And, how is it supposed to work on Windows with cp1250 encoding? Why is that special? > Well, I wonder why it is not enough to have > <?xml version=3D"1.0" encoding=3D"utf-8" standalone=3D"no"?> > as the first line of the output file?=20 Two problems: =2D Some older versions of svg.trm sort of worked, but there were bug repor= ts about poor handling of characters with octal values >127, and we changed svg.trm to escape all of these byte-by-byte into the form &#xXX; Since the individual bytes of a multi-byte UTF-8 sequence are not legal characters on their own, this comes out total garbage. =2D Even if I revert to passing the bytes through without escaping, the viewers I have access to do not correctly handle multi-byte utf-8 sequenc= es. I haven't figured out exactly what the limitations are, but mostly it doesn't come out right. Perhaps they can handle 2-byte sequences, but not 3- and 4-byte sequences? I don't know. > Thus, I don't see any advantage of the reencoding.=20 > I prefer to keep the text as-written. Well, up until now it has not worked for UTF-8. Now it does. Let's try and fix it so that the UTF-8 support doesn't break your other encodings. =2D-=20 Ethan A Merritt Biomolecular Structure Center University of Washington, Seattle 98195-7742 |
|
From: Petr M. <mi...@ph...> - 2006-10-16 06:10:50
|
>> For "set encoding iso8859_2" and the text in that encoding, it gets >> displayed wrong. > > OK. That's exactly the sort of feedback I needed. > I can't test this easily because I don't have a 8859-2 locale installed. I've just tried after your latest patch, it works with UTF-8 and 'set enco= =20 default', but not for latin2. Try this: Write b.gp in UTF-8: set encoding iso_8859_2 set term svg set out 'b.svg' set title "P=F8=EDli=B9 =BElu=BBou=E8k=FD k=F9=F2 =FAp=ECl =EF=E1belsk=E9 = =F3dy." set xlabel "P=F8=EDli=B9 =BElu=BBou=E8k=FD k=F9=F2 =FAp=ECl =EF=E1belsk=E9 = =F3dy." set ylabel "P=F8=EDli=B9 =BElu=BBou=E8k=FD k=F9=F2 =FAp=ECl =EF=E1belsk=E9 = =F3dy." plot x recode utf8..latin2 b.gp gnuplot b.gp =3D> the file is rendered incorrectly; the text appears as: <text>Pøíli¹ ¾lu»ouèký kùò=20 úpìl ïábelské ódy.</text> --- PM |
|
From: Ethan M. <merritt@u.washington.edu> - 2006-10-16 16:43:28
|
On Sunday 15 October 2006 11:10 pm, Petr Mikulik wrote:
>
> I've just tried after your latest patch, it works with UTF-8 and 'set
> enco default', but not for latin2. Try this:
>
> Write b.gp in UTF-8:
>
> set encoding iso_8859_2
> set term svg
> set out 'b.svg'
> set title "P=C5=99=C3=ADli=C5=A1 =C5=BElu=C5=A5ou=C4=8Dk=C3=BD k=C5=AF=C5=
=88 =C3=BAp=C4=9Bl =C4=8F=C3=A1belsk=C3=A9 =C3=B3dy."
> set xlabel "P=C5=99=C3=ADli=C5=A1 =C5=BElu=C5=A5ou=C4=8Dk=C3=BD k=C5=AF=
=C5=88 =C3=BAp=C4=9Bl =C4=8F=C3=A1belsk=C3=A9 =C3=B3dy."
> set ylabel "P=C5=99=C3=ADli=C5=A1 =C5=BElu=C5=A5ou=C4=8Dk=C3=BD k=C5=AF=
=C5=88 =C3=BAp=C4=9Bl =C4=8F=C3=A1belsk=C3=A9 =C3=B3dy."
> plot x
>
> recode utf8..latin2 b.gp
> gnuplot b.gp
>
>
> =3D> the file is rendered incorrectly; the text appears as:
>
> <text>Pøíli¹ ¾lu»ouèký
> kùò úpìl ïábelské ódy.</text>
But that is what it was doing already, *before* my utf-8 patch.
The change to write non-ascii characters in the form »
was due to this patch:
2006-10-03 Hans-Bernhard Broeker <br...@ph...>
* term/svg.trm: Change sprintf() and fprintf() to strcpy and
fputs, all over the place.
(SVG_put_text): Fix handling of XML reserved characters in
non-enhanced text output.
That patch broke things for me, also, which is what started me
on the path to reworking the code so that at least it works in
my utf-8 environment.
So I think we have a problem here.
Apparently Hans-Bernhard's svg viewer does not like the inline
non-ascii characters, whereas your viewers and mine require them
in order to process the encoding correctly.
The minimal reversion to pre-Oct 3 behavior is
=2D-- gnuplot/term/svg.trm 2006-10-12 10:53:40.000000000 -0700
+++ gnuplot-cvs/term/svg.trm 2006-10-16 09:37:36.000000000 -0700
@@ -945,10 +945,12 @@
fputs("&", gpoutfile);
break;
default:
+#if (0)
/* Some SVG viewers really dislike non-ascii characters */
if (symb < ' ' || symb > '~')
fprintf(gpoutfile, "&#x%2.2x;", symb);
else
+#endif
fputc(*str, gpoutfile);
break;
}
That only changes the non-enhanced mode output, however.
If you need to change the enhanced mode output also, then add
@@ -1222,12 +1224,14 @@
/* Kludge for phantom box accounting */
ENHsvg_charcount++;
=20
+#if (0)
/* My SVG viewers really dislike non-ascii characters */
if (symb < ' ' || symb > '~') {
sprintf(enhanced_cur_text, "&#x%2.2x;", symb);
enhanced_cur_text +=3D 6;
return;
}
+#endif
=20
/* Escape SVG reserved characters. Are there any besides '<' and=20
'&' ? */
switch (c) {
=2D-=20
Ethan A Merritt
Biomolecular Structure Center
University of Washington, Seattle WA
|
|
From: Petr M. <mi...@ph...> - 2006-10-16 19:53:34
|
>> I've just tried after your latest patch, it works with UTF-8 and 'set >> enco default', but not for latin2. Try this: >> >> Write b.gp in UTF-8: >> >> set encoding iso_8859_2 >> set term svg >> set out 'b.svg' >> set title "P=F8=EDli=B9 =BElu=BBou=E8k=FD k=F9=F2 =FAp=ECl =EF=E1belsk= =E9 =F3dy." >> set xlabel "P=F8=EDli=B9 =BElu=BBou=E8k=FD k=F9=F2 =FAp=ECl =EF=E1belsk= =E9 =F3dy." >> set ylabel "P=F8=EDli=B9 =BElu=BBou=E8k=FD k=F9=F2 =FAp=ECl =EF=E1belsk= =E9 =F3dy." >> plot x >> >> recode utf8..latin2 b.gp >> gnuplot b.gp >> >> >> =3D> the file is rendered incorrectly; the text appears as: >> >> <text>Pøíli¹ ¾lu»ouèký >> kùò úpìl ïábelské ódy.</text> > > But that is what it was doing already, *before* my utf-8 patch. Thus, it looks to me a "feature" of svg -- the following =09<?xml version=3D"1.0" encoding=3D"iso-8859-2" standalone=3D"no"?> =09... =09<text>--=F8-- --ø--</text> does not print =F8 twice even though F8 is its code in iso-8859-2. Cannot i= t=20 be that &#nnnn; are always interpreted as UTF-8 chars, whatever header says= ? In that case, in non-"set encoding default", gnuplot cannot do=20 transformation of 8bit chars into hex string, as the string will get all=20 wrong! (All =3D=3D without iso latin 1, which is a subset of utf-8, by desi= gn.) > The change to write non-ascii characters in the form » > was due to this patch: > > 2006-10-03 Hans-Bernhard Broeker <br...@ph...> > > * term/svg.trm: Change sprintf() and fprintf() to strcpy and > fputs, all over the place. > (SVG_put_text): Fix handling of XML reserved characters in > non-enhanced text output. > > That patch broke things for me, also, which is what started me > on the path to reworking the code so that at least it works in > my utf-8 environment. > > So I think we have a problem here. > Apparently Hans-Bernhard's svg viewer does not like the inline > non-ascii characters, whereas your viewers and mine require them > in order to process the encoding correctly. Yes, then it is a bug of that particular svg viewer. Without Hans-Bernhard's patch, SVG output is correct for both UTF-8 and=20 8859-2. I propose to remove this patch (and don't do &#nnnn; et all). --- PM |
|
From: Ethan M. <merritt@u.washington.edu> - 2006-10-17 00:30:25
|
On Monday 16 October 2006 02:49 pm, Ethan Merritt wrote:
> Petr Mikulik wrote>
> >
> > Without Hans-Bernhard's patch, SVG output is correct for both UTF-8
> > and 8859-2. I propose to remove this patch (and don't do &#nnnn; et
> > all).
>
> That works for me, after some struggles to change the default font
> paths. But I think we need to know more about what exactly was the
> problem that Hans-Bernhard's patch was originally trying to fix.
In order to facilitate testing of various combinations of character
escaping and svg viewers, I have posted two variants of the svg.trm
driver to SourceForge. (These are complete files, not patches).
Both add a new terminal option
set term svg {raw|escaped}
Variant 1 treats all encodings the same: if the "escaped" option
is selected, then non-ascii character are replaced one byte at a
time by the corresponding sequence &#xNN;
Variant 2 behaves exactly the same if an explicit encoding is set.
However, if no encoding is set then it is assumed to be UTF-8.
In this case non-ascii sequences are re-encoded by conversion from
UTF-8 to Unicode code points, rather than byte-by-byte.
Yes, we should probably have a specific S_ENC_UTF8 that can be
selected by "set encoding utf8". But we can deal with that as a
separate issue if it turns out that we need the UTF-8 specific
code.
--
Ethan A Merritt
Biomolecular Structure Center
University of Washington, Seattle WA
|
|
From: Petr M. <mi...@ph...> - 2006-10-17 07:22:01
|
> Both add a new terminal option
> set term svg {raw|escaped}
>
> Variant 1 treats all encodings the same: if the "escaped" option
> is selected, then non-ascii character are replaced one byte at a
> time by the corresponding sequence &#xNN;
I would expect "raw" writes raw 8bit chars or UTF-8 chars, without any
textual conversion to &#
Thus it is easy to "recode", "cstocs" etc to change the encoding and freely
edit the text.
Then there should be two kinds of the "escape" option, one yours and one for
the previous.
---
PM
|
|
From: Ethan A M. <merritt@u.washington.edu> - 2006-10-17 15:18:58
|
On Tuesday 17 October 2006 12:21 am, Petr Mikulik wrote:
> > Both add a new terminal option
> > set term svg {raw|escaped}
> >
> > Variant 1 treats all encodings the same: if the "escaped" option
> > is selected, then non-ascii character are replaced one byte at a
> > time by the corresponding sequence &#xNN;
>
> I would expect "raw" writes raw 8bit chars or UTF-8 chars, without any
> textual conversion to &#
Correct. That is what it does.
> Then there should be two kinds of the "escape" option, one yours and one for
> the previous.
Huh? I don't understand what you mean here.
The "escape" option leaves in place what it was doing previously.
Except that the second version knows enough to escape utf-8
sequences correctly, which the previous version didn't.
--
Ethan A Merritt
Biomolecular Structure Center
University of Washington, Seattle 98195-7742
|
|
From: Petr M. <mi...@ph...> - 2006-10-17 19:15:18
|
>> I would expect "raw" writes raw 8bit chars or UTF-8 chars, without any >> textual conversion to &# > > Correct. That is what it does. OK I've tried the two patches at [ 1578600 ] Two variants of the SVG driver and patch nb 2 works always, while nb 1 failed (utf-8, svg enha, Czech text). I propose to commit nb 2. --- PM |
|
From: Ethan M. <merritt@u.washington.edu> - 2006-10-17 21:50:34
|
On Tuesday 17 October 2006 01:43 pm, Hans-Bernhard Br=C3=B6ker wrote: > Ethan Merritt wrote: > > I have not been able to find any statement about this > > in the svg standards pages on www.w3.org. > > That's probably because this is not a property of SVG, but one of XML > in general. See http://www.w3.org/TR/REC-xml/#sec-references > ISO/IEC 10646 is what most people got to know as "Unicode". UTF-8 is > one way of encoding such characters. I understand. The specific question remains as to whether the XML tag "encoding=3D'foo'" at the head of a document is mandated to apply or not apply to octal/hex constants. We observe that it doesn't, but do not know whether this is expected behaviour or a faulty implementation in the browser plugins. > The original problem is that '&' and '<' characters were passed > unmodified into the output stream, where they can create no end of > problems. There was code to encode them as "entities" ('&' and > '<' respectively), but this was only applied for enhanced SVG. OK. Then I will commit the simpler of the 2 test versions, with no special handling of UTF-8. It maintains the & and < treatment you describe for both normal and enhanced text. =20 We may need a provision to allow people to pass a Unicode sequence as a hex code rather than as raw UTF-8 bytes: set title "This is a Unicode right arrow: ⇒ " In this case you don't want that & character replaced by & I predict many queries in the future about setting up UTF-8 fonts correctly, but font queries seem inevitable in any case. =2D-=20 Ethan A Merritt Biomolecular Structure Center University of Washington, Seattle WA |
|
From: <br...@ph...> - 2006-10-17 22:14:21
|
Ethan Merritt wrote: > On Tuesday 17 October 2006 01:43 pm, Hans-Bernhard Br=C3=B6ker wrot= e: >> Ethan Merritt wrote: >>> I have not been able to find any statement about this >>> in the svg standards pages on www.w3.org. >> That's probably because this is not a property of SVG, but one of = XML >> in general. See http://www.w3.org/TR/REC-xml/#sec-references >> ISO/IEC 10646 is what most people got to know as "Unicode". UTF-8= is >> one way of encoding such characters. >=20 > I understand. The specific question remains as to whether the XML t= ag > "encoding=3D'foo'" at the head of a document is mandated to apply > or not apply to octal/hex constants. =20 What you call octal/hex constants, XML calls a "character reference".= =20 And the paragraph of the XML spec the URL above points says quite= =20 straightforwardly that such references specify a Unicode code point. > We may need a provision to allow people to pass a Unicode sequence > as a hex code rather than as raw UTF-8 bytes: > =09set title "This is a Unicode right arrow: ⇒ " > In this case you don't want that & character replaced by & It boils down to this question: do we treat SVG like TeX, or like any= =20 other dumb format? In the former case, we should document that & is = an=20 escape character in text strings that go to the SVG driver. It may e= ven=20 be necessary to offer quoting & characters (so the above would become= =20 "\⇒" or '⇒'). |
|
From: Ethan M. <merritt@u.washington.edu> - 2006-10-17 23:22:50
|
On Tuesday 17 October 2006 03:13 pm, Hans-Bernhard Br=C3=B6ker wrote: > >> See http://www.w3.org/TR/REC-xml/#sec-references > > > > I understand. The specific question remains as to whether the XML > > tag "encoding=3D'foo'" at the head of a document is mandated to apply > > or not apply to octal/hex constants. > > What you call octal/hex constants, XML calls a "character reference". > And the paragraph of the XML spec the URL above points says quite > straightforwardly that such references specify a Unicode code point. Thanks. You are better at reading that convoluted text than I am. I had thought it gave an example of redefining all these codes based on the encoding, but on closer inspection I now see that the example pertains specifically to "entity references" in distinction to "character references". > It boils down to this question: do we treat SVG like TeX, or like any > other dumb format? In the former case, we should document that & is > an escape character in text strings that go to the SVG driver. It > may even be necessary to offer quoting & characters (so the above > would become "\⇒" or '⇒'). Indeed. It is further complicated by the fact that the & character is used in gnuplot's enhanced text mark-up. That isn't an issue for TeX, but it is for SVG. =2D-=20 Ethan A Merritt Biomolecular Structure Center University of Washington, Seattle WA |
