gnuplot-beta

cairo and character encoding issues

[Allin C. <cot...@wf...>]

I believe there are a couple of things not quite right with the 
encoding to UTF-8 in the context of the cairo-based terminals.

* In gp_cairo_convert() (gp_cairo.c) there's an *unconditional* 
conversion of an input string, from the charset returned by 
gp_cairo_get_encoding() to UTF-8.  I'd suggest that conversion 
should be conditional on failure of g_utf8_validate() on the 
string in question.

Rationale: the supposed source charset may not be correct (see 
below).  If the string validates as UTF-8 it should be passed as 
is.  You get bamboozling error messages if g_convert() is called 
on a string that's already in UTF-8, but is misrepresented as 
being in some other encoding.

* The function gp_cairo_get_encoding() doesn't work properly 
unless the right encoding has been set explicitly by the user. The 
fallback in that function is g_get_charset(), but this won't do 
anything useful, in the sense of identifying the character set 
actually in use on the given system, unless there has been a prior 
call

setlocale (LC_ALL, "");

-- 
Allin Cottrell
Department of Economics
Wake Forest University, NC

Re: cairo and character encoding issues

[Ethan A M. <merritt@u.washington.edu>]

On Friday 23 November 2007 17:27, Allin Cottrell wrote:
> I believe there are a couple of things not quite right with the 
> encoding to UTF-8 in the context of the cairo-based terminals.
> I'd suggest that conversion  should be conditional on failure of
> g_utf8_validate() on the string in question.

Could be.


> * The function gp_cairo_get_encoding() doesn't work properly 
> unless the right encoding has been set explicitly by the user. The 
> fallback in that function is g_get_charset(), but this won't do 
> anything useful, in the sense of identifying the character set 
> actually in use on the given system, unless there has been a prior 
> call 
>      setlocale (LC_ALL, "");

Isn't there one already?
Or at any rate a call to setlocale(LC_CTYPE, "")?  
When I enter gnuplot it correctly knows about the locale:

lascaux [1235] ./gnuplot

        G N U P L O T
        Version 4.3 patchlevel CVS-21Nov2007
        last modified Wed Nov 21 19:43:57 PST 2007
        System: Linux 2.6.12-24mdksmp

        Copyright (C) 1986 - 1993, 1998, 2004, 2007
        Thomas Williams, Colin Kelley and many others

        Type `help` to access the on-line reference manual.
        The gnuplot FAQ is available from
                http://www.gnuplot.info/faq/

        Send comments and help requests to  <gnu...@li...>
        Send bug reports and suggestions to <gnu...@li...>


Terminal type set to 'wxt'
gnuplot> show locale

        LC_CTYPE is en_US.UTF-8
        LC_TIME is C

But we have at least one bug report from somewhere who is having 
trouble getting gnuplot to pull LC_CTYPE from the environment, so I'm
not really sure what is going on.


-- 
Ethan A Merritt

Re: cairo and character encoding issues

[Mojca M. <moj...@gm...>]

On Nov 24, 2007 3:12 AM, Ethan A Merritt wrote:
> > * The function gp_cairo_get_encoding() doesn't work properly
> > unless the right encoding has been set explicitly by the user. The
> > fallback in that function is g_get_charset(), but this won't do
> > anything useful, in the sense of identifying the character set
> > actually in use on the given system, unless there has been a prior
> > call
> >      setlocale (LC_ALL, "");
>
> Isn't there one already?
> Or at any rate a call to setlocale(LC_CTYPE, "")?
> When I enter gnuplot it correctly knows about the locale:
>
> lascaux [1235] ./gnuplot
>
>         G N U P L O T
>         Version 4.3 patchlevel CVS-21Nov2007
>         last modified Wed Nov 21 19:43:57 PST 2007
>         System: Linux 2.6.12-24mdksmp
>
>         Copyright (C) 1986 - 1993, 1998, 2004, 2007
>         Thomas Williams, Colin Kelley and many others
>
>         Type `help` to access the on-line reference manual.
>         The gnuplot FAQ is available from
>                 http://www.gnuplot.info/faq/
>
>         Send comments and help requests to  <gnu...@li...>
>         Send bug reports and suggestions to <gnu...@li...>
>
>
> Terminal type set to 'wxt'
> gnuplot> show locale
>
>         LC_CTYPE is en_US.UTF-8
>         LC_TIME is C
>
> But we have at least one bug report from somewhere who is having
> trouble getting gnuplot to pull LC_CTYPE from the environment, so I'm
> not really sure what is going on.

Here I have:

> locale
LANG=
LC_COLLATE="sl_SI.UTF-8"
LC_CTYPE="sl_SI.UTF-8"
LC_MESSAGES="sl_SI.UTF-8"
LC_MONETARY="sl_SI.UTF-8"
LC_NUMERIC="sl_SI.UTF-8"
LC_TIME="sl_SI.UTF-8"
LC_ALL="sl_SI.UTF-8"

(I set those in .bash_profile)

> gnuplot

        G N U P L O T
        Version 4.3 patchlevel 0
        last modified February 2007
        System: Darwin 8.11.1

        Copyright (C) 1986 - 1993, 1998, 2004, 2007
        Thomas Williams, Colin Kelley and many others

        Type `help` to access the on-line reference manual.
        The gnuplot FAQ is available from
                http://www.gnuplot.info/faq/

        Send comments and help requests to  <gnu...@li...>
        Send bug reports and suggestions to <gnu...@li...>


Terminal type set to 'aqua'
gnuplot> show locale

        LC_CTYPE is C
        LC_TIME is C
        LC_NUMERIC is C

gnuplot>

Mojca

Re: cairo and character encoding issues

[Allin C. <cot...@wf...>]

On Fri, 23 Nov 2007, Ethan A Merritt wrote:

> > * The function gp_cairo_get_encoding() doesn't work properly 
> > unless the right encoding has been set explicitly by the user. The 
> > fallback in that function is g_get_charset(), but this won't do 
> > anything useful, in the sense of identifying the character set 
> > actually in use on the given system, unless there has been a prior 
> > call 
> >      setlocale (LC_ALL, "");
> 
> Isn't there one already?
> Or at any rate a call to setlocale(LC_CTYPE, "")?  
> When I enter gnuplot it correctly knows about the locale:
> 
> lascaux [1235] ./gnuplot...

> gnuplot> show locale
> 
>         LC_CTYPE is en_US.UTF-8
>         LC_TIME is C

That's what I see here too, but g_get_charset(), called in 
gp_cairo.c, is getting ANSI_X3.4-1968 (ascii).  The GLib reference 
says,

"A number of interfaces in GLib depend on the current locale in 
which an application is running. Therefore, most GLib-using 
applications should call setlocale (LC_ALL, "") to set up the 
current locale."

I tried sticking that exact invocation into gp_cairo.c, prior to 
the call to g_get_charset, and the latter then found UTF-8 OK.

Allin Cottrell

Re: cairo and character encoding issues

[Allin C. <cot...@wf...>]

On Fri, 23 Nov 2007, Ethan A Merritt wrote:
> 
> > lascaux [1235] ./gnuplot...
> 
> > gnuplot> show locale
> > 
> >         LC_CTYPE is en_US.UTF-8
> >         LC_TIME is C

This is puzzling.

1) If LC_CTYPE is en_US.UTF-8 then g_get_charset should certainly 
return UTF-8, which it is not doing in the cairo driver.

2) How did LC_CTYPE _get_ to be equal to en_US.UTF-8 at this point 
in execution, when the only command that has been issued is "show 
locale"? 

I can't see any invocation of setlocale(LC_CTYPE, "") in src/*.c 
other than in the default response to the command "set locale".

3) Anyway, by the time you plot a graph using cairo the LC_CTYPE 
setting has reverted to "C".  I've verified that by sticking a 
print before the g_get_charset() invocation:

fprintf(stderr, "gp_cairo_get_encoding: LC_CTYPE is %s\n", 
        setlocale(LC_CTYPE, NULL));

-> gp_cairo_get_encoding: LC_CTYPE is C

Allin Cottrell

Re: cairo and character encoding issues (once more)

[Allin C. <cot...@wf...>]

On Fri, 23 Nov 2007, Allin Cottrell wrote:

> On Fri, 23 Nov 2007, Ethan A Merritt wrote:
> > 
> > > lascaux [1235] ./gnuplot...
> > 
> > > gnuplot> show locale
> > > 
> > >         LC_CTYPE is en_US.UTF-8
> > >         LC_TIME is C
> 
> This is puzzling...

Still puzzling, but here's a hint: LC_CTYPE is registered 
correctly, as above, only in interactive mode.  If you put "show 
locale" into a command file and run gnuplot on it, you'll see

 LC_CTYPE is C

in an en_US.UTF-8 environment.

Allin Cottrell

Re: cairo and character encoding issues

[Ethan A M. <merritt@u.washington.edu>]

On Friday 23 November 2007 19:37, Allin Cottrell wrote:
> On Fri, 23 Nov 2007, Ethan A Merritt wrote:
> > 
> > > lascaux [1235] ./gnuplot...
> > 
> > > gnuplot> show locale
> > > 
> > >         LC_CTYPE is en_US.UTF-8
> > >         LC_TIME is C
> 
> This is puzzling.
> 
> 1) If LC_CTYPE is en_US.UTF-8 then g_get_charset should certainly 
> return UTF-8, which it is not doing in the cairo driver.
> 
> 2) How did LC_CTYPE _get_ to be equal to en_US.UTF-8 at this point 
> in execution, when the only command that has been issued is "show 
> locale"? 

Puzzling indeed.  With the help of many trace statements, I find that
the locale is loaded during the first call to readline_ipc() in rlgets(),
which is itself called from gp_get_string(). The body of this routine is listed below:

  /* get a line from stdin, and display a prompt if interactive */
  static char*
  gp_get_string(char * buffer, size_t len, const char * prompt)
  {
  # if defined(READLINE) || defined(HAVE_LIBREADLINE)
    if (interactive)
        return rlgets(buffer, len, prompt);
#              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    else
        return fgets_ipc(buffer, len);
  # else /* !(READLINE || HAVE_LIBREADLINE) */
    if (interactive)
        PUT_STRING(prompt);

    return GET_STRING(buffer, len);
  # endif /* !(READLINE || HAVE_LIBREADLINE) */
  }

So we see that the use of this input path depends on the setting of "interactive"
and the presence/absence of READLINE support.

> 3) Anyway, by the time you plot a graph using cairo the LC_CTYPE 
> setting has reverted to "C".  I've verified that by sticking a 
> print before the g_get_charset() invocation:

I can't reproduce that. Once the locale gets set, it sticks.  As you would expect.


I am thinking that regardless of any possible bugs in the cairo processing,
we should be consistent and always call setlocale(LC_CTYPE,"").

Can anyone see a problem if we load the locale from the environment immediately
on program entry?


-- 
Ethan A Merritt

Re: cairo and character encoding issues

[Allin C. <cot...@wf...>]

On Sat, 24 Nov 2007, Ethan A Merritt wrote:

> > 2) How did LC_CTYPE _get_ to be equal to en_US.UTF-8 at this point 
> > in execution, when the only command that has been issued is "show 
> > locale"? 
> 
> Puzzling indeed.  With the help of many trace statements, I find that
> the locale is loaded during the first call to readline_ipc() in rlgets(),
> which is itself called from gp_get_string()...

Nice detective work!

> > 3) Anyway, by the time you plot a graph using cairo the LC_CTYPE 
> > setting has reverted to "C".  I've verified that by sticking a 
> > print before the g_get_charset() invocation:
> 
> I can't reproduce that. Once the locale gets set, it sticks.  
> As you would expect.

Yup, I was wrong.  The active factor was interactive vs 
non-interactive, not anything that happened in between "show 
locale" and gp_cairo calling g_get_charset.

> I am thinking that regardless of any possible bugs in the cairo 
> processing, we should be consistent and always call 
> setlocale(LC_CTYPE,"").

That makes sense to me: maybe in init_locale?

Allin Cottrell

Re: cairo and character encoding issues

[Ethan A M. <merritt@u.washington.edu>]

On Friday 23 November 2007 18:21, Allin Cottrell wrote:
> > >      setlocale (LC_ALL, "");
> > 
> > Isn't there one already?
> > Or at any rate a call to setlocale(LC_CTYPE, "")?  
> > When I enter gnuplot it correctly knows about the locale:
> > 
> > lascaux [1235] ./gnuplot...
> 
> > gnuplot> show locale
> > 
> >         LC_CTYPE is en_US.UTF-8
> >         LC_TIME is C
> 
> That's what I see here too, but g_get_charset(), called in 
> gp_cairo.c, is getting ANSI_X3.4-1968 (ascii).  

Something is broken, it seems. But I really don't know what.

> The GLib reference says,
> "A number of interfaces in GLib depend on the current locale in 
> which an application is running. Therefore, most GLib-using 
> applications should call setlocale (LC_ALL, "")

We most definitely do not want LC_ALL, as that will mess with
the time/date and numerical formats also.  We have separate
commands to load LC_TIME and LC_NUMERIC. 

> I tried sticking that exact invocation into gp_cairo.c, prior to 
> the call to g_get_charset, and the latter then found UTF-8 OK.

Possibly that pins down the brokeness.
Setting LC_CTYPE should do exactly what we want - set the character encoding.
But LC_ALL may be [incorrectly] required by some library instead of the more
specific LC_CTYPE?


-- 
Ethan A Merritt

Re: cairo and character encoding issues

[<tim...@lp...>]

Allin Cottrell a écrit :
> On Sat, 24 Nov 2007, Ethan A Merritt wrote:
>
>   
>>> 2) How did LC_CTYPE _get_ to be equal to en_US.UTF-8 at this point 
>>> in execution, when the only command that has been issued is "show 
>>> locale"? 
>>>       
>> Puzzling indeed.  With the help of many trace statements, I find that
>> the locale is loaded during the first call to readline_ipc() in rlgets(),
>> which is itself called from gp_get_string()...
>>     
>
> Nice detective work!
>
>   
Please note that gtk+ (used by wxGTK, used by wxt) is another offender 
here which calls setlocale(LC_ALL,"") or something like that. For 
example, on my system where wxt is built:

> tipote@tipote-laptop:~$ echo "show locale; plot x; show locale" | gnuplot
>
>         LC_CTYPE is C
>         LC_TIME is C
>
>
>         LC_CTYPE is fr_FR.UTF-8
>         LC_TIME is fr_FR.UTF-8

When wxt is initialized, gtk+ loads the locale settings. I have already 
had to work around it for LC_NUMERIC in wxt, as you can see from the 
following lines of code taken from lines 1445-1449 in 
src/wxterminal/wxt_gui.cpp:

> #ifdef HAVE_LOCALE_H
>         /* when wxGTK is initialised, GTK+ also sets the locale of the 
> program itself;
>          * we must revert it */
>         setlocale(LC_NUMERIC, "C");
> #endif /*have_locale_h*/


At least, we know understand what happens on Mojca's box too:
>
> Here I have:
>
>   
>> > locale
>>     
> LANG=
> LC_COLLATE="sl_SI.UTF-8"
> LC_CTYPE="sl_SI.UTF-8"
> LC_MESSAGES="sl_SI.UTF-8"
> LC_MONETARY="sl_SI.UTF-8"
> LC_NUMERIC="sl_SI.UTF-8"
> LC_TIME="sl_SI.UTF-8"
> LC_ALL="sl_SI.UTF-8"
> Terminal type set to 'aqua'
> gnuplot> show locale
>
>         LC_CTYPE is C
>         LC_TIME is C
>         LC_NUMERIC is C
>
> gnuplot>
>
> Mojca

Mojca runs MacOS, so she surely doesn't have wxt, aqua is indeed the 
terminal initialized on startup, and she most likely doesn't have 
readline. That's why her LC_* are not set from her locale settings.


>>> 3) Anyway, by the time you plot a graph using cairo the LC_CTYPE 
>>> setting has reverted to "C".  I've verified that by sticking a 
>>> print before the g_get_charset() invocation:
>>>       
>> I can't reproduce that. Once the locale gets set, it sticks.  
>> As you would expect.
>>     
>
> Yup, I was wrong.  The active factor was interactive vs 
> non-interactive, not anything that happened in between "show 
> locale" and gp_cairo calling g_get_charset.
>   
Ah, I was puzzled too by that one.

>   
>> I am thinking that regardless of any possible bugs in the cairo 
>> processing, we should be consistent and always call 
>> setlocale(LC_CTYPE,"").
>>     
>
> That makes sense to me: maybe in init_locale?
>   
I agree with that. Why not LC_ALL by the way ?

Timothée

Re: cairo and character encoding issues

[Ethan A M. <merritt@u.washington.edu>]

On Saturday 24 November 2007 11:05, you wrote:
> >> I am thinking that regardless of any possible bugs in the cairo 
> >> processing, we should be consistent and always call 
> >> setlocale(LC_CTYPE,"").
> >
> > That makes sense to me: maybe in init_locale?
> >   
> I agree with that. Why not LC_ALL by the way ?

Because it sets LC_NUMERIC, which breaks all sorts of things
in locales where the decimal separator is a comma.

In 4.2 we tried to work around this by explicitly setting
LC_NUMERIC to "C" when necessary, but the list of necessary places
was long yet incomplete.  This led to more than one bug report.
(e.g. #1740017 #1707916 #1692541 #1692550).

In CVS I have inverted that.  The program runs with LC_NUMERIC set
to "C" for all purposes, except that it applies the locale when creating
a text string for display as part of the plot.

-- 
Ethan A Merritt

Re: cairo and character encoding issues

[Ethan A M. <merritt@u.washington.edu>]

On Saturday 24 November 2007 11:05, Timoth=E9e Lecomte wrote:
> Allin Cottrell a =E9crit :
> >> I am thinking that regardless of any possible bugs in the cairo=20
> >> processing, we should be consistent and always call=20
> >> setlocale(LC_CTYPE,"").
> > That makes sense to me: maybe in init_locale?
> I agree with that. Why not LC_ALL by the way ?

OK. I have added setlocale(LC_CTYPE,"") to init_locale().
I also added Allin's pre-validation of strings to see if they conform
to UTF-8. The test doesn't really prove the string is UTF-8, but I
can't think of any non-contrived cases where this would be a problem.

Neither of these patches are strictly speaking necessary. The same=20
result could be accomplished by putting "set encoding locale" in
~/.gnuplot.  But having a more consistent initial state for the program
will I hope reduce the number of confusing bug reports.

This should also resolve bug=20
#1834525 'utf8 encoding not working in pdfcairo under C locale'


=2D-=20
Ethan A Merritt

Re: cairo and character encoding issues

[Allin C. <cot...@wf...>]

On Sat, 24 Nov 2007, Ethan A Merritt wrote:

> On Saturday 24 November 2007 11:05, Timothée Lecomte wrote:
> > Allin Cottrell a écrit :
> > >> I am thinking that regardless of any possible bugs in the cairo 
> > >> processing, we should be consistent and always call 
> > >> setlocale(LC_CTYPE,"").
> > > That makes sense to me: maybe in init_locale?
> > I agree with that. Why not LC_ALL by the way ?
> 
> OK. I have added setlocale(LC_CTYPE,"") to init_locale().
> I also added Allin's pre-validation of strings to see if they conform
> to UTF-8. The test doesn't really prove the string is UTF-8, but I
> can't think of any non-contrived cases where this would be a problem.

Thank you for doing that.

> Neither of these patches are strictly speaking necessary. The 
> same result could be accomplished by putting "set encoding 
> locale" in ~/.gnuplot.

Not quite the same.  One of the attractions of a target that 
accepts UTF-8 is that files intended for that target and encoded 
in UTF-8 can be shared across platforms, regardless of the local 
encoding.  

For instance if I'm on an ISO-88959-1 platform and I download a 
UTF-8-encoded gnuplot source file to generate a plot using one of 
the cairo-based terminals, it should now work fine.  Before, it 
would fail with a confusing error message (probably a different 
confusing error message depending on whether or not I had "set 
encoding locale" in my ~/.gnuplot).

Allin Cottrell

pgnuplot for octave

[Tatsuro M. <tma...@ya...>]

Hello

The current octave for windows distribution, the Michael's modified pgnuplot was used.
That is indpendent of wgnuplot and which has both abilities of pipe receiving and the plotting system.
  That is because the current octave all plotting data to the gnuplot via pipe.   The combination of
the pgnuplot and the wgnuplot cannot be used.  The incomming data speed is fairly larger than the
wgnuplot interpret the command from the wgnuplot command terminal.

However we sometimes want use current version gnuplot like cvs one for exaple distributed by prof.
Kakuto.

Url of mirror site:

http://www.ring.gr.jp/pub/text/TeX/ptex-win32/utils/
gnuplot-43pl0w32.zip

 
The patch attached that used the test for the sucesive plot from octave.

I think that the current pgnuplot is not suitable for this purpose and not effective.
The command will be interpreted by line to line.  I set wating factor by each line.
I used the Sleep function, however even sleep(1), the wait time is enough but I this it will be enough
to be shortened.

I would like to hear the gnuplot persons' opinions.

Sincerely yours,

Tatsuro MATSUOKA





  

--------------------------------------
New Design Yahoo! JAPAN  2008/01/01
http://pr.mail.yahoo.co.jp/newdesign/

pgnuplot for octave

[Tatsuro M. <tma...@ya...>]

Hello

The previous mail included a lot of type mistakes.
Please ignore it.

The current octave for windows distribution, the Michael's modified pgnuplot was used.
That is independent of wgnuplot and which has both abilities of pipe receiving and the plotting
system.
  That is because the current octave all plotting data to the gnuplot via pipe.   The combination of
the pgnuplot and the wgnuplot cannot be used.  The incoming data speed is fairly larger than the
wgnuplot interpret the command from the wgnuplot command terminal.

However we sometimes want to use the latest version gnuplot like cvs one for example distributed by
prof.Kakuto.

Url of mirror site:

http://www.ring.gr.jp/pub/text/TeX/ptex-win32/utils/
gnuplot-43pl0w32.zip

 
The patch attached that used in the test for the successive plot from the octave.

I think that the current pgnuplot is not suitable for this purpose and not effective.
The command will be better to be interpreted by line to line.  
I have set waiting factor by each line.
I used the Sleep function, however even sleep(1), the wait time is enough but it can to be shortened.

I would like to hear the gnuplot persons' opinions.

Sincerely yours,

Tatsuro MATSUOKA


--------------------------------------
New Design Yahoo! JAPAN  2008/01/01
http://pr.mail.yahoo.co.jp/newdesign/