gnuplot-beta — Developers and beta testers of new gnuplot code

Re: Commands into datablocks

[Ethan A M. <me...@uw...>]

On Monday, 9 November 2020 15:24:19 PST Tait wrote:
> 
> Datablocks seem like a great way to avoid running slow commands 
> multiple times, as in:
>   plot $data using ..., '' using ..., '' using ...
> 
> instead of
>   plot '<cmd' using ..., '' using ..., '' using ...
> 
> But $data << '<cmd' obviously doesn't work, and the workaround 
> using load '<cmd' is hard to discover and requires modifying 
> cmd. Would it be straighttforward to add a way to populate a 
> datablock from a command the way '<' works for plot and load?

Two existing mechanisms come to mind.
Neither is a perfect match to what you ask but might serve
well enough either as-is or with a straightforward extension.

The first is the "volatile" keyword, which tells the program
not to reread the data file if the previously read values are
still available internally.  As of now it is only relevant to
"replot", but it might conceivably be extended to cover the
use of '' in a plot command.

The second works already, subject to some issues like loss of
precision and appropriate choice of string formats and/or 
field separators:

    set table $DATA
    plot "< some complicated command" using 1:2:3:4:5:(strcol(6)) with table
    unset table

    plot for [i=1:4] $DATA using 1:i with points, \
                     $DATA using 1:5:6 with labels


Does that work for your case?
If not, can you suggest ways in which the "with table" might be extended?

	cheers,

		Ethan

Commands into datablocks

[Tait <gnu...@t4...>]

Datablocks seem like a great way to avoid running slow commands 
multiple times, as in:
  plot $data using ..., '' using ..., '' using ...

instead of
  plot '<cmd' using ..., '' using ..., '' using ...

But $data << '<cmd' obviously doesn't work, and the workaround 
using load '<cmd' is hard to discover and requires modifying 
cmd. Would it be straighttforward to add a way to populate a 
datablock from a command the way '<' works for plot and load?

(Inspired by 
https://stackoverflow.com/questions/54600172/gnuplot-load-datafile-11-into-datablock 
and several similar questions. Using system() doesn't work the 
way you'd think either, as 
https://stackoverflow.com/a/60366832/11777995 mentions.)

Re: qt terminal behavior if no x11 is available

[Daniel J S. <dan...@ie...>]

On 11/8/20 4:27 AM, Dima Kogan wrote:
> Hi. Is anybody here familiar with the qt terminal init sequence? I'm
> observing that if you try to plot anything without DISPLAY defined, the
> qt terminal throws an error and hangs. It does NOT give us the prompt
> back. This confuses tools that use gnuplot to make plots.
> 
> There was a commit that changed the way this works:
> 
>    https://sourceforge.net/p/gnuplot/gnuplot-main/ci/c415285c230
> 
> Reverting that makes gnuplot give up after a few seconds. The x11 and
> wxt terminals figure it out immediately, but even a wait of several
> seconds is better than a forever hang we currently get. Is there a
> better way to handle errors in qt?
> 
> Thanks!
> 

https://doc.qt.io/qt-5/qlocalsocket.html#waitForConnected

https://doc.qt.io/qt-5/qlocalsocket.html#error

https://doc.qt.io/qt-5/qlocalsocket.html#LocalSocketError-enum

That polling loop approach isn't really what should be done.  By doing 
so it kind of obviates the error test.  In the least, what this polling 
loop should be doing checking the error() after the failure to connect 
and going to sleep to determine if the error was a 
QLocalSocket::SocketTimeoutError.  If it was some other type of error, 
don't continue waiting.

     QLocalSocket::LocalSocketError err = qt->socket.error();
     if (err != QLocalSocket::SocketTimeoutError)
         break;

and of course don't wait forever.

As for the comment:

	// The QLocalSocket::waitForConnected does not respect the time out 
argument when
	// the gnuplot_qt application is not yet started or has not yet 
self-initialized.
	// To wait for it, we need to implement the timeout ourselves

one just needs to figure out how to put this in the context of an event 
loop, i.e., start an event loop.  That could be done with threading and 
the exec() routine.

Dan

qt terminal behavior if no x11 is available

[Dima K. <gn...@di...>]

Hi. Is anybody here familiar with the qt terminal init sequence? I'm
observing that if you try to plot anything without DISPLAY defined, the
qt terminal throws an error and hangs. It does NOT give us the prompt
back. This confuses tools that use gnuplot to make plots.

There was a commit that changed the way this works:

  https://sourceforge.net/p/gnuplot/gnuplot-main/ci/c415285c230

Reverting that makes gnuplot give up after a few seconds. The x11 and
wxt terminals figure it out immediately, but even a wait of several
seconds is better than a forever hang we currently get. Is there a
better way to handle errors in qt?

Thanks!

Aw: Re: AW: filenames on MS Windows

[Bastian M. <bma...@we...>]

> Gesendet: Montag, 02. November 2020 um 21:37 Uhr
> Von: "Allin Cottrell" <cot...@wf...>
> An: "Ethan A Merritt" <me...@uw...>
> Cc: "Bastian Märkisch" <bma...@we...>, "gnuplot-beta" <gnu...@li...>
> Betreff: Re: AW: filenames on MS Windows
>
> On Mon, 2 Nov 2020, Ethan A Merritt wrote:
> 
> > On Monday, 2 November 2020 01:48:42 PST Bastian Märkisch wrote:
> >> Right now, gnuplot is able to "load" file names with Unicode encoded names, i.e. the sequence
> >> 	set encoding utf8
> >> 	load 'абвгдежзийклмнопрстуфхцчшщъыьэюяѐёђѓєѕіїјљњћќѝўџ.plt'
> >> will work just fine.
> >
> > But only if the encoding is set to utf8, right?
> > Which is a bit counter-intuitive since on Windows the
> > filename is not actually utf8.
> 
> Counter-intuitive maybe, but it's very convenient. "set encoding 
> utf8" announces that filenames in the gnuplot script will be UTF-8 
> encoded (and the script will be readable cross-platform), but thanks 
> to Bastian gnuplot knows they need to be recoded in the background 
> to UTF-16 for reading from disk on Windows.

Windows mostly uses UTF16 to support Unicode. gnuplot uses char-based
(byte) encodings only and - as many other programs from the *nix world
- uses UTF-8 for Unicode. Internally, gnuplot will use whatever 
encoding it is told to use by the user via "set encoding". The 
translation from and to UTF16 for input, output, file names, pipes,
clipboard interaction etc. is transparent to the user. This translation
is not done for command line arguments yet as pointed out by Allin.
The scheme itself is common practise to "port" applications and was
introduced in 2016.
(Note that as of now, file _content_ cannot be read or written in 
UTF16 encoding.)

  Bastian

Re: AW: filenames on MS Windows

[Allin C. <cot...@wf...>]

On Mon, 2 Nov 2020, Allin Cottrell wrote:

> On Mon, 2 Nov 2020, Bastian Märkisch wrote:
[...]
>
>> That really poses the question if we should change the default 
>> internal encoding from "ANSI" (whatever that may be depends on 
>> the Windows locale) to UTF-8.  I agree with that (and in fact my 
>> personal gnuplot.ini includes "set encoding utf8" since a long 
>> time).
>> 
>> But how do we make this backward compatible since that will 
>> inevitably break load commands in old "ANSI" encoded user 
>> scripts? (as does your current patch btw)
>
> Are you sure about that breakage, Bastian? [...]

I'm not sure I've understood the circumstances under which you 
reckon my patch would break load commands, but I just tried an 
experiment which I think is relevant. This is on Windows 10 with 
system codepage 1252, using my patched wgnuplot.exe.

I placed a gnuplot script in a directory named with Cyrillic 
characters (so not representable in CP1252), reading as follows:

# loader script
set encoding cp1252
load '<CP1252 filename>'

where <CP1252 filename> is the full path to a second script, in
a directory with a non-ASCII name representable in CP1252. (It 
contains an o-circumflex.) The second script reads thus:

# plot script
set term pngcairo
set output '<CP1252 output>'
plot sin(x)

where <CP1252 output> is again a full path including the 
non-ASCII but CP-friendly directory name.

I then called wgnuplot.exe on the "loader" script, passing its 
filename as UTF-16, and it all went fine: the loader was read OK, 
from its Cyrillic location; the load command worked; and the PNG was 
written successfully.

--
Allin Cottrell
Department of Economics
Wake Forest University

Re: AW: filenames on MS Windows

[Allin C. <cot...@wf...>]

On Mon, 2 Nov 2020, Ethan A Merritt wrote:

> On Monday, 2 November 2020 01:48:42 PST Bastian Märkisch wrote:
>> Right now, gnuplot is able to "load" file names with Unicode encoded names, i.e. the sequence
>> 	set encoding utf8
>> 	load 'абвгдежзийклмнопрстуфхцчшщъыьэюяѐёђѓєѕіїјљњћќѝўџ.plt'
>> will work just fine.
>
> But only if the encoding is set to utf8, right?
> Which is a bit counter-intuitive since on Windows the
> filename is not actually utf8.

Counter-intuitive maybe, but it's very convenient. "set encoding 
utf8" announces that filenames in the gnuplot script will be UTF-8 
encoded (and the script will be readable cross-platform), but thanks 
to Bastian gnuplot knows they need to be recoded in the background 
to UTF-16 for reading from disk on Windows.

-- 
Allin Cottrell
Department of Economics
Wake Forest University

Re: AW: filenames on MS Windows

[Ethan A M. <me...@uw...>]

On Monday, 2 November 2020 01:48:42 PST Bastian Märkisch wrote:
> Right now, gnuplot is able to "load" file names with Unicode encoded names, i.e. the sequence
> 	set encoding utf8
> 	load 'абвгдежзийклмнопрстуфхцчшщъыьэюяѐёђѓєѕіїјљњћќѝўџ.plt'
> will work just fine. 

But only if the encoding is set to utf8, right?
Which is a bit counter-intuitive since on Windows the
filename is not actually utf8.

I had not realized before this that the encoding could affect
how files are opened.  The documentation under "encoding" and/or
"utf8" could be expanded.  Maybe also "set print|out|table" or
add some general statement about filenames?

	Ethan 


> Only loading via the command line does not work and that should indeed be improved. (I don't agree to use glib for that purpose - but that is easy to change).
> 
>   Bastian
> 
> > -----Ursprüngliche Nachricht-----
> > Von: Allin Cottrell <cot...@wf...>
> > Gesendet: Sonntag, 1. November 2020 21:24
> > An: Ethan A Merritt <me...@uw...>
> > Cc: gnuplot-beta <gnu...@li...>
> > Betreff: Re: filenames on MS Windows
> > 
> > On Sun, 1 Nov 2020, Ethan A Merritt wrote:
> > 
> > > On Saturday, 31 October 2020 17:36:03 PST Allin Cottrell wrote:
> > >> On Sun, 25 Oct 2020, Allin Cottrell wrote:
> > >>
> > >>> I tried googling this but didn't find an answer -- sorry if I should
> > >>> have just tried harder! My question is: can gnuplot on Windows
> > >>> handle a unicode filename argument passed in UTF-16? As in
> > >>>
> > >>> path/to/wgnuplot.exe <UTF-16 input filename>
> > >>
> > >> OK, that question was under-researched, but now I've done my
> > >> homework. Sorry, this is a bit long but I hope I can arouse some
> > >> interest in the topic.
> > >
> > > I don't have any direct insight into this issue other than to note
> > > that the filesytem itself may be an issue.
> > 
> > In some contexts, no doubt. But if we set aside exotica such as surrogate pairs,
> > NTFS filenames are UTF-16 to a very good approximation. As such they are
> > easily converted to UTF-8 to permit handling with good old C char * APIs, and
> > easily converted back to
> > UTF-16 for _wfopen() if required.
> > 
> > > The following entry from the R developer blog is of interest
> > >
> > >
> > > https://developer.r-project.org/Blog/public/2020/05/02/utf-8-support-o
> > > n-windows/
> > >
> > > I gather from the discussion there that Windows-10 can be made to
> > > support UTF-8 as a native encoding, calling it "extended ASCII".
> > 
> > Interesting, yes, but at this point kinda science fiction. The practical issue at
> > present is whether gnuplot wants to support out-of-codepage UTF-16
> > filenames on the Windows command line. It's not terribly difficult, as I tried to
> > show.
> > 
> > Sorry if I'm being repetitive, but right now if a create, say, a Russian-language
> > filename on Windows and pass it as command-line argument to gnuplot,
> > gnuplot will not be able to open the file because its name cannot be
> > represented in my "system codepage". A program that reads the command line
> > as UTF-16, however, will have no problem opening the file.
> > 
> > Allin Cottrell
> 
> 


-- 
Ethan A Merritt
Biomolecular Structure Center,  K-428 Health Sciences Bldg
MS 357742,   University of Washington, Seattle 98195-7742

Re: AW: filenames on MS Windows

[Allin C. <cot...@wf...>]

On Mon, 2 Nov 2020, Bastian Märkisch wrote:

> Allin,
>
> Thank you for this nice contribution.  On Windows, we already replace
> fopen() (and many other functions), though.  In particular, win_fopen() in
> winmain.c already handles encodings, including UTF-8. There are also two
> functions AnsiText() and UnicodeText() to convert to/from UTF16 according to
> gnuplot's encoding.

Yes, I'm aware of that and it's a very nice feature.

> They use the simple-to-use Windows functions
> WideCharToMultiByte() MultiByteToWideChar(), so no need for glib.

Ah, OK.

> That really poses the question if we should change the default internal
> encoding from "ANSI" (whatever that may be depends on the Windows locale) to
> UTF-8.  I agree with that (and in fact my personal gnuplot.ini includes "set
> encoding utf8" since a long time).
>
> But how do we make this backward compatible since that will inevitably break
> load commands in old "ANSI" encoded user scripts? (as does your current
> patch btw)

Are you sure about that breakage, Bastian? I may be missing 
something, but here's my thinking: (1) I convert UTF-16 -> UTF-8 
only for filenames coming off the Windows command-line in winmain.c, 
and (2) in my modified loadpath_fopen() I first try fopen() on the 
filename as-is, converting UTF-8 -> UTF-16 and calling _wfopen() 
only if fopen() fails and the filename validates as UTF-8. So I 
don't _think_ my patch is going to touch filenames read via the load 
command in a gnuplot script.

Allin Cottrell

AW: filenames on MS Windows

[Bastian M. <bma...@we...>]

Allin,

Thank you for this nice contribution.  On Windows, we already replace
fopen() (and many other functions), though.  In particular, win_fopen() in
winmain.c already handles encodings, including UTF-8. There are also two
functions AnsiText() and UnicodeText() to convert to/from UTF16 according to
gnuplot's encoding. They use the simple-to-use Windows functions
WideCharToMultiByte() MultiByteToWideChar(), so no need for glib.

That really poses the question if we should change the default internal
encoding from "ANSI" (whatever that may be depends on the Windows locale) to
UTF-8.  I agree with that (and in fact my personal gnuplot.ini includes "set
encoding utf8" since a long time).

But how do we make this backward compatible since that will inevitably break
load commands in old "ANSI" encoded user scripts? (as does your current
patch btw) Possible solutions include a new command line option (like -u /
--utf8)  or  a wgnuplot.ini setting.

  Bastian


> -----Ursprüngliche Nachricht-----
> Von: Allin Cottrell <cot...@wf...>
> Gesendet: Sonntag, 1. November 2020 01:36
> An: gnuplot-beta <gnu...@li...>
> Betreff: Re: filenames on MS Windows
> 
> On Sun, 25 Oct 2020, Allin Cottrell wrote:
> 
> > I tried googling this but didn't find an answer -- sorry if I should
> > have just tried harder! My question is: can gnuplot on Windows handle
> > a unicode filename argument passed in UTF-16? As in
> >
> > path/to/wgnuplot.exe <UTF-16 input filename>
> 
> OK, that question was under-researched, but now I've done my homework.
> Sorry, this is a bit long but I hope I can arouse some interest in the
topic.
> 
> Why bother with UTF-16 filename arguments? Nowadays a fair number of
> Windows users construct paths (directory names or filenames) which are
"out
> of codepage" -- that is, unicode names which cannot be represented in the
> (retro) "system codepage", which is typically just an 8-bit encoding.
Since
> Windows has supported unicode since NT came out, it's a reasonable
> expectation that any filename one can construct on the platform should be
> accessible via any program of interest. But a program that restricts
itself to the
> "ANSI" form of filenames simply cannot access files with out-of-codepage
> paths.
> (Sane modern OSes don't have this problem because they use UTF-8
> throughout.)
> 
> So what about gnuplot? I may be wrong but it seems to me that gnuplot on
> Windows is stuck with "ANSI" filenames at present. Even with UNICODE and
> _UNICODE defined when compiling the program, the command-line arguments
> are retrieved in winmain.c using either _argv or __argv (depending on the
> compiler), and these get the ANSI-form arguments (as opposed to __wargv
> which gets the arguments in UTF-16 form).
> 
> It would be easy to swap out __argv for __wargv but by itself this would
be
> very disruptive. The subsequent code in winmain.c, and then the code in
plot.c
> (gnu_main) to which the args array is passed, all assumes the elements of
argv
> are plain "char *", not "wide char"
> arrays. Handling UTF-16, which is chock-full of NUL bytes, would require
lots of
> messy "ifdefs".
> 
> I have a proposal for fixing this. I realise it may not be acceptable as
it stands
> but maybe someone else might want to take it up. I'm attaching patches for
> src/win/winmain.c and src/misc.c for reference but here I'll try to
explain the
> strategy.
> 
> 1) In winmain.c, grab the command-line arguments as UTF-16 but immediately
> convert them to UTF-8, so they can handled by the regular string.h APIs,
both
> here and in plot.c (gnu_main).
> 
> 2) When we actually go to open a command-line file argument
> (loadpath_fopen, in misc.c, called from gnu_main), we first try opening
the file
> using the filename as-is, but it that fails (and the filename validates as
UTF-8)
> we convert it to UTF-16 and try again.
> 
> Since UTF-8 is a superset of ASCII, ASCII filename arguments should pass
> through transparently. Within-codepage non-ASCII filenames should get
> converted back to UTF-16 and opened OK. And the bonus is that out-of-
> codepage arguments should also be converted and opened OK.
> 
> I've tested this on Windows 10, with the system codepage set to Windows
> 1252 ("Western Europe"), and have successfully opened files with names in
> Russian and Greek. (I think this should also work if the user has the
system
> codepage set to UTF-8 (65001), which is a "beta" option on Windows.)
> 
> My implementation uses GLib APIs (nice and simple) to convert from
> UTF-16 to UTF-8 and back again (if needed). GLib is required anyway if one
is
> building the Cairo-based terminals. I suppose one could use native Windows
> APIs to the same purpose but I suspect it would be a lot more bother.
> 
> In my test setup this whole deal is triggered by the CFLAGS define
> 
> -DWIDE_ARGS
> 
> which is respected only when building for Windows -- and admittedly has
only
> been tested when cross-compiling for Windows from Linux using Mingw-w64.
> In my mingw Makefile, I have:
> 
> WIDE_ARGS = 1
> 
> ...
> 
> ifdef WIDE_ARGS
>    CFLAGS += -DWIDE_ARGS
>    CFLAGS += $(shell pkg-config --cflags glib-2.0) endif
> 
> --
> Allin Cottrell
> Department of Economics
> Wake Forest University

AW: filenames on MS Windows

[Bastian M. <bma...@we...>]

Right now, gnuplot is able to "load" file names with Unicode encoded names, i.e. the sequence
	set encoding utf8
	load 'абвгдежзийклмнопрстуфхцчшщъыьэюяѐёђѓєѕіїјљњћќѝўџ.plt'
will work just fine. Only loading via the command line does not work and that should indeed be improved. (I don't agree to use glib for that purpose - but that is easy to change).

  Bastian

> -----Ursprüngliche Nachricht-----
> Von: Allin Cottrell <cot...@wf...>
> Gesendet: Sonntag, 1. November 2020 21:24
> An: Ethan A Merritt <me...@uw...>
> Cc: gnuplot-beta <gnu...@li...>
> Betreff: Re: filenames on MS Windows
> 
> On Sun, 1 Nov 2020, Ethan A Merritt wrote:
> 
> > On Saturday, 31 October 2020 17:36:03 PST Allin Cottrell wrote:
> >> On Sun, 25 Oct 2020, Allin Cottrell wrote:
> >>
> >>> I tried googling this but didn't find an answer -- sorry if I should
> >>> have just tried harder! My question is: can gnuplot on Windows
> >>> handle a unicode filename argument passed in UTF-16? As in
> >>>
> >>> path/to/wgnuplot.exe <UTF-16 input filename>
> >>
> >> OK, that question was under-researched, but now I've done my
> >> homework. Sorry, this is a bit long but I hope I can arouse some
> >> interest in the topic.
> >
> > I don't have any direct insight into this issue other than to note
> > that the filesytem itself may be an issue.
> 
> In some contexts, no doubt. But if we set aside exotica such as surrogate pairs,
> NTFS filenames are UTF-16 to a very good approximation. As such they are
> easily converted to UTF-8 to permit handling with good old C char * APIs, and
> easily converted back to
> UTF-16 for _wfopen() if required.
> 
> > The following entry from the R developer blog is of interest
> >
> >
> > https://developer.r-project.org/Blog/public/2020/05/02/utf-8-support-o
> > n-windows/
> >
> > I gather from the discussion there that Windows-10 can be made to
> > support UTF-8 as a native encoding, calling it "extended ASCII".
> 
> Interesting, yes, but at this point kinda science fiction. The practical issue at
> present is whether gnuplot wants to support out-of-codepage UTF-16
> filenames on the Windows command line. It's not terribly difficult, as I tried to
> show.
> 
> Sorry if I'm being repetitive, but right now if a create, say, a Russian-language
> filename on Windows and pass it as command-line argument to gnuplot,
> gnuplot will not be able to open the file because its name cannot be
> represented in my "system codepage". A program that reads the command line
> as UTF-16, however, will have no problem opening the file.
> 
> Allin Cottrell

Re: with pointcolor

[Dima K. <gn...@di...>]

Ethan A Merritt <me...@uw...> writes:

> Plot elements are drawn using the current pen, and the keyword
> for specifying the color of the current pen is "linecolor".

Thanks for the clarification. The name of the option is un-ideal, but
I think a bit more documentation would help, specifically "help points"
and "help lines" should at least mention "linecolor". I'll write
something in the near future.

Thanks

Re: filenames on MS Windows

[Allin C. <cot...@wf...>]

On Sun, 1 Nov 2020, Ethan A Merritt wrote:

> On Saturday, 31 October 2020 17:36:03 PST Allin Cottrell wrote:
>> On Sun, 25 Oct 2020, Allin Cottrell wrote:
>>
>>> I tried googling this but didn't find an answer -- sorry if I should have
>>> just tried harder! My question is: can gnuplot on Windows handle a unicode
>>> filename argument passed in UTF-16? As in
>>>
>>> path/to/wgnuplot.exe <UTF-16 input filename>
>>
>> OK, that question was under-researched, but now I've done my
>> homework. Sorry, this is a bit long but I hope I can arouse some
>> interest in the topic.
>
> I don't have any direct insight into this issue other than to note
> that the filesytem itself may be an issue.

In some contexts, no doubt. But if we set aside exotica such as 
surrogate pairs, NTFS filenames are UTF-16 to a very good 
approximation. As such they are easily converted to UTF-8 to permit 
handling with good old C char * APIs, and easily converted back to 
UTF-16 for _wfopen() if required.

> The following entry from the R developer blog is of interest
>
>   https://developer.r-project.org/Blog/public/2020/05/02/utf-8-support-on-windows/
>
> I gather from the discussion there that Windows-10 can be made to
> support UTF-8 as a native encoding, calling it "extended ASCII".

Interesting, yes, but at this point kinda science fiction. The 
practical issue at present is whether gnuplot wants to support 
out-of-codepage UTF-16 filenames on the Windows command line. It's 
not terribly difficult, as I tried to show.

Sorry if I'm being repetitive, but right now if a create, say, a 
Russian-language filename on Windows and pass it as command-line 
argument to gnuplot, gnuplot will not be able to open the file 
because its name cannot be represented in my "system codepage". A 
program that reads the command line as UTF-16, however, will have no 
problem opening the file.

Allin Cottrell

Re: filenames on MS Windows

[Ethan A M. <me...@uw...>]

On Saturday, 31 October 2020 17:36:03 PST Allin Cottrell wrote:
> On Sun, 25 Oct 2020, Allin Cottrell wrote:
> 
> > I tried googling this but didn't find an answer -- sorry if I should have 
> > just tried harder! My question is: can gnuplot on Windows handle a unicode 
> > filename argument passed in UTF-16? As in
> >
> > path/to/wgnuplot.exe <UTF-16 input filename>
> 
> OK, that question was under-researched, but now I've done my 
> homework. Sorry, this is a bit long but I hope I can arouse some 
> interest in the topic.

I don't have any direct insight into this issue other than to note
that the filesytem itself may be an issue.  The standard Windows
filesystems impose an encoding on filenames, whereas linux filesystems
are agnostic to encoding; any null-terminated byte sequence not
containing '/' is a legal file name.  

The following entry from the R developer blog is of interest

   https://developer.r-project.org/Blog/public/2020/05/02/utf-8-support-on-windows/

I gather from the discussion there that Windows-10 can be made to
support UTF-8 as a native encoding, calling it "extended ASCII".
In that mode R (and I suppose gnuplot) can use the existing generic
linux code paths rather than multiple layers of text conversion.

	Ethan



> 
> Why bother with UTF-16 filename arguments? Nowadays a fair number of 
> Windows users construct paths (directory names or filenames) which 
> are "out of codepage" -- that is, unicode names which cannot be 
> represented in the (retro) "system codepage", which is typically 
> just an 8-bit encoding. Since Windows has supported unicode since NT 
> came out, it's a reasonable expectation that any filename one can 
> construct on the platform should be accessible via any program of 
> interest. But a program that restricts itself to the "ANSI" form of 
> filenames simply cannot access files with out-of-codepage paths. 
> (Sane modern OSes don't have this problem because they use UTF-8 
> throughout.)
> 
> So what about gnuplot? I may be wrong but it seems to me that 
> gnuplot on Windows is stuck with "ANSI" filenames at present. Even 
> with UNICODE and _UNICODE defined when compiling the program, the 
> command-line arguments are retrieved in winmain.c using either _argv 
> or __argv (depending on the compiler), and these get the ANSI-form 
> arguments (as opposed to __wargv which gets the arguments in UTF-16 
> form).
> 
> It would be easy to swap out __argv for __wargv but by itself this 
> would be very disruptive. The subsequent code in winmain.c, and then 
> the code in plot.c (gnu_main) to which the args array is passed, all 
> assumes the elements of argv are plain "char *", not "wide char" 
> arrays. Handling UTF-16, which is chock-full of NUL bytes, would 
> require lots of messy "ifdefs".
> 
> I have a proposal for fixing this. I realise it may not be 
> acceptable as it stands but maybe someone else might want to take it 
> up. I'm attaching patches for src/win/winmain.c and src/misc.c for 
> reference but here I'll try to explain the strategy.
> 
> 1) In winmain.c, grab the command-line arguments as UTF-16 but 
> immediately convert them to UTF-8, so they can handled by the 
> regular string.h APIs, both here and in plot.c (gnu_main).
> 
> 2) When we actually go to open a command-line file argument 
> (loadpath_fopen, in misc.c, called from gnu_main), we first try 
> opening the file using the filename as-is, but it that fails (and 
> the filename validates as UTF-8) we convert it to UTF-16 and try 
> again.
> 
> Since UTF-8 is a superset of ASCII, ASCII filename arguments should 
> pass through transparently. Within-codepage non-ASCII filenames 
> should get converted back to UTF-16 and opened OK. And the bonus is 
> that out-of-codepage arguments should also be converted and opened 
> OK.
> 
> I've tested this on Windows 10, with the system codepage set to 
> Windows 1252 ("Western Europe"), and have successfully opened files 
> with names in Russian and Greek. (I think this should also work if 
> the user has the system codepage set to UTF-8 (65001), which is a 
> "beta" option on Windows.)
> 
> My implementation uses GLib APIs (nice and simple) to convert from 
> UTF-16 to UTF-8 and back again (if needed). GLib is required anyway 
> if one is building the Cairo-based terminals. I suppose one could 
> use native Windows APIs to the same purpose but I suspect it would 
> be a lot more bother.
> 
> In my test setup this whole deal is triggered by the CFLAGS define
> 
> -DWIDE_ARGS
> 
> which is respected only when building for Windows -- and admittedly 
> has only been tested when cross-compiling for Windows from Linux 
> using Mingw-w64. In my mingw Makefile, I have:
> 
> WIDE_ARGS = 1
> 
> ...
> 
> ifdef WIDE_ARGS
>    CFLAGS += -DWIDE_ARGS
>    CFLAGS += $(shell pkg-config --cflags glib-2.0)
> endif
> 
> --
> Allin Cottrell
> Department of Economics
> Wake Forest University

Re: filenames on MS Windows

[Allin C. <cot...@wf...>]

On Sun, 25 Oct 2020, Allin Cottrell wrote:

> I tried googling this but didn't find an answer -- sorry if I should have 
> just tried harder! My question is: can gnuplot on Windows handle a unicode 
> filename argument passed in UTF-16? As in
>
> path/to/wgnuplot.exe <UTF-16 input filename>

OK, that question was under-researched, but now I've done my 
homework. Sorry, this is a bit long but I hope I can arouse some 
interest in the topic.

Why bother with UTF-16 filename arguments? Nowadays a fair number of 
Windows users construct paths (directory names or filenames) which 
are "out of codepage" -- that is, unicode names which cannot be 
represented in the (retro) "system codepage", which is typically 
just an 8-bit encoding. Since Windows has supported unicode since NT 
came out, it's a reasonable expectation that any filename one can 
construct on the platform should be accessible via any program of 
interest. But a program that restricts itself to the "ANSI" form of 
filenames simply cannot access files with out-of-codepage paths. 
(Sane modern OSes don't have this problem because they use UTF-8 
throughout.)

So what about gnuplot? I may be wrong but it seems to me that 
gnuplot on Windows is stuck with "ANSI" filenames at present. Even 
with UNICODE and _UNICODE defined when compiling the program, the 
command-line arguments are retrieved in winmain.c using either _argv 
or __argv (depending on the compiler), and these get the ANSI-form 
arguments (as opposed to __wargv which gets the arguments in UTF-16 
form).

It would be easy to swap out __argv for __wargv but by itself this 
would be very disruptive. The subsequent code in winmain.c, and then 
the code in plot.c (gnu_main) to which the args array is passed, all 
assumes the elements of argv are plain "char *", not "wide char" 
arrays. Handling UTF-16, which is chock-full of NUL bytes, would 
require lots of messy "ifdefs".

I have a proposal for fixing this. I realise it may not be 
acceptable as it stands but maybe someone else might want to take it 
up. I'm attaching patches for src/win/winmain.c and src/misc.c for 
reference but here I'll try to explain the strategy.

1) In winmain.c, grab the command-line arguments as UTF-16 but 
immediately convert them to UTF-8, so they can handled by the 
regular string.h APIs, both here and in plot.c (gnu_main).

2) When we actually go to open a command-line file argument 
(loadpath_fopen, in misc.c, called from gnu_main), we first try 
opening the file using the filename as-is, but it that fails (and 
the filename validates as UTF-8) we convert it to UTF-16 and try 
again.

Since UTF-8 is a superset of ASCII, ASCII filename arguments should 
pass through transparently. Within-codepage non-ASCII filenames 
should get converted back to UTF-16 and opened OK. And the bonus is 
that out-of-codepage arguments should also be converted and opened 
OK.

I've tested this on Windows 10, with the system codepage set to 
Windows 1252 ("Western Europe"), and have successfully opened files 
with names in Russian and Greek. (I think this should also work if 
the user has the system codepage set to UTF-8 (65001), which is a 
"beta" option on Windows.)

My implementation uses GLib APIs (nice and simple) to convert from 
UTF-16 to UTF-8 and back again (if needed). GLib is required anyway 
if one is building the Cairo-based terminals. I suppose one could 
use native Windows APIs to the same purpose but I suspect it would 
be a lot more bother.

In my test setup this whole deal is triggered by the CFLAGS define

-DWIDE_ARGS

which is respected only when building for Windows -- and admittedly 
has only been tested when cross-compiling for Windows from Linux 
using Mingw-w64. In my mingw Makefile, I have:

WIDE_ARGS = 1

...

ifdef WIDE_ARGS
   CFLAGS += -DWIDE_ARGS
   CFLAGS += $(shell pkg-config --cflags glib-2.0)
endif

--
Allin Cottrell
Department of Economics
Wake Forest University

Re: with pointcolor

[Ethan A M. <me...@uw...>]

On Thursday, 29 October 2020 13:26:38 PDT Dima Kogan wrote:
> Hi. I just discovered that there's no "pointcolor" property. So this
> doesn't work:
> 
>   plot x with points pt 6 ps 2 pointcolor "red"
> 
> The "right" way to do it is this:
> 
>   plot x with points pt 6 ps 2 linecolor "red"
> 
> Which is really surprising. And it means that when plotting with
> "linespoints" we can't use separate colors for the line and for the
> points. I'm not convinced that we do want to make separate line/point
> colors possible, but can "pointcolor" at the very least become an alias
> for "linecolor"?

There is nothing special about points.

Plot elements are drawn using the current pen, and the keyword
for specifying the color of the current pen is "linecolor".
Actually, prior to version 4.something the only options were
"linetype n" or "linestyle n".  The "linecolor" keyword was
added as a generic ad hoc specification that didn't require
pre-defined a linetype or linestyle.  I suppose it could have
been "pencolor" or "currentcolor" or even just "color", but
what we have is "linecolor".

So  plot ... with boxes linecolor "blue"    # not 'boxcolor'
    plot ... with impulses linecolor "blue" # not 'impulsecolor'
    plot ... with fsteps linecolor "blue"   # not "stepcolor'
    plot ... with points linecolor "blue"   # not 'pointcolor'


	Ethan


> 
> Is there some historical reason for this?
> 
> Thanks!
> dima

with pointcolor

[Dima K. <gn...@di...>]

Hi. I just discovered that there's no "pointcolor" property. So this
doesn't work:

  plot x with points pt 6 ps 2 pointcolor "red"

The "right" way to do it is this:

  plot x with points pt 6 ps 2 linecolor "red"

Which is really surprising. And it means that when plotting with
"linespoints" we can't use separate colors for the line and for the
points. I'm not convinced that we do want to make separate line/point
colors possible, but can "pointcolor" at the very least become an alias
for "linecolor"?

Is there some historical reason for this?

Thanks!
dima

filenames on MS Windows

[Allin C. <cot...@wf...>]

I tried googling this but didn't find an answer -- sorry if I should 
have just tried harder! My question is: can gnuplot on Windows 
handle a unicode filename argument passed in UTF-16? As in

path/to/wgnuplot.exe <UTF-16 input filename>

TIA.

-- 
Allin Cottrell
Department of Economics
Wake Forest University

merge for datablocks

[Daniel M. <mon...@ku...>]

Hi,

I just read about the patch for a "merge" command 
(https://sourceforge.net/p/gnuplot/patches/615/) that would allow 
merging the results of two plot commands into a single datablock that 
could then be used for a subsequent plot that needed both results. I 
don't think this was added to gnuplot. I tried "merge" in my current 
gnuplot version (5.2.8) and I get an "invalid command" reply.

It sounds it would be useful for combining data from different 
datablocks or files.
My use case for this would be creating a plot of the percentage of 
points in each interval that have a specific values from another column.

For example, for a file called "datafile.txt", containing values between 
0 and 1 in COL1 and a binary label in COL2:
COL1 COL2
0.1    0
0.2    1
0.1    1
0.33    0
0.4    1
1.0    0
0.9    0
0.12    1
0.13    1
0.13    0
0.89    1
0.88    0
....

A "merge" command would help to plot the following curve:

- percentage of points having COL1 between [0.0, 0.1] and COL2==1
- percentage of points have COL1 in the range [0.1,0.2] and COL2==1
- percentage of points have COL1 in the range [0.2,0.3] and COL2==1
and so on.

Currently, I am able to achieve this with "table" and "smooth frequency":
--------------------------------------------------------------------------------
# define function for binning data
binwidth = 0.1
bin(val) = binwidth * floor(val/binwidth)

# save total counts in each bin
set table "fileA"
plot "datafile.txt" using (bin(column("COL1"))):(1.0) smooth frequency 
with points

# save counts in each bin with a specific label
set table "fileB"
plot "datafile.txt" using (bin(column("COL1"))):(column("COL2") > 0 ? 
1.0 : 0.0) smooth frequency with points
unset table

# plot the percentage of dots with label "1" in each bin of width 0.1
plot "< paste fileA fileB" using 1:($5/$2) with linespoints
--------------------------------------------------------------------------------


Clearly this could be achieved pre-processing the data externally, but 
it looks like that the proposed "merge" command for datablocks would be 
a useful tool here.
It would simplify this without the need for writing data to files and 
reading it back.


Thank you,

__________________________________
Daniel Montezano
Center for Computational Biology
University of Kansas
Lawrence, KS 66045

Re: jitter documentation

[Allin C. <cot...@wf...>]

On Thu, 15 Oct 2020, Ethan A Merritt wrote:

> On Thursday, 15 October 2020 12:26:22 PDT Allin Cottrell wrote:
>> On Thu, 15 Oct 2020, Ethan A Merritt wrote:
>>
>>> On Thursday, 15 October 2020 08:02:27 PDT Allin Cottrell wrote:
>>>> Maybe I'm missing something, but isn't the documentation for
>>>> "jitter" backwards with respect to the swarm/square choice?
>>>>
>>>> The text reads thus: "The default jittering operation displaces
>>>> points only along x. This produces a distinctive pattern sometimes
>>>> called a "bee swarm plot". The optional keyword square adjusts the y
>>>> coordinate of displaced points in addition to their x coordinate so
>>>> that the points lie in distinct layers..."
>>>
>>> The text is correct as written.  Perhaps the attached figure,
>>> combining two plots from jitter.dem, will clarify.
>>>
>>> Left panel:
>>> 	original data, randomly distributed on y, all x values the same
>>> Center panel:
>>> 	"beeswarm" result from displacing points along x
>>> 	if they would otherwise overlap.
>>> 	Points are still randomly distributed on y.
>>> Right panel:
>>> 	"square" plot uses the x displacements that would
>>> 	generate a beeswarm plot, and adds an y displacement
>>> 	that is effectively a floor(y) operation where the unit of
>>> 	the floor operation is the "overlap" parameter to jitter.
>>>
>>>>
>>>> In the demo and in my own usage it seems the default is in fact to
>>>> displace in both the x and y dimensions while "square" limits the
>>>> scatter to the x dimension. (Except that in some cases I'm not
>>>> getting any y displacement with either choice, but that's another
>>>> issue.)
>>
>> Thanks, Ethan. I think I get it now. But this is potentially quite
>> confusing -- to understand exactly what jitter is doing one really
>> has to look closely at the y data in numerical form. So 'swarm'
>> preserves the original y values and just shoves points sideways to
>> get them off each other, while 'square' will also regularize the y
>> values to get the points into straight rows (more or less).
>
> Bee swarm plots were new to me.  I came across one in a paper I was
> reading and made a note to myself to look into it.
> The resulting gnuplot implementation was guided by what R does.
>
> A salient feature of bee swarm plots is that the jitter operation
> is reversible. If you consider any single point you can reconstruct
> its original [x,y] coordinates by projecting back onto the corresponding
> discrete x value.
>
> The "square" option loses this.  It is essentially a representation
> of binned data with a small number of points in each bin.  There is
> nothing to distinguish two points in the same bin from each other,
> as their original coordinates have been lost.
>
> As the number of points becomes large, both options are inferior
> to a violin plot.  The violinplot demo compares them and also
> shows a Gaussian jitter.
>
>> And then, given a pile-up of data points at some discrete {x,y}
>> value, there's no option to nudge them apart in both dimensions to
>> form a cloud?
>
> This sounds nice, but if x and y are both continuous I don't think
> it is a well-defined operation.  For a small-ish number of points you
> could define an energy function that has of a steep penalty gradient
> for overlap and then minimize the total energy by monte carlo.
> That rapidly becomes compute-intensive as the number of points
> increases.
>
> For x and y discrete there's a better option, sometimes called a
> bubble plot.  At each discrete [x,y], draw a circle with size
> proportional to the number of points that piled up there.
> The "size" is a sore point, however.  radius or area?
> I have not given much thought to whether that can be done with
> existing options in gnuplot or whether it would require new code
> or an external data processing stage.

Thanks again. All very clear.

It seems that what I was after in the doubly discrete case is really 
a bubble plot (and I'd vote for area!).

--
Allin Cottrell
Department of Economics
Wake Forest University

Re: jitter documentation

[Ethan A M. <me...@uw...>]

On Thursday, 15 October 2020 12:26:22 PDT Allin Cottrell wrote:
> On Thu, 15 Oct 2020, Ethan A Merritt wrote:
> 
> > On Thursday, 15 October 2020 08:02:27 PDT Allin Cottrell wrote:
> >> Maybe I'm missing something, but isn't the documentation for
> >> "jitter" backwards with respect to the swarm/square choice?
> >>
> >> The text reads thus: "The default jittering operation displaces
> >> points only along x. This produces a distinctive pattern sometimes
> >> called a "bee swarm plot". The optional keyword square adjusts the y
> >> coordinate of displaced points in addition to their x coordinate so
> >> that the points lie in distinct layers..."
> >
> > The text is correct as written.  Perhaps the attached figure,
> > combining two plots from jitter.dem, will clarify.
> >
> > Left panel:
> > 	original data, randomly distributed on y, all x values the same
> > Center panel:
> > 	"beeswarm" result from displacing points along x
> > 	if they would otherwise overlap.
> > 	Points are still randomly distributed on y.
> > Right panel:
> > 	"square" plot uses the x displacements that would
> > 	generate a beeswarm plot, and adds an y displacement
> > 	that is effectively a floor(y) operation where the unit of
> > 	the floor operation is the "overlap" parameter to jitter.
> >
> >>
> >> In the demo and in my own usage it seems the default is in fact to
> >> displace in both the x and y dimensions while "square" limits the
> >> scatter to the x dimension. (Except that in some cases I'm not
> >> getting any y displacement with either choice, but that's another
> >> issue.)
> 
> Thanks, Ethan. I think I get it now. But this is potentially quite 
> confusing -- to understand exactly what jitter is doing one really 
> has to look closely at the y data in numerical form. So 'swarm' 
> preserves the original y values and just shoves points sideways to 
> get them off each other, while 'square' will also regularize the y 
> values to get the points into straight rows (more or less).

Bee swarm plots were new to me.  I came across one in a paper I was
reading and made a note to myself to look into it.
The resulting gnuplot implementation was guided by what R does.

A salient feature of bee swarm plots is that the jitter operation
is reversible. If you consider any single point you can reconstruct
its original [x,y] coordinates by projecting back onto the corresponding
discrete x value.

The "square" option loses this.  It is essentially a representation
of binned data with a small number of points in each bin.  There is
nothing to distinguish two points in the same bin from each other,
as their original coordinates have been lost.

As the number of points becomes large, both options are inferior
to a violin plot.  The violinplot demo compares them and also
shows a Gaussian jitter.

> And then, given a pile-up of data points at some discrete {x,y}
> value, there's no option to nudge them apart in both dimensions to 
> form a cloud? 

This sounds nice, but if x and y are both continuous I don't think
it is a well-defined operation.  For a small-ish number of points you
could define an energy function that has of a steep penalty gradient
for overlap and then minimize the total energy by monte carlo.
That rapidly becomes compute-intensive as the number of points
increases.

For x and y discrete there's a better option, sometimes called a
bubble plot.  At each discrete [x,y], draw a circle with size
proportional to the number of points that piled up there.
The "size" is a sore point, however.  radius or area?
I have not given much thought to whether that can be done with
existing options in gnuplot or whether it would require new code
or an external data processing stage.

	cheers,
		Ethan

Re: jitter documentation

[Allin C. <cot...@wf...>]

On Thu, 15 Oct 2020, Ethan A Merritt wrote:

> On Thursday, 15 October 2020 08:02:27 PDT Allin Cottrell wrote:
>> Maybe I'm missing something, but isn't the documentation for
>> "jitter" backwards with respect to the swarm/square choice?
>>
>> The text reads thus: "The default jittering operation displaces
>> points only along x. This produces a distinctive pattern sometimes
>> called a "bee swarm plot". The optional keyword square adjusts the y
>> coordinate of displaced points in addition to their x coordinate so
>> that the points lie in distinct layers..."
>
> The text is correct as written.  Perhaps the attached figure,
> combining two plots from jitter.dem, will clarify.
>
> Left panel:
> 	original data, randomly distributed on y, all x values the same
> Center panel:
> 	"beeswarm" result from displacing points along x
> 	if they would otherwise overlap.
> 	Points are still randomly distributed on y.
> Right panel:
> 	"square" plot uses the x displacements that would
> 	generate a beeswarm plot, and adds an y displacement
> 	that is effectively a floor(y) operation where the unit of
> 	the floor operation is the "overlap" parameter to jitter.
>
>>
>> In the demo and in my own usage it seems the default is in fact to
>> displace in both the x and y dimensions while "square" limits the
>> scatter to the x dimension. (Except that in some cases I'm not
>> getting any y displacement with either choice, but that's another
>> issue.)

Thanks, Ethan. I think I get it now. But this is potentially quite 
confusing -- to understand exactly what jitter is doing one really 
has to look closely at the y data in numerical form. So 'swarm' 
preserves the original y values and just shoves points sideways to 
get them off each other, while 'square' will also regularize the y 
values to get the points into straight rows (more or less).

And then, given a pile-up of data points at some discrete {x,y}
value, there's no option to nudge them apart in both dimensions to 
form a cloud? I had the idea that was what "bee swarm" was about, 
but I guess I was mistaken.

--
Allin Cottrell
Department of Economics
Wake Forest University

Re: jitter documentation

[Ethan A M. <me...@uw...>]

On Thursday, 15 October 2020 08:02:27 PDT Allin Cottrell wrote:
> Maybe I'm missing something, but isn't the documentation for 
> "jitter" backwards with respect to the swarm/square choice?
> 
> The text reads thus: "The default jittering operation displaces 
> points only along x. This produces a distinctive pattern sometimes 
> called a "bee swarm plot". The optional keyword square adjusts the y 
> coordinate of displaced points in addition to their x coordinate so 
> that the points lie in distinct layers..."

The text is correct as written.  Perhaps the attached figure,
combining two plots from jitter.dem, will clarify.

Left panel:
	original data, randomly distributed on y, all x values the same
Center panel:
	"beeswarm" result from displacing points along x
	if they would otherwise overlap.
	Points are still randomly distributed on y.
Right panel:
	"square" plot uses the x displacements that would
	generate a beeswarm plot, and adds an y displacement
	that is effectively a floor(y) operation where the unit of
	the floor operation is the "overlap" parameter to jitter.

> 
> In the demo and in my own usage it seems the default is in fact to 
> displace in both the x and y dimensions while "square" limits the 
> scatter to the x dimension. (Except that in some cases I'm not 
> getting any y displacement with either choice, but that's another 
> issue.)

(figure attached)

	Ethan

bee swarm behavior

[Allin C. <cot...@wf...>]

I have a "jitter" test case with an integer-valued x variable. If I 
create a continuous y variable and plot with jitter, the swarm and 
square options give different plots (scatter in both dimensions with 
swarm, x dimension only with square). Fine (except it's the opposite 
of what "help jitter" says).

However, if I make the y variable integer-valued swarm and square 
produce the same plot, with displacement along x only. Is that 
intended? It seems contrary to the general intent. (I might mention: 
'vertical' produces y scatter OK in this case, but of course no x 
scatter.)

-- 
Allin Cottrell
Department of Economics
Wake Forest University

jitter documentation

[Allin C. <cot...@wf...>]

Maybe I'm missing something, but isn't the documentation for 
"jitter" backwards with respect to the swarm/square choice?

The text reads thus: "The default jittering operation displaces 
points only along x. This produces a distinctive pattern sometimes 
called a "bee swarm plot". The optional keyword square adjusts the y 
coordinate of displaced points in addition to their x coordinate so 
that the points lie in distinct layers..."

In the demo and in my own usage it seems the default is in fact to 
displace in both the x and y dimensions while "square" limits the 
scatter to the x dimension. (Except that in some cases I'm not 
getting any y displacement with either choice, but that's another 
issue.)

-- 
Allin Cottrell
Department of Economics
Wake Forest University