gnuplot-beta — Developers and beta testers of new gnuplot code

Re: question about binary plot data

[Daniel J S. <dan...@ie...>]

Daniel J Sebald wrote:
> [Ethan, please review patch for bug fix and apply if you agree with 
> changes.  Thanks.]

Oops, one of the bug fixes introduced a different bug... this new patch doesn't remove end of line reset, but resets only for first record.  Please review.

Dan

Re: gnuplot question

[Daniel J S. <dan...@ie...>]

Petr Mikulik wrote:
>> How do I set palette to read in a binary rgb palette file for a 3d
>> image (pm3d) ?
> 
> 
> According to
>     help palette file
> only ascii files are accepted, so convert your binary file in advance.

Petr,

Is there any objection to binary data files for the palette?

I see in set.c that the code uses "df_open()" and "df_readline()" just like in plot2d.c and plot3d.c.  That's good, the idea was to make that combination common for data input, and now I see that it is also used for palettes, so there is good consistency there.

If the following were removed from "set.c", binary data files might work just fine for the palette.

    if (df_binary)
	int_error( c_token, "Binary palette files not implemented");

Dan

Announcing alpha of port.trm (patch on sourceforge)

[Chris K <gnu...@li...>]

Hello,

The port.trm has reached a temporary feature plateau.  So I have posted it to
sourceforge's patch tracker:

http://sourceforge.net/tracker/?group_id=2055&atid=302055

The next things it needs are image support and a binary protocol.

The text metrics are hard coded at the moment, but could be made into terminal
options.

The oversampling may also become an option and/or part of the protocol.

Enhanced text support really needs two-way communication, and the best way to do
that would be pass in a fifo name or a file descriptor number as a terminal or
command line option.  A hard-coded default might work, but is less flexible.

I have not really handled text encoding issues.  My haskell back end likes
unicode, which should mean it is iso-latin-1 (iso-8859-1) for the 128-255 range.
 Going to a binary protocol will likely mean that I use a smarter IO library
which will also give me encoding options for text.

-- 
Chris K

Re: Arrow head filling patch, and "port.trm"

[Chris K <gnu...@li...>]

Petr Mikulik wrote:
>> 7) The magic number of the protocol are in "src/gplt_x11.h" instead of
>> x11.trm
>>
>> Looking at pm.trm : It use a pipe of some kind
> 
> It is just
>     FILE *PM_pipe
> 
>> and include "os2/pm_msgs.h" for the magic numbers of the binary
>> protocol, a file that is not part of gnuplot.
> 
> ?? This file IS part of gnuplot sources: src/os2/pm_msgs.h
> 
> BTW, pm.trm is used for a long time in a perl graphics by Ilya
> Zakharevich for what I think looks exactly what you wish to achieve
> (parsing and drawing those fprintf'ed commands instead of gnupmdrv).
> 
> ---
> PM
> 

Hmm...It seems that pm.trm does produce a reasonable binary protocol.

One things that looks nasty to understand from the code is the way it starts
communication with the other process.

And pm.trm is not an option on unix.  The DosRead command, for instance, is not
defined in any of the files in gnuplot's code.


-- 
Chris

Re: question about binary plot data

[Chris K <gnu...@li...>]

Daniel J Sebald wrote:
> Chris K wrote:
> 
>> outd.data is exactly 10 floats long (40 bytes)
>>
>> cat test-2.gp outd.data out.data | gnuplot  WORKS
>> cat test-3.gp outd.data out.data | gnuplot  WORKS
>> cat test-4.gp outd.data out.data | gnuplot  FAILS
>> cat test-4.gp outd.data out.data outd.data | gnuplot  WORKS
> 
> If you mean that it plots, then yes "WORKS" is correct.  However, I
> think this isn't working the manner you intend.

That's what I meant.

> 
> It's subtle because of the fact the data within the file looks to be a
> simple linear or affine relationship.  Thus, the shape of the curve when
> gnuplot does or doesn't generate the index data is about the same.  Read
> on...

Yup, just 0..9

> 
>>> This works:
>>>
>>> set title "test-2.gp"
>>> plot "-" binary record=10 using 0:1 with linespoints,\
>>>     "-" binary record=10 using 0:($1)**2 with linespoints
> 
> It specifies record of 10 and since "0" is a special qualifier in
> gnuplot meaning to generate the index data 10x1=10 floats for the first
> line and 10*1=10 floats for the second line.  No problem; a plot with
> red and green lines each containing 10 points.  (I'll send you the PNGs
> in a separate email.)
> 

My plots look like your PNGs

> There may be a bug here however.  I notice that with "0" the index
> starts at "-1" on the plot.  That doesn't seem a reasonable place to
> start.  It should be either "0" or "1".  I will check the documentation
> to see if the start value is specified and will fix it accordingly.

Yeah, I did not pay attention to the (-1) starting x-scale.  That is even
weirder than I thought.  Definitely a bug from my POV.

>>>
>>> This works:
>>>
>>> set title "test-3.gp"
>>> plot "-" binary record=10 using 0:1 with linespoints,\
>>>     "-" binary record=5x2 using 1:($2)**2 with linespoints
> 
> Again, there is a "0" on the first line so that is 10x1=10 floats.  But
> now the second line is saying that the record inside the binary file is
> 5 by 2 or five pairs (or five 2 tuples) and the using syntax says to use
> the first coordinate of the pair and the second coordinate of that
> pair.  The "record" and "using" do no conflict, but the specification is
> that there are only 5 points.
> 
> So if you look at the graphs you should see there are only 5 points for
> "test-3.png".  As I said before, because of the linear nature of the
> data, you may have seen the two and thought that "test-2" and "test-3"
> plotted the same but in fact they are different.

I knew they were different.  I was mainly worried about crashing.

> 
>>>
>>> This fails, saying "line 0: warning: Skipping data file with no valid
>>> points"
>>>
>>> set title "test-4.gp"
>>> plot "-" binary record=5x2 using 1:2 with linespoints,\
>>>     "-" binary record=5x2 using 1:($2)**2 with linespoints
>>>
>>> The use of 5x2 instead of 10 in the first plot triggers the error.  I
>>> am going
>>> to hypothesize that the first use of "record=5x2" causes it to read
>>> an amount of
>>> data greater than "record=10".
> 
> You may be correct.  But I don't see why if fails.  This could be a
> strange "loose end" kind of bug (an unaccounted for boundary value or
> something) because the "record" and "using" make sense on the plot.  The
> data in the file is 0 1 2 3 4 5 6 7 8 9 and both the red and green plots
> are obviously treating the data as
> 
> (0,1)
> (2,3)
> (4,5)
> (6,7)
> (8,9)
> 
> in "test-4.png".
> 
> OK, so a couple strange things there, using "0" starting at -1 and a
> complaint about running out of data when there really shouldn't be.  Let
> me work on this...
> 
> Dan
> 

And since "cat test-4.gp outd.data out.data outd.data | gnuplot  WORKS" it
*seems* to be (a) reading exactly twice as much data for the first 5x2 record as
it should and then (b) is happy with a normal amount of data in the second 5x2
record.  This behavior is strange enough that I can't really understand how to
work around it. And I looked at datafile.c and the parsing is very powerful and
very hairy.  Trying to think in "clever c" and "clever haskell" in the same day
is giving me mental whiplash.

-- 
Chris

Re: gnuplot question

[Petr M. <mi...@ph...>]

> How do I set palette to read in a binary rgb palette file for a 3d
> image (pm3d) ?

According to
 	help palette file
only ascii files are accepted, so convert your binary file in advance.

---
PM

Re: Arrow head filling patch, and "port.trm"

[Petr M. <mi...@ph...>]

> 7) The magic number of the protocol are in "src/gplt_x11.h" instead of x11.trm
>
> Looking at pm.trm : It use a pipe of some kind

It is just
 	FILE *PM_pipe

> and include "os2/pm_msgs.h" for the magic numbers of the binary protocol, 
> a file that is not part of gnuplot.

?? This file IS part of gnuplot sources: src/os2/pm_msgs.h

BTW, pm.trm is used for a long time in a perl graphics by Ilya Zakharevich 
for what I think looks exactly what you wish to achieve (parsing and drawing 
those fprintf'ed commands instead of gnupmdrv).

---
PM

Re: question about binary plot data

[Daniel J S. <dan...@ie...>]

Chris K wrote:

> This is passing strange.  I am attaching my files.
> 
> (Endian is correct for Mac OS X on PPC, you may need to specify).

OK, "endian=big" works.> I almost see what is happening now.


> outd.data is exactly 10 floats long (40 bytes)
> 
> cat test-2.gp outd.data out.data | gnuplot  WORKS
> cat test-3.gp outd.data out.data | gnuplot  WORKS
> cat test-4.gp outd.data out.data | gnuplot  FAILS
> cat test-4.gp outd.data out.data outd.data | gnuplot  WORKS

If you mean that it plots, then yes "WORKS" is correct.  However, I think this isn't working the manner you intend.

It's subtle because of the fact the data within the file looks to be a simple linear or affine relationship.  Thus, the shape of the curve when gnuplot does or doesn't generate the index data is about the same.  Read on...

>>This works:
>>
>>set title "test-2.gp"
>>plot "-" binary record=10 using 0:1 with linespoints,\
>>     "-" binary record=10 using 0:($1)**2 with linespoints

It specifies record of 10 and since "0" is a special qualifier in gnuplot meaning to generate the index data 10x1=10 floats for the first line and 10*1=10 floats for the second line.  No problem; a plot with red and green lines each containing 10 points.  (I'll send you the PNGs in a separate email.)

Note that "0" and "-1" as special column numbers isn't the most intuitive.  Ease of programming probably trumped in that case.

There may be a bug here however.  I notice that with "0" the index starts at "-1" on the plot.  That doesn't seem a reasonable place to start.  It should be either "0" or "1".  I will check the documentation to see if the start value is specified and will fix it accordingly.

>>
>>This works:
>>
>>set title "test-3.gp"
>>plot "-" binary record=10 using 0:1 with linespoints,\
>>     "-" binary record=5x2 using 1:($2)**2 with linespoints

Again, there is a "0" on the first line so that is 10x1=10 floats.  But now the second line is saying that the record inside the binary file is 5 by 2 or five pairs (or five 2 tuples) and the using syntax says to use the first coordinate of the pair and the second coordinate of that pair.  The "record" and "using" do no conflict, but the specification is that there are only 5 points.

So if you look at the graphs you should see there are only 5 points for "test-3.png".  As I said before, because of the linear nature of the data, you may have seen the two and thought that "test-2" and "test-3" plotted the same but in fact they are different.

>>
>>This fails, saying "line 0: warning: Skipping data file with no valid points"
>>
>>set title "test-4.gp"
>>plot "-" binary record=5x2 using 1:2 with linespoints,\
>>     "-" binary record=5x2 using 1:($2)**2 with linespoints
>>
>>The use of 5x2 instead of 10 in the first plot triggers the error.  I am going
>>to hypothesize that the first use of "record=5x2" causes it to read an amount of
>>data greater than "record=10".

You may be correct.  But I don't see why if fails.  This could be a strange "loose end" kind of bug (an unaccounted for boundary value or something) because the "record" and "using" make sense on the plot.  The data in the file is 0 1 2 3 4 5 6 7 8 9 and both the red and green plots are obviously treating the data as

(0,1)
(2,3)
(4,5)
(6,7)
(8,9)

in "test-4.png".

OK, so a couple strange things there, using "0" starting at -1 and a complaint about running out of data when there really shouldn't be.  Let me work on this...

Dan

Re: Arrow head filling patch, and "port.trm"

[Chris K <gnu...@li...>]

I sit corrected.

Hans-Bernhard Br=F6ker wrote:
> Chris K wrote:
>> Hans-Bernhard Br=F6ker wrote:
>=20
>>> Looks like you're re-inventing the 'xlib' terminal.  And/or tkcanvas.
>=20
>> I mildly disagree.  I re-invented debug.trm.
>=20
> No.  You may have started out from debug.trm, by you ended up with
> something that's really a good deal closer to xlib.trm than to
> debug.trm, in functionality.  Debug.trm is for humans, not machines, to
> read.  Xlib.trm is basically not more than a stream serialization of th=
e
> gnuplot terminal API, to be read by a program like gnuplot_x11.

Okay...now I actually read and tested it.  And learned more about the sem=
antics,
which was helpful.

> The x11 commands *are*, for almost all intents and purposes, the
> original commands.  PM.trm and WIN.trm are very similar to this, in
> approach, although their implementation is somewhat different.
>=20

For the most part, that is true.

>> This is highly specialized to x11.
>=20
> Not really.  All the X11-specific code is outside x11.trm.  Up until
> about version 3.7 or so, this even extended to the point that the main
> gnuplot programs wasn't even linked to the X11 libraries.
>=20
>> In the end, gnuplot gains to ability to drive a rendering program via
>> the output
>> file, which I will redirect to a pipe (/dev/fd/5). =20
>=20
> It already has that ability, in the shape of xlib.trm.  Only the syntax
> is different from your work, but not the semantics.
>=20

Observations from reading the x11 syntax and useful things I have learned=
:

0) It is a mixed text and binary protocol.  IMHO, this is ugly enough tha=
t I
won't make the same choice.  All or nothing, probably via a "set term por=
t
binary|text" option.

1) The strings sent by gnuplot to put_text do not have '\0' or '\n' chara=
cters
in them.  This is a useful property to be certain about.  This is also tr=
ue for
enhanced text.  Now I don't have to print the text characters as hex byte=
s to
avoid strange escaping rules.

2) The coordinates are printf'd with "%4d" which limits their size.  This=
 limit
to a size of 10000 should be added to the help text.  I want to oversampl=
e, so I
 exceed this limit, and thus I have no chance of using xlib's output.

3) The binary encoding uses something I had not seen before. It shifts th=
e byte
values to minimize the number of '\0' and '\n' characters, and then escap=
e them
by sending an escape code (0x05) (and escape the escape).  These strange
escaping rules have different magic numbers depending on the command, and=
 the
escaping changes the number of bytes written.  Going to a 100% binary pro=
tocol
would simplify this.

4) The endian issue is handled at runtime in a smooth way, by sending a k=
nown
value at the start (like in UTF-16/32 encodings).  I like it.  Alternativ=
ely I
could force network byte order.

5) Drawing points is punted to the downstream renderer.  Drawing arrows i=
s
punted to gnuplot's do_arrow.  I intend to make these two binary choices =
options
to "set term port".

6) x11 does what my haskell code does:  It creates a pipe then calls fork=
 and
exec and fdopen.  The only difference with the haskell code is which proc=
ess
creates which.  The x11 driver also can make a second pipe for getting
information back, which I have not done yet.

7) The magic number of the protocol are in "src/gplt_x11.h" instead of x1=
1.trm

Looking at pm.trm : It use a pipe of some kind, and include "os2/pm_msgs.=
h" for
the magic numbers of the binary protocol, a file that is not part of gnup=
lot.

Looking at win.trm : Uses "src/win/wgnuplib.h" for the magic constants.

--=20
Chris

Re: Arrow head filling patch, and "port.trm"

[<br...@ph...>]

Chris K wrote:
> Hans-Bernhard Bröker wrote:

>> Looks like you're re-inventing the 'xlib' terminal.  And/or tkcanvas.

> I mildly disagree.  I re-invented debug.trm.

No.  You may have started out from debug.trm, by you ended up with 
something that's really a good deal closer to xlib.trm than to 
debug.trm, in functionality.  Debug.trm is for humans, not machines, to 
read.  Xlib.trm is basically not more than a stream serialization of the 
gnuplot terminal API, to be read by a program like gnuplot_x11.

> I still have not read through each and every terminal driver.  I do see that x11
> is among the largest drivers, and is interactive.  The xlib driver merely
> redirects the x11 commands to a file, which is a slightly different layer than
> port.trm which just sends the original commands.  

The x11 commands *are*, for almost all intents and purposes, the 
original commands.  PM.trm and WIN.trm are very similar to this, in 
approach, although their implementation is somewhat different.

 > This is highly specialized to x11.

Not really.  All the X11-specific code is outside x11.trm.  Up until 
about version 3.7 or so, this even extended to the point that the main 
gnuplot programs wasn't even linked to the X11 libraries.

> In the end, gnuplot gains to ability to drive a rendering program via the output
> file, which I will redirect to a pipe (/dev/fd/5).  

It already has that ability, in the shape of xlib.trm.  Only the syntax 
is different from your work, but not the semantics.

Re: question about binary plot data

[Chris K <gnu...@li...>]

I almost see what is happening now.

> This works:
> 
> set title "test-2.gp"
> plot "-" binary record=10 using 0:1 with linespoints,\
>      "-" binary record=10 using 0:($1)**2 with linespoints
> 
> This works:
> 
> set title "test-3.gp"
> plot "-" binary record=10 using 0:1 with linespoints,\
>      "-" binary record=5x2 using 1:($2)**2 with linespoints
> 
> This fails, saying "line 0: warning: Skipping data file with no valid points"
> 
> set title "test-4.gp"
> plot "-" binary record=5x2 using 1:2 with linespoints,\
>      "-" binary record=5x2 using 1:($2)**2 with linespoints
> 
> The use of 5x2 instead of 10 in the first plot triggers the error.  I am going
> to hypothesize that the first use of "record=5x2" causes it to read an amount of
> data greater than "record=10".  But I have not read the code to check.
> 

outd.data is exactly 10 floats long (40 bytes)

cat test-2.gp outd.data out.data | gnuplot  WORKS
cat test-3.gp outd.data out.data | gnuplot  WORKS
cat test-4.gp outd.data out.data | gnuplot  FAILS
cat test-4.gp outd.data out.data outd.data | gnuplot  WORKS

This is passing strange.  I am attaching my files.

(Endian is correct for Mac OS X on PPC, you may need to specify).

-- 
Chris

Re: question about binary plot data

[Chris K <gnu...@li...>]

Daniel J Sebald wrote:
> Daniel J Sebald wrote:
>>> plot "-" binary record=5x2 using 1:2 with linespoints,\
>>>      "-" binary record=5x2 using 1:2 with linespoints
>>>
>>> cat "command-2.gp" "binary.dat" "binary.dat" | gnuplot
>>>
>>> The error message seems to indicated the second "-" is the problem:
>>>
>>> gnuplot> plot "-" binary record=5x2 using 1:($2) with
>>> linespoints,     "-"
>>> binary record=5x2 using 1:($2)**2 with linespoints
>>>
>>>                                   ^
>>>          line 0: warning: Skipping data file with no valid points
>>>
> 
> ... I wonder why the above is failing.  The warning makes me think that
> gnuplot believes it has run out of data.  Might you have an extra CR or
> LF as a result of the cat process?
> 
> Actually, this seems to work for me.  For example, try the attached
> files with the command
> 
> cat "command" "bin1.bin" "bin2.bin" | gnuplot
> 
> and you should see a PNG file appear in the directory.

Your example worked.  So I have written more test cases, and I have isolated the
trigger for failure.

This works:

set title "test-2.gp"
plot "-" binary record=10 using 0:1 with linespoints,\
     "-" binary record=10 using 0:($1)**2 with linespoints

This works:

set title "test-3.gp"
plot "-" binary record=10 using 0:1 with linespoints,\
     "-" binary record=5x2 using 1:($2)**2 with linespoints

This fails, saying "line 0: warning: Skipping data file with no valid points"

set title "test-4.gp"
plot "-" binary record=5x2 using 1:2 with linespoints,\
     "-" binary record=5x2 using 1:($2)**2 with linespoints

The use of 5x2 instead of 10 in the first plot triggers the error.  I am going
to hypothesize that the first use of "record=5x2" causes it to read an amount of
data greater than "record=10".  But I have not read the code to check.

I see in datafile.c :

/* Daniel Sebald: added general binary 2d data support. (20 August 2004)
 */

So thanks for the help with this.

-- 
Chris

Re: question about binary plot data

[Daniel J S. <dan...@ie...>]

Daniel J Sebald wrote:
> Chris,
> 
> I've read your list entry and will investigate.  I can't recall thinking 
> to try an example with multiple plots and "-", so there may be a problem.

Actually, I did give an example of this.  If you look in 'image.dem' there is a plot titled:

"Binary data specified at the command line, intended for use through pipe"

in which the binary data is inside the file itself (i.e., as though one somehow typed binary data at the command line, which will never happen but it is simply meant to illustrate).

So...


>> plot "-" binary record=5x2 using 1:2 with linespoints,\
>>      "-" binary record=5x2 using 1:2 with linespoints
>>
>> cat "command-2.gp" "binary.dat" "binary.dat" | gnuplot
>>
>> The error message seems to indicated the second "-" is the problem:
>>
>> gnuplot> plot "-" binary record=5x2 using 1:($2) with linespoints,     
>> "-"
>> binary record=5x2 using 1:($2)**2 with linespoints
>>
>>                                   ^
>>          line 0: warning: Skipping data file with no valid points
>>

... I wonder why the above is failing.  The warning makes me think that gnuplot believes it has run out of data.  Might you have an extra CR or LF as a result of the cat process?

Actually, this seems to work for me.  For example, try the attached files with the command

cat "command" "bin1.bin" "bin2.bin" | gnuplot

and you should see a PNG file appear in the directory.


>> The documentation page explains that:
>>
>>
>>> General binary data can be entered at the command line via 
>>
>>
>> the special file name '-'. However, this is intended for use through
>> a pipe where programs can exchange binary data, not for keyboards.
>> There is no "end of record" character for binary data.
>> Gnuplot continues reading from a pipe until it has read the
>> number of points declared in the array qualifier.
>>
>> But I used a record qualifier.  And I don't know how to simulate the 
>> array using
>> records (even given that I can resort the order of the binary data).  
>> Both
>> "array=5,5 index0" and "array=5,5 index 1" work like I expect.  But I 
>> don't see
>> how i can make an (x,y) plot from index 0 and index 1.
>>
>> Can gnuplot do the same ascii data tricks with binary data?

I'm not exactly sure what you mean to do here.  However, look through the examples in image.dem and see if there is anything close to what you want.  (I'm thinking there should be.)  I tried giving enough examples of every feature.

Dan

Re: question about binary plot data

[Daniel J S. <dan...@ie...>]

Chris,

I've read your list entry and will investigate.  I can't recall thinking to try an example with multiple plots and "-", so there may be a problem.

Dan


Chris K wrote:
> As I am trying to send data to gnuplot, I have been experimenting to understand
> how to pipe data to stdin.
> 
> 
> Short version: It is possible to put ascii data after a plot command with more
> than one blob (several sections that end with the line "e") Can I send more that
> one binary blob?
> 
> In particular, I was seeing if I could send binary data via stdin:
> 
> cat "command-1.gp" "binary.dat" | gnuplot
> 
> where the command-1.gp file is one line:
> 
> plot "-" binary record=5x2 using 1:2 with linespoints
> 
> and the binary.dat file is 10 floats (40 bytes)
> 
> That worked like I expected it to, parsing the binary file as "x1 y1 x2 y3 x3 y3
> x4 y4 x5 y5" and plotting markers at (x1,y1) (x2,y2),...
> 
> When I tried something more complicated, it failed:
> 
> plot "-" binary record=5x2 using 1:2 with linespoints,\
>      "-" binary record=5x2 using 1:2 with linespoints
> 
> cat "command-2.gp" "binary.dat" "binary.dat" | gnuplot
> 
> The error message seems to indicated the second "-" is the problem:
> 
> gnuplot> plot "-" binary record=5x2 using 1:($2) with linespoints,     "-"
> binary record=5x2 using 1:($2)**2 with linespoints
> 
>                                   ^
>          line 0: warning: Skipping data file with no valid points
> 
> The documentation page explains that:
> 
> 
>>General binary data can be entered at the command line via 
> 
> the special file name '-'. However, this is intended for use through
> a pipe where programs can exchange binary data, not for keyboards.
> There is no "end of record" character for binary data.
> Gnuplot continues reading from a pipe until it has read the
> number of points declared in the array qualifier.
> 
> But I used a record qualifier.  And I don't know how to simulate the array using
> records (even given that I can resort the order of the binary data).  Both
> "array=5,5 index0" and "array=5,5 index 1" work like I expect.  But I don't see
> how i can make an (x,y) plot from index 0 and index 1.
> 
> Can gnuplot do the same ascii data tricks with binary data?
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by xPML, a groundbreaking scripting language
> that extends applications into web and mobile media. Attend the live webcast
> and join the prime developer group breaking into this new coding territory!
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
> _______________________________________________
> gnuplot-beta mailing list
> gnu...@li...
> https://lists.sourceforge.net/lists/listinfo/gnuplot-beta
> 


-- 

Dan Sebald
phone: 608 256 7718
email: daniel DOT sebald AT ieee DOT org
URL: http://webpages DOT charter DOT net/dsebald/

question about binary plot data

[Chris K <gnu...@li...>]

As I am trying to send data to gnuplot, I have been experimenting to understand
how to pipe data to stdin.


Short version: It is possible to put ascii data after a plot command with more
than one blob (several sections that end with the line "e") Can I send more that
one binary blob?

In particular, I was seeing if I could send binary data via stdin:

cat "command-1.gp" "binary.dat" | gnuplot

where the command-1.gp file is one line:

plot "-" binary record=5x2 using 1:2 with linespoints

and the binary.dat file is 10 floats (40 bytes)

That worked like I expected it to, parsing the binary file as "x1 y1 x2 y3 x3 y3
x4 y4 x5 y5" and plotting markers at (x1,y1) (x2,y2),...

When I tried something more complicated, it failed:

plot "-" binary record=5x2 using 1:2 with linespoints,\
     "-" binary record=5x2 using 1:2 with linespoints

cat "command-2.gp" "binary.dat" "binary.dat" | gnuplot

The error message seems to indicated the second "-" is the problem:

gnuplot> plot "-" binary record=5x2 using 1:($2) with linespoints,     "-"
binary record=5x2 using 1:($2)**2 with linespoints

                                  ^
         line 0: warning: Skipping data file with no valid points

The documentation page explains that:

> General binary data can be entered at the command line via 
the special file name '-'. However, this is intended for use through
a pipe where programs can exchange binary data, not for keyboards.
There is no "end of record" character for binary data.
Gnuplot continues reading from a pipe until it has read the
number of points declared in the array qualifier.

But I used a record qualifier.  And I don't know how to simulate the array using
records (even given that I can resort the order of the binary data).  Both
"array=5,5 index0" and "array=5,5 index 1" work like I expect.  But I don't see
how i can make an (x,y) plot from index 0 and index 1.

Can gnuplot do the same ascii data tricks with binary data?

Re: Arrow head filling patch, and "port.trm"

[Chris K <gnu...@li...>]

Ethan A Merritt wrote:
> On Saturday 01 April 2006 02:24 pm, you wrote:
> 
> I think the thread messages arrived here out of order.
> After I responded to the first in which I saw the "set term fd <fd>"
> proposal, I saw two more with the same proposal and no mention of
> my first response.  But they may well have been sent earlier and
> arrived later.

I wrote the response to your first message last night, and did not push send
until today, alongside me second response.  I plead staying up till 3AM and
temporary incapacity.

>  
>>> If you need a communication channel aside from stdin/stdout,
>>> create a named pipe and use "set output" to pass its name.
>> Why on earth might it matter whether I use mkfifo or a pipe like that?
> 
> Because if it's a named pipe, it can be used with the existing
> "set output" mechanism for 2-way communication. There are several
> old threads discussing this, and I think at least one bit of sample
> code on the web site showing how this can be used to drive gnuplot
> from python code.
>  

Now that is something that might be very useful in the future.  Right now
"port.trm" gets no feedback from the client.   The client can only send gnuplot
"set term port option badger 10 snake 1 badger 3" to communicate back.

A two-way pipe from the port.trm code and the client would be a much more
complicated protocol.  I have no clear idea what I want to "say", so I can't
implement that yet.  But it could be needed for getting font metrics or the
extent of text blocks.  I still have not done enhanced text.

I have used gnuplot.py and scipy to drive gnuplot to display my data.  But they
did not use a special terminal.  They were more of a front end than a back end.
 Cairo has made it easy to do both at once.

The use of a file descriptor / pipe / named fifo allows the front-end and
back-end to be separable concepts, loosely coupled.  This flexibility was not
provided by the other terminals, so I wrote port.trm.

>> It's not like I'm doing something crazy like serving the results via https.
> 
> Passing a file descriptor by number via stdin to another process sounds
> crazy to me.  If it's a child process you might as well just
> let it inherit the file descriptor directly, similar to the piping 
> between gnuplot and gnuplot_x11.  If it's not a child  process, let's just
> say there is some question whether the number will be of any use.

I was not sufficiently clear.  It will be a child process.  It does just inherit
the open file descriptor.  But rather than hard-code which integer, or use an
environment variable, or a command line parameter, I had thought to make the
port.trm take a new option "set term port fd 5".  But now I know "set out
'/dev/fd/5'" works just as well.

> If for some reason you don't want to create a named pipe,
> then on current linux systems you could try
> 	set output "/proc/${pid}/fd/whatever"

With the pid...that would be ugly.  Thus the child process idea.

> To me that seems ugly also, but less ugly than passing a bare number
> via stdin. I haven't ever tried this however, so I am not sure what
> extra wrinkles (buffering? permissions?) might arise.

I did the experiment before I wrote port.trm; it works for haskell calling a
test program "hi-n".  This is the haskell code (which sends the bare number 'w'
as ascii):

> main = do
>   (r,w) <- createPipe
>   (toChild,fromChild,fromChildErr,child) <- runInteractiveProcess "./hi-n" []
Nothing Nothing
>   hPutStrLn toChild (show w)
>   hFlush toChild

I did indeed need to call hFlush.  But that was not hard to figure out.  The c
code is a snippet that reads 'w' and uses write/fdopen/fprintf to verify that it
can send text to the parent process.

> 
>> It's not like I'm doing something crazy like encoding the results in
>> XML.  That's svg.trm's job.
> 
> Heh.
> So, given the date, you think someone should start writing
> a flash.trm instead?
> 

That would be an excellent project for someone else.  I have never created any
flash / director / shockwave / blink-tag code.

-- 
Chris

Re: Arrow head filling patch, and "port.trm"

[Chris K <gnu...@li...>]

Ethan A Merritt wrote:
> On Saturday 01 April 2006 02:22 pm, Chris K wrote:
>>> That's fine, but perhaps your new code should just replace the existing
>>> sadly out of date debug terminal rather than creating a new name.
>> Well, the debugging term has a lot of sanity checking chatter on the output
>> which I don't want.  I think if I were debugging with the debug term I would be
>> adding all kinds of printf(stderr,...) statements to investigate
> 
> Why not have both?

Everything takes time, that is why not both.

And I think the additional printf's could be put in debug.trm and achieve the
same or better effect.

Normally, debug.trm is compiled when DEBUG is defined, which turns on printf
code throughout gnuplot.

> Give your revised terminal an option 
> 	set term debug verbose
> to generate all the extra sanity checking output.
> 
If such checks get written, it will be done while I am actually debugging.

The *ideal* way to extend the protocol would be patching gnuplot to add semantic
information about each part of the output.  "X Axis Tics", "Line for second
plot", "key", "Y2 Grid lines", ...

That would still be easier than re-writing the parts of gnuplot that will be
used.  But this is a hobby, not a job.

Once I think I understand the protocol well enough, then I may be able to
implement a binary version.

A few interesting things that might be done easily from the recipient's side
(i.e. in haskell) based on the data, without calling gnuplot again:

Global changes:
* Rescale the plot, e.g. to match the window size.
* Crop and Zoom in.
* Arrange multiple plots.
* Re-render to a PNG file (that's how I made the image I linked to).
* Perhaps eventually re-render to future cairo backends (e.g. pdf)
Internal changes:
* Change the linetype or marker points appearances.
* Edit the text / nudge text position
* Pick out individual moveto(lineto+) lines and change their properties
separately.  But for this I may as well export to fig.trm
And by this time next year:
* Add a pony

-- 
Chris

Re: Arrow head filling patch, and "port.trm"

[Ethan A M. <merritt@u.washington.edu>]

On Saturday 01 April 2006 02:24 pm, you wrote:
> Also, "obsession" is a hostile term to use.  Despite this being April 1st, I
> will try and keep the snark escalation under control.

Sorry. No offense intended.

I think the thread messages arrived here out of order.
After I responded to the first in which I saw the "set term fd <fd>"
proposal, I saw two more with the same proposal and no mention of
my first response.  But they may well have been sent earlier and
arrived later.
 
> > If you need a communication channel aside from stdin/stdout,
> > create a named pipe and use "set output" to pass its name.
> 
> Why on earth might it matter whether I use mkfifo or a pipe like that?

Because if it's a named pipe, it can be used with the existing
"set output" mechanism for 2-way communication. There are several
old threads discussing this, and I think at least one bit of sample
code on the web site showing how this can be used to drive gnuplot
from python code.
 
> It's not like I'm doing something crazy like serving the results via https.

Passing a file descriptor by number via stdin to another process sounds
crazy to me.  If it's a child process you might as well just
let it inherit the file descriptor directly, similar to the piping 
between gnuplot and gnuplot_x11.  If it's not a child  process, let's just
say there is some question whether the number will be of any use.
If for some reason you don't want to create a named pipe,
then on current linux systems you could try
	set output "/proc/${pid}/fd/whatever"
To me that seems ugly also, but less ugly than passing a bare number
via stdin. I haven't ever tried this however, so I am not sure what
extra wrinkles (buffering? permissions?) might arise.

> It's not like I'm doing something crazy like encoding the results in
> XML.  That's svg.trm's job.

Heh.
So, given the date, you think someone should start writing
a flash.trm instead?

-- 
Ethan A Merritt
Biomolecular Structure Center
University of Washington, Seattle 98195-7742

Re: Arrow head filling patch, and "port.trm"

[Ethan A M. <merritt@u.washington.edu>]

On Saturday 01 April 2006 02:22 pm, Chris K wrote:
> > 
> > That's fine, but perhaps your new code should just replace the existing
> > sadly out of date debug terminal rather than creating a new name.
> 
> Well, the debugging term has a lot of sanity checking chatter on the output
> which I don't want.  I think if I were debugging with the debug term I would be
> adding all kinds of printf(stderr,...) statements to investigate

Why not have both?
Give your revised terminal an option 
	set term debug verbose
to generate all the extra sanity checking output.


-- 
Ethan A Merritt
Biomolecular Structure Center
University of Washington, Seattle 98195-7742

Re: Arrow head filling patch, and "port.trm"

[Chris K <gnu...@li...>]

Ethan A Merritt wrote:
> On Saturday 01 April 2006 01:05 pm, Chris K wrote:
>> Second: Where is the best documentation for the enhanced text format a=
nd protocol?
>=20
> .../docs/psdoc/ps_guide.ps=09
> 	gives the UI
> .../term/README		=09
> 	gives the required driver entry points to support it
> .../term/dumb.trm	=09
> 	contains a minimal implementation to be used as a model
>=20
>> In the end, gnuplot gains to ability to drive a rendering program via =
the output
>> file, which I will redirect to a pipe (/dev/fd/5).
>=20
> Please give up on this file descriptor obsession.
>

I am removing the "fd #" option to the terminal.  I will be using "set ou=
t"

Also, "obsession" is a hostile term to use.  Despite this being April 1st=
, I
will try and keep the snark escalation under control.

> If you need a communication channel aside from stdin/stdout,
> create a named pipe and use "set output" to pass its name.

Why on earth might it matter whether I use mkfifo or a pipe like that?

It's not like I'm doing something crazy like serving the results via http=
s.

>> Then gnuplot has the ability=20
>> to drive any program that parses the protocol without recompiling gnup=
lot.  None
>> of the terminal drivers that I have read (besides debug) approached it=
 this way.
>=20
> Sure. I agree it's a useful thing to have, if only for debugging.
> But for any extensive set of data, the extra bandwidth required by the=20
> ascii intermediate will kill you. That's why a binary protocol mode
> was added to the x11 communications channel.

True.  I am not concerned about speed.  Since I did not even know what th=
e
commands were or what the protocol should look like, I started with what =
was
simplest.  It's not like I'm doing something crazy like encoding the resu=
lts in
XML.  That's svg.trm's job.

>=20
>> I am ultimately using a Cairo backend which has commands like are
>> nearly isomorphic to what gnuplot sends to the terminal.
>=20
> You should have a look at Timoth=E9e Lecomte's wxWidgets driver,
> which also ends up driving Pango+Cairo.
>=20

I only found that recently.  I still have not gotten their code.  I will =
soon.

--=20
Chris

Re: Arrow head filling patch, and "port.trm"

[Chris K <gnu...@li...>]

Ethan Merritt wrote:
> On Friday 31 March 2006 01:39 pm, Chris K wrote:
>> Pre-Announcing "port.trm" :

After reading the thread of the wxWidget terminal patch on sourceforge, I
realized I had the same sampling/anti-aliasing problem.  I now oversample and
this fixes the wobbly appearance of "plot [-10:10] x".  The text emitted now
uses doubles for the coordinates, reflecting the added precision.  The amount of
oversampling is the largest integer multiple of max(width,height) than keeps it
smaller than 2^15. That way (x*x+y*y) is still about 2^31 and nothing weird
should happen.

>>
>> The ability to write the commands as ascii text to a file descriptor port is why
>> it is called "port.trm".  The idea to for the parent process to read the
>> commands and render/process them.
> 
> It sounds like this is a substantial update to the existing "debug.trm".
> That's fine, but perhaps your new code should just replace the existing
> sadly out of date debug terminal rather than creating a new name.

Well, the debugging term has a lot of sanity checking chatter on the output
which I don't want.  I think if I were debugging with the debug term I would be
adding all kinds of printf(stderr,...) statements to investigate, which would
break the text protocol that "port.trm" implicitly defines.

> 
>> So the new "port.trm" device uses fprintf to send a legible
>> ascii description of the rendering commands to either the output file or to an
>> integer file descriptor passed as a terminal option.
> 
> Ugh. Forget it.  No way.

Okay.  Consider it forgotten.

> Just set the output file to wherever you want the output to go.
>  

If you don't set the file descriptor than it does send it to gpoutfile like you
prefer.  And I can kludge the same effect for any terminal with:

set out "/dev/fd/5"

or, assuming bash:

set out "| cat >&5"

The only thing I worry about is if gnuplot closes the file descriptor and breaks
the connection.

>> When it is a bit more polished, I will post a copy of "port.trm" to sourceforge.
> 
> That's fine. It sounds like a worthwhile update to the set
> of terminal capabilities.
>  

Re: Arrow head filling patch, and "port.trm"

[Ethan A M. <merritt@u.washington.edu>]

On Saturday 01 April 2006 01:05 pm, Chris K wrote:
>=20
> Second: Where is the best documentation for the enhanced text format and =
protocol?

=2E../docs/psdoc/ps_guide.ps=09
	gives the UI
=2E../term/README		=09
	gives the required driver entry points to support it
=2E../term/dumb.trm	=09
	contains a minimal implementation to be used as a model

> In the end, gnuplot gains to ability to drive a rendering program via the=
 output
> file, which I will redirect to a pipe (/dev/fd/5).

Please give up on this file descriptor obsession.
If you need a communication channel aside from stdin/stdout,=20
create a named pipe and use "set output" to pass its name.

> Then gnuplot has the ability=20
> to drive any program that parses the protocol without recompiling gnuplot=
=2E  None
> of the terminal drivers that I have read (besides debug) approached it th=
is way.

Sure. I agree it's a useful thing to have, if only for debugging.
But for any extensive set of data, the extra bandwidth required by the=20
ascii intermediate will kill you. That's why a binary protocol mode
was added to the x11 communications channel.

> I am ultimately using a Cairo backend which has commands like are
> nearly isomorphic to what gnuplot sends to the terminal.

You should have a look at Timoth=E9e Lecomte's wxWidgets driver,
which also ends up driving Pango+Cairo.

=2D-=20
Ethan A Merritt
Biomolecular Structure Center
University of Washington, Seattle 98195-7742

Re: Arrow head filling patch, and "port.trm"

[Chris K <gnu...@li...>]

First: The result of rendering the commands sent by port.trm using
haskell/gtk2hs/cairo:

http://img102.imageshack.us/img102/2040/woo0nn.png

This is a 640x480 rendering of the test command.

Second: Where is the best documentation for the enhanced text format and =
protocol?

Hans-Bernhard Br=F6ker wrote:
> Chris K wrote:
>=20
>> Pre-Announcing "port.trm" :
>>
>> The ability to write the commands as ascii text to a file descriptor
>> port is why
>> it is called "port.trm".  The idea to for the parent process to read t=
he
>> commands and render/process them.
>=20
> Looks like you're re-inventing the 'xlib' terminal.  And/or tkcanvas.
>=20

I mildly disagree.  I re-invented debug.trm.

I still have not read through each and every terminal driver.  I do see t=
hat x11
is among the largest drivers, and is interactive.  The xlib driver merely
redirects the x11 commands to a file, which is a slightly different layer=
 than
port.trm which just sends the original commands.  This is highly speciali=
zed to x11.

The tkcanvas driver is stranger.  It is a tcl/tk code generator that emit=
s the
executable code to operate gui.  This is highly specialize to just tcl/tk=
.

=93Any problem in computer science can be solved with another layer of
indirection=94 -- David Wheeler.

What I have done is derive port.trm from debug.trm (Literally -- I starte=
d by
searching and replacing DEBUG with PORT).  The port.trm driver has to imp=
lement
a little policy, but is mainly concerned with creating a precise descript=
ion of
the commands as data.

In the end, gnuplot gains to ability to drive a rendering program via the=
 output
file, which I will redirect to a pipe (/dev/fd/5).  Then gnuplot has the =
ability
to drive any program that parses the protocol without recompiling gnuplot=
.  None
of the terminal drivers that I have read (besides debug) approached it th=
is way.
     This was an attractive idea because, like the experimental wxWidgets
terminal, I am ultimately using a Cairo backend which has commands like a=
re
nearly isomorphic to what gnuplot sends to the terminal.

The ascii format is very easily read by Haskell, but also very easily rea=
d by
people (Well..the text to display is hex encoded 0-9a-f, so that is less =
human
readable).  Each command is exactly one line.  Everything is in the
][a-zA-Z0-9(_)- character set (plus newline), and there are no escaping o=
r
endian issues.  I don't use them, but I expect every command could be par=
sed
with a scanf or a simple regular expression.

The ability to save the output to a file and replay or edit it makes for =
strange
possibilities.

--=20
Chris

Re: Arrow head filling patch, and "port.trm"

[<br...@ph...>]

Chris K wrote:

> Pre-Announcing "port.trm" :
> 
> The ability to write the commands as ascii text to a file descriptor port is why
> it is called "port.trm".  The idea to for the parent process to read the
> commands and render/process them.

Looks like you're re-inventing the 'xlib' terminal.  And/or tkcanvas.

Re: Arrow head filling patch, and "port.trm"

[Ethan M. <merritt@u.washington.edu>]

On Friday 31 March 2006 01:39 pm, Chris K wrote:
> Pre-Announcing "port.trm" :
> 
> The ability to write the commands as ascii text to a file descriptor port is why
> it is called "port.trm".  The idea to for the parent process to read the
> commands and render/process them.

It sounds like this is a substantial update to the existing "debug.trm".
That's fine, but perhaps your new code should just replace the existing
sadly out of date debug terminal rather than creating a new name.

> So the new "port.trm" device uses fprintf to send a legible
> ascii description of the rendering commands to either the output file or to an
> integer file descriptor passed as a terminal option.

Ugh. Forget it.  No way.
Just set the output file to wherever you want the output to go.
 
> When it is a bit more polished, I will post a copy of "port.trm" to sourceforge.

That's fine. It sounds like a worthwhile update to the set
of terminal capabilities.
 

-- 
Ethan A Merritt
Biomolecular Structure Center
University of Washington, Seattle WA