gnuplot-beta

multiple plot lines with plot '-'

[<pl...@pi...>]

HI,

I am plotting some data from a data file with four columns to produce 
three plot lines.

set timefmt "%d.%m.%y:%H:%M"; set xdata time
plot  "fast.log" using 1:2 t "systolic" w l,  "fast.log" using 1:3 t 
"diastolic" w l, "fast.log" using 1:4 t "pulse" w l


To save having a separate gnuplot script file I was trying to put these 
two lines at the head of the data and use the plot '-' feature. But this 
involves reading the data three times and I can't see how to do it.

Before suggesting ways this could be improved I was wondering whether I 
was missing a trick.

Thanks. Peter.

Re: multiple plot lines with plot '-'

[Mojca M. <moj...@gm...>]

On Mon, May 14, 2012 at 8:21 AM,  <pl...@pi...> wrote:
> HI,
>
> I am plotting some data from a data file with four columns to produce
> three plot lines.
>
> set timefmt "%d.%m.%y:%H:%M"; set xdata time
> plot  "fast.log" using 1:2 t "systolic" w l,  "fast.log" using 1:3 t
> "diastolic" w l, "fast.log" using 1:4 t "pulse" w l
>
>
> To save having a separate gnuplot script file I was trying to put these
> two lines at the head of the data and use the plot '-' feature. But this
> involves reading the data three times and I can't see how to do it.
>
> Before suggesting ways this could be improved I was wondering whether I
> was missing a trick.

Simply use empty quotation marks:
   plot  "fast.log" using 1:2 t "systolic" w l,  "" using ...

Mojca

Re: multiple plot lines with plot '-'

[<pl...@pi...>]

On 05/14/12 09:51, Mojca Miklavec wrote:
> On Mon, May 14, 2012 at 8:21 AM,<pl...@pi...>  wrote:
>> HI,
>>
>> I am plotting some data from a data file with four columns to produce
>> three plot lines.
>>
>> set timefmt "%d.%m.%y:%H:%M"; set xdata time
>> plot  "fast.log" using 1:2 t "systolic" w l,  "fast.log" using 1:3 t
>> "diastolic" w l, "fast.log" using 1:4 t "pulse" w l
>>
>>
>> To save having a separate gnuplot script file I was trying to put these
>> two lines at the head of the data and use the plot '-' feature. But this
>> involves reading the data three times and I can't see how to do it.
>>
>> Before suggesting ways this could be improved I was wondering whether I
>> was missing a trick.
>
> Simply use empty quotation marks:
>     plot  "fast.log" using 1:2 t "systolic" w l,  "" using ...
>
> Mojca
>

Thanks but I think you missed the point .

The line I posted was using the datafile as data , that could be made 
more concise by using the empty quotes syntax.

My problem is to use the inline data format with  plot "-"  followed by 
the data then the termination line 'e' .

This does not work using the empty bracket syntax because it does not 
reread the data lines that follow. Only the first line gets plotted , 
the rest throw an error about "skipping line with no data".

Thx.

Re: multiple plot lines with plot '-'

[Allin C. <cot...@wf...>]

On Mon, 14 May 2012, Mojca Miklavec wrote:

> On Mon, May 14, 2012 at 8:21 AM,  <pl...@pi...> wrote:
>> HI,
>>
>> I am plotting some data from a data file with four columns to produce
>> three plot lines.
>>
>> set timefmt "%d.%m.%y:%H:%M"; set xdata time
>> plot  "fast.log" using 1:2 t "systolic" w l,  "fast.log" using 1:3 t
>> "diastolic" w l, "fast.log" using 1:4 t "pulse" w l
>>
>> To save having a separate gnuplot script file I was trying to put these
>> two lines at the head of the data and use the plot '-' feature. But this
>> involves reading the data three times and I can't see how to do it.
>>
>> Before suggesting ways this could be improved I was wondering whether I
>> was missing a trick.
>
> Simply use empty quotation marks:
>   plot  "fast.log" using 1:2 t "systolic" w l,  "" using ...

This doesn't work in the context plotter specified, namely 
when you replace the data file "fast.log" with inline data in 
the plot file, using "plot '-'".

I too would be interested to know if there's any way to avoid 
repeating the inline data when plotting multiple lines.

Allin Cottrell

Re: multiple plot lines with plot '-'

[<pl...@pi...>]

On 05/14/12 14:52, Allin Cottrell wrote:
> On Mon, 14 May 2012, Mojca Miklavec wrote:
>
>> On Mon, May 14, 2012 at 8:21 AM, <pl...@pi...> wrote:
>>> HI,
>>>
>>> I am plotting some data from a data file with four columns to produce
>>> three plot lines.
>>>
>>> set timefmt "%d.%m.%y:%H:%M"; set xdata time
>>> plot  "fast.log" using 1:2 t "systolic" w l,  "fast.log" using 1:3 t
>>> "diastolic" w l, "fast.log" using 1:4 t "pulse" w l
>>>
>>> To save having a separate gnuplot script file I was trying to put these
>>> two lines at the head of the data and use the plot '-' feature. But this
>>> involves reading the data three times and I can't see how to do it.
>>>
>>> Before suggesting ways this could be improved I was wondering whether I
>>> was missing a trick.
>>
>> Simply use empty quotation marks:
>> plot  "fast.log" using 1:2 t "systolic" w l,  "" using ...
>
> This doesn't work in the context plotter specified, namely when you
> replace the data file "fast.log" with inline data in the plot file,
> using "plot '-'".
>
> I too would be interested to know if there's any way to avoid repeating
> the inline data when plotting multiple lines.
>
> Allin Cottrell

Thanks Allin,

unless someone points out a trick we're missing perhaps the plot '-' 
feature could be extended to act in same way as a named file.

It seems that  plot '-', '-'  already has a defined behaviour of reading 
two subsequent sets of inline data so I guess that has to stay as it is. 
This seems to be the effect of using plot "" after plot '-' too.

This an unfortunate asymmetry in the behaviour when compared to named 
files but it's long established and probably should stay as it is.

So could another flag like plot '--' be used to indicate rereading of 
the inline data rather than continuing at the next block of inline data ?

regards, Peter.

Re: multiple plot lines with plot '-'

[Ethan A M. <sf...@us...>]

On Monday, May 14, 2012 08:12:25 am pl...@pi... wrote:
> On 05/14/12 14:52, Allin Cottrell wrote:
> > On Mon, 14 May 2012, Mojca Miklavec wrote:
> >
> >> Simply use empty quotation marks:
> >> plot  "fast.log" using 1:2 t "systolic" w l,  "" using ...
> >
> > This doesn't work in the context plotter specified, namely when you
> > replace the data file "fast.log" with inline data in the plot file,
> > using "plot '-'".
> >
> > I too would be interested to know if there's any way to avoid repeating
> > the inline data when plotting multiple lines.
> >
> > Allin Cottrell
> 
> Thanks Allin,
> 
> unless someone points out a trick we're missing perhaps the plot '-' 
> feature could be extended to act in same way as a named file.

It _does_ act in the same way as a named file.  The empty quotes indicate
that the previous source of data should be read again.

> It seems that  plot '-', '-'  already has a defined behaviour of reading 
> two subsequent sets of inline data so I guess that has to stay as it is. 
> This seems to be the effect of using plot "" after plot '-' too.

Yes. "" means "read the same input source again".

> So could another flag like plot '--' be used to indicate rereading of 
> the inline data rather than continuing at the next block of inline data ?

This wouldn't work in the general case - you can't reread from a pipe.

The next obvious thought is "can't we remember the values we just read
and re-use them if necessary?". But that doesn't work either.
Consider:
   plot '-' using 1:2, '' using 3:4, '' using 99:100
The first plot uses only columns 1 and 2; these values are stored
internally and could be used again without rereading.  That's what the
"refresh"  command does.  But the 2nd and 3rd plots use data values from
columns we didn't use the first time and hence didn't keep. 
So those values can only be recovered by rereading.

The closest I can think of still requires the use of a temporary file.
The script would do:
  set print "tempfile.dat'
  print "first data line"
  print "second data line"
    ...
  print "last data line"

  plot 'tempfile.dat' using 1:2, '' using 3:4, '' using 99:100

Re: multiple plot lines with plot '-'

[<pl...@pi...>]

On 05/16/12 06:56, sfeam (Ethan Merritt) wrote:
> On Monday, 14 May 2012, su...@pi... wrote:
>> On 05/14/12 18:27, Ethan A Merritt wrote:
>>>>>   So could another flag like plot '--' be used to indicate rereading of
>>>>>   the inline data rather than continuing at the next block of inline data ?
>>> This wouldn't work in the general case - you can't reread from a pipe.
>>
>> hmm. I hadn't thought of "inline" data actually being a pipe. Good point.
>>
>> perhaps that could be accepted as a limitation on reading from pipes.
>> It would not reduce the functionality that is already available to pipes.
>
> Up till now, the following have all been equivalent:
>      gnuplot file.in
>      cat file.in | gnuplot
>      cat file.in | gnuplot -
>      gnuplot -e 'load file.in'
>
> I think it would be a bad idea to introduce new syntax that breaks
> this equivalence.

I agree, I like that sort of symmetry.

Part of the problem here is that plot '-' does not work in quite the 
same way as plot datafile

plot '-','-'   plots two consecutive segments from stdin  whereas plot 
datafile , datafile  rereads the file and plots the same thing twice.

This is presumably because you can't reread a pipe but it means plot 
command is acting differently depending on the data source specified.


 >>Consider:
 >>   plot '-' using 1:2, '' using 3:4, '' using 99:100
 >> The first plot uses only columns 1 and 2; these values are stored
 >> internally and could be used again without rereading.  That's what the
  >> "refresh"  command does.  But the 2nd and 3rd plots use data values 
from
 >> columns we didn't use the first time and hence didn't keep.
 >> So those values can only be recovered by rereading.

 > Since this is all in one plot command , couldn't the whole line be
 > parsed to find all the columns to be required by all the using clauses?
 >
 > In fact isn't this already done? Are you implying that
 >
 >   plot datafile using 1:2, '' using 3:4, '' using 99:100
 >
 > actually reads the data file three times looking at different columns :?

 >> Yes.

So if I get this,

gnuplot>  plot datafile using 1:2, '' using 3:4

opens and reads same lines of the file twice. Filling two sets of slots


  gnuplot>  plot '-' using 1:2, '' using 3:4

reads consecutively from stdin , twice. Filling two sets of slots


It would be more efficient if the first version parsed the whole line to 
find all required columns and did one read. I do plots with 5 or 6 
lines, this could be a useful improvement.

Then the suggested '--' could work with no extra effort.
gnuplot>  plot '--' using 1:2, '' using 3:4


The asymmetrical behaviour of the existing plot '-'  would still need a 
different code path.




> If the previous work-around I suggested is too
> long-winded, requiring a "print" and matching quotes on each line,
> how about introducing an idea from perl:
>
> set print 'tempfile.dat'
> print<<  EOF
> 1st line of data
> 2nd line of data
> 3rd line of data
> EOF
>

Yes , I was looking for something like that yesterday. I think that 
would be a useful addition in a number of situations.

However, rewriting all the data in the current file to another 
superfluous copy just to get around the syntax seems a bit clunky.

regards, Peter.

> 	Ethan
>

Re: multiple plot lines with plot '-'

[Daniel J S. <dan...@ie...>]

On 05/16/2012 02:43 PM, pl...@pi... wrote:
> On 05/16/12 18:35, Daniel J Sebald wrote:
>> Perhaps there should be a mode switch, "use the same data" or "don't
>> reread". But I'm not sure how crucial that is.
>
>
> Isn't that what I was suggesting using plot '--' (double minus sign) ?

Sure, but it is slightly confusing because...

> If the read-in data is buffered, that would fix Ethan's worry about this
> not working the same from pipes.

Your previous email pointed out a problem with 'datafile' reread on ARM 
at a particular instant.  There would then need to be something 
analogous to '-' and '--' but for data files.  OR, the default for 
datafiles could be to buffer the data (no easy task given gnuplot wasn't 
built that way from the ground up).  But if the default for datafiles is 
to buffer data, then '--' becomes analogous to 'datafile' as the default 
behavior, and '-' becomes the special case.

With a mode switch, there is a bit less confusion, but that too has its 
drawbacks.

Dan

Re: multiple plot lines with plot '-'

[<pl...@pi...>]

On 05/17/12 19:56, Tait wrote:
>
> I was going to make a point about GB-sized data sets and memory usage
> of trying to cache such things, but Ethan helpfully already did so.
> Some of the data I work with is intentionally plotted in a raster
> format because plotting in a vector format creates file sizes that
> overflow memory and filesystem limits.
>
>> Allin's suggestion would likely be simple to do but could potentially be
>> very wasteful. With the increasing importance of mobile and embedded
>> devices this may not be too desirable
>
> I like Allin's suggestion best out of what I've seen suggested here. It
> doesn't demand a backward-incompatible change to syntax, does not need
> to penalize users with large data sets, and still allows small-dataset
> users to cache their data in memory.
>
>> As I understand it so far, data for each plot line in read in with a
>> separate reading of the input data source.
>
> This is fundamental and unavoidable, because the input to be read might
> depend on the data itself. The following is perfectly legal, for example,
> and I've even used it before:
>
> f(x,y,z)=<some function>
> g(x)=<some other function>
> plot 'datafile' using (column(f($1,$2,$3))):(column(g($4)), \
> 	'data2file' using ($1>$2 ? column(g($4)) : 1/0)

Hi Tait,

I think your example needs to be stripped down to the bare essentials of 
making the point you are trying to make. I'll have a guess about what 
you are trying to say, sorry if I miss the mark, and I think you are 
mistaken.

plot 'data2file' using ($1>$2 ? column(g($4)) : 1/0)

this plot line requires reading of columns 1,2 and 4 (all lines). When 
it comes to be processed the plotted points  will be evaluated and some 
of them will be NaN. All these data pairs (including the NaNs)  will be 
stored in slots for later reuse.

I also use this kind of construction frequently, I don't see that it has 
any bearing on what I suggested.


>
>> 1. unnecessary re-reading of input file
>
> The file is re-read, yes, but it's not unnecessary.
>
>> 2. possible plotting of different states of the data in one graph.
>
> I think any user would not be surprised that a file being modified while
> gnuplot is plotting would have undefined results. If this is a concern
> for your application, it is incumbent on you to cache or copy-on-write,
> or whatever works for your scenario.
>
>> 3. plot '-' using 1:2 , '' using 1:3	requires two separate datasets (or
>> duplication) to be supplied and thus has different behaviour to a
>> similar command using a data file.
>
> I don't understand this. The behavior of '-' is currently the same as
> for a data file. What you're suggesting is to make '-' act different than
> a normal input file would act.

No, you need to read up on how '-' works.  This has been covered in a 
number of posts in this thread.

  I can see the advantage of specifying a
> single data set inline somehow and then referring repeatedly to that one
> data set, but that seems like a new and different feature, not really an
> extension of "plot '-'".
>
>
regards.

Re: multiple plot lines with plot '-'

[Tait <gnu...@t4...>]

> > f(x,y,z)=<some function>
> > g(x)=<some other function>
> > plot 'datafile' using (column(f($1,$2,$3))):(column(g($4)), \
> > 	'data2file' using ($1>$2 ? column(g($4)) : 1/0)
>
> I think your example needs to be stripped down to the bare essentials of
> making the point you are trying to make. I'll have a guess about what
> you are trying to say, sorry if I miss the mark, and I think you are
> mistaken.

I was trying to make two separate points in that example, and you missed
the first -- and more important -- of them. To simplify:

plot 'datafile' using (column($1))

The column to be used for data is the column specified in the first
column of the data file. This is not and cannot be known until the file
(that column, at least) is actually read. The point of including the
functions is that we might not even know based on a single line of the
input what column we would need to preserve. The only way to know what
must be preserved is to actually evaluate the using conditions on the
input, see what results, and then reread the file for the next plot
because it, too, may have the exact same dereferencing behavior that
the first using statement had. And even more so, to the extent the first
using evaluation had side-effects that alter the meaning of the second
using statement, these side-effects cannot be anticipated ahead-of
time without actually doing the calculations demanded by the user.
These kinds of side effects are already suggested by demos like
http://gnuplot.sourceforge.net/demo_4.6/running_avg.html.

In short, the current behavior of gnuplot makes it impossible to
do a single-pass read of the data file. We could break the current
behavior, of course, but then we explicitly decrease the flexibility
we have for the sake of introducing other inconsistent and backward-
incompatible magic behavior.

At risk of repeating myself, caching may be worthwhile, but not in
the way and with the syntax you've suggested.

> >			 ... The behavior of '-' is currently the same as
> > for a data file. ...
>
> No, you need to read up on how '-' works.  This has been covered in a
> number of posts in this thread.

I know exactly how it works, and it works just like data files work.
I use and rely on this behavior. The behavior you want, and that
you (incorrectly, imo) describe as being the same as what's done for
data files is different. When you say plot 'file1', '', gnuplot does
not save 'file1' and rewind through this cache to find the data it
needs when plotting the ''. It plots 'file1', then when it sees ''
it reads 'file1' (again). When gnuplot plots '-', it reads the inline
data, and when it sees '' after, it reads the inline data again, not
a cached copy that it rewinds and replays. The fact that '-' might be
different on the first and second read is exactly the same as 'file1'
might be different on the first and second read. This describes the
same behavior for both 'file1' and '-'. What you're asking for is that
unlike the 'file1' case, '-' be handled differently, so that it saves
or caches the data and rewinds back and forth through this cache
instead of reading anew. This sort of special-case handling of '-'
might be a worthwhile option, but it's a separate feature, and one
that must be kept optional so as to not break backward compatibility
and important use cases for some users (like me).

Re: multiple plot lines with plot '-'

[<pl...@pi...>]

On 18/05/12 01:32, Tait wrote:
>>> f(x,y,z)=<some function>
>>> g(x)=<some other function>
>>> plot 'datafile' using (column(f($1,$2,$3))):(column(g($4)), \
>>> 	'data2file' using ($1>$2 ? column(g($4)) : 1/0)
>>
>> I think your example needs to be stripped down to the bare essentials of
>> making the point you are trying to make. I'll have a guess about what
>> you are trying to say, sorry if I miss the mark, and I think you are
>> mistaken.
>
> I was trying to make two separate points in that example, and you missed
> the first -- and more important -- of them. To simplify:
>
> plot 'datafile' using (column($1))
>

Yes, I had not thought of that trick. Nice.

One case where this could produce a change in behaviour is if a variable 
was assigned from the data in the first plot line and used to determine 
which columns were read later. This would prevent pre-emptive reading 
unless Allin's "read everything" solution was adopted.



> The column to be used for data is the column specified in the first
> column of the data file. This is not and cannot be known until the file
> (that column, at least) is actually read. The point of including the
> functions is that we might not even know based on a single line of the
> input what column we would need to preserve. The only way to know what
> must be preserved is to actually evaluate the using conditions on the
> input, see what results, and then reread the file for the next plot
> because it, too, may have the exact same dereferencing behavior that
> the first using statement had. And even more so, to the extent the first
> using evaluation had side-effects that alter the meaning of the second
> using statement, these side-effects cannot be anticipated ahead-of
> time without actually doing the calculations demanded by the user.
> These kinds of side effects are already suggested by demos like
> http://gnuplot.sourceforge.net/demo_4.6/running_avg.html.
>
> In short, the current behavior of gnuplot makes it impossible to
> do a single-pass read of the data file. We could break the current
> behavior, of course, but then we explicitly decrease the flexibility
> we have for the sake of introducing other inconsistent and backward-
> incompatible magic behavior.
>
> At risk of repeating myself, caching may be worthwhile, but not in
> the way and with the syntax you've suggested.
>
>>> 			... The behavior of '-' is currently the same as
>>> for a data file. ...
>>
>> No, you need to read up on how '-' works.  This has been covered in a
>> number of posts in this thread.
>
> I know exactly how it works, and it works just like data files work.
> I use and rely on this behavior. The behavior you want, and that
> you (incorrectly, imo) describe as being the same as what's done for
> data files is different. When you say plot 'file1', '', gnuplot does
> not save 'file1' and rewind through this cache to find the data it
> needs when plotting the ''. It plots 'file1', then when it sees ''
> it reads 'file1' (again). When gnuplot plots '-', it reads the inline
> data, and when it sees '' after, it reads the inline data again, not
> a cached copy that it rewinds and replays. The fact that '-' might be
> different on the first and second read is exactly the same as 'file1'
> might be different on the first and second read. This describes the
> same behavior for both 'file1' and '-'.

If you are suggesting that associating a file descriptor with a file; 
opening; reading; closing; reopening (with implies a seek(0) ) ; and 
rereading   is the same as opening ; reading upto 'e' marker *in the 
data* then continuing to read the same opened device,  I have to say 
your reasoning is getting a bit contrived.

I find it hard to see that as being the same behaviour.

What you're asking for is that
> unlike the 'file1' case, '-' be handled differently, so that it saves
> or caches the data and rewinds back and forth through this cache
> instead of reading anew. This sort of special-case handling of '-'
> might be a worthwhile option, but it's a separate feature, and one
> that must be kept optional so as to not break backward compatibility
> and important use cases for some users (like me).
>
>
open '-' is already a special case. That is the source of the "problem". 
However, I totally agree with you about backwards compatibility and it 
would need a pretty substantial problem for me to suggest breaking that.

Since this idea just looked like a nice improvement when I suggested it 
I don't see any pressing case. I'm always looking for speed improvements 
and rereading my data file six times seemed wasteful.

There is clearly a lot of divergent ideas of what would (not) be useful 
so I'm dropping the idea.

Thanks for all replies and comments, it's been interesting and has 
brought up some good ideas.

regards, Peter.

Re: multiple plot lines with plot '-'

[Ethan A M. <sf...@us...>]

On Friday, May 18, 2012 02:25:15 am pl...@pi... wrote:
> If you are suggesting that associating a file descriptor with a file; 
> opening; reading; closing; reopening (with implies a seek(0) ) ; and 
> rereading   is the same as opening ; reading upto 'e' marker *in the 
> data* then continuing to read the same opened device,  I have to say 
> your reasoning is getting a bit contrived.
> 
> I find it hard to see that as being the same behaviour.

Nevertheless, from the point of view of the program itself, or of the
programmers working on it, these cases are handled exactly the same way.
You can find the code near the top of the routine df_open() in datafile.c

In pseudocode:

   if (*filename == '-')
		data_fp = stdin;
   else if (*filename == '<')
		data_fp = fpopen( some piped command stored in filename );
   else 
		data_fp = fopen( filename );

Other routines then read line by line from data_fp as needed.
In two of the three cases, pipe and stdin, backing up to the beginning
in order to re-read is not possible. The code never uses fseek().

	Ethan

Re: multiple plot lines with plot '-'

[<pl...@pi...>]

On 05/18/12 22:49, Ethan A Merritt wrote:
> On Friday, May 18, 2012 02:25:15 am pl...@pi... wrote:
>> If you are suggesting that associating a file descriptor with a file;
>> opening; reading; closing; reopening (with implies a seek(0) ) ; and
>> rereading   is the same as opening ; reading upto 'e' marker *in the
>> data* then continuing to read the same opened device,  I have to say
>> your reasoning is getting a bit contrived.
>>
>> I find it hard to see that as being the same behaviour.
>
> Nevertheless, from the point of view of the program itself, or of the
> programmers working on it, these cases are handled exactly the same way.
> You can find the code near the top of the routine df_open() in datafile.c
>
> In pseudocode:
>
>     if (*filename == '-')
> 		data_fp = stdin;
>     else if (*filename == '<')
> 		data_fp = fpopen( some piped command stored in filename );
>     else
> 		data_fp = fopen( filename );
>
> Other routines then read line by line from data_fp as needed.
> In two of the three cases, pipe and stdin, backing up to the beginning
> in order to re-read is not possible. The code never uses fseek().
>
> 	Ethan
>

stdin is already open, the other two cases call an open function. 
Calling fopen implicitly calls seek(0) as I pointed out earlier.

How are these "the same" ?

regards.

Re: multiple plot lines with plot '-'

[Tait <gnu...@t4...>]

> If you are suggesting that associating a file descriptor with a file; 
> opening; reading; closing; reopening (with implies a seek(0) ) ; and 
> rereading   is the same as opening ; reading up to 'e' marker *in the 
> data* then continuing to read the same opened device,  I have to say 
> your reasoning is getting a bit contrived.
> 
> I find it hard to see that as being the same behavior.

Maybe we're drifting beyond the point of useful distinction. I mean in
terms of behavior the users see, plot consumes inputs. Each data set
given to plot is consumed by that plot. Plot 'file' opens and reads
file; it's consumed in generating the plot, and everything is reset
for the next plot, whether in the same plot command or in a
subsequent. For instance...
	plot 'file1'
	plot 'file2'
is the same -- for purposes of reading input -- as
	plot 'file1', 'file2'

And...
	plot '-'
	...
	e
	plot '-'
	...
	e
is the same, in terms of reading input, as
	plot '-', '-'
	...
	e
	...
	e

In both cases, the data is consumed by the plot command, and the
next plot command begins consuming a new set of data (even though
file1 and file2 may be the same, and thus consume the same data
multiple times).

That's what I mean when I say the behavior of '-' and 'file'
are the same. I think I see the model you suggest, though,
which is more like saying the lines between plot and e create
a named record called '-', which would then be treated the same
as a named record called 'file1' and referenced multiple times.
If those lines could be given a name explicitly -- other than
'-' -- and could exist outside of an appendage to a plot
command, then I think that model would make more sense. And
that's a little closer to what Allin was suggesting, which is
why I liked that idea.

Re: multiple plot lines with plot '-'

[<pl...@pi...>]

On 05/19/12 00:19, Tait wrote:
>> If you are suggesting that associating a file descriptor with a file;
>> opening; reading; closing; reopening (with implies a seek(0) ) ; and
>> rereading   is the same as opening ; reading up to 'e' marker *in the
>> data* then continuing to read the same opened device,  I have to say
>> your reasoning is getting a bit contrived.
>>
>> I find it hard to see that as being the same behavior.
>
> Maybe we're drifting beyond the point of useful distinction. I mean in
> terms of behavior the users see, plot consumes inputs. Each data set
> given to plot is consumed by that plot. Plot 'file' opens and reads
> file; it's consumed in generating the plot, and everything is reset
> for the next plot, whether in the same plot command or in a
> subsequent. For instance...
> 	plot 'file1'
> 	plot 'file2'
> is the same -- for purposes of reading input -- as
> 	plot 'file1', 'file2'
>
> And...
> 	plot '-'
> 	...
> 	e
> 	plot '-'
> 	...
> 	e
> is the same, in terms of reading input, as
> 	plot '-', '-'
> 	...
> 	e
> 	...
> 	e
>
> In both cases, the data is consumed by the plot command, and the
> next plot command begins consuming a new set of data (even though
> file1 and file2 may be the same, and thus consume the same data
> multiple times).

A test of whether the behaviour is the same would be to use a variable 
to define the data source:

datasrc='file1.dat'
plot datasrc, datasrc


datasrc='-'
plot datasrc, datasrc

It would be nice if gnuplot was data source agnostic in that way, but I 
can see why it was thought to be more useful to treat '-' as a special case.

That's the way it was done and there is not sufficient problems to 
suggest breaking compatibility.


regards. Peter.



>
> That's what I mean when I say the behavior of '-' and 'file'
> are the same. I think I see the model you suggest, though,
> which is more like saying the lines between plot and e create
> a named record called '-', which would then be treated the same
> as a named record called 'file1' and referenced multiple times.
> If those lines could be given a name explicitly -- other than
> '-' -- and could exist outside of an appendage to a plot
> command, then I think that model would make more sense. And
> that's a little closer to what Allin was suggesting, which is
> why I liked that idea.
>
>

Re: multiple plot lines with plot '-'

[sfeam (E. Merritt) <eam...@gm...>]

On Saturday, 19 May 2012, pl...@pi... wrote:
> A test of whether the behaviour is the same would be to use a variable 
> to define the data source:
> 
> datasrc='file1.dat'
> plot datasrc, datasrc
> 
> datasrc='-'
> plot datasrc, datasrc
> 
> It would be nice if gnuplot was data source agnostic in that way, but I 
> can see why it was thought to be more useful to treat '-' as a special case.

You should probably add to the mix:

 datasrc="< date +%N"

All three of these work.  Does that make the program "source agnostic"?  
The first will probably produce the same result twice, 
though only if the file contents are unchanged.  
The second may or may not produce the same result twice,
depending on what is in the input stream.
The third definitely will not find the same number twice.   

It's fair enough to say that the second option, '-', 
doesn't do what you want.  But I don't see the logic for
calling it a special case.  It's one of several options
for defining a data source.   Use it only if it is suitable.

Empty quotes '' mean "use the previous data _source_",
not "use the previous _data_".

Anyhow, I would find it helpful to hear more about the use cases
where reading data from a separate file is not suitable.
Is it just a concern that the script and the data will get
separated?  Are there technical issues about multiple data files?
Speed?  Would it solve the problem if you could define a separate
datablock in the same input file (separate from the 'plot' command
line)?

Re: multiple plot lines with plot '-'

[Allin C. <cot...@wf...>]

On Sat, 19 May 2012, sfeam (Ethan Merritt) wrote:

> Anyhow, I would find it helpful to hear more about the use cases
> where reading data from a separate file is not suitable.
> Is it just a concern that the script and the data will get
> separated?

In some contexts I think it's fine to have the gnuplot spec and the 
data in separate files, but in other contexts it's preferable to 
have them integrated (in case the data get deleted, moved or 
inadvertently modified).

> Would it solve the problem if you could define a separate 
> datablock in the same input file (separate from the 'plot' command 
> line)?

For me, yes, the ability to define a reusable data block before 
the "plot" lines would be a nice enhancement.

Allin Cottrell

Re: multiple plot lines with plot '-'

[Tait <gnu...@t4...>]

> ...
> Anyhow, I would find it helpful to hear more about the use cases
> where reading data from a separate file is not suitable.

For me, the utility is in convenience, not necessity. If I want to
embed a file in a presentation or email a file in a way that lets the
receiver recreate the graph, it's just easier to have a single file
instead of needing to manipulate multiple files, or combine them into
a container format (e.g. *.zip) then undo that on the receiving side.

Re: multiple plot lines with plot '-'

[<pl...@pi...>]

On 05/19/12 17:37, sfeam (Ethan Merritt) wrote:
> On Saturday, 19 May 2012, pl...@pi... wrote:
>> A test of whether the behaviour is the same would be to use a variable
>> to define the data source:
>>
>> datasrc='file1.dat'
>> plot datasrc, datasrc
>>
>> datasrc='-'
>> plot datasrc, datasrc
>>
>> It would be nice if gnuplot was data source agnostic in that way, but I
>> can see why it was thought to be more useful to treat '-' as a special case.
>
> You should probably add to the mix:
>
>   datasrc="<  date +%N"
>
> All three of these work.  Does that make the program "source agnostic"?

Clearly not , if it does not produce the same result.


> The first will probably produce the same result twice,
> though only if the file contents are unchanged.
> The second may or may not produce the same result twice,
> depending on what is in the input stream.

The second case will 100% _not_ produce the same result , it is plotting 
two different sets of data. Even if the numbers are the same, it is 
plotting different data sets. The behaviour is different.

> The third definitely will not find the same number twice.
>
> It's fair enough to say that the second option, '-',
> doesn't do what you want.

Which is why I was suggesting a new syntax with more compatible 
behaviour. I've now dropped that idea having seen the divergence of 
requirements.

But I don't see the logic for
> calling it a special case.  It's one of several options
> for defining a data source.   Use it only if it is suitable.
>
> Empty quotes '' mean "use the previous data _source_",
> not "use the previous _data_".
>
> Anyhow, I would find it helpful to hear more about the use cases
> where reading data from a separate file is not suitable.
> Is it just a concern that the script and the data will get
> separated?

My initial motivation here was that I have a data file with several 
columns of data. To plot it I need a few lines of gnuplot script to set 
axes labels etc. but it can be done in a couple of lines.

It's too complex to be typed in every time but it is a pain to maintain 
a separate script and data file for such a trivial situation. More than 
one file becomes an 'archive' and requires version matching, etc. A 
single file is much simpler to maintain reliably.

I have gnuplot scripts that run to two pages, there it warrants being a 
file in it's own right, for two lines it's a headache.

The work around you suggested , writing the data out to a second, 
temporary file, would work if we had the  <<EOF perl-like syntax. ( I 
still like that syntax idea. ) But making a file dupe itself seems a bit 
of a  kludge.

Are there technical issues about multiple data files?
> Speed?  Would it solve the problem if you could define a separate
> datablock in the same input file (separate from the 'plot' command
> line)?
>

Yes, maybe defining the data may provide a solution to the case I 
outlined above and may fit in with what Allin is suggesting.

Since the other three input sources produce incompatible results, 
perhaps this one could provide the cached , matrix capabilities that 
Allin was looking for.

Peter.

Re: multiple plot lines with plot '-'

[Juhász P. <pet...@gm...>]

On Sun, 2012-05-20 at 10:36 +0200, pl...@pi... wrote:
> On 05/19/12 17:37, sfeam (Ethan Merritt) wrote:
> > On Saturday, 19 May 2012, pl...@pi... wrote:

> > Speed?  Would it solve the problem if you could define a separate
> > datablock in the same input file (separate from the 'plot' command
> > line)?
> >
> 
> Yes, maybe defining the data may provide a solution to the case I 
> outlined above and may fit in with what Allin is suggesting.
> 
> Since the other three input sources produce incompatible results, 
> perhaps this one could provide the cached , matrix capabilities that 
> Allin was looking for.
> 
> Peter.

I haven't followed the discussion, but to this specific point I have a
suggestion:

In Perl there is a special __DATA__ marker that you can use to tell the
interpreter that anything coming after it is data, not code; and there
is a special filehandle to access this data. Example:

#!/usr/bin/perl

while (my $line = <DATA>) {
	do_something_with($line);
}

__DATA__
foo
bar
baz


We could adapt this kind of functionality to gnuplot, e.g. like this:

#!gnuplot

plot GPVAL_DATA using 1:2

__DATA__
1 10
2 13
3 17

I'd envision this working by GPVAL_DATA behaving as a normal string
variable most of the time (with the value "__DATA__"), except when it is
used in a plot command, it would magically refer to the inline data
block. The GPVAL prefix would ensure (and signify) that it is a
read-only special variable.

This would 
1) provide the data-inlined-in-gnuplot-scripts functionality described
by some in this thread,
2) be familiar to at least some of the users,
3) be relatively easy to implement within the current framework of
gnuplot, without the need of having to (re)write a lot of code or
introduce radically new syntax.

How would this work internally:
if the magical variable (or magical file name) appears in a plot
command, the program would open the script file, scan for the magical
__DATA__ string, and from then on use the usual datafile processing
routines.

I'm not sure about the cases where the commands themselves are streamed
to gnuplot (gnuplot -e 'plot ...'; echo 'plot ...' | gnuplot -; ...),
perhaps this functionality does not make sense in those cases, and
besides, "plot '-'" is already there for those cases.

Péter Juhász

Re: multiple plot lines with plot '-'

[Allin C. <cot...@wf...>]

On Mon, 14 May 2012, Ethan A Merritt wrote:

> On Monday, May 14, 2012 08:12:25 am pl...@pi... wrote:
>> On 05/14/12 14:52, Allin Cottrell wrote:
>>> On Mon, 14 May 2012, Mojca Miklavec wrote:
>>>
>>>> Simply use empty quotation marks:
>>>> plot  "fast.log" using 1:2 t "systolic" w l,  "" using ...
>>>
>>> This doesn't work in the context plotter specified, namely when you
>>> replace the data file "fast.log" with inline data in the plot file,
>>> using "plot '-'".
>>>
>>> I too would be interested to know if there's any way to avoid repeating
>>> the inline data when plotting multiple lines. [...]
>>
>> unless someone points out a trick we're missing perhaps the plot '-'
>> feature could be extended to act in same way as a named file.
>
> It _does_ act in the same way as a named file.  The empty quotes indicate
> that the previous source of data should be read again.

With the source in this case being stdin, which has "moved on" 
by the time you attempt a second read, so you come up 
empty-handed. I see your point. On the other hand, with the 
plot-file syntax

plot '-' using ...
<inline-data>
e

it's sort of a fiction that the source is stdin, isn't it? 
(Albeit a well established fiction.) I wonder if it would be 
worth adding another "dummy" input, something like

plot inline using ...
<inline-data>
e

where the inline data would be read into an array and could be
reused at will.

> The closest I can think of still requires the use of a temporary file.
> The script would do:
>  set print "tempfile.dat'
>  print "first data line"
>  print "second data line"
>    ...
>  print "last data line"
>
>  plot 'tempfile.dat' using 1:2, '' using 3:4, '' using 99:100

Ah, that's clever. Not quite as nice as "inline" (if it were 
possible) but it serves the purpose of storing the data and 
the plot spec in a single file.

Allin Cottrell

Re: multiple plot lines with plot '-'

[<pl...@pi...>]

On 05/14/12 18:27, Ethan A Merritt wrote:
> The next obvious thought is "can't we remember the values we just read
> and re-use them if necessary?". But that doesn't work either.
> Consider:
>     plot '-' using 1:2, '' using 3:4, '' using 99:100
> The first plot uses only columns 1 and 2; these values are stored
> internally and could be used again without rereading.  That's what the
> "refresh"  command does.  But the 2nd and 3rd plots use data values from
> columns we didn't use the first time and hence didn't keep.
> So those values can only be recovered by rereading.

Since this is all in one plot command , couldn't the whole line be 
parsed to find all the columns to be required by all the using clauses?

In fact isn't this already done? Are you implying that


  plot datafile using 1:2, '' using 3:4, '' using 99:100

actually reads the data file three times looking at different columns :?

thx, Peter.

Re: multiple plot lines with plot '-'

[Ethan A M. <sf...@us...>]

On Monday, May 14, 2012 10:19:01 am pl...@pi... wrote:
> On 05/14/12 18:27, Ethan A Merritt wrote:
> > The next obvious thought is "can't we remember the values we just read
> > and re-use them if necessary?". But that doesn't work either.
> > Consider:
> >     plot '-' using 1:2, '' using 3:4, '' using 99:100
> > The first plot uses only columns 1 and 2; these values are stored
> > internally and could be used again without rereading.  That's what the
> > "refresh"  command does.  But the 2nd and 3rd plots use data values from
> > columns we didn't use the first time and hence didn't keep.
> > So those values can only be recovered by rereading.
> 
> Since this is all in one plot command , couldn't the whole line be 
> parsed to find all the columns to be required by all the using clauses?
> 
> In fact isn't this already done? Are you implying that
>  
>   plot datafile using 1:2, '' using 3:4, '' using 99:100
> 
> actually reads the data file three times looking at different columns :?

Yes. 

The internal structures that hold data reserve 7 slots for each input
data point.  These 7 slots are just barely enough to hold 
x/y/width/xerr/yerr/color/pointsize/etc of the various plot modes.
There is no mechanism or structure to store an arbitrary number of
additional raw data columns for each line of input.

	Ethan

Re: multiple plot lines with plot '-'

[Hans-Bernhard B. <HBB...@t-...>]

On 14.05.2012 19:13, Allin Cottrell wrote:

> On the other hand, with the plot-file syntax

> plot '-' using ...
> <inline-data>
> e

> it's sort of a fiction that the source is stdin, isn't it?

No.  Because that's not "plot file syntax".  That's gnuplot command 
syntax, which can and quite frequently will be entered on stdin --- 
either by actually typing it in, or copy-pasting into a console session, 
or by stdin being redirected from a file, or from some other program.

Re: multiple plot lines with plot '-'

[Allin C. <cot...@wf...>]

On Mon, 14 May 2012, Hans-Bernhard Bröker wrote:

> On 14.05.2012 19:13, Allin Cottrell wrote:
>
>> On the other hand, with the plot-file syntax
>
>> plot '-' using ...
>> <inline-data>
>> e
>
>> it's sort of a fiction that the source is stdin, isn't it?
>
> No.  Because that's not "plot file syntax".  That's gnuplot command syntax, 
> which can and quite frequently will be entered on stdin --- either by 
> actually typing it in, or copy-pasting into a console session, or by stdin 
> being redirected from a file, or from some other program.

Insofar as the snippet I posted is actually entered on stdin, or by 
the other methods you cite, then Yes, the "stdin" quality of the 
data is not in question. But plotter and I were talking about the 
use of this syntax in an integrated plot file, and in that context 
the "stdin-ness" is a fiction; the data are supplied "as if" via
stdin.

Allin Cottrell