gnuplot-beta — Developers and beta testers of new gnuplot code

Re: comment handling

[Ethan A M. <sf...@us...>]

On Wednesday, May 23, 2012 01:50:43 pm pl...@pi... wrote:
> Hi,
> 
> I am getting some inconsistent behaviour from comment handling.
> 
> gnuplot> set timefmt "%d.%m.%y:%H:%M"; set xdata time; set ylab "mm Hg" 
> ; set grid
> 
> 21.05.12:23:20  96 72  65           # wetday , not walking
> 22.05.12:09:00                      # rise 9am 81kg
> 23.05.12:04:30 120 87  49           # work late,bed
> 
> 
> help says:
> 
>   Comments are supported as follows: a `#` may appear in most places in 
> a line
>   and `gnuplot` will ignore the rest of the line.  It will not have this 
> effect
>   inside quotes, inside numbers (including complex numbers), inside command
>   substitutions, etc.  In short, it works anywhere it makes sense to work.

That section is refering to gnuplot command lines, not to data files.
The very next line says "See also `set datafile commentschars` 
for specifying comment characters in data files".

gnuplot> help set datafile comment
 The `set datafile commentschars` tells `gnuplot` what characters are used in a
 data file to begin comment lines. If the first non-blank character on a line is
 one of the specified characters then the rest of the input line is ignored.
 Default value of the string is "#!" on VMS and "#" otherwise.



> 
> However the middle line with the missing data tries to plot the comment !
> 
> I'm using a plot that is essentially:
> 
> plot datafile using 1:2, '' using 1:3 , '' using 1:4
> 
> the first line joins 96 and120 with a straight line.
> the second gets a break (this is what I expected for all three)
> the last line tries to plot "9am" and so I get a line 65 9 49
> 
> All this seems to indicate that it is parsing "#" and "rise" as NaN 
> rather than seeing the comment as a comment.
> 
> Is this expected? If so the help needs to be changed.
> 
> regards, Peter.
> 
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and 
> threat landscape has changed and how IT managers can respond. Discussions 
> will include endpoint security, mobile security and the latest in malware 
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> gnuplot-beta mailing list
> gnu...@li...
> https://lists.sourceforge.net/lists/listinfo/gnuplot-beta
> 

comment handling

[<pl...@pi...>]

Hi,

I am getting some inconsistent behaviour from comment handling.

gnuplot> set timefmt "%d.%m.%y:%H:%M"; set xdata time; set ylab "mm Hg" 
; set grid

21.05.12:23:20  96 72  65           # wetday , not walking
22.05.12:09:00                      # rise 9am 81kg
23.05.12:04:30 120 87  49           # work late,bed


help says:

  Comments are supported as follows: a `#` may appear in most places in 
a line
  and `gnuplot` will ignore the rest of the line.  It will not have this 
effect
  inside quotes, inside numbers (including complex numbers), inside command
  substitutions, etc.  In short, it works anywhere it makes sense to work.


However the middle line with the missing data tries to plot the comment !

I'm using a plot that is essentially:

plot datafile using 1:2, '' using 1:3 , '' using 1:4

the first line joins 96 and120 with a straight line.
the second gets a break (this is what I expected for all three)
the last line tries to plot "9am" and so I get a line 65 9 49

All this seems to indicate that it is parsing "#" and "rise" as NaN 
rather than seeing the comment as a comment.

Is this expected? If so the help needs to be changed.

regards, Peter.

gnuplot and octave for android

[Mojca M. <moj...@gm...>]

There is a project/funding proposal on Kickstarter which might be
interesting to the gnuplot community - getting gnuplot & octave with
some GUI working on Android:
     http://www.kickstarter.com/projects/6438588/sombreros-for-the-android-world

(Even though I'm afraid that the goal was set "a bit" to high to
actually get any funding at all.)

Mojca

Re: multiple plot lines with plot '-'

[Juhász P. <pet...@gm...>]

On Sun, 2012-05-20 at 10:36 +0200, pl...@pi... wrote:
> On 05/19/12 17:37, sfeam (Ethan Merritt) wrote:
> > On Saturday, 19 May 2012, pl...@pi... wrote:

> > Speed?  Would it solve the problem if you could define a separate
> > datablock in the same input file (separate from the 'plot' command
> > line)?
> >
> 
> Yes, maybe defining the data may provide a solution to the case I 
> outlined above and may fit in with what Allin is suggesting.
> 
> Since the other three input sources produce incompatible results, 
> perhaps this one could provide the cached , matrix capabilities that 
> Allin was looking for.
> 
> Peter.

I haven't followed the discussion, but to this specific point I have a
suggestion:

In Perl there is a special __DATA__ marker that you can use to tell the
interpreter that anything coming after it is data, not code; and there
is a special filehandle to access this data. Example:

#!/usr/bin/perl

while (my $line = <DATA>) {
	do_something_with($line);
}

__DATA__
foo
bar
baz


We could adapt this kind of functionality to gnuplot, e.g. like this:

#!gnuplot

plot GPVAL_DATA using 1:2

__DATA__
1 10
2 13
3 17

I'd envision this working by GPVAL_DATA behaving as a normal string
variable most of the time (with the value "__DATA__"), except when it is
used in a plot command, it would magically refer to the inline data
block. The GPVAL prefix would ensure (and signify) that it is a
read-only special variable.

This would 
1) provide the data-inlined-in-gnuplot-scripts functionality described
by some in this thread,
2) be familiar to at least some of the users,
3) be relatively easy to implement within the current framework of
gnuplot, without the need of having to (re)write a lot of code or
introduce radically new syntax.

How would this work internally:
if the magical variable (or magical file name) appears in a plot
command, the program would open the script file, scan for the magical
__DATA__ string, and from then on use the usual datafile processing
routines.

I'm not sure about the cases where the commands themselves are streamed
to gnuplot (gnuplot -e 'plot ...'; echo 'plot ...' | gnuplot -; ...),
perhaps this functionality does not make sense in those cases, and
besides, "plot '-'" is already there for those cases.

Péter Juhász

Re: multiple plot lines with plot '-'

[<pl...@pi...>]

On 05/19/12 17:37, sfeam (Ethan Merritt) wrote:
> On Saturday, 19 May 2012, pl...@pi... wrote:
>> A test of whether the behaviour is the same would be to use a variable
>> to define the data source:
>>
>> datasrc='file1.dat'
>> plot datasrc, datasrc
>>
>> datasrc='-'
>> plot datasrc, datasrc
>>
>> It would be nice if gnuplot was data source agnostic in that way, but I
>> can see why it was thought to be more useful to treat '-' as a special case.
>
> You should probably add to the mix:
>
>   datasrc="<  date +%N"
>
> All three of these work.  Does that make the program "source agnostic"?

Clearly not , if it does not produce the same result.


> The first will probably produce the same result twice,
> though only if the file contents are unchanged.
> The second may or may not produce the same result twice,
> depending on what is in the input stream.

The second case will 100% _not_ produce the same result , it is plotting 
two different sets of data. Even if the numbers are the same, it is 
plotting different data sets. The behaviour is different.

> The third definitely will not find the same number twice.
>
> It's fair enough to say that the second option, '-',
> doesn't do what you want.

Which is why I was suggesting a new syntax with more compatible 
behaviour. I've now dropped that idea having seen the divergence of 
requirements.

But I don't see the logic for
> calling it a special case.  It's one of several options
> for defining a data source.   Use it only if it is suitable.
>
> Empty quotes '' mean "use the previous data _source_",
> not "use the previous _data_".
>
> Anyhow, I would find it helpful to hear more about the use cases
> where reading data from a separate file is not suitable.
> Is it just a concern that the script and the data will get
> separated?

My initial motivation here was that I have a data file with several 
columns of data. To plot it I need a few lines of gnuplot script to set 
axes labels etc. but it can be done in a couple of lines.

It's too complex to be typed in every time but it is a pain to maintain 
a separate script and data file for such a trivial situation. More than 
one file becomes an 'archive' and requires version matching, etc. A 
single file is much simpler to maintain reliably.

I have gnuplot scripts that run to two pages, there it warrants being a 
file in it's own right, for two lines it's a headache.

The work around you suggested , writing the data out to a second, 
temporary file, would work if we had the  <<EOF perl-like syntax. ( I 
still like that syntax idea. ) But making a file dupe itself seems a bit 
of a  kludge.

Are there technical issues about multiple data files?
> Speed?  Would it solve the problem if you could define a separate
> datablock in the same input file (separate from the 'plot' command
> line)?
>

Yes, maybe defining the data may provide a solution to the case I 
outlined above and may fit in with what Allin is suggesting.

Since the other three input sources produce incompatible results, 
perhaps this one could provide the cached , matrix capabilities that 
Allin was looking for.

Peter.

Re: multiple plot lines with plot '-'

[Tait <gnu...@t4...>]

> ...
> Anyhow, I would find it helpful to hear more about the use cases
> where reading data from a separate file is not suitable.

For me, the utility is in convenience, not necessity. If I want to
embed a file in a presentation or email a file in a way that lets the
receiver recreate the graph, it's just easier to have a single file
instead of needing to manipulate multiple files, or combine them into
a container format (e.g. *.zip) then undo that on the receiving side.

Re: multiple plot lines with plot '-'

[Allin C. <cot...@wf...>]

On Sat, 19 May 2012, sfeam (Ethan Merritt) wrote:

> Anyhow, I would find it helpful to hear more about the use cases
> where reading data from a separate file is not suitable.
> Is it just a concern that the script and the data will get
> separated?

In some contexts I think it's fine to have the gnuplot spec and the 
data in separate files, but in other contexts it's preferable to 
have them integrated (in case the data get deleted, moved or 
inadvertently modified).

> Would it solve the problem if you could define a separate 
> datablock in the same input file (separate from the 'plot' command 
> line)?

For me, yes, the ability to define a reusable data block before 
the "plot" lines would be a nice enhancement.

Allin Cottrell

Re: multiple plot lines with plot '-'

[sfeam (E. Merritt) <eam...@gm...>]

On Saturday, 19 May 2012, pl...@pi... wrote:
> A test of whether the behaviour is the same would be to use a variable 
> to define the data source:
> 
> datasrc='file1.dat'
> plot datasrc, datasrc
> 
> datasrc='-'
> plot datasrc, datasrc
> 
> It would be nice if gnuplot was data source agnostic in that way, but I 
> can see why it was thought to be more useful to treat '-' as a special case.

You should probably add to the mix:

 datasrc="< date +%N"

All three of these work.  Does that make the program "source agnostic"?  
The first will probably produce the same result twice, 
though only if the file contents are unchanged.  
The second may or may not produce the same result twice,
depending on what is in the input stream.
The third definitely will not find the same number twice.   

It's fair enough to say that the second option, '-', 
doesn't do what you want.  But I don't see the logic for
calling it a special case.  It's one of several options
for defining a data source.   Use it only if it is suitable.

Empty quotes '' mean "use the previous data _source_",
not "use the previous _data_".

Anyhow, I would find it helpful to hear more about the use cases
where reading data from a separate file is not suitable.
Is it just a concern that the script and the data will get
separated?  Are there technical issues about multiple data files?
Speed?  Would it solve the problem if you could define a separate
datablock in the same input file (separate from the 'plot' command
line)?

Re: multiple plot lines with plot '-'

[<pl...@pi...>]

On 05/19/12 00:19, Tait wrote:
>> If you are suggesting that associating a file descriptor with a file;
>> opening; reading; closing; reopening (with implies a seek(0) ) ; and
>> rereading   is the same as opening ; reading up to 'e' marker *in the
>> data* then continuing to read the same opened device,  I have to say
>> your reasoning is getting a bit contrived.
>>
>> I find it hard to see that as being the same behavior.
>
> Maybe we're drifting beyond the point of useful distinction. I mean in
> terms of behavior the users see, plot consumes inputs. Each data set
> given to plot is consumed by that plot. Plot 'file' opens and reads
> file; it's consumed in generating the plot, and everything is reset
> for the next plot, whether in the same plot command or in a
> subsequent. For instance...
> 	plot 'file1'
> 	plot 'file2'
> is the same -- for purposes of reading input -- as
> 	plot 'file1', 'file2'
>
> And...
> 	plot '-'
> 	...
> 	e
> 	plot '-'
> 	...
> 	e
> is the same, in terms of reading input, as
> 	plot '-', '-'
> 	...
> 	e
> 	...
> 	e
>
> In both cases, the data is consumed by the plot command, and the
> next plot command begins consuming a new set of data (even though
> file1 and file2 may be the same, and thus consume the same data
> multiple times).

A test of whether the behaviour is the same would be to use a variable 
to define the data source:

datasrc='file1.dat'
plot datasrc, datasrc


datasrc='-'
plot datasrc, datasrc

It would be nice if gnuplot was data source agnostic in that way, but I 
can see why it was thought to be more useful to treat '-' as a special case.

That's the way it was done and there is not sufficient problems to 
suggest breaking compatibility.


regards. Peter.



>
> That's what I mean when I say the behavior of '-' and 'file'
> are the same. I think I see the model you suggest, though,
> which is more like saying the lines between plot and e create
> a named record called '-', which would then be treated the same
> as a named record called 'file1' and referenced multiple times.
> If those lines could be given a name explicitly -- other than
> '-' -- and could exist outside of an appendage to a plot
> command, then I think that model would make more sense. And
> that's a little closer to what Allin was suggesting, which is
> why I liked that idea.
>
>

Re: multiple plot lines with plot '-'

[Tait <gnu...@t4...>]

> If you are suggesting that associating a file descriptor with a file; 
> opening; reading; closing; reopening (with implies a seek(0) ) ; and 
> rereading   is the same as opening ; reading up to 'e' marker *in the 
> data* then continuing to read the same opened device,  I have to say 
> your reasoning is getting a bit contrived.
> 
> I find it hard to see that as being the same behavior.

Maybe we're drifting beyond the point of useful distinction. I mean in
terms of behavior the users see, plot consumes inputs. Each data set
given to plot is consumed by that plot. Plot 'file' opens and reads
file; it's consumed in generating the plot, and everything is reset
for the next plot, whether in the same plot command or in a
subsequent. For instance...
	plot 'file1'
	plot 'file2'
is the same -- for purposes of reading input -- as
	plot 'file1', 'file2'

And...
	plot '-'
	...
	e
	plot '-'
	...
	e
is the same, in terms of reading input, as
	plot '-', '-'
	...
	e
	...
	e

In both cases, the data is consumed by the plot command, and the
next plot command begins consuming a new set of data (even though
file1 and file2 may be the same, and thus consume the same data
multiple times).

That's what I mean when I say the behavior of '-' and 'file'
are the same. I think I see the model you suggest, though,
which is more like saying the lines between plot and e create
a named record called '-', which would then be treated the same
as a named record called 'file1' and referenced multiple times.
If those lines could be given a name explicitly -- other than
'-' -- and could exist outside of an appendage to a plot
command, then I think that model would make more sense. And
that's a little closer to what Allin was suggesting, which is
why I liked that idea.

Re: multiple plot lines with plot '-'

[<pl...@pi...>]

On 05/18/12 22:49, Ethan A Merritt wrote:
> On Friday, May 18, 2012 02:25:15 am pl...@pi... wrote:
>> If you are suggesting that associating a file descriptor with a file;
>> opening; reading; closing; reopening (with implies a seek(0) ) ; and
>> rereading   is the same as opening ; reading upto 'e' marker *in the
>> data* then continuing to read the same opened device,  I have to say
>> your reasoning is getting a bit contrived.
>>
>> I find it hard to see that as being the same behaviour.
>
> Nevertheless, from the point of view of the program itself, or of the
> programmers working on it, these cases are handled exactly the same way.
> You can find the code near the top of the routine df_open() in datafile.c
>
> In pseudocode:
>
>     if (*filename == '-')
> 		data_fp = stdin;
>     else if (*filename == '<')
> 		data_fp = fpopen( some piped command stored in filename );
>     else
> 		data_fp = fopen( filename );
>
> Other routines then read line by line from data_fp as needed.
> In two of the three cases, pipe and stdin, backing up to the beginning
> in order to re-read is not possible. The code never uses fseek().
>
> 	Ethan
>

stdin is already open, the other two cases call an open function. 
Calling fopen implicitly calls seek(0) as I pointed out earlier.

How are these "the same" ?

regards.

Re: multiple plot lines with plot '-'

[Ethan A M. <sf...@us...>]

On Friday, May 18, 2012 02:25:15 am pl...@pi... wrote:
> If you are suggesting that associating a file descriptor with a file; 
> opening; reading; closing; reopening (with implies a seek(0) ) ; and 
> rereading   is the same as opening ; reading upto 'e' marker *in the 
> data* then continuing to read the same opened device,  I have to say 
> your reasoning is getting a bit contrived.
> 
> I find it hard to see that as being the same behaviour.

Nevertheless, from the point of view of the program itself, or of the
programmers working on it, these cases are handled exactly the same way.
You can find the code near the top of the routine df_open() in datafile.c

In pseudocode:

   if (*filename == '-')
		data_fp = stdin;
   else if (*filename == '<')
		data_fp = fpopen( some piped command stored in filename );
   else 
		data_fp = fopen( filename );

Other routines then read line by line from data_fp as needed.
In two of the three cases, pipe and stdin, backing up to the beginning
in order to re-read is not possible. The code never uses fseek().

	Ethan

Re: multiple plot lines with plot '-'

[<pl...@pi...>]

On 18/05/12 01:32, Tait wrote:
>>> f(x,y,z)=<some function>
>>> g(x)=<some other function>
>>> plot 'datafile' using (column(f($1,$2,$3))):(column(g($4)), \
>>> 	'data2file' using ($1>$2 ? column(g($4)) : 1/0)
>>
>> I think your example needs to be stripped down to the bare essentials of
>> making the point you are trying to make. I'll have a guess about what
>> you are trying to say, sorry if I miss the mark, and I think you are
>> mistaken.
>
> I was trying to make two separate points in that example, and you missed
> the first -- and more important -- of them. To simplify:
>
> plot 'datafile' using (column($1))
>

Yes, I had not thought of that trick. Nice.

One case where this could produce a change in behaviour is if a variable 
was assigned from the data in the first plot line and used to determine 
which columns were read later. This would prevent pre-emptive reading 
unless Allin's "read everything" solution was adopted.



> The column to be used for data is the column specified in the first
> column of the data file. This is not and cannot be known until the file
> (that column, at least) is actually read. The point of including the
> functions is that we might not even know based on a single line of the
> input what column we would need to preserve. The only way to know what
> must be preserved is to actually evaluate the using conditions on the
> input, see what results, and then reread the file for the next plot
> because it, too, may have the exact same dereferencing behavior that
> the first using statement had. And even more so, to the extent the first
> using evaluation had side-effects that alter the meaning of the second
> using statement, these side-effects cannot be anticipated ahead-of
> time without actually doing the calculations demanded by the user.
> These kinds of side effects are already suggested by demos like
> http://gnuplot.sourceforge.net/demo_4.6/running_avg.html.
>
> In short, the current behavior of gnuplot makes it impossible to
> do a single-pass read of the data file. We could break the current
> behavior, of course, but then we explicitly decrease the flexibility
> we have for the sake of introducing other inconsistent and backward-
> incompatible magic behavior.
>
> At risk of repeating myself, caching may be worthwhile, but not in
> the way and with the syntax you've suggested.
>
>>> 			... The behavior of '-' is currently the same as
>>> for a data file. ...
>>
>> No, you need to read up on how '-' works.  This has been covered in a
>> number of posts in this thread.
>
> I know exactly how it works, and it works just like data files work.
> I use and rely on this behavior. The behavior you want, and that
> you (incorrectly, imo) describe as being the same as what's done for
> data files is different. When you say plot 'file1', '', gnuplot does
> not save 'file1' and rewind through this cache to find the data it
> needs when plotting the ''. It plots 'file1', then when it sees ''
> it reads 'file1' (again). When gnuplot plots '-', it reads the inline
> data, and when it sees '' after, it reads the inline data again, not
> a cached copy that it rewinds and replays. The fact that '-' might be
> different on the first and second read is exactly the same as 'file1'
> might be different on the first and second read. This describes the
> same behavior for both 'file1' and '-'.

If you are suggesting that associating a file descriptor with a file; 
opening; reading; closing; reopening (with implies a seek(0) ) ; and 
rereading   is the same as opening ; reading upto 'e' marker *in the 
data* then continuing to read the same opened device,  I have to say 
your reasoning is getting a bit contrived.

I find it hard to see that as being the same behaviour.

What you're asking for is that
> unlike the 'file1' case, '-' be handled differently, so that it saves
> or caches the data and rewinds back and forth through this cache
> instead of reading anew. This sort of special-case handling of '-'
> might be a worthwhile option, but it's a separate feature, and one
> that must be kept optional so as to not break backward compatibility
> and important use cases for some users (like me).
>
>
open '-' is already a special case. That is the source of the "problem". 
However, I totally agree with you about backwards compatibility and it 
would need a pretty substantial problem for me to suggest breaking that.

Since this idea just looked like a nice improvement when I suggested it 
I don't see any pressing case. I'm always looking for speed improvements 
and rereading my data file six times seemed wasteful.

There is clearly a lot of divergent ideas of what would (not) be useful 
so I'm dropping the idea.

Thanks for all replies and comments, it's been interesting and has 
brought up some good ideas.

regards, Peter.

Re: multiple plot lines with plot '-'

[Allin C. <cot...@wf...>]

On Thu, 17 May 2012, pl...@pi... wrote:

> I know [Ethan's] comment here was in the context of the matrix 
> discussion, but in relation to what I was suggesting (reading once 
> only) am I correct in thinking that this would not mean increasing 
> storage, merely filling the same storage a bit earlier?
>
> With regards to the changes that would be needed to parse all plot using
> clauses in one hit, could you point me to the code that would be
> affected so I can get a better idea of what would be involved?

My impression from looking at the gnuplot code is that the answer to 
your first question is No, you're not correct; and the answer to 
your second question is that this would involve a radical redesign 
of gnuplot, which currently performs no look-ahead when parsing and 
executing lines of input.

Allin Cottrell

Re: multiple plot lines with plot '-'

[Tait <gnu...@t4...>]

> > f(x,y,z)=<some function>
> > g(x)=<some other function>
> > plot 'datafile' using (column(f($1,$2,$3))):(column(g($4)), \
> > 	'data2file' using ($1>$2 ? column(g($4)) : 1/0)
>
> I think your example needs to be stripped down to the bare essentials of
> making the point you are trying to make. I'll have a guess about what
> you are trying to say, sorry if I miss the mark, and I think you are
> mistaken.

I was trying to make two separate points in that example, and you missed
the first -- and more important -- of them. To simplify:

plot 'datafile' using (column($1))

The column to be used for data is the column specified in the first
column of the data file. This is not and cannot be known until the file
(that column, at least) is actually read. The point of including the
functions is that we might not even know based on a single line of the
input what column we would need to preserve. The only way to know what
must be preserved is to actually evaluate the using conditions on the
input, see what results, and then reread the file for the next plot
because it, too, may have the exact same dereferencing behavior that
the first using statement had. And even more so, to the extent the first
using evaluation had side-effects that alter the meaning of the second
using statement, these side-effects cannot be anticipated ahead-of
time without actually doing the calculations demanded by the user.
These kinds of side effects are already suggested by demos like
http://gnuplot.sourceforge.net/demo_4.6/running_avg.html.

In short, the current behavior of gnuplot makes it impossible to
do a single-pass read of the data file. We could break the current
behavior, of course, but then we explicitly decrease the flexibility
we have for the sake of introducing other inconsistent and backward-
incompatible magic behavior.

At risk of repeating myself, caching may be worthwhile, but not in
the way and with the syntax you've suggested.

> >			 ... The behavior of '-' is currently the same as
> > for a data file. ...
>
> No, you need to read up on how '-' works.  This has been covered in a
> number of posts in this thread.

I know exactly how it works, and it works just like data files work.
I use and rely on this behavior. The behavior you want, and that
you (incorrectly, imo) describe as being the same as what's done for
data files is different. When you say plot 'file1', '', gnuplot does
not save 'file1' and rewind through this cache to find the data it
needs when plotting the ''. It plots 'file1', then when it sees ''
it reads 'file1' (again). When gnuplot plots '-', it reads the inline
data, and when it sees '' after, it reads the inline data again, not
a cached copy that it rewinds and replays. The fact that '-' might be
different on the first and second read is exactly the same as 'file1'
might be different on the first and second read. This describes the
same behavior for both 'file1' and '-'. What you're asking for is that
unlike the 'file1' case, '-' be handled differently, so that it saves
or caches the data and rewinds back and forth through this cache
instead of reading anew. This sort of special-case handling of '-'
might be a worthwhile option, but it's a separate feature, and one
that must be kept optional so as to not break backward compatibility
and important use cases for some users (like me).

Re: multiple plot lines with plot '-'

[sfeam (E. Merritt) <eam...@gm...>]

On Thursday, 17 May 2012, pl...@pi... wrote:
> On 05/16/12 23:05, Ethan A Merritt wrote:
> > Clearly there are benefits on both sides, but up until now we have
> > considered it a priority to minimize internal data storage.
> > 
> Hi Ethan,
> 
> I know your comment here was in the context of the matrix discussion, 
> but in relation to what I was suggesting (reading once only) am I 
> correct in thinking that this would not mean increasing storage, merely 
> filling the same storage a bit earlier?

No, that is not correct.  Gnuplot keeps no history of past plots.
All the data structures pertain only to the current plot, where
"plot" means one clause of the plot or splot statement.  
The data for each clause overwrites the data for the previous plot.
Furthermore, the data that is stored internally does not correspond
1-to-1 to columns of the input data file. Instead what is stored is
the result of evaluating each specifier in the "using" option.
So  
   plot data using (sum [i=1:100] column(i))
stores only two values for each point:
   the index (implicit column(0)) is stored as the x coord
   the sum of columns 1 through 100 is stored as the y coord
Two things to note here are that the x coordinate isn't even
in the data file, and even though 100 numbers are read from
each line in order to calculate a sum, none are stored for later
use.

	Ethan 

> With regards to the changes that would be needed to parse all plot using 
> clauses in one hit, could you point me to the code that would be 
> affected so I can get a better idea of what would be involved? 
> Thanks , Peter.

Re: multiple plot lines with plot '-'

[<pl...@pi...>]

On 05/16/12 23:05, Ethan A Merritt wrote:
> Clearly there are benefits on both sides, but up until now we have
> considered it a priority to minimize internal data storage.
>
>> >

Hi Ethan,

I know your comment here was in the context of the matrix discussion, 
but in relation to what I was suggesting (reading once only) am I 
correct in thinking that this would not mean increasing storage, merely 
filling the same storage a bit earlier?

With regards to the changes that would be needed to parse all plot using 
clauses in one hit, could you point me to the code that would be 
affected so I can get a better idea of what would be involved?

Thanks , Peter.

Re: multiple plot lines with plot '-'

[<pl...@pi...>]

On 05/17/12 19:56, Tait wrote:
>
> I was going to make a point about GB-sized data sets and memory usage
> of trying to cache such things, but Ethan helpfully already did so.
> Some of the data I work with is intentionally plotted in a raster
> format because plotting in a vector format creates file sizes that
> overflow memory and filesystem limits.
>
>> Allin's suggestion would likely be simple to do but could potentially be
>> very wasteful. With the increasing importance of mobile and embedded
>> devices this may not be too desirable
>
> I like Allin's suggestion best out of what I've seen suggested here. It
> doesn't demand a backward-incompatible change to syntax, does not need
> to penalize users with large data sets, and still allows small-dataset
> users to cache their data in memory.
>
>> As I understand it so far, data for each plot line in read in with a
>> separate reading of the input data source.
>
> This is fundamental and unavoidable, because the input to be read might
> depend on the data itself. The following is perfectly legal, for example,
> and I've even used it before:
>
> f(x,y,z)=<some function>
> g(x)=<some other function>
> plot 'datafile' using (column(f($1,$2,$3))):(column(g($4)), \
> 	'data2file' using ($1>$2 ? column(g($4)) : 1/0)

Hi Tait,

I think your example needs to be stripped down to the bare essentials of 
making the point you are trying to make. I'll have a guess about what 
you are trying to say, sorry if I miss the mark, and I think you are 
mistaken.

plot 'data2file' using ($1>$2 ? column(g($4)) : 1/0)

this plot line requires reading of columns 1,2 and 4 (all lines). When 
it comes to be processed the plotted points  will be evaluated and some 
of them will be NaN. All these data pairs (including the NaNs)  will be 
stored in slots for later reuse.

I also use this kind of construction frequently, I don't see that it has 
any bearing on what I suggested.


>
>> 1. unnecessary re-reading of input file
>
> The file is re-read, yes, but it's not unnecessary.
>
>> 2. possible plotting of different states of the data in one graph.
>
> I think any user would not be surprised that a file being modified while
> gnuplot is plotting would have undefined results. If this is a concern
> for your application, it is incumbent on you to cache or copy-on-write,
> or whatever works for your scenario.
>
>> 3. plot '-' using 1:2 , '' using 1:3	requires two separate datasets (or
>> duplication) to be supplied and thus has different behaviour to a
>> similar command using a data file.
>
> I don't understand this. The behavior of '-' is currently the same as
> for a data file. What you're suggesting is to make '-' act different than
> a normal input file would act.

No, you need to read up on how '-' works.  This has been covered in a 
number of posts in this thread.

  I can see the advantage of specifying a
> single data set inline somehow and then referring repeatedly to that one
> data set, but that seems like a new and different feature, not really an
> extension of "plot '-'".
>
>
regards.

Re: multiple plot lines with plot '-'

[Allin C. <cot...@wf...>]

On Thu, 17 May 2012, Ethan A Merritt wrote:

> On Wednesday, May 16, 2012 04:41:25 pm Allin Cottrell wrote:
>> I'm envisaging, say,
>>
>> m = {1,2,3; 4,5,6; 7,8,9}
>>
>> where (just for illustration) I'm assuming a matrix syntax in which
>> ',' is the column separator and ';' is the row separator.
>
> Only numerical data?
> That really seems like it's a separate request - to support numerical
> arrays as a variable type.  In certain cases it might substitute for
> a mechanism of caching the input stream, but it would be less general.

OK, point taken.

> My inclination is to think about caching multiline character input
> rather than numerical arrays.  Somewhat along the lines of the patch
> that Dan Sebald mentioned, although as I recall that dealt with caching
> input from a normal data file rather than offering a new mechanism
> of providing data in-line.
>
> Something like
>
> gnuplot> store DATA1 <<EOF
> Paris   France  2110420 48.86°  2.34°
> Marseille       France  820729  43.31°  5.37°
> Lyon    France  443859  45.76°  4.83°
> Toulouse        France  411768  43.62°  1.45°
> EOF
> gnuplot> plot DATA1 using 5:4:($3 < 5000 ? "-" : strcol(1) with labels

Hmm, I think I like it.

> Questions
> - Would we need sanity checks limiting the size (number of lines?) of
>  the in-line data?

IMO sanity is up to the user. Maybe I have 32 GB RAM (actually I 
don't).

> - How persistant would DATA1 be?  Would we require an explicit command
>  to discard it and free the memory?

I think so (require explicit command).

> - Would all the existing "set datafile" options apply to this in-line
>  input also, or would it get a separate set of options?

I'd say, apply the existing options.

>  [aside] It's really hard to type <tab> characters into the gnuplot
>  input stream, because readline grabs them. So tab-separated data
>  fields could be a problem.

But would anyone really want to do this sort of thing interactively? 
(And even if so, would it really be necessary to support that?)

> - What about binary data?

Ugh, forbidden (should always be forbidden, IMO).

> - Would the stored data act as a normal string for the purpose of
>  string operations?
> - What about the keyword "every"?

Hmm, I pass on those for lack of knowledge of what is done to date.

Allin Cottrell

Re: multiple plot lines with plot '-'

[Ethan A M. <sf...@us...>]

On Wednesday, May 16, 2012 04:41:25 pm Allin Cottrell wrote:
> I'm envisaging, say,
> 
> m = {1,2,3; 4,5,6; 7,8,9}
> 
> where (just for illustration) I'm assuming a matrix syntax in which 
> ',' is the column separator and ';' is the row separator.

Only numerical data?
That really seems like it's a separate request - to support numerical
arrays as a variable type.  In certain cases it might substitute for
a mechanism of caching the input stream, but it would be less general.
And, as I noted before, people would immediate start requesting 
support for inv(m) and det(m) and m2 = m * m and so on.

My inclination is to think about caching multiline character input
rather than numerical arrays.  Somewhat along the lines of the patch
that Dan Sebald mentioned, although as I recall that dealt with caching
input from a normal data file rather than offering a new mechanism
of providing data in-line.

Something like

gnuplot> store DATA1 <<EOF
Paris   France  2110420 48.86°  2.34°
Marseille       France  820729  43.31°  5.37°
Lyon    France  443859  45.76°  4.83°
Toulouse        France  411768  43.62°  1.45°
EOF
gnuplot> plot DATA1 using 5:4:($3 < 5000 ? "-" : strcol(1) with labels

Questions
- Would we need sanity checks limiting the size (number of lines?) of
  the in-line data?
- How persistant would DATA1 be?  Would we require an explicit command
  to discard it and free the memory?
- Would all the existing "set datafile" options apply to this in-line
  input also, or would it get a separate set of options?
  [aside] It's really hard to type <tab> characters into the gnuplot
  input stream, because readline grabs them. So tab-separated data
  fields could be a problem.
- What about binary data?
- Would the stored data act as a normal string for the purpose of 
  string operations?  
- What about the keyword "every"? 

 
> > It's been suggested before.  For example, Feature Request #1957568
> > https://sourceforge.net/tracker/index.php?func=detail&aid=1957568&group_id=2055&atid=352055
> 
> OK, I see what you mean, but please note that I'm _not_ requesting
> that "calculations can be done with the [matrix] data". That, it 
> seems to me, is not gnuplot's job.
> 
> > I'd like to see a more fully-developed proposal before seriously
> > considering it.
> 
> Or a patch; I'm not just requesting that someone else does this for 
> me.
> 
> > You can bet that the moment we provided a way to read in a matrix, 
> > people would start clamoring for math library support to do matrix 
> > operations.
> 
> FWIW you'd have my full support in resisting that sort of bloat. All 
> I'm interested in is a more efficient way to "inline" plottable data 
> in a gnuplot command file.
> 
> Allin Cottrell

Re: multiple plot lines with plot '-'

[Tait <gnu...@t4...>]

I was going to make a point about GB-sized data sets and memory usage
of trying to cache such things, but Ethan helpfully already did so.
Some of the data I work with is intentionally plotted in a raster
format because plotting in a vector format creates file sizes that
overflow memory and filesystem limits.

> Allin's suggestion would likely be simple to do but could potentially be
> very wasteful. With the increasing importance of mobile and embedded
> devices this may not be too desirable

I like Allin's suggestion best out of what I've seen suggested here. It
doesn't demand a backward-incompatible change to syntax, does not need
to penalize users with large data sets, and still allows small-dataset
users to cache their data in memory.

> As I understand it so far, data for each plot line in read in with a
> separate reading of the input data source.

This is fundamental and unavoidable, because the input to be read might
depend on the data itself. The following is perfectly legal, for example,
and I've even used it before:

f(x,y,z)=<some function>
g(x)=<some other function>
plot 'datafile' using (column(f($1,$2,$3))):(column(g($4)), \
	'data2file' using ($1>$2 ? column(g($4)) : 1/0)

> 1. unnecessary re-reading of input file

The file is re-read, yes, but it's not unnecessary.

> 2. possible plotting of different states of the data in one graph.

I think any user would not be surprised that a file being modified while
gnuplot is plotting would have undefined results. If this is a concern
for your application, it is incumbent on you to cache or copy-on-write,
or whatever works for your scenario.

> 3. plot '-' using 1:2 , '' using 1:3	requires two separate datasets (or
> duplication) to be supplied and thus has different behaviour to a
> similar command using a data file.

I don't understand this. The behavior of '-' is currently the same as
for a data file. What you're suggesting is to make '-' act different than
a normal input file would act. I can see the advantage of specifying a
single data set inline somehow and then referring repeatedly to that one
data set, but that seems like a new and different feature, not really an
extension of "plot '-'".

Re: multiple plot lines with plot '-'

[<pl...@pi...>]

On 05/16/12 21:56, Daniel J Sebald wrote:
>
> Your previous email pointed out a problem with 'datafile' reread on ARM
> at a particular instant.  There would then need to be something
> analogous to '-' and '--' but for data files.  OR, the default for
> datafiles could be to buffer the data (no easy task given gnuplot wasn't
> built that way from the ground up).  But if the default for datafiles is
> to buffer data, then '--' becomes analogous to 'datafile' as the default
> behavior, and '-' becomes the special case.

Thanks all.

Allin's suggestion would likely be simple to do but could potentially be 
very wasteful. With the increasing importance of mobile and embedded 
devices this may not be too desirable


As I understand it so far, data for each plot line in read in with a 
separate reading of the input data source. The points are stored in a 
slot that enables refresh to do it's job without re-reading the data.

It seems it is the calculated plotted points that are stored, not the 
actual input data.
.
eg plot datafile using (2*$1:$2/10.)

Each plot clause is processed separately and sequentially , hence the 
need for re-reading the file because the needs of the second plot line 
are not known on the first reading of the file.

I can see how this could have grown out of the constant evolution of 
features. However, it seems suboptimal for (at least) three reasons:

1. unnecessary re-reading of input file
2. possible plotting of different states of the data in one graph.
3. plot '-' using 1:2 , '' using 1:3  requires two separate datasets (or 
duplication) to be supplied and thus has different behaviour to a 
similar command using a data file.


Now the obvious technical solution would be to parse the whole plot line 
to work out what is needed and do it once.

This implies some buffering until the actual calculations implied by 
each 'using' clause get processed. However, the slots about to be 
attributed.

To avoid major restructuring of the code , the data columns needed for 
all lines could be stored in the slots and processed to actual plotted 
point data in the usual sequence.

All this requires is parsing the whole plot command before the first 
read of data and allocation of the slots a bit earlier. This would not 
take any extra storage

Again , because plot '-' using  is doing something different it will 
have a different code path but that's already the case.

A new plot '--'  could provide the same functionality that is available 
for data files, for pipe input. This would be more consistent and 
interchangeable. This would make gnuplot behaviour agnostic as to 
whether the input was a file or a pipe.

That is appealing.

Pre parsing the plot command would not seem too onerous and , unless I'm 
wrong about how this is done, it would not seem to involve a paradigm 
shift in the coding.


regards, Peter.

Re: multiple plot lines with plot '-'

[Allin C. <cot...@wf...>]

On Wed, 16 May 2012, Ethan A Merritt wrote:

> On Wednesday, May 16, 2012 01:17:22 pm Allin Cottrell wrote:
>
>>>> Just a notion, but what if gnuplot's syntax permitted defining a matrix
>>>> (either "inline" with some nice, simple syntax or by reading from a
>>>> file), and you could then say something like
>>>>
>>>> plot matrix "foo" using ...
>>>>
>>>> as many times as you wanted, using whichever columns?
>>
>> The way I'm thinking about this, no look-ahead is needed. All
>> gnuplot has to do is store the given matrix (all of it), then draw
>> on it as and when specified.
>
> There is a significant subset of gnuplot users for whom it is a key
> feature that gnuplot can handle large data sets.  Here "large"
> means millions of points.  It can do this exactly because it does
> not store the entire contents of the data file prior to processing.

Fine, understood. But that need not rule out the possibility of 
storing a reasonably sized matrix to serve as the target for 
repeated plot commands, for those who want to do that sort of thing.

>> I'm mooting this as an extension of the way that gnuplot can
>> currently store scalars (and complex numbers) for future use,
>> without having any idea at the time what they'll be used for.
>
> Sorry, you lost me there.  What are you refering to?

Nothing mysterious, just

d = 1.3257
c = {1.5, 3.1}

and so on. I'm envisaging, say,

m = {1,2,3; 4,5,6; 7,8,9}

where (just for illustration) I'm assuming a matrix syntax in which 
',' is the column separator and ';' is the row separator.

> It's been suggested before.  For example, Feature Request #1957568
> https://sourceforge.net/tracker/index.php?func=detail&aid=1957568&group_id=2055&atid=352055

OK, I see what you mean, but please note that I'm _not_ requesting
that "calculations can be done with the [matrix] data". That, it 
seems to me, is not gnuplot's job.

> I'd like to see a more fully-developed proposal before seriously
> considering it.

Or a patch; I'm not just requesting that someone else does this for 
me.

> You can bet that the moment we provided a way to read in a matrix, 
> people would start clamoring for math library support to do matrix 
> operations.

FWIW you'd have my full support in resisting that sort of bloat. All 
I'm interested in is a more efficient way to "inline" plottable data 
in a gnuplot command file.

Allin Cottrell

Re: multiple plot lines with plot '-'

[Ethan A M. <sf...@us...>]

On Wednesday, May 16, 2012 01:17:22 pm Allin Cottrell wrote:

> >> Just a notion, but what if gnuplot's syntax permitted defining a matrix
> >> (either "inline" with some nice, simple syntax or by reading from a
> >> file), and you could then say something like
> >> 
> >> plot matrix "foo" using ...
> >> 
> >> as many times as you wanted, using whichever columns?
> 
> The way I'm thinking about this, no look-ahead is needed. All 
> gnuplot has to do is store the given matrix (all of it), then draw 
> on it as and when specified.

There is a significant subset of gnuplot users for whom it is a key
feature that gnuplot can handle large data sets.  Here "large"
means millions of points.  It can do this exactly because it does
not store the entire contents of the data file prior to processing.
This makes it very different from alternative programs such as
R, which do not handle such large data sets gracefully, if at all.

Clearly there are benefits on both sides, but up until now we have
considered it a priority to minimize internal data storage.

> I'm mooting this as an extension of the way that gnuplot can 
> currently store scalars (and complex numbers) for future use, 
> without having any idea at the time what they'll be used for.

Sorry, you lost me there.  What are you refering to?
We don't currently have a generic method for reading in and storing
scalars either.  You can sort of fake it using commands like
    foo = int(system("cat data-file-containing-a-number"))
but that is really ugly.

> This strikes me as a lot more straightforward than messing with the
> way datafiles and stdin currently work.

It's been suggested before.  For example, Feature Request #1957568
https://sourceforge.net/tracker/index.php?func=detail&aid=1957568&group_id=2055&atid=352055

I'd like to see a more fully-developed proposal before seriously
considering it.   It's not just a question of making up some syntax for
plot commands; that's the easy part.   You can bet that the moment we
provided a way to read in a matrix, people would start clamoring for 
math library support to do matrix operations.

Of course, it might have the advantage of replacing a lot of very
messy code that currently handles ascii and binary "matrix" data
during plotting.  It might turn out to be simpler overall to separate
the operation into two steps: 1) read in the matrix data 2) plot it.
Or maybe not.  I'm reasonably happy with the way "with image" works now.
But I'm less happy that the interpretation and handling for matrix 
data currently differs depending on whether the input was ascii or binary. 

	Ethan

Re: multiple plot lines with plot '-'

[Ethan A M. <sf...@us...>]

On Wednesday, May 16, 2012 12:41:08 pm pl...@pi... wrote:
> On 05/16/12 18:35, Daniel J Sebald wrote:
> > However, this isn't as different as you think.  I would guess that if
> > "datafile" were changed at an instant between when gnuplot does the two
> > plots, the two plots would look different.  The analogy is typing the
> > same thing at the keyboard or pipe when using '-'.
>
> This is not so hypothetical and improbably as it may seem.
> 
> Some of my embedded plots take 16-20s to run the ARM hardware I'm using. 
> It is totally possible (maybe probable) that the source file could 
> change in that time.

That is exactly why I added the "volatile" keyword and the option to
use "refresh" rather than "replot".

The original gnuplot behavior was to always read the file again when
replotting.  If the contents of the file had changed, you got a different
plot.   "volatile" and "refresh" tell it to try to replot using the input
data that was stored internally during the original read, so even if the
file contents have changed you get the same plot.

> plot datafile using 1:2 , '' using 1:3
> 
> Now both these lines are coming out on the same plot , I would expect it 
> to be a snapshot of the data at the same point in time. We are almost 
> certainly going to be comparing them. Currently there is a possibility 
> that the two lines represent different states of the data set.  If they 
> were not coincident in time it could lead to misrepresentation or 
> misinterpretation.

True. There is currently no protection against that, although there may
be ways of capturing all the needed information in a single plot clause.
E.g.
   plot datafile using 1:2:3 with yerrorbars pt 0
If this is a serious concern, it would help to have an example of the
real-world plot that is desired.