gnuplot-beta

Re: multiple plot lines with plot '-'

[Daniel J S. <dan...@ie...>]

On 05/16/2012 02:35 AM, pl...@pi... wrote:
> On 05/16/12 06:56, sfeam (Ethan Merritt) wrote:
>> On Monday, 14 May 2012, su...@pi... wrote:
>>> On 05/14/12 18:27, Ethan A Merritt wrote:
>>>>>>    So could another flag like plot '--' be used to indicate rereading of
>>>>>>    the inline data rather than continuing at the next block of inline data ?
>>>> This wouldn't work in the general case - you can't reread from a pipe.
>>>
>>> hmm. I hadn't thought of "inline" data actually being a pipe. Good point.
>>>
>>> perhaps that could be accepted as a limitation on reading from pipes.
>>> It would not reduce the functionality that is already available to pipes.
>>
>> Up till now, the following have all been equivalent:
>>       gnuplot file.in
>>       cat file.in | gnuplot
>>       cat file.in | gnuplot -
>>       gnuplot -e 'load file.in'
>>
>> I think it would be a bad idea to introduce new syntax that breaks
>> this equivalence.
>
> I agree, I like that sort of symmetry.
>
> Part of the problem here is that plot '-' does not work in quite the
> same way as plot datafile
>
> plot '-','-'   plots two consecutive segments from stdin  whereas plot
> datafile , datafile  rereads the file and plots the same thing twice.

I see the point.


> This is presumably because you can't reread a pipe but it means plot
> command is acting differently depending on the data source specified.

It wouldn't be difficult to program this so that the same data is used 
multiple times.  There has always been incentive to store the original 
data for redrawing purposes other than this, and I think I had written a 
patch some time ago that does that.

However, this isn't as different as you think.  I would guess that if 
"datafile" were changed at an instant between when gnuplot does the two 
plots, the two plots would look different.  The analogy is typing the 
same thing at the keyboard or pipe when using '-'.

Perhaps there should be a mode switch, "use the same data" or "don't 
reread".  But I'm not sure how crucial that is.


>   >>Consider:
>   >>    plot '-' using 1:2, '' using 3:4, '' using 99:100
>   >>  The first plot uses only columns 1 and 2; these values are stored
>   >>  internally and could be used again without rereading.  That's what the
>    >>  "refresh"  command does.  But the 2nd and 3rd plots use data values
> from
>   >>  columns we didn't use the first time and hence didn't keep.
>   >>  So those values can only be recovered by rereading.
>
>   >  Since this is all in one plot command , couldn't the whole line be
>   >  parsed to find all the columns to be required by all the using clauses?
>   >
>   >  In fact isn't this already done? Are you implying that
>   >
>   >    plot datafile using 1:2, '' using 3:4, '' using 99:100
>   >
>   >  actually reads the data file three times looking at different columns :?
>
>   >>  Yes.
>
> So if I get this,
>
> gnuplot>   plot datafile using 1:2, '' using 3:4
>
> opens and reads same lines of the file twice. Filling two sets of slots
>
>
>    gnuplot>   plot '-' using 1:2, '' using 3:4
>
> reads consecutively from stdin , twice. Filling two sets of slots
>
>
> It would be more efficient if the first version parsed the whole line to
> find all required columns and did one read. I do plots with 5 or 6
> lines, this could be a useful improvement.

This creates some logistical problems of storing the data or accessing 
the data at different times between different sections of code.  It is 
all dependent upon the way gnuplot is programmed internally.  When 
gnuplot started long ago, that was sort of the paradigm; linear run the 
data through various steps.  I did write a patch at some point that 
stores the unprocessed data and can feed it through a second, third, 
etc. time.  That is still the same paradigm so not much change to 
gnuplots innards (requires lots of memory so it would be nice to 
deactivate such a feature).


> Then the suggested '--' could work with no extra effort.
> gnuplot>   plot '--' using 1:2, '' using 3:4
>
>
> The asymmetrical behaviour of the existing plot '-'  would still need a
> different code path.

Have to think about this, because again it isn't clear that '-' and 
'datafile' are currently acting differently.


>> If the previous work-around I suggested is too
>> long-winded, requiring a "print" and matching quotes on each line,
>> how about introducing an idea from perl:
>>
>> set print 'tempfile.dat'
>> print<<   EOF
>> 1st line of data
>> 2nd line of data
>> 3rd line of data
>> EOF
>>
>
> Yes , I was looking for something like that yesterday. I think that
> would be a useful addition in a number of situations.
>
> However, rewriting all the data in the current file to another
> superfluous copy just to get around the syntax seems a bit clunky.

Yes, but that is the nature of gnuplot going back to the beginning. 
What you are suggesting is a pretty fundamental change.

Dan

Re: multiple plot lines with plot '-'

[<pl...@pi...>]

On 05/16/12 18:35, Daniel J Sebald wrote:
> However, this isn't as different as you think.  I would guess that if
> "datafile" were changed at an instant between when gnuplot does the two
> plots, the two plots would look different.  The analogy is typing the
> same thing at the keyboard or pipe when using '-'.
>
> Perhaps there should be a mode switch, "use the same data" or "don't
> reread".  But I'm not sure how crucial that is.

This is not so hypothetical and improbably as it may seem.

Some of my embedded plots take 16-20s to run the ARM hardware I'm using. 
It is totally possible (maybe probable) that the source file could 
change in that time.

Since the time between the first plot and the second one is a fairly 
arbitrary one, depending up on the hardware and the length of data , I 
can't see it being a useful feature.  Maybe someone else can.

The trivial case I gave was intentionally simplistic. Consider this 
variation:

plot datafile using 1:2 , '' using 1:3

Now both these lines are coming out on the same plot , I would expect it 
to be a snapshot of the data at the same point in time. We are almost 
certainly going to be comparing them. Currently there is a possibility 
that the two lines represent different states of the data set.  If they 
were not coincident in time it could lead to misrepresentation or 
misinterpretation.

I would consider it a defect.

Peter.

Re: multiple plot lines with plot '-'

[Ethan A M. <sf...@us...>]

On Wednesday, May 16, 2012 12:41:08 pm pl...@pi... wrote:
> On 05/16/12 18:35, Daniel J Sebald wrote:
> > However, this isn't as different as you think.  I would guess that if
> > "datafile" were changed at an instant between when gnuplot does the two
> > plots, the two plots would look different.  The analogy is typing the
> > same thing at the keyboard or pipe when using '-'.
>
> This is not so hypothetical and improbably as it may seem.
> 
> Some of my embedded plots take 16-20s to run the ARM hardware I'm using. 
> It is totally possible (maybe probable) that the source file could 
> change in that time.

That is exactly why I added the "volatile" keyword and the option to
use "refresh" rather than "replot".

The original gnuplot behavior was to always read the file again when
replotting.  If the contents of the file had changed, you got a different
plot.   "volatile" and "refresh" tell it to try to replot using the input
data that was stored internally during the original read, so even if the
file contents have changed you get the same plot.

> plot datafile using 1:2 , '' using 1:3
> 
> Now both these lines are coming out on the same plot , I would expect it 
> to be a snapshot of the data at the same point in time. We are almost 
> certainly going to be comparing them. Currently there is a possibility 
> that the two lines represent different states of the data set.  If they 
> were not coincident in time it could lead to misrepresentation or 
> misinterpretation.

True. There is currently no protection against that, although there may
be ways of capturing all the needed information in a single plot clause.
E.g.
   plot datafile using 1:2:3 with yerrorbars pt 0
If this is a serious concern, it would help to have an example of the
real-world plot that is desired.

Re: multiple plot lines with plot '-'

[<pl...@pi...>]

On 05/16/12 18:35, Daniel J Sebald wrote:
> Perhaps there should be a mode switch, "use the same data" or "don't
> reread".  But I'm not sure how crucial that is.


Isn't that what I was suggesting using plot '--'   (double minus sign) ?

If the read-in data is buffered, that would fix Ethan's worry about this 
not working the same from pipes.

Peter.

Re: multiple plot lines with plot '-'

[Allin C. <cot...@wf...>]

On Wed, 16 May 2012, Daniel J Sebald wrote:

> Perhaps there should be a mode switch, "use the same data" 
> or "don't reread".  But I'm not sure how crucial that is.

Just a notion, but what if gnuplot's syntax permitted defining 
a matrix (either "inline" with some nice, simple syntax or by 
reading from a file), and you could then say something like

   plot matrix "foo" using ...

as many times as you wanted, using whichever columns?

Allin Cottrell

Re: multiple plot lines with plot '-'

[Daniel J S. <dan...@ie...>]

On 05/16/2012 11:50 AM, Allin Cottrell wrote:
> On Wed, 16 May 2012, Daniel J Sebald wrote:
>
>> Perhaps there should be a mode switch, "use the same data" or "don't
>> reread". But I'm not sure how crucial that is.
>
> Just a notion, but what if gnuplot's syntax permitted defining a matrix
> (either "inline" with some nice, simple syntax or by reading from a
> file), and you could then say something like
>
> plot matrix "foo" using ...
>
> as many times as you wanted, using whichever columns?

Well, that's a paradigm change as far as programming because the whole 
plot command would need to be processed to figure out what columns are 
required for _all_ plots not just the current one.  That also expands 
the number of possible storage elements creates difficult programming in 
this design.

Currently, the data file could be (exaggerating) 1000 columns wide but 
for any given plot gnuplot selects and stores only the few columns of 
data it needs for individual plots (say column 1, column 50 and column 
51 on the first plot).  It does the same for the next plot, but it 
rereads the file and might select a different few columns (say column 1, 
column 120, column 121).

Attempting an explanation, gnuplot has been programmed in such a way as 
to not "look ahead" at what is to come.  Changing to a system that needs 
to "look ahead" for plots that are to come really creates challenges 
internally.  gnuplot started in an era in which memory wasn't so readily 
available, but at the same time I think it should probably remain close 
to the parsimonious paradigm as possible.  In my opinion, software 
generally has become a little too bloated.  I did write a patch a while 
back that stored the raw data, sort of bucking that paradigm, but that 
was storing just the columns of data an individual plot uses, not the 
whole possible data file, i.e., no "look ahead" but instead "store what 
has been used".

Dan

Re: multiple plot lines with plot '-'

[Allin C. <cot...@wf...>]

On Wed, 16 May 2012, Daniel J Sebald wrote:

> On 05/16/2012 11:50 AM, Allin Cottrell wrote:
>> On Wed, 16 May 2012, Daniel J Sebald wrote:
>> 
>>> Perhaps there should be a mode switch, "use the same data" or "don't
>>> reread". But I'm not sure how crucial that is.
>> 
>> Just a notion, but what if gnuplot's syntax permitted defining a matrix
>> (either "inline" with some nice, simple syntax or by reading from a
>> file), and you could then say something like
>> 
>> plot matrix "foo" using ...
>> 
>> as many times as you wanted, using whichever columns?
>
> Well, that's a paradigm change as far as programming because the whole plot 
> command would need to be processed to figure out what columns are required 
> for _all_ plots not just the current one.

The way I'm thinking about this, no look-ahead is needed. All 
gnuplot has to do is store the given matrix (all of it), then draw 
on it as and when specified.

I'm mooting this as an extension of the way that gnuplot can 
currently store scalars (and complex numbers) for future use, 
without having any idea at the time what they'll be used for. This 
strikes me as a lot more straightforward than messing with the
way datafiles and stdin currently work.

Allin Cottrell

Re: multiple plot lines with plot '-'

[Ethan A M. <sf...@us...>]

On Wednesday, May 16, 2012 01:17:22 pm Allin Cottrell wrote:

> >> Just a notion, but what if gnuplot's syntax permitted defining a matrix
> >> (either "inline" with some nice, simple syntax or by reading from a
> >> file), and you could then say something like
> >> 
> >> plot matrix "foo" using ...
> >> 
> >> as many times as you wanted, using whichever columns?
> 
> The way I'm thinking about this, no look-ahead is needed. All 
> gnuplot has to do is store the given matrix (all of it), then draw 
> on it as and when specified.

There is a significant subset of gnuplot users for whom it is a key
feature that gnuplot can handle large data sets.  Here "large"
means millions of points.  It can do this exactly because it does
not store the entire contents of the data file prior to processing.
This makes it very different from alternative programs such as
R, which do not handle such large data sets gracefully, if at all.

Clearly there are benefits on both sides, but up until now we have
considered it a priority to minimize internal data storage.

> I'm mooting this as an extension of the way that gnuplot can 
> currently store scalars (and complex numbers) for future use, 
> without having any idea at the time what they'll be used for.

Sorry, you lost me there.  What are you refering to?
We don't currently have a generic method for reading in and storing
scalars either.  You can sort of fake it using commands like
    foo = int(system("cat data-file-containing-a-number"))
but that is really ugly.

> This strikes me as a lot more straightforward than messing with the
> way datafiles and stdin currently work.

It's been suggested before.  For example, Feature Request #1957568
https://sourceforge.net/tracker/index.php?func=detail&aid=1957568&group_id=2055&atid=352055

I'd like to see a more fully-developed proposal before seriously
considering it.   It's not just a question of making up some syntax for
plot commands; that's the easy part.   You can bet that the moment we
provided a way to read in a matrix, people would start clamoring for 
math library support to do matrix operations.

Of course, it might have the advantage of replacing a lot of very
messy code that currently handles ascii and binary "matrix" data
during plotting.  It might turn out to be simpler overall to separate
the operation into two steps: 1) read in the matrix data 2) plot it.
Or maybe not.  I'm reasonably happy with the way "with image" works now.
But I'm less happy that the interpretation and handling for matrix 
data currently differs depending on whether the input was ascii or binary. 

	Ethan

Re: multiple plot lines with plot '-'

[<pl...@pi...>]

On 05/16/12 23:05, Ethan A Merritt wrote:
> Clearly there are benefits on both sides, but up until now we have
> considered it a priority to minimize internal data storage.
>
>> >

Hi Ethan,

I know your comment here was in the context of the matrix discussion, 
but in relation to what I was suggesting (reading once only) am I 
correct in thinking that this would not mean increasing storage, merely 
filling the same storage a bit earlier?

With regards to the changes that would be needed to parse all plot using 
clauses in one hit, could you point me to the code that would be 
affected so I can get a better idea of what would be involved?

Thanks , Peter.

Re: multiple plot lines with plot '-'

[sfeam (E. Merritt) <eam...@gm...>]

On Thursday, 17 May 2012, pl...@pi... wrote:
> On 05/16/12 23:05, Ethan A Merritt wrote:
> > Clearly there are benefits on both sides, but up until now we have
> > considered it a priority to minimize internal data storage.
> > 
> Hi Ethan,
> 
> I know your comment here was in the context of the matrix discussion, 
> but in relation to what I was suggesting (reading once only) am I 
> correct in thinking that this would not mean increasing storage, merely 
> filling the same storage a bit earlier?

No, that is not correct.  Gnuplot keeps no history of past plots.
All the data structures pertain only to the current plot, where
"plot" means one clause of the plot or splot statement.  
The data for each clause overwrites the data for the previous plot.
Furthermore, the data that is stored internally does not correspond
1-to-1 to columns of the input data file. Instead what is stored is
the result of evaluating each specifier in the "using" option.
So  
   plot data using (sum [i=1:100] column(i))
stores only two values for each point:
   the index (implicit column(0)) is stored as the x coord
   the sum of columns 1 through 100 is stored as the y coord
Two things to note here are that the x coordinate isn't even
in the data file, and even though 100 numbers are read from
each line in order to calculate a sum, none are stored for later
use.

	Ethan 

> With regards to the changes that would be needed to parse all plot using 
> clauses in one hit, could you point me to the code that would be 
> affected so I can get a better idea of what would be involved? 
> Thanks , Peter.

Re: multiple plot lines with plot '-'

[Allin C. <cot...@wf...>]

On Thu, 17 May 2012, pl...@pi... wrote:

> I know [Ethan's] comment here was in the context of the matrix 
> discussion, but in relation to what I was suggesting (reading once 
> only) am I correct in thinking that this would not mean increasing 
> storage, merely filling the same storage a bit earlier?
>
> With regards to the changes that would be needed to parse all plot using
> clauses in one hit, could you point me to the code that would be
> affected so I can get a better idea of what would be involved?

My impression from looking at the gnuplot code is that the answer to 
your first question is No, you're not correct; and the answer to 
your second question is that this would involve a radical redesign 
of gnuplot, which currently performs no look-ahead when parsing and 
executing lines of input.

Allin Cottrell

Re: multiple plot lines with plot '-'

[Allin C. <cot...@wf...>]

On Wed, 16 May 2012, Ethan A Merritt wrote:

> On Wednesday, May 16, 2012 01:17:22 pm Allin Cottrell wrote:
>
>>>> Just a notion, but what if gnuplot's syntax permitted defining a matrix
>>>> (either "inline" with some nice, simple syntax or by reading from a
>>>> file), and you could then say something like
>>>>
>>>> plot matrix "foo" using ...
>>>>
>>>> as many times as you wanted, using whichever columns?
>>
>> The way I'm thinking about this, no look-ahead is needed. All
>> gnuplot has to do is store the given matrix (all of it), then draw
>> on it as and when specified.
>
> There is a significant subset of gnuplot users for whom it is a key
> feature that gnuplot can handle large data sets.  Here "large"
> means millions of points.  It can do this exactly because it does
> not store the entire contents of the data file prior to processing.

Fine, understood. But that need not rule out the possibility of 
storing a reasonably sized matrix to serve as the target for 
repeated plot commands, for those who want to do that sort of thing.

>> I'm mooting this as an extension of the way that gnuplot can
>> currently store scalars (and complex numbers) for future use,
>> without having any idea at the time what they'll be used for.
>
> Sorry, you lost me there.  What are you refering to?

Nothing mysterious, just

d = 1.3257
c = {1.5, 3.1}

and so on. I'm envisaging, say,

m = {1,2,3; 4,5,6; 7,8,9}

where (just for illustration) I'm assuming a matrix syntax in which 
',' is the column separator and ';' is the row separator.

> It's been suggested before.  For example, Feature Request #1957568
> https://sourceforge.net/tracker/index.php?func=detail&aid=1957568&group_id=2055&atid=352055

OK, I see what you mean, but please note that I'm _not_ requesting
that "calculations can be done with the [matrix] data". That, it 
seems to me, is not gnuplot's job.

> I'd like to see a more fully-developed proposal before seriously
> considering it.

Or a patch; I'm not just requesting that someone else does this for 
me.

> You can bet that the moment we provided a way to read in a matrix, 
> people would start clamoring for math library support to do matrix 
> operations.

FWIW you'd have my full support in resisting that sort of bloat. All 
I'm interested in is a more efficient way to "inline" plottable data 
in a gnuplot command file.

Allin Cottrell

Re: multiple plot lines with plot '-'

[<pl...@pi...>]

On 05/16/12 21:56, Daniel J Sebald wrote:
>
> Your previous email pointed out a problem with 'datafile' reread on ARM
> at a particular instant.  There would then need to be something
> analogous to '-' and '--' but for data files.  OR, the default for
> datafiles could be to buffer the data (no easy task given gnuplot wasn't
> built that way from the ground up).  But if the default for datafiles is
> to buffer data, then '--' becomes analogous to 'datafile' as the default
> behavior, and '-' becomes the special case.

Thanks all.

Allin's suggestion would likely be simple to do but could potentially be 
very wasteful. With the increasing importance of mobile and embedded 
devices this may not be too desirable


As I understand it so far, data for each plot line in read in with a 
separate reading of the input data source. The points are stored in a 
slot that enables refresh to do it's job without re-reading the data.

It seems it is the calculated plotted points that are stored, not the 
actual input data.
.
eg plot datafile using (2*$1:$2/10.)

Each plot clause is processed separately and sequentially , hence the 
need for re-reading the file because the needs of the second plot line 
are not known on the first reading of the file.

I can see how this could have grown out of the constant evolution of 
features. However, it seems suboptimal for (at least) three reasons:

1. unnecessary re-reading of input file
2. possible plotting of different states of the data in one graph.
3. plot '-' using 1:2 , '' using 1:3  requires two separate datasets (or 
duplication) to be supplied and thus has different behaviour to a 
similar command using a data file.


Now the obvious technical solution would be to parse the whole plot line 
to work out what is needed and do it once.

This implies some buffering until the actual calculations implied by 
each 'using' clause get processed. However, the slots about to be 
attributed.

To avoid major restructuring of the code , the data columns needed for 
all lines could be stored in the slots and processed to actual plotted 
point data in the usual sequence.

All this requires is parsing the whole plot command before the first 
read of data and allocation of the slots a bit earlier. This would not 
take any extra storage

Again , because plot '-' using  is doing something different it will 
have a different code path but that's already the case.

A new plot '--'  could provide the same functionality that is available 
for data files, for pipe input. This would be more consistent and 
interchangeable. This would make gnuplot behaviour agnostic as to 
whether the input was a file or a pipe.

That is appealing.

Pre parsing the plot command would not seem too onerous and , unless I'm 
wrong about how this is done, it would not seem to involve a paradigm 
shift in the coding.


regards, Peter.

Re: multiple plot lines with plot '-'

[Tait <gnu...@t4...>]

I was going to make a point about GB-sized data sets and memory usage
of trying to cache such things, but Ethan helpfully already did so.
Some of the data I work with is intentionally plotted in a raster
format because plotting in a vector format creates file sizes that
overflow memory and filesystem limits.

> Allin's suggestion would likely be simple to do but could potentially be
> very wasteful. With the increasing importance of mobile and embedded
> devices this may not be too desirable

I like Allin's suggestion best out of what I've seen suggested here. It
doesn't demand a backward-incompatible change to syntax, does not need
to penalize users with large data sets, and still allows small-dataset
users to cache their data in memory.

> As I understand it so far, data for each plot line in read in with a
> separate reading of the input data source.

This is fundamental and unavoidable, because the input to be read might
depend on the data itself. The following is perfectly legal, for example,
and I've even used it before:

f(x,y,z)=<some function>
g(x)=<some other function>
plot 'datafile' using (column(f($1,$2,$3))):(column(g($4)), \
	'data2file' using ($1>$2 ? column(g($4)) : 1/0)

> 1. unnecessary re-reading of input file

The file is re-read, yes, but it's not unnecessary.

> 2. possible plotting of different states of the data in one graph.

I think any user would not be surprised that a file being modified while
gnuplot is plotting would have undefined results. If this is a concern
for your application, it is incumbent on you to cache or copy-on-write,
or whatever works for your scenario.

> 3. plot '-' using 1:2 , '' using 1:3	requires two separate datasets (or
> duplication) to be supplied and thus has different behaviour to a
> similar command using a data file.

I don't understand this. The behavior of '-' is currently the same as
for a data file. What you're suggesting is to make '-' act different than
a normal input file would act. I can see the advantage of specifying a
single data set inline somehow and then referring repeatedly to that one
data set, but that seems like a new and different feature, not really an
extension of "plot '-'".

Re: multiple plot lines with plot '-'

[Ethan A M. <sf...@us...>]

On Wednesday, May 16, 2012 04:41:25 pm Allin Cottrell wrote:
> I'm envisaging, say,
> 
> m = {1,2,3; 4,5,6; 7,8,9}
> 
> where (just for illustration) I'm assuming a matrix syntax in which 
> ',' is the column separator and ';' is the row separator.

Only numerical data?
That really seems like it's a separate request - to support numerical
arrays as a variable type.  In certain cases it might substitute for
a mechanism of caching the input stream, but it would be less general.
And, as I noted before, people would immediate start requesting 
support for inv(m) and det(m) and m2 = m * m and so on.

My inclination is to think about caching multiline character input
rather than numerical arrays.  Somewhat along the lines of the patch
that Dan Sebald mentioned, although as I recall that dealt with caching
input from a normal data file rather than offering a new mechanism
of providing data in-line.

Something like

gnuplot> store DATA1 <<EOF
Paris   France  2110420 48.86°  2.34°
Marseille       France  820729  43.31°  5.37°
Lyon    France  443859  45.76°  4.83°
Toulouse        France  411768  43.62°  1.45°
EOF
gnuplot> plot DATA1 using 5:4:($3 < 5000 ? "-" : strcol(1) with labels

Questions
- Would we need sanity checks limiting the size (number of lines?) of
  the in-line data?
- How persistant would DATA1 be?  Would we require an explicit command
  to discard it and free the memory?
- Would all the existing "set datafile" options apply to this in-line
  input also, or would it get a separate set of options?
  [aside] It's really hard to type <tab> characters into the gnuplot
  input stream, because readline grabs them. So tab-separated data
  fields could be a problem.
- What about binary data?
- Would the stored data act as a normal string for the purpose of 
  string operations?  
- What about the keyword "every"? 

 
> > It's been suggested before.  For example, Feature Request #1957568
> > https://sourceforge.net/tracker/index.php?func=detail&aid=1957568&group_id=2055&atid=352055
> 
> OK, I see what you mean, but please note that I'm _not_ requesting
> that "calculations can be done with the [matrix] data". That, it 
> seems to me, is not gnuplot's job.
> 
> > I'd like to see a more fully-developed proposal before seriously
> > considering it.
> 
> Or a patch; I'm not just requesting that someone else does this for 
> me.
> 
> > You can bet that the moment we provided a way to read in a matrix, 
> > people would start clamoring for math library support to do matrix 
> > operations.
> 
> FWIW you'd have my full support in resisting that sort of bloat. All 
> I'm interested in is a more efficient way to "inline" plottable data 
> in a gnuplot command file.
> 
> Allin Cottrell

Re: multiple plot lines with plot '-'

[Allin C. <cot...@wf...>]

On Thu, 17 May 2012, Ethan A Merritt wrote:

> On Wednesday, May 16, 2012 04:41:25 pm Allin Cottrell wrote:
>> I'm envisaging, say,
>>
>> m = {1,2,3; 4,5,6; 7,8,9}
>>
>> where (just for illustration) I'm assuming a matrix syntax in which
>> ',' is the column separator and ';' is the row separator.
>
> Only numerical data?
> That really seems like it's a separate request - to support numerical
> arrays as a variable type.  In certain cases it might substitute for
> a mechanism of caching the input stream, but it would be less general.

OK, point taken.

> My inclination is to think about caching multiline character input
> rather than numerical arrays.  Somewhat along the lines of the patch
> that Dan Sebald mentioned, although as I recall that dealt with caching
> input from a normal data file rather than offering a new mechanism
> of providing data in-line.
>
> Something like
>
> gnuplot> store DATA1 <<EOF
> Paris   France  2110420 48.86°  2.34°
> Marseille       France  820729  43.31°  5.37°
> Lyon    France  443859  45.76°  4.83°
> Toulouse        France  411768  43.62°  1.45°
> EOF
> gnuplot> plot DATA1 using 5:4:($3 < 5000 ? "-" : strcol(1) with labels

Hmm, I think I like it.

> Questions
> - Would we need sanity checks limiting the size (number of lines?) of
>  the in-line data?

IMO sanity is up to the user. Maybe I have 32 GB RAM (actually I 
don't).

> - How persistant would DATA1 be?  Would we require an explicit command
>  to discard it and free the memory?

I think so (require explicit command).

> - Would all the existing "set datafile" options apply to this in-line
>  input also, or would it get a separate set of options?

I'd say, apply the existing options.

>  [aside] It's really hard to type <tab> characters into the gnuplot
>  input stream, because readline grabs them. So tab-separated data
>  fields could be a problem.

But would anyone really want to do this sort of thing interactively? 
(And even if so, would it really be necessary to support that?)

> - What about binary data?

Ugh, forbidden (should always be forbidden, IMO).

> - Would the stored data act as a normal string for the purpose of
>  string operations?
> - What about the keyword "every"?

Hmm, I pass on those for lack of knowledge of what is done to date.

Allin Cottrell

In-line data [was multiple plot lines with plot '-']

[Ethan A M. <sf...@us...>]

Revisiting a discussion from last month...

I had a go at implemented data blocks defined as a "here document".
The patch is on SourceForge 
<https://sourceforge.net/tracker/?func=detail&aid=3534646&group_id=2055&atid=302055>

In brief it adds perl-like syntax

$Mydata << EOD
1 2 3
4 5 ?
7 8 9
# comments or anything else legal in a data file
10 11 12
EOD

stats $Mydata using 3
plot $Mydata using 1:3, $Mydata using 0:2, '' using 1:2



Notes:

- The data block name must begin with a '$'.  This allows the
  named data blocks to be tracked internally using the same code
  as other user-defined variables, but prevents a data block name
  from being used in an expression, function, or other places
  that a normal variable would be accepted.

- The end-of-data marker (EOD in the example) can be any
  arbitrary string of alphanumerics beginning with a letter.
  I'm don't know what advantage this gives over requiring a
  fixed character sequence like EOD, but this seems to be how
  here documents are defined in other programming languages.

- The in-line data is read with no interpretation or processing.
  It is simply stored for later use in a "plot" "splot" or "stats"
  command.

- Any plot command that works with a normal text input file should
  work also with an in-line data block.

- The "show variables" command will report
            $Mydata = <5 line datablock>

- Do you think there needs to be a special command to clear all
  existing datablocks?  Right now the only way to clear one is
  to redefine it as a zero-length block:
  $Mydata << EOD
  EOD

Known problems:

- It doesn't work to include a here document definition inside a
  curly-brace delimited clause.  Not that it should ever be necessary
  to do so, but the program should at least not crash if you try it.

- I didn't look into using a data block as input to "fit".
  Do you think this would be useful?

	- Ethan

Re: In-line data [was multiple plot lines with plot '-']

[Allin C. <cot...@wf...>]

On Tue, 12 Jun 2012, Ethan A Merritt wrote:

> Revisiting a discussion from last month...
>
> I had a go at implemented data blocks defined as a "here document".
> The patch is on SourceForge
> <https://sourceforge.net/tracker/?func=detail&aid=3534646&group_id=2055&atid=302055>

This looks great to me. And it's impressive that your patch is 
only 18K, nice and compact. I haven't tested it yet, but will 
do so soon.

Allin Cottrell

Re: In-line data [was multiple plot lines with plot '-']

[Allin C. <cot...@wf...>]

On Tue, 12 Jun 2012, Allin Cottrell wrote:

> On Tue, 12 Jun 2012, Ethan A Merritt wrote:
>
>> Revisiting a discussion from last month...
>> 
>> I had a go at implemented data blocks defined as a "here document".
>> The patch is on SourceForge
>> <https://sourceforge.net/tracker/?func=detail&aid=3534646&group_id=2055&atid=302055>
>
> This looks great to me. And it's impressive that your patch is only 18K, nice 
> and compact. I haven't tested it yet, but will do so soon.

I've now given this mechanism a "live" test, and it worked just 
fine. One of the reasons I like this idea I forgot to mention in the 
previous discussion: my program gretl offers a GUI for configuring 
gnuplot graphs, and for this to work the plot command file must be 
fairly readily parseable. This data-block setup makes it easier and 
more efficient to parse the data to be plotted out of such a file, 
as compared with chasing down a bunch of e-terminated "stdin" 
blocks.

Allin Cottrell