gnuplot-beta — Developers and beta testers of new gnuplot code

Re: disabling 'insecure' parts of gnuplot file

[Ethan M. <merritt@u.washington.edu>]

On Tuesday 31 August 2004 04:31 pm, Edward Peschko wrote:
> 
> Because the whole idea was to make it so:
> 
> 	a) other people can reproduce the graph
> 	b) other people can see the data used to generate the graph
> 	c) other people can modify the data used to generate the graph, or modify
>            the layout.
> 
> You lose lots in the translation to the PNG - just like you lose lots when you
> edit an illustrator or PSP file and don't save it in their native format.

True.
There is another option that I have found useful in the past.
It would achieve exactly what you describe above, but it 
does assume that the script is trusted. What you can do  
is to associate the mime type "application/gnuplot" with 
a gnuplot wrapper script, and have the web site return the
script itself via a browser click.  This is the ultimate in not
losing anything to translation.  For details, see
	
http://www.bmsc.washington.edu/people/merritt/gnuplot_test.html

In my case the script on the server side is trusted, so the
issues you are worried about are not of primary concern.
The data being plotted can either be local to the user's machine,
or can be pulled from the web site via http protocols.
If the data were on a wiki site, my procedure would allow 
your (a), (b) and (c) above.


-- 
Ethan A Merritt       merritt@u.washington.edu
Biomolecular Structure Center
Mailstop 357742
University of Washington, Seattle, WA 98195

Re: disabling 'insecure' parts of gnuplot file

[Edward P. <es...@pg...>]

On Tue, Aug 31, 2004 at 04:29:04PM -0700, Ethan Merritt wrote:
> On Monday 30 August 2004 12:56 pm, Edward Peschko wrote:
> > 
> > I'm writing an application whose intent is to take a gnuplot file, turn it into a
> > PNG graph, and then display that graph online.
> 
> > IE: it would be up to the user to use programs
> > to create data, etc. which would then be uploaded to mediawiki.
> 
> I took a step back to think about what you said you want to do,
> and now I am puzzled.  If the external user is writing the gnuplot script,
> then I presume they know how to use gnuplot.  You also say that the
> external user will be generating the data.  So why can't they just run
> the script themselves and upload the resulting png file to the wiki,
> rather than uploading the script and having the wiki's host machine
> run it there?

Because the whole idea was to make it so:

	a) other people can reproduce the graph
	b) other people can see the data used to generate the graph
	c) other people can modify the data used to generate the graph, or modify
           the layout.

You lose lots in the translation to the PNG - just like you lose lots when you
edit an illustrator or PSP file and don't save it in their native format.

Ed

Re: disabling 'insecure' parts of gnuplot file

[Edward P. <es...@pg...>]

On Tue, Aug 31, 2004 at 02:41:28PM -0700, Ethan Merritt wrote:
> On Tuesday 31 August 2004 12:47 pm, Edward Peschko wrote:
> > 
> > > you could disable pipes, but the program you were left with wouldn't
> > > be very useful.
> > 
> > not really true, IMO. In mediawiki we'd probably want to limit plotting to inline,
> > which I asked about the other time. IE: it would be up to the user to use programs
> > to create data, etc. which would then be uploaded to mediawiki.
> 
> ???
> What do you mean by "in line"?
> I normally interpret that to mean "via pipe", but that's exactly what
> you would be disabling.

plot '-' index 0, '-' index1
..
..
..
..
e

via 

help plot special

I'm assuming this isn't implemented via pipes, but I to be fair I don't know.

> > > I think the only possible mechanism would be to create a 
> > > wrapper script that set the UID/EID to a non-privileged user
> > > with no permission to write outside of a captive directory tree.
> > 
> > Its barely possible, but its still pretty ugly... You'd need a separate 
> > user/etc for each graph.
> 
> I don't think you would.  The wrapper script itself could save the
> output graph back to the user's own area.  Its flow would look like:
> 	stdin = open input
> 	stdout = open output
> 	drop privileges
> 	mkdir /tmp_<process_id>
> 	chroot /tmp_<process_id> gnuplot

yes, I could do this (I guess). It sort of sucks that I need to bend the 
tool in order to get something done though.

And chroot itself is a pain in the #!% to get working correctly. You need
to get the entire library structure, all associated files with the command
put underneath /tmp/_<process_id> in order to run it (remember - the 
OS doesn't have access to them as soon as you run chroot). And you'd have
to do this with *every call to chroot*.


Plus, portability would go out the window because different systems would
require different files to be copied for chroot, and some systems chroot
doesn't work under anything but 'root' - so you'd need root perms to run
it. 

And trying to sell the idea of making it standard functionality to 
include with mediawiki would be well-nigh impossible.

I think you should take the effort to make it compilably secure, but 
that's just me...  I would think that you would *want* the ability to 
run gnuplot using users' input. It would make the tool that much more 
powerful.

Ed

Re: disabling 'insecure' parts of gnuplot file

[Ethan M. <merritt@u.washington.edu>]

On Monday 30 August 2004 12:56 pm, Edward Peschko wrote:
> 
> I'm writing an application whose intent is to take a gnuplot file, turn it into a
> PNG graph, and then display that graph online.

> IE: it would be up to the user to use programs
> to create data, etc. which would then be uploaded to mediawiki.

I took a step back to think about what you said you want to do,
and now I am puzzled.  If the external user is writing the gnuplot script,
then I presume they know how to use gnuplot.  You also say that the
external user will be generating the data.  So why can't they just run
the script themselves and upload the resulting png file to the wiki,
rather than uploading the script and having the wiki's host machine
run it there?

-- 
Ethan A Merritt       merritt@u.washington.edu
Biomolecular Structure Center
Mailstop 357742
University of Washington, Seattle, WA 98195

Re: disabling 'insecure' parts of gnuplot file

[Arun P. <ape...@lb...>]

Ethan Merritt wrote:
> On Tuesday 31 August 2004 12:47 pm, Edward Peschko wrote:
>>>[disabled pipes]
 >>not really true, IMO. In mediawiki we'd probably want to limit 
plotting to inline,
>>which I asked about the other time. IE: it would be up to the user to use programs
>>to create data, etc. which would then be uploaded to mediawiki.
> 
> ???
> What do you mean by "in line"?

I guess something like
plot "-"
followed by the data...

>>>[wrapper script issues]
> I don't think you would.  The wrapper script itself could save the
> output graph back to the user's own area.  Its flow would look like:
> 	stdin = open input
> 	stdout = open output
> 	drop privileges
> 	mkdir /tmp_<process_id>
> 	chroot /tmp_<process_id> gnuplot

this and a having the wiki (per php or similar) check the gnuplot input 
file for "!" "set output","print" and other commands which could be 
harmfull and not needed for just simple ploting etc. might be a good way 
to do it then...

perhaps you even want to be very restrictive anyway to get the same look 
for most of the graphs in the wiki and so the user would only need to 
supply the data and the plotstyle and then you would give gnuplot most 
of the input (e.g. use of grid yes/no, set terminal, etc) and
your plot command would always look like:

plot "-" $userplotstyle
$data

in that case you just have to scan those two variables for dangerous 
items or just allow for example numbers in $data and certain words in 
$userplotstyle

just my two cents

ARUN

Re: disabling 'insecure' parts of gnuplot file

[Ethan M. <merritt@u.washington.edu>]

On Tuesday 31 August 2004 12:47 pm, Edward Peschko wrote:
> 
> > you could disable pipes, but the program you were left with wouldn't
> > be very useful.
> 
> not really true, IMO. In mediawiki we'd probably want to limit plotting to inline,
> which I asked about the other time. IE: it would be up to the user to use programs
> to create data, etc. which would then be uploaded to mediawiki.

???
What do you mean by "in line"?
I normally interpret that to mean "via pipe", but that's exactly what
you would be disabling.


> > I think the only possible mechanism would be to create a 
> > wrapper script that set the UID/EID to a non-privileged user
> > with no permission to write outside of a captive directory tree.
> 
> Its barely possible, but its still pretty ugly... You'd need a separate 
> user/etc for each graph.

I don't think you would.  The wrapper script itself could save the
output graph back to the user's own area.  Its flow would look like:
	stdin = open input
	stdout = open output
	drop privileges
	mkdir /tmp_<process_id>
	chroot /tmp_<process_id> gnuplot


-- 
Ethan A Merritt       merritt@u.washington.edu
Biomolecular Structure Center
Mailstop 357742
University of Washington, Seattle, WA 98195

Re: disabling 'insecure' parts of gnuplot file

[Hans-Bernhard B. <br...@ph...>]

Arun Persaud wrote:

> How about disabling the "!" command completely and checking that the 
> output file is in the right directory... deleting the "< ..."  option 
> for input files might also be a good thing...

As Ethan already pointed out, that would run into serious usability 
problems.

> shouldn't be too much work to delete the right lines in the source code...

Deleting them: no, that wouldn't be hard.  But *finding* them would be,
esp. if you want to be sure you found not just some of the relevant 
lines, but strictly *all* of them.  Miss just one, and you've turned a 
known-insecure program into a presumably secure one with a hole.  That 
would be rather the opposite of an improvement.

> I'm not sure though in what other ways you could trick gnuplot to do 
> dangerous things...

Neither are we.  That's why I wrote that there are no way of doing that 
is foreseen in the source.  This would require a full-blown audit of the 
entire code, and IMHO neither the team nor the code is in the condition 
that would warrant trying that.

Re: disabling 'insecure' parts of gnuplot file

[Arun P. <ape...@lb...>]

Edward Peschko wrote:
> Would a 'taint' gnuplot be possible? (ie: compile it such that dangerous 
> behaviors are not allowed? Or perhaps a switch to gnuplot that doesn't allow them?
> 
> And is the above the only dangerous behaviour that is possible from gnuplot?

How about disabling the "!" command completely and checking that the 
output file is in the right directory... deleting the "< ..."  option 
for input files might also be a good thing...

I also would just including one terminal e.g. for png

shouldn't be too much work to delete the right lines in the source code...

if you don't want to mess with the gnuplot source code just write a 
php-file that checks the gnuplot input file before you call gnuplot and 
  disallow the use of "^\s*!" ".*plot.*<  ", etc...

I'm not sure though in what other ways you could trick gnuplot to do 
dangerous things...

HTH
	ARUN

Re: disabling 'insecure' parts of gnuplot file

[Edward P. <es...@pg...>]

On Tue, Aug 31, 2004 at 12:30:38PM -0700, Ethan Merritt wrote:
> On Tuesday 31 August 2004 11:50 am, Edward Peschko wrote:
> > And is the above the only dangerous behaviour that is possible from gnuplot?
> 
> No.  Not by a long shot.
> 
> Consider:
> 
> set output "~/.login"       # trash the user's login file
> print `dd if=/bin/zero of=/somewhere/bad`    # abuse the back-tic syntax
> plot '< rm -rf .'        #abuse the pipe mechanism
> sh "bad command" # abuse the shell escape mechanism
> 
> You'd have to disable all of these, and probably a few more
> that I'm not thinking of at the moment.   

> Disabling shell escapes and back-tics would not harm most
> applications. But disabling pipes, in particular,
> would drastically cripple gnuplot's flexibility and ability to work with
> other programs.   That's probably a deal-breaker right there;

well, of course I wouldn't expect a safe version of gnuplot to come standard -
I could live with compiling gnuplot with a separate flag that gets rid of these
things.

> you could disable pipes, but the program you were left with wouldn't
> be very useful.

not really true, IMO. In mediawiki we'd probably want to limit plotting to inline,
which I asked about the other time. IE: it would be up to the user to use programs
to create data, etc. which would then be uploaded to mediawiki.

> And what could you possibly do to limit "set output <foo>"?

disallow it... let output only go to STDOUT..

> I think the only possible mechanism would be to create a 
> wrapper script that set the UID/EID to a non-privileged user
> with no permission to write outside of a captive directory tree.

Its barely possible, but its still pretty ugly... You'd need a separate 
user/etc for each graph. Suppose we had a chroot jail and someone did
'rm -rf /'? The fact that it only eliminated all of the mediawiki graphs
is pretty small consolation..

Ed

Re: disabling 'insecure' parts of gnuplot file

[Ethan M. <merritt@u.washington.edu>]

On Tuesday 31 August 2004 11:50 am, Edward Peschko wrote:
> And is the above the only dangerous behaviour that is possible from gnuplot?

No.  Not by a long shot.

Consider:

set output "~/.login"       # trash the user's login file
print `dd if=/bin/zero of=/somewhere/bad`    # abuse the back-tic syntax
plot '< rm -rf .'        #abuse the pipe mechanism
sh "bad command" # abuse the shell escape mechanism

You'd have to disable all of these, and probably a few more
that I'm not thinking of at the moment.   

Disabling shell escapes and back-tics would not harm most
applications. But disabling pipes, in particular,
would drastically cripple gnuplot's flexibility and ability to work with
other programs.   That's probably a deal-breaker right there;
you could disable pipes, but the program you were left with wouldn't
be very useful.

And what could you possibly do to limit "set output <foo>"?

I think the only possible mechanism would be to create a 
wrapper script that set the UID/EID to a non-privileged user
with no permission to write outside of a captive directory tree.

-- 
Ethan A Merritt       merritt@u.washington.edu
Biomolecular Structure Center
Mailstop 357742
University of Washington, Seattle, WA 98195

Re: disabling 'insecure' parts of gnuplot file

[Edward P. <es...@pg...>]

On Tue, Aug 31, 2004 at 10:09:03AM +0200, Hans-Bernhard Broeker wrote:
> Edward Peschko wrote:
> 
> >Anyway, I would like to therefore disable anything that gnuplot can do 
> >outside of drawing graphs, for example:
> >
> >!rm -rf /
> 
> No way of doing that is foreseen inside gnuplot --- you'll have to use
 external means, e.g. run it in a chroot or jail environment.

well, the main idea was to integrate it into wiki (to be used for preparing 
graphs). Hence a chroot/jail environment might not be possible since the server
would be running the software in producing the graph.  Just making a chroot
environment for the sake of gnuplot is not going to happen.

Would a 'taint' gnuplot be possible? (ie: compile it such that dangerous 
behaviors are not allowed? Or perhaps a switch to gnuplot that doesn't allow them?

And is the above the only dangerous behaviour that is possible from gnuplot?

> look up help plot datafile special

thanks, that worked..

Ed

Re: Propositions to Improve Gnuplot

[Hans-Bernhard B. <br...@ph...>]

On Tue, 31 Aug 2004, Bernhard [utf-8] Kr=C3=A4mer wrote:

> 1) Introduce the Unix "ls" command to the gnuplot shell?

What's wrong with the existing
=09
=09!ls

?  Remember the Unix philosophy: one tool <--> one task.=20

> 2) Introduce a tab-completion feature as in xterm?

Exists already --- you just have to link with GNU libreadline.

> 3) Get rid of the "bug" that is when you type a command over more t=
han one=20
> line, and return to the first line in order to insert something, it=
 is not=20
> readable what you are typing?

Should be fixed if you use libreadline, too.

Or just use shorter lines --- that's what you backslash-newline=20
continuation for.

--=20
Hans-Bernhard Broeker (br...@ph...)
Even if all the snow were burnt, ashes would remain.

Propositions to Improve Gnuplot

[Bernhard <Ber...@we...>]

Hello,

I don't know if these propositions have been posted yet, (and if it is the 
right place to make such propositions,) but they would help improving the yet 
great Gnuplot software.

It is possible to:

1) Introduce the Unix "ls" command to the gnuplot shell?
2) Introduce a tab-completion feature as in xterm?
3) Get rid of the "bug" that is when you type a command over more than one 
line, and return to the first line in order to insert something, it is not 
readable what you are typing?

These three things would greatly improve gnuplot and facilitate its use. 

Yours, 

 Bernhard Krämer

(To contact me, please reply to my mail address)

Re: image/binary patch complete

[Daniel J S. <dan...@ie...>]

Ethan A Merritt wrote:

>On Friday 27 August 2004 12:05 am, mi...@ph... wrote:
>  
>
>>>The patch has been uploaded to the SourceForge site.
>>>("with_image_26aug2004.patch" and "with_image_data_24aug2004.patch" are
>>>      
>>>
>>It works very well for me. I propose to commit the patch into CVS.
>>    
>>
>
>I like the layout of this version much better than the earlier ones.
>I think there are some places still that can be cleaned up, but I hope
>there will be adequate opportunity for that afterwards.
>So adding it to CVS is OK with me.
>  
>

Yes, there are several things that can be cleaned up yet, especially if 
the #ifdef .. #else .. #endif constructs are untangled.  And as always, 
one codes something up and then later realizes a way of improving 
things.  Passing necessary information into the datafile.c routines is 
the one thing that clearly needs a standard.  I'm indifferent about 
that, as it can be done several ways.

Dan

Re: disabling 'insecure' parts of gnuplot file

[Hans-Bernhard B. <br...@ph...>]

Edward Peschko wrote:

> Anyway, I would like to therefore disable anything that gnuplot can do outside of 
> drawing graphs, for example:
> 
> !rm -rf /

No way of doing that is foreseen inside gnuplot --- you'll have to use
external means, e.g. run it in a chroot or jail environment.

> How can you make it so that the plot statement refers to itself, ie a dataset in the 
> file itself?

See 'help plot datafile special'

disabling 'insecure' parts of gnuplot file

[Edward P. <es...@pg...>]

hey all,

I'm writing an application whose intent is to take a gnuplot file, turn it into a
PNG graph, and then display that graph online.

Anyway, I would like to therefore disable anything that gnuplot can do outside of 
drawing graphs, for example:

!rm -rf /

considering that I don't want someone to upload something to wipe out my 
system (or even try)..

Also - I was wondering - is there a good format that people would suggest to 
make any and all plots self-contained? I'd like something like:

	- one plot per graph file

	- forcing of png format

	- data and plot statement in the same file

yet the plot statemnts (at least for data) tend to be of the form

plot [x1:x2] [y1:y2] "filename"

How can you make it so that the plot statement refers to itself, ie a dataset in the 
file itself?

Ed

Re: image/binary patch complete

[<mi...@ph...>]

>> > The patch has been uploaded to the SourceForge site.
>> > ("with_image_26aug2004.patch" and "with_image_data_24aug2004.patch"
>>
>> It works very well for me. I propose to commit the patch into CVS.
>
> I like the layout of this version much better than the earlier ones. I
> think there are some places still that can be cleaned up, but I hope
> there will be adequate opportunity for that afterwards.
> So adding it to CVS is OK with me.

Thus, unless there is a strong vote against, I will commit the patch to
cvs on September 1.
---
PM

Re: image/binary patch complete

[Ethan A M. <merritt@u.washington.edu>]

On Friday 27 August 2004 12:05 am, mi...@ph... wrote:
> >
> > The patch has been uploaded to the SourceForge site.
> > ("with_image_26aug2004.patch" and "with_image_data_24aug2004.patch" are
>
> It works very well for me. I propose to commit the patch into CVS.

I like the layout of this version much better than the earlier ones.
I think there are some places still that can be cleaned up, but I hope
there will be adequate opportunity for that afterwards.
So adding it to CVS is OK with me.

-- 
Ethan A Merritt
Department of Biochemistry & Biomolecular Structure Center
University of Washington, Seattle

Re: image/binary patch complete

[<mi...@ph...>]

>  OK, so here it is:  the image/binary patch with all the alterations to
>
> reading binary files that I had envisioned.  I think it works pretty
> well, and it is as far as I wish to take the code at this time.  In
> this  last version, it is the data file stuff that has changed.  So
> only that  is addressed here.
>
> The patch has been uploaded to the SourceForge site.
> ("with_image_26aug2004.patch" and "with_image_data_24aug2004.patch" are

It works very well for me. I propose to commit the patch into CVS.

Petr

image/binary patch complete

[Daniel J S. <dan...@ie...>]

  OK, so here it is:  the image/binary patch with all the alterations to 
reading binary files that I had envisioned.  I think it works pretty 
well, and it is as far as I wish to take the code at this time.  In this 
last version, it is the data file stuff that has changed.  So only that 
is addressed here.

The patch has been uploaded to the SourceForge site. 
 ("with_image_26aug2004.patch" and "with_image_data_24aug2004.patch" are 
required.)  If you want to see just the results, I've updated the PDF 
output of the demo at

http://acer-access.com/~ds...@ac.../gnuplot/image.pdf

There are a few more plots, dealing mainly with matrix (gnuplot) binary 
and matrix ASCII files and how they are integrated with the new 
"df_readbinary()" variant of "df_readline()".

I'll be out of town the rest of this week, and I won't have much time to 
work on this anymore once September rolls around.  But as I've said, 
I've taken this now to the point of what I imagined.  Minor details 
about the syntax might not be agreeable with everyone, but I think in 
terms of coding, it would be fairly straightforward to alter bits of 
code to achieve any conceivable alterations...  As for incremental 
features, like data compression in PostScript files, maybe a project for 
next year.

Dan

PS:  If you want to read on, here are the main points:

** I've put map3d_xy() back to the way it was so that creating a 
PostScript file for `all.dem` under the CVS version and the patched 
version differs only in the dates.  This means that map3d_xy_double() is 
pretty much a replication of map3d_xy().  I've left in, and #defined 
out, the method where map3d_xy() is derived from map3d_xy_double(), 
along with a descriptive note in case that can be simplified in the future.

**  I've moved all binary (and ASCII matrix) file reads into the new 
routine df_readbinary().  With some simple alterations to this routine, 
it can now read in binary matrix (gnuplot binary) files in a "one at a 
time" fashion.  This achieves several things:

1)  It makes for only one way to read data from a file, i.e., 
df_readline().  This is very nice because it makes plot2d.c and plot3d.c 
very similar in program flow.  Good for future development.  Also, it 
allows matrix binary to be used in 2d plots.  Useful for images, or 
perhaps arrow style.  [Of course, it is limited in that there is only 
one "information" column beyond the first two coordinate columns.]

2)  There are now just two spots for user spec interpretation code, in 
df_readascii() and df_readbinary().  Hence both matrix binary and 
general binary use the same using code (very nice!) and furthermore the 
features for general binary may also be used for matrix binary.  [Note 
however, that I've not combined the using code for df_readascii() with 
that of df_readbinary()... maybe a short subroutine to fill in the v[] 
vector would be a good idea after all.]  I've added a few additional 
plots in the `image.dem` script to illustrate and test that capability. 
 [One that I added includes `binary3` four times, translated to 
different locations... gnuplot really crunches away on that one, 
computing the hidden lines.  But I'm impressed it works; it looks 
funky...  Slightly different color scheme between X11 and PDF for the 
last plot, dark blue in X11, peach in PDF.]

3)  Another nice thing about this is that `binary` documentation becomes 
simplified.  Now all the additional syntax for general binary applies to 
matrix binary as well.  So no more of this clumsy documentation for 
`binary` where there is one set of rules for `splot` and another set of 
rules for all other cases.  Hence, I've reorganized the documentation 
slightly so that, for example, 'help binary' gives
__________

 The `binary` keyword allows a data file to be binary as opposed to ASCII.
 There are two formats for binary--matrix binary and general binary.  Matrix
 binary is a fixed format in which data appears in a 2D array with an extra
 row and column for coordinate values.  General binary is a flexible format
 for which details about the file must be given at the command line.

 See `binary matrix` or `binary general` for more details.

Subtopics available for binary:
    general           matrix
__________

and 'help matrix' gives
__________

 The `matrix` flag indicates that the file data (ASCII or binary) are stored
 in matrix format.  The formats are slightly different amongst these 
two.  For
 details, see `matrix ascii` or `matrix binary`.

Subtopics available for matrix:
    ascii             binary
___________

The descriptions under this sub-categories then are pretty much what 
existed before, but I've weeded out anything tying the file types to 
`plot` or `splot`.

4)  df_3dmatrix() is no longer required and is commented out of the 
code.  Similarly, the code inside binary.c is no longer required.  That 
is conditionally commented out as well.  But it is a bit tricky because 
the code inside binary.c is required for the program bf_test.  Since the 
contents inside `binary.c` can't be compiled two different ways at once, 
and I can't alter the Makefile.am file because that would not work for 
when BINARY_DATA_FILE is not defined.  So, I've created a new file 
called `bin_hook.c` as part of the gnuplot_SOURCES, and inside of there 
is only the lines

#ifndef BINARY_DATA_FILE
#include "binary.c"
#endif

** I'll say a bit here about how the integration of matrix binary and 
matrix ASCII into df_readbinary() works.  In the case of matrix binary, 
there is only one data set that can be in a binary matrix file, such 
that after opening the file, a few uses of fread(), ftell() and fseek() 
can figure out the dimensions of the data and locations of the grid 
corners.  The information is then put in the "binary_record" array that 
df_readbinary() uses. There is no need to read through all the data in 
the file.  If df_readline() knows the dimensions of the array, it can 
handle reading in the first row, first column, etc. on an "as you go" basis.

The story is slightly different for ASCII matrix data.  There one can't 
figure out the size of the matrix until _all_ the data has been read in. 
 There was a routine for doing that, df_read_matrix().  I wanted to 
re-use that.  So the strategy is to read in all the data, store it as 
floats in memory, then inside df_readline() rather than pulling data 
from a FILE * is comes from memory as a byte stream as though it were 
from a file.  (There are assurances in the code that the endianess is 
properly observed.)  Pretty simple.  I slightly altered df_read_matrix() 
so that it returns after getting a blank line indicating the end of a 
record.  That is, here is the core of the ASCII matrix routine:

    /* Keep reading matrices until file is empty. */
    while (1) {
        if ((matrix = df_read_matrix(&nr, &nc)) != NULL) {
            int index = df_num_bin_records;
            /* *** Careful!  Could error out in next step.  "matrix" should
             * be static and test next time. ***
             */
            df_add_binary_records(1, DF_CURRENT_RECORDS);
            df_bin_record[index].memory_data = (char *) matrix;
            matrix = NULL;
            df_bin_record[index].scan_dim[0] = nc;
            df_bin_record[index].scan_dim[1] = nr;
            df_bin_record[index].scan_dim[2] = 0;
            df_bin_file_endianess = THIS_COMPILER_ENDIAN;
        } else
            break;
    }

This sets up df_readline() to do its thing.  Note that with this setup, 
`index` works for ASCII matrix (see demos), which I'm not it did before, 
in reading some comments in datafile.c.

So ASCII matrix reads in all the data to memory, binary matrix doesn't. 
 (I know, but the point[] array in the plot is much more than the 
original data file.  But good programming is good practice.  Also, if 
decimation were used, then the contents that end up in memory will be a 
fraction of the original file size.)  But, generally, ASCII matrix won't 
be such big files, so reading all the contents of an ASCII data file 
into memory isn't the worst of methods.  (The alternative would be to 
read the ASCII file twice... don't like that one!)

** Internal to datafile.c, I've altered the variable df_matrix slightly. 
 Near the end of df_open(), df_matrix is set to TRUE if the data came 
from a matrix format (either binary or ASCII) _or_ if it came from 
binary data when the `array` keyword were used (i.e., two dimensions 
with consistent coordinates).  This makes df_matrix still accessible to 
the rest of gnuplot code, but it ensures that no outside code can alter 
the value and change inadvertently how the datafile.c code might behave 
for successive calls to df_readline().

A similar idea is used for the df_binary variable.

So inside of datafile.c now there are a variety of similar variables, 
that at first seem like name overload:

/* Logical variables indicating information about data file. */
TBOOLEAN df_binary_file;
TBOOLEAN df_matrix_file;

/* Binary *read* variables used by df_readbinary().  The difference 
between matrix
 * binary and general binary is that matrix binary requires an extra 
first column
 * and extra first row giving the sample coordinates.  Furthermore, note 
that if
 * ASCII matrix data is converted to floats (i.e., binary) then it 
really falls in
 * the general binary class, not the matrix binary class.
 */
TBOOLEAN df_read_binary;
TBOOLEAN df_matrix_binary;

As the note says, the fact that ASCII matrix and binary matrix are 
slightly different complicates matters.  I think it is easier going with 
a few more variables than trying to assign so much meaning to df_matrix 
conditioned on what other setting might be.

** I propose that eventually "df_binary" be removed from datafile.h and 
datafile.c.  There is only one instance where df_binary is used outside 
of datafile.c:

        if (this_plot->plot_type == DATA3D && df_binary==TRUE && 
end_token==start_token+1)
            /* let default title for  splot 'a.dat' binary  is 'a.dat'
             * while for  'a.dat' binary using 2:1:3  will be all 4 words */
            m_capture(&(this_plot->title), start_token, start_token);
        else

But, as I've argued before, there is no reason for the rest of gnuplot 
code to know whether or not data came from a binary file.  Can the above 
bit of code be changed so that it keys off the presence of a using string?

I've left a note in plot3d.c near the above code as a reminder to 
address this eventually.  However, I didn't alter the code because that 
would probably cause the output PostScript files for 'all.dem' for old 
and patched versions of gnuplot to be different.

** You may be wondering about `df_matrix`, why should that variable be 
made available to code outside datafile.c?  Well, that is useful 
information to some parts of gnuplot, specifically the scan line, grid 
code.  Also, the image code can make use of such a variable, but one 
that indicates _uniform_ grid, which df_matrix doesn't necessarily do. 
 (However, in the case of image code, it is just a shortcut to save some 
computations on the "grid check" code that the image routine uses 
otherwise.)

** Why is ASCII matrix data not consistent with binary matrix data in 
the sense that the x and y coordinates may be derived from the first row 
and column of the matrix?  Matrix data with row indeces used for 
coordinates has some use, but having something similar to binary matrix 
is more useful.  I'm not sure which came first, but if ASCII matrix is 
fairly recent, would it be possible to change the format to the same as 
gnuplot binary?  I know this sort of thing is not a nice thing to do to 
users, but if use of ASCII matrix isn't widespread, the repercussions 
might not be too bad.  I inquire because I notice that there is no use 
of `matrix` in any of the demo scripts.

I'm not too concerned about this, however.  I think there is some 
flexibility there with being able to pass ASCII matrix coordinates 
(i.e., the indeces) through a function.  E.g., "using (2*$1):(0.5*$2):3".

** The last item here has to do with not being able to pass coordinates 
generated for general binary through a function, just as the 0, -1, -2 
fields indicating datum, line, block can't go through a function.  I 
don't think this is of immediate concern because I can't think of too 
many ways that the occasion would arise to pass uniformly spaced samples 
through some kind of function.  If you want the gory details about the 
issue, here is a copy of note I sent to Petr:

>I don't know whether I've got the point, but I think that in the (new)
>binary image matrix you cannot address x- and y- indices, i.e.  splot 'a.edf' using -2:-1:3 with image
>to index the axes via column number and row number, or can you?
>

Yes, one can select just a particular column,

plot 'blutux.rgb' binary array=128x128 flip=y format='%uchar' using 3 
with image

But you are onto the point I was trying to make.

One point is that I don't want to have the notation so that the example 
you give, would be

splot 'a.edf' using 1:2:5


Where 1 and 2 are the generated, or implicit, columns... like in the 
case of `matrix`...  I think that is downright confusing.  The user sets 
up their columns in a file, then they have to remember to add two to the 
front of it.  Just confusing I think.

> What was
>the point of your discussion?  Actually I would like these -1 and -2. Just few days earlier I wanted to
>  find the measurement number at a given angle (x-axis of the image), and
>  I had to recalculate the image (from experimental data).
>Petr
>  
>

Plus, you know you should be able to override any in file settings, at 
the command line, if that helps your situation any.

Here the 0, -1 and -2 are like in the datum, line count and index.  They 
are not the _generated_ values associated with array=50x50, for example. 
 That was another of my points.

And my last point is that the 0, -1 and -2 can't be passed through an 
expression the way columns can, i.e., $1.

You see, in order for something to be used in an expression, I believe 
that quantity needs to exist in the df_column[] array.  But the 0, -1, 
and -2 information are not put into a column, they are just drawn from a 
variable, i.e.,

        } else if (column == -2) {
            v[output] = df_current_index;
        } else if (column == -1) {
            v[output] = line_count;
        } else if (column == 0) {
            v[output] = df_datum;    /* using 0 */

And similarly, the generated coordinates that I added also don't go into 
df_column[]:

        } else if (column == -5) {      /* Perhaps try using a switch 
statement to avoid so many tests. */
            v[output] = o_value*delta[2];
        } else if (column == -4) {
            v[output] = n_value*delta[1];
        } else if (column == -3) {
            v[output] = m_value*delta[0];

Now, I can easily change that by stuffing those values instead into 
df_column[].  Say, under the hood I extend the df_column[] array by 
three and put those values in there.  Then they could be passed through 
an expression, if only the thing that parses the expressions knows that 
it should treat a -1, say, as (df_no_cols + 1).  See my point?  In the 
case of `matrix`, the indeces are immediately stuffed into df_column[0] 
and df_column[1]:

        /* Fill backward so that current read value is not overwritten. */
        for (j=df_no_bin_cols-1; j >= 0; j--) {
            if (j == 0)
            df_column[j].datum = df_matrix_binary ? 
scanned_matrix_row[df_M_count] : df_M_count;
            else if (j == 1)
            df_column[j].datum = df_matrix_binary ? first_matrix_column 
: df_N_count;
            else
            df_column[j].datum = df_column[i].datum;
            df_column[j].good = DF_GOOD;
            df_column[j].position = NULL;
        }

and from that point forward can be used in expressions as $1 and $2.

Dan

Re: maximing/resizing multiple x11: change of palette

[Daniel J S. <dan...@ie...>]

I believe that may be tied into the same problem area.  I think I 
described something in that recursive code where removing one 
conditional statement would allow the palette to be stored and then 
reinstalled upon refreshing the graph.  But the problem was that with 
that alteration the palette wasn't orignally coming out with the proper 
number of colors.  The behavior seemed similar to this example.

Dan


mi...@ph... wrote:

>I have found yet another bug of X11 terminal which may (or may not?) have
>a common base with the bug in $Subj. Please try this:
>set palette maxcolors 4
>set pm3d
>set term x11 1
>splot x
>set term x11 2
>splot x
>
>=> only the 1st x11 window shows the palette correctly, the following
>ignore 'set palette' options. It seems like x11 window structure is either
>not copying the palette information, or ignores it.
>---
>Petr Mikulik
>
>
>
>
>
>-------------------------------------------------------
>SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
>100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
>Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
>http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
>_______________________________________________
>gnuplot-beta mailing list
>gnu...@li...
>https://lists.sourceforge.net/lists/listinfo/gnuplot-beta
>
>
>  
>

-- 

Dan Sebald
email: daniel DOT sebald AT ieee DOT org
URL: http://acer-access DOT com/~dsebald AT acer-access DOT com/

Re: maximing/resizing multiple x11: change of palette

[<mi...@ph...>]

I have found yet another bug of X11 terminal which may (or may not?) have
a common base with the bug in $Subj. Please try this:
set palette maxcolors 4
set pm3d
set term x11 1
splot x
set term x11 2
splot x

=> only the 1st x11 window shows the palette correctly, the following
ignore 'set palette' options. It seems like x11 window structure is either
not copying the palette information, or ignores it.
---
Petr Mikulik

Re: gnuplot manual misprint (no.14)

[Lars H. <lhe...@us...>]

Shigeharu TAKENO writes:
> shige 08/22 2004
> ----------------
> 
> In gnuplot.doc of current CVS version
> 
>  C RCS $Id: gnuplot.doc,v 1.235 2004/08/20 05:59:46 sfeam Exp $
> 
> we found some points that seem misprints or differ from real 
> feauture of gnuplot-current.
> 
> I send the context diff file for it.
 
 Thanks, these changes are in cvs now.

gnuplot manual misprint (no.14)

[Shigeharu T. <sh...@ie...>]

shige 08/22 2004
----------------

In gnuplot.doc of current CVS version

 C RCS $Id: gnuplot.doc,v 1.235 2004/08/20 05:59:46 sfeam Exp $

we found some points that seem misprints or differ from real 
feauture of gnuplot-current.

I send the context diff file for it.

----- From here -----
*** gnuplot.doc.orig	Sun Aug 22 18:57:45 2004
--- gnuplot.doc	Sun Aug 22 19:15:42 2004
***************
*** 164,170 ****
   The list of those interested in beta-test versions is:
         gnu...@li...
  
!  There is also the canonical (if occassionally out-of-date) gnuplot web page at
  
  ^ <a href="http://www.gnuplot.info">
             http://www.gnuplot.info
--- 164,170 ----
   The list of those interested in beta-test versions is:
         gnu...@li...
  
!  There is also the canonical (if occasionally out-of-date) gnuplot web page at
  
  ^ <a href="http://www.gnuplot.info">
             http://www.gnuplot.info
***************
*** 1550,1556 ****
         pause mouse keypress
         print "Keystroke ", MOUSE_KEY, " at ", MOUSE_X, " ", MOUSE_Y
  
!  If 'pause mouse keypress' is terminated by a mouse click rather than by a
   keypress, then MOUSE_KEY will contain the button number, otherwise it will
   contain the ascii character value of the key that was pressed. If the
   pause command is terminated abnormally (e.g. by ctrl-C or by externally
--- 1550,1556 ----
         pause mouse keypress
         print "Keystroke ", MOUSE_KEY, " at ", MOUSE_X, " ", MOUSE_Y
  
!  If `pause mouse keypress` is terminated by a mouse click rather than by a
   keypress, then MOUSE_KEY will contain the button number, otherwise it will
   contain the ascii character value of the key that was pressed. If the
   pause command is terminated abnormally (e.g. by ctrl-C or by externally
***************
*** 3087,3105 ****
  5 x2ticlabels
  ?using x2ticlabels
  ?plot using x2ticlabels
!  See plot using xticlabels.
  5 yticlabels
  ?using yticlabels
  ?plot using yticlabels
!  See plot using xticlabels.
  5 y2ticlabels
  ?using y2ticlabels
  ?plot using y2ticlabels
!  See plot using xticlabels.
  5 zticlabels
  ?using zticlabels
  ?plot using zticlabels
!  See plot using xticlabels.
  3 errorbars
  ?commands plot errorbars
  ?commands splot errorbars
--- 3087,3105 ----
  5 x2ticlabels
  ?using x2ticlabels
  ?plot using x2ticlabels
!  See `plot using xticlabels`.
  5 yticlabels
  ?using yticlabels
  ?plot using yticlabels
!  See `plot using xticlabels`.
  5 y2ticlabels
  ?using y2ticlabels
  ?plot using y2ticlabels
!  See `plot using xticlabels`.
  5 zticlabels
  ?using zticlabels
  ?plot using zticlabels
!  See `plot using xticlabels`.
  3 errorbars
  ?commands plot errorbars
  ?commands splot errorbars
***************
*** 4067,4073 ****
  ?show boxwidth
  ?boxwidth
   The `set boxwidth` command is used to set the default width of boxes in the
!  `boxes`, `boxerrorbars`, `candlesticks` and `histogram` styles.
  
   Syntax:
         set boxwidth {<width>} {absolute|relative}
--- 4067,4073 ----
  ?show boxwidth
  ?boxwidth
   The `set boxwidth` command is used to set the default width of boxes in the
!  `boxes`, `boxerrorbars`, `candlesticks` and `histograms` styles.
  
   Syntax:
         set boxwidth {<width>} {absolute|relative}
***************
*** 5856,5868 ****
   to give the `set size` and `set origin` commands before each plot:  Those
   are generated automatically, but can be overridden at any time.  With
   `layout` the display will be devided by a grid with <cols> colums and 
!  <rows> rows.  This grid is filled rows first or colums first depending wether
!  `rowmajor` or `columnmajor` is given by the subsequent plot commands.  
!  Default is `rowmajor`.
   Each plot can be scaled by `scale` and shifted with `offset`; if the y-values
!  for shift or offset are ommited, the x-value will be used.
   `unset multiplot` will turn off the automatic layout.  You should manually
!  `set size` and `set offset` to usefull values (i.e. 1,1 and 0,0) after
   `unset multiplot` if you wish to make more plots in non-multiplot-mode
   as `unset multiplot` will leave those values as they were in the automatic
   layout mode.
--- 5856,5868 ----
   to give the `set size` and `set origin` commands before each plot:  Those
   are generated automatically, but can be overridden at any time.  With
   `layout` the display will be devided by a grid with <cols> colums and 
!  <rows> rows.  This grid is filled rows first or colums first depending whether
!  `rowmajor` or `columnmajor` is given by the subsequent option of the 
!  multiplot command. Default is `columnmajor`.
   Each plot can be scaled by `scale` and shifted with `offset`; if the y-values
!  for scale or offset are ommited, the x-value will be used.
   `unset multiplot` will turn off the automatic layout.  You should manually
!  `set size` and `set origin` to usefull values (i.e. 1,1 and 0,0) after
   `unset multiplot` if you wish to make more plots in non-multiplot-mode
   as `unset multiplot` will leave those values as they were in the automatic
   layout mode.
***************
*** 5875,5882 ****
         unset multiplot
  
   Above example will a produce 6 plots in 3 colmns filled top to bottom, left
!  to right.  Each plot will have a horizontal size of 1.1/3 and a vertical
!  size of 0.9/2. 
     
   See also
  ^ <a href="http://gnuplot.sourceforge.net/demo/multiplt.html">multiplot demo (multiplt.dem)</a>
--- 5875,5882 ----
         unset multiplot
  
   Above example will a produce 6 plots in 3 colmns filled top to bottom, left
!  to right.  Each plot will have a horizontal size of 1.1/2 and a vertical
!  size of 0.9/3. 
     
   See also
  ^ <a href="http://gnuplot.sourceforge.net/demo/multiplt.html">multiplot demo (multiplt.dem)</a>
***************
*** 7418,7425 ****
   `histeps` is only a plotting style; `gnuplot` does not have the ability to
   create bins and determine their population from some data set.
  5 histograms
! ?commands set style histograms
! ?set style histograms
  ?style histograms
  ?histograms
   The `histograms` style is only relevant to 2-d plotting.  It produces a bar
--- 7418,7425 ----
   `histeps` is only a plotting style; `gnuplot` does not have the ability to
   create bins and determine their population from some data set.
  5 histograms
! ?commands set style histogram
! ?set style histogram
  ?style histograms
  ?histograms
   The `histograms` style is only relevant to 2-d plotting.  It produces a bar
----- To here -----

+========================================================+
 Shigeharu TAKENO     NIigata Institute of Technology
                       kashiwazaki,Niigata 945-1195 JAPAN
 sh...@ie...   TEL(&FAX): +81-257-22-8161
+========================================================+

Re: demo stuff

[Ethan M. <merritt@u.washington.edu>]

On Saturday 21 August 2004 01:23 am, you wrote:
>  > term->option() does not actually *do*
> > anything; it just loads some fields in the terminal data structures.
>
> Which term->option() in particular are you speaking of?  There are
> dozens of them, each a bit different.

I looked at all of them, and saw a potential problem only in svg.trm.
But I freely admit that it's possible I missed something; that's why I
asked if you had a particular terminal type or bit of code in mind as
a problem area.

All the critical (order-dependent) code seems to be in term->init(), 
rather than term->options().   If this wasn't intentional, then it's a 
lucky coincidence.  Either way, calling term->options looks safe to me.   

Furthermore, I have modified the code in set_termoptions() to call
only those terminal drivers that actually support the option in 
question ('set term font ..' or 'set term {no}enhanced').   
So the term->options() of most drivers will not be called even if
you do issue a 'set termoption' command.

Nevertheless, if there is indeed a fundamental problem with
this approach I can back off to some other mechanism that is
specific to enabling enhanced text mode.

-- 
Ethan A Merritt