gnuplot-beta — Developers and beta testers of new gnuplot code

demo stuff

[Daniel J S. <dan...@ie...>]

Some observations about demos:

borders.dem isn't very useful.  Too fast to see what is on the page. 
 Plus requires CNTRL-C to exit.

The following demos are not in all.dem:
    borders.dem
    candlesticks.dem
    charset.dem
    enhancedtext.dem
    multiplt.dem (such a nice demo)
    starmap.dem (another nice demo)
    vector.dem (another nice demo)

mousevariables.dem, fontfile.dem and fontfile_latex.dem aren't in 
all.dem, but on purpose I think.  Not finding the font files will cause 
gnuplot to error out.  Maybe they could be put at the very end of 
all.dem so that if they do crash, nothing will be missed.  (Include them 
might be good because "all.dem" is a way of testing that nothing is 
broke in gnuplot.)

Re: maximing/resizing multiple x11: change of palette

[Daniel J S. <dan...@ie...>]

Ethan Merritt wrote:

>On Monday 16 August 2004 01:42 pm, Dave Denholm wrote:
>  
>
>>(Come to think of it - I may not even have the latest sources, in
>> which case what I just wrote may have been rubbish)
>>    
>>
>
>The particular code bit you posted applies to line colors, not
>to pm3d rectangles.   But there is a similar bit of code in 
>PaletteSetColor(plot_struct * plot, double gray) that applies
>to pm3d:
>
>        index = gray * (plot->cmap->allocated - 1);
>        if (index >= plot->cmap->allocated)
>                index = plot->cmap->allocated -1;
>        XSetForeground(dpy, gc_pm3d, plot->cmap->pixels[index]);
>
>I don't really have a grasp of the big picture.
>I *think* the problem is that plot->cmap itself is still only
>a list of color indices, and since the time it was first set up the 
>true colormap stored in the gc has changed.  So the above
>code updates the index, but it is an index into an obsolete table.
>That's why I thought we might have to store a copy of the gc
>in use at the time the original color mapping was set up.
>

I believe this is correct.  I looked at the array of GC values, it 
doesn't look to have anything to do with the color map.

In

process_configure_notify_event(XEvent *event)

is where the mouse event comes for resizing the window.  Now, I'm pretty 
sure that a problem here is that in this routine is

    plot = Find_Plot_In_Linked_List_By_Window(event->xconfigure.window);

near the top and if there is a change in the window size and a few other 
things, eventually there is a

        display(plot);

So, nothing in that code looks as though a window color scheme was 
activated or reinstalled between those two commands.  My thinking is 
that the fix should be as simple as that attached diff file.  But there 
is that "small" detail that Ethan describes.

The complete color map needs to be stored as part of the structure. 
 That is, plot->cmap needs to point at something that is dynamically 
allocated independent of the current color map.  I'll work on that fix, 
but if this doesn't sound correct, let me know.  Also, maybe think if 
what I've added in this patch is adequate to fix the problem when I have 
the plot->cmap problem fixed.  That is, are there any other details 
about the GC that also must be updated?

Dan

Re: maximing/resizing multiple x11: change of palette

[Daniel J S. <dan...@ie...>]

Is joze out there?

> The complete color map needs to be stored as part of the structure. 
> That is, plot->cmap needs to point at something that is dynamically 
> allocated independent of the current color map.  I'll work on that 
> fix, but if this doesn't sound correct, let me know.  Also, maybe 
> think if what I've added in this patch is adequate to fix the problem 
> when I have the plot->cmap problem fixed.  That is, are there any 
> other details about the GC that also must be updated? 


I started on this one and soon realized that this code is already 
present.  (The default "cmap" is used similar to zero when doing dynamic 
memory allocation.)  So, I tracked down this problem to a line of code. 
 The small change in the attached diff will solve the problem Petr 
found, but it also creates another.

Here is the situation.  That update of plot->cmap to point to the proper 
colormap is done inside PaletteMake.  Here is the block of code:

    if (plot->cmap->allocated < min_colors && !recursion) {
        ReleaseColormap(plot);
        /* create a private colormap. */
        fprintf(stderr, "switching to private colormap\n");
        plot->cmap = (cmap_t *) malloc(sizeof(cmap_t));
        assert(plot->cmap);
        CmapClear(plot->cmap);
        plot->cmap->colormap = XCreateColormap(dpy, root, vis, AllocNone);
        assert(plot->cmap->colormap);
        pr_color(plot->cmap);    /* set default colors for lines */
        RecolorWindow(plot);
        recursion = 1;
        PaletteMake(plot, (t_sm_palette *) 0);
    } else {

This allocates memory for the map and then sets plot->cmap to point to 
it.  If all things are hunky dory then, whenever plot->cmap is refered 
to in the future plot->cmap will point to the proper colormap.

However, that test at the start:

plot->cmap->allocated < min_colors

I think that is the problem.  The first time in, the min_colors will be 
2 or 10 in one of the examples Petr gives.  However, the default cmap, 
what plot->cmap points to by default has more than 10 allocated values. 
 Hence, that test never passes and the code which dynamically allocates 
the cmap never gets called.

So, remove that test as I did in the diff file.  That fixes the problem 
of the palette being restored to the proper palette.  Unfortunately, 
it's not creating the proper palette in the first place.  (I has far 
more than the 2 or 10 colors requested.)

I could look at it further, but the recursive nature of this routine 
causes me to initially punt to joze.

Dan

Re: maximing/resizing multiple x11: change of palette

[Ethan M. <merritt@u.washington.edu>]

On Monday 16 August 2004 11:47 pm, mi...@ph... wrote:
> > "set palette; replot" must be done to affect a change in
> > the plot appearance.
> 
> Yes, it should work this way.

I don't understand what you guys are saying.
You cannot do a "replot" on a previous plot, only on the current plot.


-- 
Ethan A Merritt       merritt@u.washington.edu
Biomolecular Structure Center
Mailstop 357742
University of Washington, Seattle, WA 98195

Re: maximing/resizing multiple x11: change of palette

[Daniel J S. <dan...@ie...>]

Ethan Merritt wrote:

>On Monday 16 August 2004 11:47 pm, mi...@ph... wrote:
>  
>
>>>"set palette; replot" must be done to affect a change in
>>>the plot appearance.
>>>      
>>>
>>Yes, it should work this way.
>>    
>>
>
>I don't understand what you guys are saying.
>You cannot do a "replot" on a previous plot, only on the current plot.
>

Right, I think it is already set up that way.

Dan

Re: maximing/resizing multiple x11: change of palette

[<mi...@ph...>]

> "set palette; replot" must be done to affect a change in
> the plot appearance.

Yes, it should work this way.

--
PM

Slightly suspect memory allocation

[Daniel J S. <dan...@ie...>]

Looking through some code, I see the following line

        df_column = (df_column_struct *) gp_realloc(df_column,
                            (df_max_cols += (df_max_cols < 20 ? 20 : 
df_max_cols)) * sizeof(df_column_struct),
                            "datafile column");

Does anyone else find this suspicious?  Particularly the part where 
df_max_cols is changed *before* the actual allocation takes place.

Here is a far fetched scenario that might cause a problem.  Say someone 
runs gnuplot and has a bunch of other programs running.  The person 
tries a plot and it fails to allocate memory because there is no room. 
 Gnuplot gives an error in the gp_realloc() function.  However, 
df_max_cols has already been updated.  The person then thinks to exit 
some program to free memory, runs his or her plot command and it 
crashes.  The reason being, as I'm thinking, df_max_cols was adjusted 
previously and the limit test thinks there is enough room for 
df_max_cols when in fact that memory couldn't be allocated last time the 
plot command was attempted.

There may be other similar instances of this and I'd say hold off 
addressing it for a while.

Dan

Re: maximing/resizing multiple x11: change of palette

[Daniel J S. <dan...@ie...>]

Ethan Merritt wrote:

>On Monday 16 August 2004 01:42 pm, Dave Denholm wrote:
>  
>
>>(Come to think of it - I may not even have the latest sources, in
>> which case what I just wrote may have been rubbish)
>>

<snip>

>That's why I thought we might have to store a copy of the gc
>in use at the time the original color mapping was set up.
>  
>

I'll give that a try this evening.  I know exactly where the memory 
pointer initialization and alloc's should go.

Dan

Re: maximing/resizing multiple x11: change of palette

[Ethan M. <merritt@u.washington.edu>]

On Monday 16 August 2004 01:42 pm, Dave Denholm wrote:
> 
> (Come to think of it - I may not even have the latest sources, in
>  which case what I just wrote may have been rubbish)

The particular code bit you posted applies to line colors, not
to pm3d rectangles.   But there is a similar bit of code in 
PaletteSetColor(plot_struct * plot, double gray) that applies
to pm3d:

        index = gray * (plot->cmap->allocated - 1);
        if (index >= plot->cmap->allocated)
                index = plot->cmap->allocated -1;
        XSetForeground(dpy, gc_pm3d, plot->cmap->pixels[index]);

I don't really have a grasp of the big picture.
I *think* the problem is that plot->cmap itself is still only
a list of color indices, and since the time it was first set up the 
true colormap stored in the gc has changed.  So the above
code updates the index, but it is an index into an obsolete table.
That's why I thought we might have to store a copy of the gc
in use at the time the original color mapping was set up.

Maybe.



-- 
Ethan A Merritt       merritt@u.washington.edu
Biomolecular Structure Center
Mailstop 357742
University of Washington, Seattle, WA 98195

Re: maximing/resizing multiple x11: change of palette

[Dave D. <dde...@es...>]

Daniel J Sebald <dan...@ie...> writes:

> Dave,
>
> If you want to look into this one, let me know.  (Otherwise I can do
> it one of these evenings.)  I can picture in my mind how to do it.  A

Sorry, I'm really out of touch with what it's *supposed* to do :-(

(Come to think of it - I may not even have the latest sources, in
 which case what I just wrote may have been rubbish)


dd
-- 
Dave Denholm              <dde...@es...>       http://www.esmertec.com

Re: maximing/resizing multiple x11: change of palette

[Daniel J S. <dan...@ie...>]

And when the plot structure memory is freed, must first test whether the 
palette pointer is valid and if so free that as well.

Dan


Daniel J Sebald wrote:

> Dave,
>
> If you want to look into this one, let me know.  (Otherwise I can do 
> it one of these evenings.)  I can picture in my mind how to do it.  A 
> palette pointer needs to be placed in the plot array.  There is going 
> to be some alloc() and realloc() for that pointer and the palette when 
> it is changed, or in things like prepare_plot().  The one careful 
> detail is immediately upon creating the memory for the plot structure 
> itself as part of the linked-list of plots, be sure to set the pointer 
> for the palette to 0 (!!! much emphasis).  That could also be done in 
> prepare_plot() but I advise putting it at the spot where memory is 
> created, because doing it in prepare_plot() leaves open the door for 
> someone to inadvertently call prepare_plot() and mash a valid palette 
> pointer, creating a memory leak.
>
> Dan

Re: maximing/resizing multiple x11: change of palette

[Daniel J S. <dan...@ie...>]

Dave,

If you want to look into this one, let me know.  (Otherwise I can do it 
one of these evenings.)  I can picture in my mind how to do it.  A 
palette pointer needs to be placed in the plot array.  There is going to 
be some alloc() and realloc() for that pointer and the palette when it 
is changed, or in things like prepare_plot().  The one careful detail is 
immediately upon creating the memory for the plot structure itself as 
part of the linked-list of plots, be sure to set the pointer for the 
palette to 0 (!!! much emphasis).  That could also be done in 
prepare_plot() but I advise putting it at the spot where memory is 
created, because doing it in prepare_plot() leaves open the door for 
someone to inadvertently call prepare_plot() and mash a valid palette 
pointer, creating a memory leak.

Dan


Dave Denholm wrote:

>Ethan Merritt <merritt@u.washington.edu> writes:
>
>  
>
>>On Monday 16 August 2004 01:50 am, Petr Mikulik <mi...@ph...> wrote:
>>    
>>
>>>There is a bug in the color palette treatment in the X11 terminal: when
>>>using multiple X11 terminals, window redraw (requested e.g. by a window
>>>manager) will change its palette.
>>>
>>>Is it possible to fix it?
>>>      
>>>
>>I am not very familiar with this, and I may have it wrong.
>>But It looks to me that the PM3D code would have to be re-written.
>>
>>As it is, gnuplot_x11 only maintains two GC (= "Graphics Context")
>>structures. One of these is used for PM3D operations, and the other
>>is used for everything else.   There is not currently any notion of
>>keeping a separate GC for each plot window.  So if you redefine
>>the PM3D GC to use a different palette, it affects all PM3D operations
>>in all windows.
>>
>>I imagine it would be possible to create a new GC for each window,
>>but I am not sure what pitfalls that would expose.
>>
>>    
>>
>
>
>I had a quick look, and decided that the problem was that colors[] is
>global, so that any change will affect all subsequent
>repaints. Including the colors[] in the (gnuplot_x11) plot structure
>may help.
>
>I don't think the single GC is an issue - the values in the GC are
>reset as required.
>
>
>Eg in
>
>/*-----------------------------------------------------------------------------
> *   display - display a stored plot
> *---------------------------------------------------------------------------*/
>
>void
>display(plot)
>plot_struct *plot;
>{
>
>we have
>
>...
>
>    /* loop over accumulated commands from inboard driver */
>    for (n = 0; n < plot->ncommands; n++) {
>
>    ...
>
>	/*   X11_linetype(type) - set line type  */
>	else if (*buffer == 'L') {
>	    sscanf(buffer, "L%4d", &lt);
>	    lt = (lt % 8) + 2;
>	    /* default width is 0 {which X treats as 1} */
>	    width = widths[lt] ? user_width * widths[lt] : user_width;
>	    if (dashes[lt][0]) {
>		type = LineOnOffDash;
>		XSetDashes(dpy, gc, 0, dashes[lt], strlen(dashes[lt]));
>	    } else {
>		type = LineSolid;
>	    }
>	    XSetForeground(dpy, gc, colors[lt + 3]);
>	    XSetLineAttributes(dpy, gc, width, type, CapButt, JoinBevel);
>	}
>
>
>which is updating the Gc foreground colour, but from the global
>colors[] array, which may have changed since the plot was created.
>
>(I also decided that line widths and dashes could suffer from the same
> problem, but couldn't immediately reproduce any problem
>)
>
>
>
>dd
>  
>

-- 

Dan Sebald
email: daniel DOT sebald AT ieee DOT org
URL: http://acer-access DOT com/~dsebald AT acer-access DOT com/

Re: maximing/resizing multiple x11: change of palette

[Daniel J S. <dan...@ie...>]

Daniel J Sebald wrote:

> What this means, I think, is that when you issue a palette command to 
> a certain X11 plot, the graphics primitives can still be used to draw 
> under that new palette.  The X11 code could certainly be redrawn upon 
> getting that palette command.  Do you want that?  I.e., for X11 do you 
> want to have to have to do "set palette; replot"?  Or should the "set 
> palette" change the X11 plot right away?  Note, that for non-GUI 
> terminals you would always have to do a "set palette; replot" to 
> affect a change. 


Hold on.  There may be a problem with that scheme.  We're not sure the 
primitives in the plot memory match the same size of the new palette 
installed.  "set palette; replot" must be done to affect a change in the 
plot appearance.  Yes?  No?

Dan

Re: maximing/resizing multiple x11: change of palette

[Daniel J S. <dan...@ie...>]

Ethan Merritt wrote:

>On Monday 16 August 2004 01:50 am, Petr Mikulik <mi...@ph...> wrote:
>  
>
>>There is a bug in the color palette treatment in the X11 terminal: when
>>using multiple X11 terminals, window redraw (requested e.g. by a window
>>manager) will change its palette.
>>
>>Is it possible to fix it?
>>    
>>
>
>I am not very familiar with this, and I may have it wrong.
>But It looks to me that the PM3D code would have to be re-written.
>
>As it is, gnuplot_x11 only maintains two GC (= "Graphics Context")
>structures. One of these is used for PM3D operations, and the other
>is used for everything else.   There is not currently any notion of
>keeping a separate GC for each plot window.  So if you redefine
>the PM3D GC to use a different palette, it affects all PM3D operations
>in all windows.
>
>I imagine it would be possible to create a new GC for each window,
>but I am not sure what pitfalls that would expose.
>
>I believe that Johannes Zellner wrote this code, so maybe he has
>some insight into the issue.
>
>Otherwise, maintaining per-window information is related to code
>that Daniel Sebald added recently, so he might be interested in 
>experimenting with this.
>

I didn't look at the code, but as soon as I tried Petr's example it 
struck me that what Ethan's describing is the case.

I'll have a look this evening to assess how much it would involve.  But 
it seems logical that since an X11 plot keeps all information about the 
plotting primitives so that it can be redrawn, it should also keep track 
of the palette.  If things are organized as I suspect, that palette 
should be a separate entity in the plot structure, not go into the list 
of graphics primitives.  That's easy enough to add.  Then, when 
reactivated, just install the palette.

What this means, I think, is that when you issue a palette command to a 
certain X11 plot, the graphics primitives can still be used to draw 
under that new palette.  The X11 code could certainly be redrawn upon 
getting that palette command.  Do you want that?  I.e., for X11 do you 
want to have to have to do "set palette; replot"?  Or should the "set 
palette" change the X11 plot right away?  Note, that for non-GUI 
terminals you would always have to do a "set palette; replot" to affect 
a change.

Dan

Re: maximing/resizing multiple x11: change of palette

[Dave D. <dde...@es...>]

Ethan Merritt <merritt@u.washington.edu> writes:

> On Monday 16 August 2004 01:50 am, Petr Mikulik <mi...@ph...> wrote:
>> There is a bug in the color palette treatment in the X11 terminal: when
>> using multiple X11 terminals, window redraw (requested e.g. by a window
>> manager) will change its palette.
>> 
>> Is it possible to fix it?
>
> I am not very familiar with this, and I may have it wrong.
> But It looks to me that the PM3D code would have to be re-written.
>
> As it is, gnuplot_x11 only maintains two GC (= "Graphics Context")
> structures. One of these is used for PM3D operations, and the other
> is used for everything else.   There is not currently any notion of
> keeping a separate GC for each plot window.  So if you redefine
> the PM3D GC to use a different palette, it affects all PM3D operations
> in all windows.
>
> I imagine it would be possible to create a new GC for each window,
> but I am not sure what pitfalls that would expose.
>


I had a quick look, and decided that the problem was that colors[] is
global, so that any change will affect all subsequent
repaints. Including the colors[] in the (gnuplot_x11) plot structure
may help.

I don't think the single GC is an issue - the values in the GC are
reset as required.


Eg in

/*-----------------------------------------------------------------------------
 *   display - display a stored plot
 *---------------------------------------------------------------------------*/

void
display(plot)
plot_struct *plot;
{

we have

...

    /* loop over accumulated commands from inboard driver */
    for (n = 0; n < plot->ncommands; n++) {

    ...

	/*   X11_linetype(type) - set line type  */
	else if (*buffer == 'L') {
	    sscanf(buffer, "L%4d", &lt);
	    lt = (lt % 8) + 2;
	    /* default width is 0 {which X treats as 1} */
	    width = widths[lt] ? user_width * widths[lt] : user_width;
	    if (dashes[lt][0]) {
		type = LineOnOffDash;
		XSetDashes(dpy, gc, 0, dashes[lt], strlen(dashes[lt]));
	    } else {
		type = LineSolid;
	    }
	    XSetForeground(dpy, gc, colors[lt + 3]);
	    XSetLineAttributes(dpy, gc, width, type, CapButt, JoinBevel);
	}


which is updating the Gc foreground colour, but from the global
colors[] array, which may have changed since the plot was created.

(I also decided that line widths and dashes could suffer from the same
 problem, but couldn't immediately reproduce any problem
)



dd
-- 
Dave Denholm              <dde...@es...>       http://www.esmertec.com

Re: maximing/resizing multiple x11: change of palette

[Ethan M. <merritt@u.washington.edu>]

On Monday 16 August 2004 01:50 am, Petr Mikulik <mi...@ph...> wrote:
> There is a bug in the color palette treatment in the X11 terminal: when
> using multiple X11 terminals, window redraw (requested e.g. by a window
> manager) will change its palette.
> 
> Is it possible to fix it?

I am not very familiar with this, and I may have it wrong.
But It looks to me that the PM3D code would have to be re-written.

As it is, gnuplot_x11 only maintains two GC (= "Graphics Context")
structures. One of these is used for PM3D operations, and the other
is used for everything else.   There is not currently any notion of
keeping a separate GC for each plot window.  So if you redefine
the PM3D GC to use a different palette, it affects all PM3D operations
in all windows.

I imagine it would be possible to create a new GC for each window,
but I am not sure what pitfalls that would expose.

I believe that Johannes Zellner wrote this code, so maybe he has
some insight into the issue.

Otherwise, maintaining per-window information is related to code
that Daniel Sebald added recently, so he might be interested in 
experimenting with this.

-- 
Ethan A Merritt       merritt@u.washington.edu
Biomolecular Structure Center
Mailstop 357742
University of Washington, Seattle, WA 98195

Re: Lengthy discussion about datafile.c...

[Daniel J S. <dan...@ie...>]

mi...@ph... wrote:

>>>Well, I'd not thought of strings in the binary file, but perhaps
>>>something like "%s" in the format string?  I would probably make the
>>>restriction that the strings within the file need to be NULL
>>>terminated.
>>>I'm not sure what you mean by a matrix of strings.
>>>      
>>>
>
>Isn't "binary file with strings" the same as "ascii file with strings",
>just with \0 instead of \n -- thus, "tr" filter would do all. Aha, that
>won't help if binary data and strings are mixed in one file. Do you mean
>this case?
>

Yes, that case.  But I think they still are the same.  If you look at 
binary files with headers in an editor, the strings near the top are 
just as readable as if it were an ascii file.

>>>The problem is that an image of say 500 x 500 pixels gets very big in
>>>ASCII.
>>>      
>>>
>>I think that is a non-issue.  You don't have to store this anywhere;
>>you're  just piping it in.
>>    
>>
>
>Piping is not as fast as direct read.
>
>For example, I have recently benchmarked reading a big .gz files by a C
>program with (1) popen("gzip -c -d"), and with (2) linking it with zlib.
>The case (2) was 15% faster -- quite interesting speed-up if you have to
>read 2 GB of data.
>  
>
>>When disks got cheap we all heaved a huge sigh of
>>relief and for the most part stopped using binary output files.
>>    
>>
>
>I don't think this is right.
>
>- You can work on a computer where you are not authorized to replace hard
>disk (case of many companies).
>

Good point.  That sort of thing has been in U.S. news lately, i.e., 
misplaced drives that shouldn't be misplaced.

> Also notebook hard disks are not cheap,
>fast, and easily replaceable.
>- Digital detectors are improving, and nowadays I have to deal with
>2048x2048x16bit image series. That's plenty of data -- it grows
>quadratically with improvements in detector technology.
>- I guess image processing will never switch to ascii data -- it will
>always be too big and slow.
>

Petr has it exactly right with image processing.  As computers get 
faster, the technology seems to fill the void.  That is a point I wanted 
to make before.  (Call me the "neo-geezer".)

>>Many users may not be
>>up to this, but those same users won't be able to figure out the
>>endian business anyhow.
>>

We've included an option "swap" for which a person doesn't need to know 
what big/little endian mean.  Just swap the order and see how it turns out.

>In the "with image" patch, parameters and thus command line options for
>reading binary (matrix) files were designed carefully so that you can read
>any type of data. Command line options cover the same range of options you
>have to fill for any, even GUI-like, binary image reader, to read an
>arbitrary image data. The user must always know its data, that's it, and
>he is not bother that he has to pass this information to gnuplot or
>whichever other image drawer.
>  
>
>>Simplicity is worth *a lot*.  Far more than saving a little bandwidth
>>in the input pipe.
>>    
>>
>
>The patch reading and drawing binary data is a major speedup. Try to
>compare drawing big (>512x512) traditional gnuplot binary data file and an
>binary image file.
>  
>
>>Input of binary files containing regular arrays may be worth it
>>for convenience.   But more complicated input requiring flags for
>>bit order, word size, floating point format, and pre-announcement
>>of the file structure?
>>

There are all kinds of data files out there.  I guess the question is, 
should the user be obligated to write something to make their files 
conform to gnuplot, or should the syntax exist for the user to finagle 
gnuplot into reading his or her file?  Take the moderately proficient 
linux user, like myself.  I can toil with  a program's syntax to a 
certain extent.  But to write a linux utility to convert data file 
formats, that's more trouble.

But I acknowledge Ethan's point; perhaps a bit of "code bloat".  (Let's 
see if I can help that, see below.)

>>All that strikes me as being more trouble
>>than it is worth.  Will your code work on an Amiga?  On a 64-bit
>>VMS machine?
>>    
>>
>
>I think so. You can specify Float32, Float64 etc.
>
>Probably you cannot draw binary floats saved by Turbo Pascal v 5--6,
>because these are 6 B -- I don't know about recent Pascals. But you cannot
>read in any other image programs I guess.
>  
>
>>Who is going to explain to users how to set
>>all the right flags to make it work?
>>    
>>
>
>Users working with image processing know their format structure. That's
>the user who explains gnuplot what to draw via command line options to
>"plot ... with image".
>Otherwise, you or somebody else writes a reader for Octave, and from there
>you draw your matrix via imagegp.m included in the patch.
>  
>
>>I believe that Petr had some specific applications in mind, so
>>maybe he can step in and clarify exactly what pieces of this
>>code he wanted, and why.
>>    
>>
>
>Yes, I want to fastly image binary image data with axes x and y of
>physical units (not pixel numbers).
>  
>
>>I myself plot many sorts of data in
>>gnuplot, but I've  never felt a need for direct binary input.
>>    
>>
>
>I need it always when drawing an image larger than >=128x128. Otherwise,
>the drawing speed is very low (and especially on X11) and memory
>consumption is high (I remember that gnuplot eats at about 130 B for a
>point read+drawn from a column-wise file).
>The current version of Daniel's patch is fully satisfying my needs.
>

That's correct.  We'd thought the current format with gnuplot's 
eight-field-wide point was sort of inefficient for large data files. 
 However, tacking in a new scheme for more compact storage would be too 
much of a paradigm shift.

In that same vein, I'd like to address that df_3dmatrix() routine again. 
 Now, inside of there is code that looks very similar to the 
df_readbinary() I've created.  So, this df_3dmatrix() started out in 
some ways similar to df_readline(), with the use_specs and all.   And 
Ethan and I have discussed now this problem of code re-use or similar 
functionality for binary and ascii, without intermixing them in the same 
routine to create a mess.  So, that df_3dmatrix() has in some sense 
already has not kept up in functionality to df_readline().

I'd like to propose you let me take a bit of time to move the important 
parts of df_3dmatrix() that already aren't in df_readbinary(), which I 
think are very few, and move them into df_readbinary().  I could easily 
make that df_readbinary() routine read gnuplot binary files().  Then 
df_3dmatrix() and it's helper routine read_file() could be discarded. 
 That would make the innards of plot2d.c and plot3d.c only use the 
"df_readline()" approach to bringing in data.  Perhaps one doesn't like 
the "df_readline()" approach, but I think there is advantage to having 
just one paradigm *and* to having df_readascii() and df_readbinary() in 
the same file where they share their many similarities and it is a good 
reminder that if someone adds functionality to df_readascii() there is 
always that df_readbinary() to be aware of.

Basically df_3dmatrix() and read_matrix() read in the whole data file at 
once, then go through a short loop to store the array into the "point 
structure".  Is that the direction that Gnuplot should head?  If you 
think not, and agree that having only just a "df_readline()" form of 
input is good, then that right there will free up some code space and 
assuage concerns about code bloat.

Dan

maximing/resizing multiple x11: change of palette

[<mi...@ph...>]

There is a bug in the color palette treatment in the X11 terminal: when
using multiple X11 terminals, window redraw (requested e.g. by a window
manager) will change its palette.

Try this script:


set pm3d map

set term x11 10
set title '10 gray levels'
set palette gray
set palette maxcolors 10
splot x*x

set term x11 2
set title '2 colors'
set palette color
set palette maxcolors 2
splot x


Now, maximize or resize window #10 by mouse => it will change from gray map
with 10 gray levels to color map with 2 colors.

Is it possible to fix it?

---
PM

Re: Lengthy discussion about datafile.c...

[<mi...@ph...>]

>> Well, I'd not thought of strings in the binary file, but perhaps
>> something like "%s" in the format string?  I would probably make the
>> restriction that the strings within the file need to be NULL
>> terminated.
>> I'm not sure what you mean by a matrix of strings.

Isn't "binary file with strings" the same as "ascii file with strings",
just with \0 instead of \n -- thus, "tr" filter would do all. Aha, that
won't help if binary data and strings are mixed in one file. Do you mean
this case?
>> The problem is that an image of say 500 x 500 pixels gets very big in
>> ASCII.
>
> I think that is a non-issue.  You don't have to store this anywhere;
> you're  just piping it in.

Piping is not as fast as direct read.

For example, I have recently benchmarked reading a big .gz files by a C
program with (1) popen("gzip -c -d"), and with (2) linking it with zlib.
The case (2) was 15% faster -- quite interesting speed-up if you have to
read 2 GB of data.
> When disks got cheap we all heaved a huge sigh of
> relief and for the most part stopped using binary output files.

I don't think this is right.

- You can work on a computer where you are not authorized to replace hard
disk (case of many companies). Also notebook hard disks are not cheap,
fast, and easily replaceable.
- Digital detectors are improving, and nowadays I have to deal with
2048x2048x16bit image series. That's plenty of data -- it grows
quadratically with improvements in detector technology.
- I guess image processing will never switch to ascii data -- it will
always be too big and slow.

> Many users may not be
> up to this, but those same users won't be able to figure out the
> endian business anyhow.

In the "with image" patch, parameters and thus command line options for
reading binary (matrix) files were designed carefully so that you can read
any type of data. Command line options cover the same range of options you
have to fill for any, even GUI-like, binary image reader, to read an
arbitrary image data. The user must always know its data, that's it, and
he is not bother that he has to pass this information to gnuplot or
whichever other image drawer.
> Simplicity is worth *a lot*.  Far more than saving a little bandwidth
> in the input pipe.

The patch reading and drawing binary data is a major speedup. Try to
compare drawing big (>512x512) traditional gnuplot binary data file and an
binary image file.
> Input of binary files containing regular arrays may be worth it
> for convenience.   But more complicated input requiring flags for
> bit order, word size, floating point format, and pre-announcement
> of the file structure?


> All that strikes me as being more trouble
> than it is worth.  Will your code work on an Amiga?  On a 64-bit
> VMS machine?

I think so. You can specify Float32, Float64 etc.

Probably you cannot draw binary floats saved by Turbo Pascal v 5--6,
because these are 6 B -- I don't know about recent Pascals. But you cannot
read in any other image programs I guess.
> Who is going to explain to users how to set
> all the right flags to make it work?

Users working with image processing know their format structure. That's
the user who explains gnuplot what to draw via command line options to
"plot ... with image".
Otherwise, you or somebody else writes a reader for Octave, and from there
you draw your matrix via imagegp.m included in the patch.
> I believe that Petr had some specific applications in mind, so
> maybe he can step in and clarify exactly what pieces of this
> code he wanted, and why.

Yes, I want to fastly image binary image data with axes x and y of
physical units (not pixel numbers).
> I myself plot many sorts of data in
> gnuplot, but I've  never felt a need for direct binary input.

I need it always when drawing an image larger than >=128x128. Otherwise,
the drawing speed is very low (and especially on X11) and memory
consumption is high (I remember that gnuplot eats at about 130 B for a
point read+drawn from a column-wise file).
The current version of Daniel's patch is fully satisfying my needs.

---
PM

Re: Binary syntax reduction [Was: Lengthy discussion...]

[Daniel J S. <dan...@ie...>]

Daniel J Sebald wrote:

> binary {3 | xy | xyz | xyzc} {format="string"} {endian=little}


Or perhaps a better way of doing this would be

binary {#} {columns} {format="string"} {endian=big}

Here

binary #  :  Matrix format, # specifies the number of components per 
matrix element.

binary # columns  :  Binary data has # columns of "string" data format.

How's that?  No user functions, no strings, but I think that would cover 
most applications that use temporary data as opposed to, say, archived 
ascii where strings etc. are useful.

I could implement that as df_readbinary() fairly easily.  That would 
allow binary data for 2D data.


So, how about 3D data?  It already exists.  But let's examine its 
structure.  Basically, inside plot3d.c is the following:

    if (df_matrix)
        xdatum = df_3dmatrix(this_plot, NEED_PALETTE(this_plot));
    else {
<snip>
        while ((j = df_readline(v,MAXDATACOLS)) != DF_EOF) {

Inside of df_3dmatrix is some code that looks very similar similar to 
df_readline (with using specs, etc.... maybe "using" can't be discarded 
afterall) , but instead df_3dmatrix does the work of storing the data in 
the plot structure, rather than passing data back to plot3d.c for 
interpretting.

The question I pose is could df_3dmatrix be replaced by df_readline() 
which uses df_readbinary internally?  The advantage of df_3dmatrix() is 
that it has a short little loop to do the storage rather than passing 
back a v[] vector.  However, it isn't consistent with the most 
predominant gnuplot "paradigm" for interpretting and storing data. 
 Plus, if there is a df_readline() which is very similar to df_3dmatrix, 
it is a bit of code repetition.

Now, I know Ethan would like to pass in the plot structure pointer to 
the reading routine.  Perhaps then the idea is to move away from that 
strategy, and df_3dmatrix is more the desired model.

Here are some interesting comments from the code and online help:

/* FIXME HBB 20001207: doesn't respect 'index' at all, even though it
 * could, and probably should. */
static float **
df_read_matrix(int *rows, int *cols)

 The `index` keyword is not supported, since the file format allows only one
 surface per file.  The `every` and `using` filters are supported.  `using`
 operates as if the data were read in the above triplet form.

So, it seems maybe its desirable to have 'using' as part of the binary 
command.  But, I don't think 'index' works from what is in a 
gnuplot-binary file.  There is only the value <N+1> at the start of the 
file, not the other dimension.  So there is no way of knowing when to 
end one record without something  like "record=120x150".  But, I'm 
drifting to the geezer camp, not allowing more than one record per file.

Dan

Binary syntax reduction [Was: Lengthy discussion...]

[Daniel J S. <dan...@ie...>]

Ethan Merritt wrote:

>(wandering a bit off topic)
>
>On Sunday 15 August 2004 01:03 am, Daniel J Sebald wrote:
>  
>
>>My geezerness only goes back to the days of PDP 11-70 and the 8 inch
>>floppy platter.  But I can't recall binary files ever having special
>>characters to serve as the end of a record.
>>    
>>
>
>PDP 11/xx used the FILES-11 filesystem, in which meta-information
>about record type, disk allocation, ACLs, etc were stored in a 
>separate meta-file, not as in-line info.
>
>These filesystems supported very complicated record structures
>for database work (still in use today), but also had 3 main "simple"
>file structures:
>	Fixed-length records:
>		What it sounds like.  The record length was specified in
>		meta-data.  A read operation returned 1 whole record.
>	Variable-length records:
>		Each record began with an integer specifying how long
>		the record was.
>	CR/LF:
>		Unix-like stream-of-bytes, with end of record signalled by
>		either a CR or a LF.
>
>On top of that, Fortran used carriage-control characters at the beginning
>of a record.
>

OK, you win the geezer challenge...  Anyway, gnuplot binary then is 
similar to variable-length record.

>But it's not a file. It never hits the disk, so I/O speed is not an issue.
>And at current memory bandwidths, transferring 10 MB of data should
>take only about 0.01 sec (if I haven't dropped a decimal point somewhere).
>That will be totally dominated by the I/O time to read the original binary
>data from a disk file.  So it may be unaesthetic to have an intermediate
>ascii stream, but I doubt it will be noticeable in terms of interactive
>response.  
>

Here is a test.  Lets say a 500 x 500 image is processed in Octave and 
is to be plotted.  I don't think 500 x 500 is unreasonable, x-ray 
angiography images, telescopic space images, they're usually pretty big. 
 If you have octave, try the following to simulate the amount of data 
that would be transferred through the pipe.  (Granted, we have no idea 
what kind of bottle necks might exist in how Octave is programmed for 
the pipe--perhaps it could be improved--but we'll use this as a rough test.)

t = [1:500*500]/100;
s = sin(t);
plot(t,s);

On my machine, a three year old Dell with a Pentium 4, 900-1000 MHz 
system bus, that plot takes 8 seconds.  After the 4th second the octave 
command line returns, and 4 seconds after that the gnuplot plot appears. 
 To me, that time is unacceptable.  (Imagine the derision...  No, the 
solution is not to buy a faster computer.)  There are probably a couple 
things going on.  First, the pipe may not transfer data at the rate you 
suggest, due to time sharing perhaps.  Who knows?  Second there is also 
the issue of this being formatted I/O, meaning that every value has to 
go through the scanf function.  Does that slow things down?

Now an example in Octave using the m-file designed to use the image and 
binary features added to gnuplot.

A = 1./hilb(500);
imagegp(A);

This takes 3/4 to 1 second.  Tolerable.  There is a difference here 
though.  The binary data goes through a file.  So maybe the file is 
faster than the pipe.

Let's try one last test.  Sending the image data to a file in ascii 
form.  I'll put an "if 1" around the instructions to ensure they are all 
executed as fast as possible one after the other.

X = ones(size(A,2),1)* [1:size(A,1)];
Y = [1:size(A,2)]'*ones(1,size(A,1));
N = size(A,1)*size(A,2);
B = [reshape(X,N,1) reshape(Y,N,1) reshape(A,N,1)]';

if 1
    fid = fopen("junk.dat","w");
    fprintf(fid, "%f %f %f\n", B);
    fclose(fid);
    graw("plot \'junk.dat\' using 1:2:3 w image\n");
end

This takes 6 or 7 seconds.  So files and a pipe are roughly the same in 
this crude test.  Perhaps the file is even faster because more data is 
being transferred in that case.  However, there are other things within 
gnuplot, i.e., reading from a file and reading from '-' may be different.

Anyway, rough test.  But, the conclusion is that it is probably the 
"fprintf' and 'scanf', i.e., formatted I/O, that slows things down, and 
binary data is a nice feature to have with images.

>>I would add that I myself am deterred from implementing general binary
>>if the df_readline() is going to continue to grow with functionality
>>from within.  Unless, say, the use_spec processing is converted to a
>>function that can be called from multiple places, trying to maintain two
>>"analogous", or "parallel", routines is too much for anyone, whether he
>>or she is the original author or not.
>>    
>>
>
>You mean changing use_spec[] from an array into a function?
>If that turns out to be useful then I suppose it would be reasonable.
>

Yeah, but I'm not advocating that.  You are persuading me that perhaps 
"binary" should be simpler.  The question is, how many people will use 
Gnuplot, from the command line for processing images.  Not many; so I 
would say that passing data through a function isn't that necessary, as 
in this example

plot 'blutux.rgb' binary array=128x128 flipy format='%uchar' using 
(1.5*$1):2:3 with rgbimage

The primary use I have in mind for this "large data set plotting" is 
something done by an application in an ephemeral way.  Just send some 
data over, plot it, and discard the data.  So, perhaps the ability to 
skip data within a binary file isn't necessary.  That is, no '%*uchar%' 
kind of stuff, or skipping a number of bytes at the head of the file.

How about tossing out the multiple records per file feature.  If there 
is more than one big data set to plot, just create multiple files.

How about tossing the implicit sampling interval?  That would mean that 
all data must appear in the file, for example the (x,y) coordinates for 
each pixel of an image must be along with the pixel value.  That means a 
sample image for the 'image.dem' program would increase in size by a 
factor of 5/3.  No problem.

Translations, toss that in the case where coordinates are in the file.

All of this stuff would reduce a lot of the code, much of which is for 
interpreting the keywords.  With no "using" there can be no functions. 
 Also, let's say with binary, no strings, no time data, etc.  Again, 
this kind of stuff will be small in quantity if ever it is plotted, in 
which case ASCII can be used.  What I mean is there is no need to plot 
500 strings.

I'd hesitate to toss '%uchar', etc.  Although I could give on that one. 
 But let's rule out multiple data types per file.  Maybe just one 
%float, etc. inside the format string.  The code that does the 
transformation inside the df_readbinary() routine is fairly 
straightforward.  There is a set of tables to compute datasizes upon 
compilation.  Looks nasty but once it is compiled, it probably isn't too 
big.

I'd hesitate to toss the endian information too.  That code inside 
df_readbinary() also isn't too bad.  The thing is, octave has a 
qualifier associated with its fopen() routine "ieee-le" and "ieee-be". 
 They pay attention to endianess, so maybe gnuplot binary should too.

So, in order to get functionality, here is a possible reduced syntax.

binary {3 | xy | xyz | xyzc} {format="string"} {endian=little}

Now if we want to toss the format, and require "all floats, all the 
time", fine.

But the first part of that syntax is to allow entry for both images and 
long linear records such as speech waveforms or whatever.

binary  :  The current gpbin file

binary 3  :  Very similar to current gpbin, what I call gpbin3.  That 
is, it is the matrix format, but each element of the matrix has 3 
components.  (Could make that an arbitrary number, 1 up to max columns.)

Now that covers images, i.e, a matrix format.  But what about sampling 
in one dimension?  Perhaps that could be done with gpbin if one sets N 
(the number of columns and first number in the file) to one.  But that 
is tricky from the user's perspective.  Hence the following:

binary xy  :  Two "columns" of data.  Would be useful for 2D plots.

binary xyz  :  Three "columns" of data.  Would be useful for 3D plots.

binary xyzc  :  Four "columns" of data.  Would be useful for 3D plot 
doing color.

This wouldn't have to be the exact syntax.  For example, it would be 
nice if one could just specify the number of columns with 2, 3, 4, 5, 
..., max_cols, but that would conflict with trying to introduce multiple 
components per element of matrix binary.

Dan

Re: Lengthy discussion about datafile.c...

[Daniel J S. <dan...@ie...>]

Ethan A Merritt wrote:

>On Saturday 14 August 2004 08:54 pm, Daniel Sebald wrote:
>  
>
>>>Your docs say
>>>	+ Gnuplot will retrieve a number of binary
>>>	+ variables equal to the largest column specified in the `<using list>`.
>>>	+ For example, `using 1:3` would cause three columns to be read, of which
>>>	+ the second will be ignored.
>>>So how do you handle the case of 10 logical columns of data in the file,
>>>of which you only want to read the 2nd and 4th?  How do you skip "columns"
>>>5 to 10 of each "line"?
>>>      
>>>
>>"format" is supposed to be analogous to the "using" format string, so
>>something like the following should work
>>
>>plot "datafile.dat" binary format ="%10float" using 2:4
>>    
>>
>
>But according to the documentation I quoted above, that would only
>read in 4 logical columns, leaving 6 more unread values in the file
>before you get to the next set of input values.  How do you tell it
>to skip the next 6 columns?
>

Well, the documentation is a bit misleading.  If all one specifies is 
"using", without a format string, then the assumption is that the 
highest number is the number of columns.  But when the format string is 
there, it tells that a line is 10 floats in this case.  I guess the idea 
was that one could leave out the format string and just type "using 
1:2:3" for example.  I'd not be against the "format" and "using" 
required to appear together.  A lot of code goes to these special 
assumptions and whatnot.

>>>What constitutes the logical equivalent of a "blank line" in your binary
>>>files? Or is there no equivalent to the auto-determination of scan lines?
>>>      
>>>
>>A blank line occurs when the scan line reaches its end.  For example,
>>here is the scatter2 example from the image demo
>>
>>splot 'scatter2.bin' binary record=30,30,29,26 endian=little using 1:2:3
>>
>>which means blank lines occur at the 30th line, 60th line, etc.
>>    
>>
>
>But there you have told it on the command line what the structure is.
>The thing about blank lines in an ascii input file is that they define
>a structure on the fly; you don't need to specify it on the command line.
>I would much rather require a file format that indicates what each
>logical line contains.  A blank line is then indicated *in the file* by some
>designated code (probably some number of 0s, but whatever).
>

My geezerness only goes back to the days of PDP 11-70 and the 8 inch 
floppy platter.  But I can't recall binary files ever having special 
characters to serve as the end of a record.  (If so, what are they that 
they wouldn't clash with a valid IEEE float?  Is there a NAN in IEEE 
float that could serve as an end of record?)


>>Well, I'd not thought of strings in the binary file, but perhaps
>>something like "%s" in the format string?  I would probably make the
>>restriction that the strings within the file need to be NULL terminated.
>> That's not an unrealistic expectation, is it?  Or wait, maybe "%s"
>>could be general length but NULL terminated; "%[#]s" could be a fixed
>>length of # characters.
>>    
>>
>
>Fixed length strings are not interesting.  You could use NULL-termination,
>but only if you specify everything on the command line because otherwise
>the input routine doesn't know whether it's reading a string at all.
>

Right.  I guess all this sort of information would be in some type of 
header.


>>I'm not sure what you mean by a matrix of strings.
>>    
>>
>
>I mean like an input file consisting of 10 lines of 5 strings each.
>Only in this case it would be a binary file containing 50 NULL-terminated
>strings that you have somehow flagged as being in a 10x5 matrix.
>
>  
>
>>What is the matrix variant?  
>>    
>>
>
>Like your example above. (At least I *think* that's what your example
>was doing).   A regular array of values all of the same sort.  E.g.
>a  100x200x300 grid with x varying faster than y faster than z.
>But since it's regular and all the entries are the same length you
>know exactly where to find every element without any funky format
>stuff.
>

Yes, I'd certainly be content with something simple to get data across 
from an application to gnuplot.  I mean, the expectation is not that one 
has a thousand columns of binary data of which you want to pick out only 
two of them.  In most cases one could say something like

format="%int" using 1:2:3

or

format="%float" using 1:2:3

I think originally this started as a slight variation on the using 
command, but it got too confusing from that.  So "binary" developed it's 
own using string.

>>The problem is that an image of say 500 x 500 pixels gets very big in ASCII.
>>    
>>
>
>I think that is a non-issue.  You don't have to store this anywhere; you're 
>just piping it in.    But this is the very straightforward case that I called 
>a matrix.  You know in advance it's a 500x500 array, and you know how big
>each element is.  No need for format statements, using specs, or any of that.
>
>[EAM puts on geezer hat again]  Back in the old days of limited disk space
>it was a big win to store numeric data in binary files.  This caused 
>man-centuries of time to be wasted in dealing with cross-platform conversions
>and uncertainty about the exact format of the binary files.  When disks got
>cheap we all heaved a huge sigh of relief and for the most part stopped 
>using binary output files.  It's just not worth it.  So what if the ascii 
>equivalent is big?  Just compress it and it goes back to being about the
>same size as the original binary (OK, that depends a bit on what sort of
>data it is).  
>

[Battle of the geezers coming]  Point taken, when we're talking a few 
hundred data points.  But when we're talking images, it could be a 
megabyte file, and converting that to ASCII yields a 10 megabyte file. 
 (Each data point gets expanded to a floating point ASCII number.  In 
addition the (x,y) locations have to be added as columns.)  Then it has 
to be read in using the formatted I/O.  (I assume scanf is slightly 
slower than raw data.)  There gets to be this delay between hitting the 
return key and an image popping up in what looks like Octave, but is 
really Gnuplot.

Another similar situation might be a speech waveform.  I understand that 
the person using the software should really appropriately down-sample 
the data so that one isn't sending all kinds of data and extraneous high 
resolution to gnuplot, but people don't do that unfortunately.  They've 
got fast computers and 120 Gbyte hard drives.

>Bottom line is I really don't like this general binary input format.
>If you know enough about your binary format to write a cryptic 
>description like
>    plot "datafile.dat" binary format="%*int16%float32%*float32%" \
>    record=30,30,29,26 endian=little 
>then by gum, you know enough to write a jiffy filter routine and
>pipe normal ascii input into gnuplot.   Many users may not be
>up to this, but those same users won't be able to figure out the
>endian business anyhow.  Where exactly is the big gain?
>Simplicity is worth *a lot*.  Far more than saving a little bandwidth
>in the input pipe.
>
>Input of binary files containing regular arrays may be worth it
>for convenience.   But more complicated input requiring flags for
>bit order, word size, floating point format, and pre-announcement
>of the file structure? ---- All that strikes me as being more trouble
>than it is worth.  Will your code work on an Amiga?  On a 64-bit
>VMS machine?   Who is going to explain to users how to set
>all the right flags to make it work?
>

Hey, I'm not going to fight you on that one.  I'm all for simplicity. 
 No one ever offered up a simple solution.  It started as a slight 
variation on the current implementation of user/etc.  I probably figured 
at the time why not treat binary just like ascii so that all the 
functionality that ascii input has is also present for binary, e.g., 
passing through a function, etc.?  It grew from there.

I would add that I myself am deterred from implementing general binary 
if the df_readline() is going to continue to grow with functionality 
from within.  Unless, say, the use_spec processing is converted to a 
function that can be called from multiple places, trying to maintain two 
"analogous", or "parallel", routines is too much for anyone, whether he 
or she is the original author or not.

>I believe that Petr had some specific applications in mind, so
>maybe he can step in and clarify exactly what pieces of this 
>code he wanted, and why.  I myself plot many sorts of data in
>gnuplot, but I've  never felt a need for direct binary input.
>  
>

... but I would say that binary input has to exist if one is going to 
display images.  The faster the response between hitting the return key 
and the image popping up on the screen, the better.  If it gets too slow 
for relatively small images, the user's response won't be favorable.

Offer up a simple way of doing it...

Would a syntax where there is _no_ format string and _no_ using string 
simplify matters?  That is, everything must be binary floats and there 
cannot be any discarded columns.  It would remove a lot of bits and 
pieces of code that would add up to pretty much I guess.  How about the 
sample intervals?  Are those useful?  Again, the main structure of a 
binary data file from an application would just a solid string of raw 
data, but it isn't unreasonable to require everything about (x,y) 
positions be explicit, rather than implicit.

Is there a subset of the syntax we've offered up which will work 
sufficiently and provide some flexibility so that both large image files 
and large linear files, like speech or other forms of lengthy time 
records can be transferred efficiently?

If one wants to rule out lengthy binary linear records, and still allow 
large binary image files, how about an extension to the gnuplot binary 
format?  That is, would you allow a variable to follow "binary" as 
binary presently exists in the CVS version?  This purpose of the 
variable would be to indicate how many "channels" or "entries" or 
whatever are associated with a location in the grid.  This would allow 
the use of grayscale and RGB images.  For example, "binary 3" would be

N                   x1                   x2              x3   ...    xN
y1       <r11 g11 b11>   <r21 g21 b21>    <rN1 gN1 bN1>
y2       <r12 g12 b12>   <r22 g22 b22>    <rN2 gN2 bN2>
etc.

The x and y wouldn't necessarily have to be Cartesian.  They could be 
radial, if ever one gets adventurous enough to attempt circular images 
like sonograms, CT...  which probably won't happen.  I do acknowledge 
that gnuplot binary is limiting though.  But it works for me.  Petr may 
have feelings otherwise.  (But Petr, it may be possible to take a lot of 
the binary code and write a little app that converts ESRF to gpbin or 
gpbin3, then have a little awk script so that gnuplot behaves almost 
exactly like "plot 'image.edf' with rgbimage".)

Seriously, some consensus on a simple, acceptable approach and I can 
toss it together in a matter of hours and be done.  From my perspective, 
so long as I can use Octave to get images, in this case spectrograms, 
into a PostScript or PS/Latex form, that has axes and tics and labels, 
and can be imported to a LaTeX document, I'm happy.  What I have now 
works for me, but I won't be motivated to do a simpler design without 
some consensus, as opposed to offering up some other alternative for 
evaluation.  I'm happy for the feedback and willing to change things if 
it will go somewhere.

Dan

Re: Lengthy discussion about datafile.c...

[Ethan A M. <merritt@u.washington.edu>]

On Saturday 14 August 2004 08:54 pm, Daniel Sebald wrote:
>
> >Your docs say
> >	+ Gnuplot will retrieve a number of binary
> >	+ variables equal to the largest column specified in the `<using list>`.
> >	+ For example, `using 1:3` would cause three columns to be read, of which
> >	+ the second will be ignored.
> >So how do you handle the case of 10 logical columns of data in the file,
> >of which you only want to read the 2nd and 4th?  How do you skip "columns"
> >5 to 10 of each "line"?
>
> "format" is supposed to be analogous to the "using" format string, so
> something like the following should work
>
> plot "datafile.dat" binary format ="%10float" using 2:4

But according to the documentation I quoted above, that would only
read in 4 logical columns, leaving 6 more unread values in the file
before you get to the next set of input values.  How do you tell it
to skip the next 6 columns?

> >What constitutes the logical equivalent of a "blank line" in your binary
> >files? Or is there no equivalent to the auto-determination of scan lines?
>
> A blank line occurs when the scan line reaches its end.  For example,
> here is the scatter2 example from the image demo
>
> splot 'scatter2.bin' binary record=30,30,29,26 endian=little using 1:2:3
>
> which means blank lines occur at the 30th line, 60th line, etc.

But there you have told it on the command line what the structure is.
The thing about blank lines in an ascii input file is that they define
a structure on the fly; you don't need to specify it on the command line.
I would much rather require a file format that indicates what each
logical line contains.  A blank line is then indicated *in the file* by some
designated code (probably some number of 0s, but whatever).


> Well, I'd not thought of strings in the binary file, but perhaps
> something like "%s" in the format string?  I would probably make the
> restriction that the strings within the file need to be NULL terminated.
>  That's not an unrealistic expectation, is it?  Or wait, maybe "%s"
> could be general length but NULL terminated; "%[#]s" could be a fixed
> length of # characters.

Fixed length strings are not interesting.  You could use NULL-termination,
but only if you specify everything on the command line because otherwise
the input routine doesn't know whether it's reading a string at all.

> I'm not sure what you mean by a matrix of strings.

I mean like an input file consisting of 10 lines of 5 strings each.
Only in this case it would be a binary file containing 50 NULL-terminated
strings that you have somehow flagged as being in a 10x5 matrix.

> What is the matrix variant?  

Like your example above. (At least I *think* that's what your example
was doing).   A regular array of values all of the same sort.  E.g.
a  100x200x300 grid with x varying faster than y faster than z.
But since it's regular and all the entries are the same length you
know exactly where to find every element without any funky format
stuff.

> The problem is that an image of say 500 x 500 pixels gets very big in ASCII.

I think that is a non-issue.  You don't have to store this anywhere; you're 
just piping it in.    But this is the very straightforward case that I called 
a matrix.  You know in advance it's a 500x500 array, and you know how big
each element is.  No need for format statements, using specs, or any of that.

[EAM puts on geezer hat again]  Back in the old days of limited disk space
it was a big win to store numeric data in binary files.  This caused 
man-centuries of time to be wasted in dealing with cross-platform conversions
and uncertainty about the exact format of the binary files.  When disks got
cheap we all heaved a huge sigh of relief and for the most part stopped 
using binary output files.  It's just not worth it.  So what if the ascii 
equivalent is big?  Just compress it and it goes back to being about the
same size as the original binary (OK, that depends a bit on what sort of
data it is).  

Bottom line is I really don't like this general binary input format.
If you know enough about your binary format to write a cryptic 
description like
    plot "datafile.dat" binary format="%*int16%float32%*float32%" \
    record=30,30,29,26 endian=little 
then by gum, you know enough to write a jiffy filter routine and
pipe normal ascii input into gnuplot.   Many users may not be
up to this, but those same users won't be able to figure out the
endian business anyhow.  Where exactly is the big gain?
Simplicity is worth *a lot*.  Far more than saving a little bandwidth
in the input pipe.

Input of binary files containing regular arrays may be worth it
for convenience.   But more complicated input requiring flags for
bit order, word size, floating point format, and pre-announcement
of the file structure? ---- All that strikes me as being more trouble
than it is worth.  Will your code work on an Amiga?  On a 64-bit
VMS machine?   Who is going to explain to users how to set
all the right flags to make it work?

I believe that Petr had some specific applications in mind, so
maybe he can step in and clarify exactly what pieces of this 
code he wanted, and why.  I myself plot many sorts of data in
gnuplot, but I've  never felt a need for direct binary input.

-- 
Ethan A Merritt
Department of Biochemistry & Biomolecular Structure Center
University of Washington, Seattle

Re: Lengthy discussion about datafile.c...

[Daniel J S. <dan...@ie...>]

Ethan Merritt wrote:

>On Saturday 14 August 2004 03:50 pm, Daniel J Sebald wrote:
>  
>
>>I know that for ASCII files the number of columns can be determined by
>>the file itself and gnuplot readjusts accordingly.
>>    
>>
>
>Which brings up another issue.  The description of your binary read
>"format" commands looks *really* fragile.  I mean the stuff being parsed
>in plot_option_binary_format(). I am seriously worried that
>it won't transfer well across 32/64 bit machines, that it won't handle
>string data, and worst of all that it requires too much user-knowledge
>of file and data types.  Basically I don't like it.
>

Don't have a 64 bit machine to try this on.  But the question as to how 
it will transfer is a matter of how data is stored in the file.  Is 
there a 64-bit IEEE floating point format?  There probably is.  32-bit 
floats in files are still certainly readable.  64-bit should work so 
long as the native file byte order matches the CPU/compiler order byte 
order.

>[EAM puts on geezer hat] In the old days of Fortran programming and 
>VMS file systems, binary files had actual "records".   In those days
>there was an obvious parallel between "columns" in an ascii file
>and "records" in a binary file.  But that approach has been drowned
>by the unix notion that "everything is a stream of bytes".  
>

I know, that's the crux.

>It's *really hard* to figure out what data is in a binary stream, and I
>am dubious that it is worth spending thousands of lines of code in
>gnuplot trying to do so.  The unix way in such a case would be to
>run the input binary data through a tailored filter on its way into
>gnuplot.  That way gnuplot only has to know about ascii input, and
>you can debug a suitable filter for your application without having
>to recode gnuplot.  
>

The problem is that an image of say 500 x 500 pixels gets very big in ASCII.

>Your docs say  
>	+ Gnuplot will retrieve a number of binary
>	+ variables equal to the largest column specified in the `<using list>`.
>	+ For example, `using 1:3` would cause three columns to be read, of which
>	+ the second will be ignored. 
>So how do you handle the case of 10 logical columns of data in the file,
>of which you only want to read the 2nd and 4th?  How do you skip "columns"
>5 to 10 of each "line"?
>

"format" is supposed to be analogous to the "using" format string, so 
something like the following should work

plot "datafile.dat" binary format ="%10float" using 2:4

(But actually, I see there is a bug because 10 is greater than MAX_COLS, 
which is a silly restriction in the code... I'll fix that.)

Or, if there were a mix of variable types

plot "datafile.dat" binary format = 
"%*int16%float32%*float32%int16%3*int%3*float"

I agree that unless one uses this a lot it is a bit arcane.  But recall, 
one of the primary uses is for automation.  For example, passing an 
image from Octave to Gnuplot in binary is an example.  Once the image() 
script in Octave is written with the proper format string, there is no 
need to deal with that again in Octave.

>What constitutes the logical equivalent of a "blank line" in your binary
>files? Or is there no equivalent to the auto-determination of scan lines?
>

A blank line occurs when the scan line reaches its end.  For example, 
here is the scatter2 example from the image demo

splot 'scatter2.bin' binary record=30,30,29,26 endian=little using 1:2:3

which means blank lines occur at the 30th line, 60th line, etc. 
 Whatever application that is sending the data to gnuplot must know the 
quantity being sent.  If that information is stored within the datafile 
and must be interpretted, then that requires additional routines, an 
example of which Petr has supplied.  Such routines are easy to link in, 
but I'm not enthusiastic about writing all sorts of binary file routines 
for the bazillion different formats in the computer world.  My original 
goal with all this was to quickly pump raw image data across to Octave.

>Do you plan to handle strings?  How? Would you require a full "binary format"
>description in this case?  Is there such a thing as a matrix of strings?
>

Well, I'd not thought of strings in the binary file, but perhaps 
something like "%s" in the format string?  I would probably make the 
restriction that the strings within the file need to be NULL terminated. 
 That's not an unrealistic expectation, is it?  Or wait, maybe "%s" 
could be general length but NULL terminated; "%[#]s" could be a fixed 
length of # characters.

I'm not sure what you mean by a matrix of strings.

>The matrix variant is far more straight-forward.  I would think this will be 
>by far the most common use anyhow, and it would cover the pixel images
>that you obviously have fondness for. Could we maybe have a first cut
>version of this patch that only deals with matrix format binary data?
>

What is the matrix variant?  Gnuplot binary?  That was available all 
along.  However, gnuplot binary doesn't work for color images.  (Need 
three channels for that.)  The switch BINARY_DATA_FILE can be undefined 
to remove binary datafiles from the code.  Gnuplot binary would still 
work with the swicth off.

Dan

Re: Lengthy discussion about datafile.c...

[Daniel J S. <dan...@ie...>]

Ethan Merritt wrote:

>On Saturday 14 August 2004 03:50 pm, you wrote:
>  
>
>>OK, let me back up here.  I think I see now the more important issue
>>here is that the data to be plotted, the imigration.dat file for
>>example, won't work because it has more columns than allowed by max_cols
>>passed into the df_readline routine.
>>    
>>
>
>No, that's completely wrong.  The data is being plotted 1 column at
>a time.  Sure there are lots of columns in the data file, but there's 
>nothing special about that.
>

OK, got it.

>Hint:  In the current code every requested column is returned
>twice, once as a number and once as a string.  The caller can 
>choose whether it wants the string value or the numeric value.
>I don't know how this fits in with your binary data files, but I
>assure you it is fully generic.  The histogram code doesn't use
>this anyhow; histogramming is not about strings, it's about
>columns of numbers.   You are, I am guessing, thinking about
>my *other* new plotting mode - 'with labels'.
>

Inside df_readline() is the if statement:

        if (use_spec[output].expected_type >= CT_XTICLABEL) {

and inside of that case is an instruction:

            if (df_current_plot) xpos += df_current_plot->histogram->start;

and inside of graphics.h is the definition of histogram for curve_points:

    struct histogram_style *histogram;    /* Only used if plot_style == 
HISTOGRAM */

If I follow, only the histogram plot style, then, can make use of that 
particular block of code inside df_readline().  That, or one needs to 
realize that the histogram->start can be used in a generic fashion.

I guess it isn't that big of a deal, but it just seems like the 
use_spec[] portion of df_readline() is growing very large and doing 
lot's of specific stuff.

>>Does this get around some problems?  Am I understanding the big issue
>>now, that there are more columns now than max_cols?
>>    
>>
>
>No. Nothing at all like that.  I want to get rid of max_cols not because
>I want lots of columns, but just because it is not used for anything that
>really has to do with columns.  It is only used to try to deduce back to
>what the plot type is - which I think is nuts. If you need to know the plot
>type then just pass the plot type.
>

Oh, well yeah guessing the structure from the max cols is kind of silly.

>>Well, here is the thing.  There is a certain element of this that can't
>>be disentangled (if that's a word).  A lot of the parameters for reading
>>from a file are set up by df_open() because it is there that the
>>keywords from the command line are processed.  So, at the point of
>>df_open() it isn't known yet whethere the file is ascii or binary.  That
>>could be fixed by first, at the start of df_open, checking all the
>>keywords to see if one is "binary", but that's not graceful.
>>    
>>
>
>Hunh?  Now I'm the one who is confused.  I really have not been
>looking at that part of your patch because I have no use for binary
>input.  But I assumed you told the program *somehow* that this
>was a binary data file.  How does this work at all if there isn't a
>keyword on the command line?
>

What I'm saying is if all the parameters that are controlled by the 
command line keywords are to have there own "local instance" in a 
different file, say "binfile.c" that will mean there has to be a variant 
of df_open() just for binary files, which resides inside "binfile.c". 
 It will look extremely similar to the current df_open().  I'd like to 
avoid that sort of thing, i.e., code repetition.

Dan