Re: Lengthy discussion about datafile.c...

[Ethan M. <merritt@u.washington.edu>]

On Saturday 14 August 2004 03:50 pm, Daniel J Sebald wrote:
> Ethan Merritt wrote:
> >Didn't we already have this discussion a few months ago?
> >I proposed that the whole notion of tracking input data by how
> >many columns were read in has outlived its usefulness.
> >I think we should get rid of max_cols and all the various
> >tests that depend on it, and instead pass explicit information
> >about the requested input data.  Hans-Bernard disagreed

Oh, and apologies for the typo in Hans-Bernhard's name.

> I know that for ASCII files the number of columns can be determined by
> the file itself and gnuplot readjusts accordingly.

Which brings up another issue.  The description of your binary read
"format" commands looks *really* fragile.  I mean the stuff being parsed
in plot_option_binary_format(). I am seriously worried that
it won't transfer well across 32/64 bit machines, that it won't handle
string data, and worst of all that it requires too much user-knowledge
of file and data types.  Basically I don't like it.

[EAM puts on geezer hat] In the old days of Fortran programming and 
VMS file systems, binary files had actual "records".   In those days
there was an obvious parallel between "columns" in an ascii file
and "records" in a binary file.  But that approach has been drowned
by the unix notion that "everything is a stream of bytes".  
It's *really hard* to figure out what data is in a binary stream, and I
am dubious that it is worth spending thousands of lines of code in
gnuplot trying to do so.  The unix way in such a case would be to
run the input binary data through a tailored filter on its way into
gnuplot.  That way gnuplot only has to know about ascii input, and
you can debug a suitable filter for your application without having
to recode gnuplot.  


Your docs say  
	+ Gnuplot will retrieve a number of binary
	+ variables equal to the largest column specified in the `<using list>`.
	+ For example, `using 1:3` would cause three columns to be read, of which
	+ the second will be ignored. 
So how do you handle the case of 10 logical columns of data in the file,
of which you only want to read the 2nd and 4th?  How do you skip "columns"
5 to 10 of each "line"?

What constitutes the logical equivalent of a "blank line" in your binary
files? Or is there no equivalent to the auto-determination of scan lines?

Do you plan to handle strings?  How? Would you require a full "binary format"
description in this case?  Is there such a thing as a matrix of strings?

The matrix variant is far more straight-forward.  I would think this will be 
by far the most common use anyhow, and it would cover the pixel images
that you obviously have fondness for. Could we maybe have a first cut
version of this patch that only deals with matrix format binary data?