Re: [RFC] Proposal to re-think binary file handling

[Ethan M. <merritt@u.washington.edu>]

On Tuesday 02 December 2008 15:17:14 Ralf Juengling wrote:
> 
> On Tue, 2 Dec 2008, Ethan Merritt wrote:
> 
> >   set binary datafile array=Infx5 format="%double%double"
> >
> > As best as I can understand it, this is supposed to describe a rectangular
> > array of data whose extent in x is however many records there are in the
> > data file  (keyword "Inf") and whose extent in y is 5.
> >
> > Needless to say, there is no way that gnuplot's normal tokenising and
> > parsing routines (in scanner.c) can figure out that the string of
> > characters "Infx5" is supposed to represent three entities:
> >  "Inf"	- not the number "infinity" but a magic keyword
> >  "x"	- a magic-character field separator
> 
> Yes, but so is ":".

Sorry, I was not clear enough.
When gnuplot reads an input line, it separates the character stream into
a list of "tokens",  each of which is one or more characters that function
as a  single unit in the syntax.  Tokens must be separated by whitespace or
by a limited set of special characters.  "x" is not one of those special 
characters, because it is needed as a normal alphabetic character.
Placing quotes around any character sequence turns it into a single
token, a string constant.

The character sequence   10,12 is 3 tokens:    "10" "," and "12"
The character sequence   a1=22 is 3 tokens:    "a1"  "=" and "22"
The character sequence   5x5   is two tokens   "5"  and "x5"  
The character sequence   5xInf is two tokens   "5" and "xInf"
The character sequence   Infx5 is a single token  "Infx5"

Infx5 would be a legal variable name, or a possible keyword, but there
are only 2 ways to break it down into more than one unit of information.
One way is to bypass the existing gnuplot scanning/parsing mechanism and
write special-purpose code.  Worse yet, after treating this one sequence
of characters as a special case, you have to dummy up a modified list of
tokens that hides the characters you just processed separately.
That is what the current code does.

A second way is to put the whole thing in quotes, so that it is treated
as a single token.  In that case you still have to parse it specially,
but it does not require breaking and then repairing the existing token
scanning process.



> 
> >  "5"	- an actual integer
> >
> > If this specification instead used some rational syntax, perhaps
> >   array="[-1:5]"
> 
> I think this syntax is not a good idea as "[a:b]" means something
> completely different in other places. And with more dimensions you
> could not reuse existing parsing code anyway, could you?

That may or may not be the case, but it becomes irrelevant if we make
the required parameter be a string.  If it's a string, then it will be
handled as a single token.  


> > or the even more minimal
> >   array="-1 5"
>
> That's better (but I don't see how this is easier to parse than
> "x" as a separator). Why not a separator token other than space,
> a comma, for instance? With a comma one could write ",5" instead
> of "-1,5" to indicate that the number of records is unknown.

I don't really care much what is in the string, so long as it is
easy to interpret, ideally via a single sscanf().  

And it doesn't really have to be a string either, so long as the choice
of separators and syntax results in something that can be parsed unambiguously.
We already do that for "offset". Consider:
    set label "foo" offset 1
    set label "foo" offset 1,2
    set label "foo" offset 1,2,3

Having looked at the code, I think currently it would be easier to
store a single string for later interpretation than it would be to
store a variable number of separate dimensions.  But I'm proposing
to throw out the current code anyhow, so that argument doesn't have
much weight.



> As for breaking compatibility, I am in favor if it means it fixes
> problems. Binary data is a pretty recent feature, better make the
> change now than never.
> 
> Ralf
> 
> 
> 
> 
> 
> >
> > then we would gain at least the following:
> >
> > 1) It could be parsed by a single sscanf call
> > 	sscanf(string, "[%d:%d]", &xdim, &ydim]")
> > 2) The normal input line tokenise+parse routines could handle it
> > 3) The use of a string would allow substitution of alternative formats:
> > 	A = "[5:4]"; B = "[4:5]"
> > 	set binary datafile array=( typeA ? A : B )
> > 4) We could throw away hundreds of lines of unreadable code in
> >   plot_option_array() and associated routines
> > 5) Future maintenance would be vastly easier
> >
> >
> > What is the downside?
> > ---------------------
> >
> > It will break all current scripts, including the image-handling demos,
> > that use the keywords "array" or "record".
> > Then again, as Shige has been pointing out, many things are broken in
> > the current version already.
> >
> >
> > Should we go even further?
> > --------------------------
> >
> > Someone please remind me why we need oddball format specs like
> > 	format="%double%double"
> > rather than
> > 	format="%lf%lf"
> > I'm sure I must have asked this before, when the code went into CVS,
> > but I have forgotten the answer.  Can we get rid of additional hundreds
> > of lines of obscure code simply by requiring that users provide
> > a valid C format statement for reading their own binary data files?
> >
> > 	Ethan (sf...@us...)
> >
> > --
> > Ethan A Merritt
> >
> > -------------------------------------------------------------------------
> > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> > Build the coolest Linux based applications with Moblin SDK & win great prizes
> > Grand prize is a trip for two to an Open Source event anywhere in the world
> > http://moblin-contest.org/redirect.php?banner_id=100&url=/
> > _______________________________________________
> > gnuplot-beta mailing list
> > gnu...@li...
> > https://lists.sourceforge.net/lists/listinfo/gnuplot-beta
> >
> 



-- 
Ethan A Merritt
Biomolecular Structure Center
University of Washington, Seattle 98195-7742