gnuplot-beta

[RFC] Proposal to re-think binary file handling

[Ethan M. <merritt@u.washington.edu>]

This is a serious proposal to throw out and re-implement the
current binary file handling code.

Yes, this will break backwards compatibility.  I suppose this means
that current CVS will eventually become version 5.0 rather than 4.4.

Why do it?
----------

The code in datafile.c that handles binary files is horrible.
So horrible that I think it is unmaintainable as currently written.
Some of the documented behavior was never implemented, and some of
the implemented code does not work. I have managed to fix a couple of
simple errors, but the  amount of time I had to spend slogging
through obscure code was ridiculous.  The gnuplot project does not
have enough developer time available to maintain this code in its
current state.

For specific examples of problems, please see Shige Takeno's recent
series of error reports.  

For an example of the poor design and consequent ugly code that
tries to support it, consider the following command:

   set binary datafile array=Infx5 format="%double%double"

As best as I can understand it, this is supposed to describe a rectangular 
array of data whose extent in x is however many records there are in the
data file  (keyword "Inf") and whose extent in y is 5.

Needless to say, there is no way that gnuplot's normal tokenising and
parsing routines (in scanner.c) can figure out that the string of
characters "Infx5" is supposed to represent three entities:
  "Inf"	- not the number "infinity" but a magic keyword
  "x"	- a magic-character field separator
  "5"	- an actual integer       

If this specification instead used some rational syntax, perhaps
   array="[-1:5]"
or the even more minimal
   array="-1 5"

then we would gain at least the following:

1) It could be parsed by a single sscanf call
	sscanf(string, "[%d:%d]", &xdim, &ydim]")
2) The normal input line tokenise+parse routines could handle it
3) The use of a string would allow substitution of alternative formats:
	A = "[5:4]"; B = "[4:5]"
	set binary datafile array=( typeA ? A : B )
4) We could throw away hundreds of lines of unreadable code in 
   plot_option_array() and associated routines
5) Future maintenance would be vastly easier


What is the downside?
---------------------

It will break all current scripts, including the image-handling demos,
that use the keywords "array" or "record".
Then again, as Shige has been pointing out, many things are broken in
the current version already.


Should we go even further?
--------------------------

Someone please remind me why we need oddball format specs like
	format="%double%double"
rather than 
	format="%lf%lf"
I'm sure I must have asked this before, when the code went into CVS,
but I have forgotten the answer.  Can we get rid of additional hundreds
of lines of obscure code simply by requiring that users provide
a valid C format statement for reading their own binary data files?

	Ethan (sf...@us...)

-- 
Ethan A Merritt

Re: [RFC] Proposal to re-think binary file handling

[Ralf J. <jue...@cs...>]

On Tue, 2 Dec 2008, Ethan Merritt wrote:

>   set binary datafile array=Infx5 format="%double%double"
>
> As best as I can understand it, this is supposed to describe a rectangular
> array of data whose extent in x is however many records there are in the
> data file  (keyword "Inf") and whose extent in y is 5.
>
> Needless to say, there is no way that gnuplot's normal tokenising and
> parsing routines (in scanner.c) can figure out that the string of
> characters "Infx5" is supposed to represent three entities:
>  "Inf"	- not the number "infinity" but a magic keyword
>  "x"	- a magic-character field separator

Yes, but so is ":".

>  "5"	- an actual integer
>
> If this specification instead used some rational syntax, perhaps
>   array="[-1:5]"

I think this syntax is not a good idea as "[a:b]" means something
completely different in other places. And with more dimensions you
could not reuse existing parsing code anyway, could you?


> or the even more minimal
>   array="-1 5"

That's better (but I don't see how this is easier to parse than
"x" as a separator). Why not a separator token other than space,
a comma, for instance? With a comma one could write ",5" instead
of "-1,5" to indicate that the number of records is unknown.

As for breaking compatibility, I am in favor if it means it fixes
problems. Binary data is a pretty recent feature, better make the
change now than never.

Ralf





>
> then we would gain at least the following:
>
> 1) It could be parsed by a single sscanf call
> 	sscanf(string, "[%d:%d]", &xdim, &ydim]")
> 2) The normal input line tokenise+parse routines could handle it
> 3) The use of a string would allow substitution of alternative formats:
> 	A = "[5:4]"; B = "[4:5]"
> 	set binary datafile array=( typeA ? A : B )
> 4) We could throw away hundreds of lines of unreadable code in
>   plot_option_array() and associated routines
> 5) Future maintenance would be vastly easier
>
>
> What is the downside?
> ---------------------
>
> It will break all current scripts, including the image-handling demos,
> that use the keywords "array" or "record".
> Then again, as Shige has been pointing out, many things are broken in
> the current version already.
>
>
> Should we go even further?
> --------------------------
>
> Someone please remind me why we need oddball format specs like
> 	format="%double%double"
> rather than
> 	format="%lf%lf"
> I'm sure I must have asked this before, when the code went into CVS,
> but I have forgotten the answer.  Can we get rid of additional hundreds
> of lines of obscure code simply by requiring that users provide
> a valid C format statement for reading their own binary data files?
>
> 	Ethan (sf...@us...)
>
> --
> Ethan A Merritt
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> gnuplot-beta mailing list
> gnu...@li...
> https://lists.sourceforge.net/lists/listinfo/gnuplot-beta
>

Re: [RFC] Proposal to re-think binary file handling

[Ethan M. <merritt@u.washington.edu>]

On Tuesday 02 December 2008 15:17:14 Ralf Juengling wrote:
> 
> On Tue, 2 Dec 2008, Ethan Merritt wrote:
> 
> >   set binary datafile array=Infx5 format="%double%double"
> >
> > As best as I can understand it, this is supposed to describe a rectangular
> > array of data whose extent in x is however many records there are in the
> > data file  (keyword "Inf") and whose extent in y is 5.
> >
> > Needless to say, there is no way that gnuplot's normal tokenising and
> > parsing routines (in scanner.c) can figure out that the string of
> > characters "Infx5" is supposed to represent three entities:
> >  "Inf"	- not the number "infinity" but a magic keyword
> >  "x"	- a magic-character field separator
> 
> Yes, but so is ":".

Sorry, I was not clear enough.
When gnuplot reads an input line, it separates the character stream into
a list of "tokens",  each of which is one or more characters that function
as a  single unit in the syntax.  Tokens must be separated by whitespace or
by a limited set of special characters.  "x" is not one of those special 
characters, because it is needed as a normal alphabetic character.
Placing quotes around any character sequence turns it into a single
token, a string constant.

The character sequence   10,12 is 3 tokens:    "10" "," and "12"
The character sequence   a1=22 is 3 tokens:    "a1"  "=" and "22"
The character sequence   5x5   is two tokens   "5"  and "x5"  
The character sequence   5xInf is two tokens   "5" and "xInf"
The character sequence   Infx5 is a single token  "Infx5"

Infx5 would be a legal variable name, or a possible keyword, but there
are only 2 ways to break it down into more than one unit of information.
One way is to bypass the existing gnuplot scanning/parsing mechanism and
write special-purpose code.  Worse yet, after treating this one sequence
of characters as a special case, you have to dummy up a modified list of
tokens that hides the characters you just processed separately.
That is what the current code does.

A second way is to put the whole thing in quotes, so that it is treated
as a single token.  In that case you still have to parse it specially,
but it does not require breaking and then repairing the existing token
scanning process.



> 
> >  "5"	- an actual integer
> >
> > If this specification instead used some rational syntax, perhaps
> >   array="[-1:5]"
> 
> I think this syntax is not a good idea as "[a:b]" means something
> completely different in other places. And with more dimensions you
> could not reuse existing parsing code anyway, could you?

That may or may not be the case, but it becomes irrelevant if we make
the required parameter be a string.  If it's a string, then it will be
handled as a single token.  


> > or the even more minimal
> >   array="-1 5"
>
> That's better (but I don't see how this is easier to parse than
> "x" as a separator). Why not a separator token other than space,
> a comma, for instance? With a comma one could write ",5" instead
> of "-1,5" to indicate that the number of records is unknown.

I don't really care much what is in the string, so long as it is
easy to interpret, ideally via a single sscanf().  

And it doesn't really have to be a string either, so long as the choice
of separators and syntax results in something that can be parsed unambiguously.
We already do that for "offset". Consider:
    set label "foo" offset 1
    set label "foo" offset 1,2
    set label "foo" offset 1,2,3

Having looked at the code, I think currently it would be easier to
store a single string for later interpretation than it would be to
store a variable number of separate dimensions.  But I'm proposing
to throw out the current code anyhow, so that argument doesn't have
much weight.



> As for breaking compatibility, I am in favor if it means it fixes
> problems. Binary data is a pretty recent feature, better make the
> change now than never.
> 
> Ralf
> 
> 
> 
> 
> 
> >
> > then we would gain at least the following:
> >
> > 1) It could be parsed by a single sscanf call
> > 	sscanf(string, "[%d:%d]", &xdim, &ydim]")
> > 2) The normal input line tokenise+parse routines could handle it
> > 3) The use of a string would allow substitution of alternative formats:
> > 	A = "[5:4]"; B = "[4:5]"
> > 	set binary datafile array=( typeA ? A : B )
> > 4) We could throw away hundreds of lines of unreadable code in
> >   plot_option_array() and associated routines
> > 5) Future maintenance would be vastly easier
> >
> >
> > What is the downside?
> > ---------------------
> >
> > It will break all current scripts, including the image-handling demos,
> > that use the keywords "array" or "record".
> > Then again, as Shige has been pointing out, many things are broken in
> > the current version already.
> >
> >
> > Should we go even further?
> > --------------------------
> >
> > Someone please remind me why we need oddball format specs like
> > 	format="%double%double"
> > rather than
> > 	format="%lf%lf"
> > I'm sure I must have asked this before, when the code went into CVS,
> > but I have forgotten the answer.  Can we get rid of additional hundreds
> > of lines of obscure code simply by requiring that users provide
> > a valid C format statement for reading their own binary data files?
> >
> > 	Ethan (sf...@us...)
> >
> > --
> > Ethan A Merritt
> >
> > -------------------------------------------------------------------------
> > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> > Build the coolest Linux based applications with Moblin SDK & win great prizes
> > Grand prize is a trip for two to an Open Source event anywhere in the world
> > http://moblin-contest.org/redirect.php?banner_id=100&url=/
> > _______________________________________________
> > gnuplot-beta mailing list
> > gnu...@li...
> > https://lists.sourceforge.net/lists/listinfo/gnuplot-beta
> >
> 



-- 
Ethan A Merritt
Biomolecular Structure Center
University of Washington, Seattle 98195-7742

Re: [RFC] Proposal to re-think binary file handling

[Petr M. <mi...@ph...>]

>    set binary datafile array=Infx5 format="%double%double"
> 
> Someone please remind me why we need oddball format specs like
> 	format="%double%double"
> rather than 
> 	format="%lf%lf"
> I'm sure I must have asked this before, when the code went into CVS,
> but I have forgotten the answer.  Can we get rid of additional hundreds
> of lines of obscure code simply by requiring that users provide
> a valid C format statement for reading their own binary data files?

Implementation of general binary files comes from 2003. We exchanged a lot 
of e-mails with Daniel Sebald. Nobody came with such a proposal as you have 
now.

The main reason for format="%double%double" and not "%lf%lf" is that textual 
headers of image or volume files have keywords like char, byte, double, int, 
float, etc. That's why we had used the same keywords for "format" as well.

On the other hand, your proposal "%lf%lf" reflects the formating from 
programmer's point of view. However, in C, there is no special formatting 
for signed and unsigned char, so we are out of luck for this notation.

---
PM

Re: [RFC] Proposal to re-think binary file handling

[Ethan M. <merritt@u.washington.edu>]

On Wednesday 03 December 2008 09:28:18 Ralf Juengling wrote:
> > Having looked at the code, I think currently it would be easier to
> > store a single string for later interpretation than it would be to
> > store a variable number of separate dimensions.  But I'm proposing
> > to throw out the current code anyhow, so that argument doesn't have
> > much weight.
> 
> I have a slight preference for not making it a string. On a second
> thought a comma might not be such a good choice for separator, though. 
> The comma is also used to separate different lines in a single plot
> command, so there is potential for ambiguity.
> 
> I vote for ':' instead, like
> 
> array=4:23:100
> 
> but not
> 
> array=[4:23:100]
> 
> as you initially suggested.
> 
> Ralf
> 

Please have a look at the patch now attached to Bug #2378535
 https://sourceforge.net/tracker/index.php?func=detail&aid=2378535&group_id=2055&atid=102055

The best I have come up with is to replace 
   array=5x10
with
   array=(5,10)

That cleans up the parsing, and also allows you to use variables and expressions: 
   array=(A, 2+NVAR)

A bare comma with no parentheses would not work, because it makes the following
plot command ambiguous:

   plot foo binary array=5,baz    # Is baz a second dimension or a second graph?

Colon is already in use to separate fields within a record, and may exist in the
same specification as an array:

   record=(5,10):2:4


	thanks for your feedback,

		Ethan


-- 
Ethan A Merritt

Re: [RFC] Proposal to re-think binary file handling

[Daniel S. <dan...@ie...>]

Petr Mikulik <mikulik <at> physics.muni.cz> writes:

> 
> >    set binary datafile array=Infx5 format="%double%double"
> > 
> > Someone please remind me why we need oddball format specs like
> > 	format="%double%double"
> > rather than 
> > 	format="%lf%lf"
> > I'm sure I must have asked this before, when the code went into CVS,
> > but I have forgotten the answer.  Can we get rid of additional hundreds
> > of lines of obscure code simply by requiring that users provide
> > a valid C format statement for reading their own binary data files?
> 
> Implementation of general binary files comes from 2003. We exchanged a lot 
> of e-mails with Daniel Sebald. Nobody came with such a proposal as you have 
> now.
> 
> The main reason for format="%double%double" and not "%lf%lf" is that textual 
> headers of image or volume files have keywords like char, byte, double, int, 
> float, etc. That's why we had used the same keywords for "format" as well.
> 
> On the other hand, your proposal "%lf%lf" reflects the formating from 
> programmer's point of view. However, in C, there is no special formatting 
> for signed and unsigned char, so we are out of luck for this notation.

Hi Folks,

The bugs found by Shigeharu Takeno (I think there were two separate bugs)
seemed insignificant.  If they are still outstanding, point me to the bug
reports.

'set datafile binary format' was implemented from what I recall, but that
is where one of the bugs may lie.

Generally, I don't think the binary code is the awful mess claimed to be,
but sure the syntax and layout could perhaps use improvement.  I'm certain
I asked time and again for feedback five years ago, but Petr seemed the
only one to take interest.  In fact, I agreed it should be labeled
'EXPERIMENTAL', and I'm open for changes if others are.

Petr has described the reason for the more varied format specifiers; it's
for the programmer trying to communicate with gnuplot.  That is, 'double',
'uint', etc. is more the language of binary programming.  There are
machine independent and machine dependent sizes for various instances.
Anyway, parsing that code isn't too bad.  The '10x5' dimension specifier
is the inelegant one.

It wasn't fun to program the binary array specification, mostly because--
as Ethan pointed out--almost anything one can dream up causes confusion
with already-used tokens.  The comma, the colon, etc.  They cause the
parser to think it is an end-of-line, so on.  'Inf' isn't particularly
graceful, is it?  I suppose I chose that as a symbol because it was the
only thing I could think of to mean "indefinite".  However, how about a
syntax that is similar to the way variables are declared in C?

array [10][5]

Would that parse a little better?  That would be six tokens, I think, that
the parser would send back.  The syntax would allow for the 'indefinite'
dimension length

array [10][5][]

which has the look of C.

Dan