Re: Lengthy discussion about datafile.c...

[<mi...@ph...>]

>> Well, I'd not thought of strings in the binary file, but perhaps
>> something like "%s" in the format string?  I would probably make the
>> restriction that the strings within the file need to be NULL
>> terminated.
>> I'm not sure what you mean by a matrix of strings.

Isn't "binary file with strings" the same as "ascii file with strings",
just with \0 instead of \n -- thus, "tr" filter would do all. Aha, that
won't help if binary data and strings are mixed in one file. Do you mean
this case?
>> The problem is that an image of say 500 x 500 pixels gets very big in
>> ASCII.
>
> I think that is a non-issue.  You don't have to store this anywhere;
> you're  just piping it in.

Piping is not as fast as direct read.

For example, I have recently benchmarked reading a big .gz files by a C
program with (1) popen("gzip -c -d"), and with (2) linking it with zlib.
The case (2) was 15% faster -- quite interesting speed-up if you have to
read 2 GB of data.
> When disks got cheap we all heaved a huge sigh of
> relief and for the most part stopped using binary output files.

I don't think this is right.

- You can work on a computer where you are not authorized to replace hard
disk (case of many companies). Also notebook hard disks are not cheap,
fast, and easily replaceable.
- Digital detectors are improving, and nowadays I have to deal with
2048x2048x16bit image series. That's plenty of data -- it grows
quadratically with improvements in detector technology.
- I guess image processing will never switch to ascii data -- it will
always be too big and slow.

> Many users may not be
> up to this, but those same users won't be able to figure out the
> endian business anyhow.

In the "with image" patch, parameters and thus command line options for
reading binary (matrix) files were designed carefully so that you can read
any type of data. Command line options cover the same range of options you
have to fill for any, even GUI-like, binary image reader, to read an
arbitrary image data. The user must always know its data, that's it, and
he is not bother that he has to pass this information to gnuplot or
whichever other image drawer.
> Simplicity is worth *a lot*.  Far more than saving a little bandwidth
> in the input pipe.

The patch reading and drawing binary data is a major speedup. Try to
compare drawing big (>512x512) traditional gnuplot binary data file and an
binary image file.
> Input of binary files containing regular arrays may be worth it
> for convenience.   But more complicated input requiring flags for
> bit order, word size, floating point format, and pre-announcement
> of the file structure?


> All that strikes me as being more trouble
> than it is worth.  Will your code work on an Amiga?  On a 64-bit
> VMS machine?

I think so. You can specify Float32, Float64 etc.

Probably you cannot draw binary floats saved by Turbo Pascal v 5--6,
because these are 6 B -- I don't know about recent Pascals. But you cannot
read in any other image programs I guess.
> Who is going to explain to users how to set
> all the right flags to make it work?

Users working with image processing know their format structure. That's
the user who explains gnuplot what to draw via command line options to
"plot ... with image".
Otherwise, you or somebody else writes a reader for Octave, and from there
you draw your matrix via imagegp.m included in the patch.
> I believe that Petr had some specific applications in mind, so
> maybe he can step in and clarify exactly what pieces of this
> code he wanted, and why.

Yes, I want to fastly image binary image data with axes x and y of
physical units (not pixel numbers).
> I myself plot many sorts of data in
> gnuplot, but I've  never felt a need for direct binary input.

I need it always when drawing an image larger than >=128x128. Otherwise,
the drawing speed is very low (and especially on X11) and memory
consumption is high (I remember that gnuplot eats at about 130 B for a
point read+drawn from a column-wise file).
The current version of Daniel's patch is fully satisfying my needs.

---
PM