RFC: a proposed alternative solution to the "plotting columns from multiple files" problem

[Juhász P. <pet...@gm...>]

Dear gnuplot developers,

Ethan has a patch on Sourceforge that aims to solve the old FAQ "how can
I combine values from columns in multiple input files?"

http://sourceforge.net/p/gnuplot/patches/615/

First some observations to the patch and its description:

I don't like the name "merge" for this operation, because simply,
merging is not what it does. If we were to go with it, I'd propose
"store" or "stash" (the latter is inspired by "git stash")...

...but I don't really like it as it is. I found the new command and its
usage pattern quite hard to understand and the whole thing comes across
as a hack.

So, I thought, if we wanted to solve the original FAQ problem, why not
attack it directly, by extending the plot command to allow plotting from
multiple files -- but I couldn't find an acceptable solution, with the
plot command being quite complicated as it is, both in its user
interface and its implementation.

Then a new idea struck: introduce a new concept called "datasource", in
effect a layer that comes between low-level datafile reading and
plotting. In this new mechanism, a datasource would be a kind of a
"virtual file" that would define how the contents from one or more real
data files are to be combined, transformed and filtered, and their
output fed to the plot command. 

For example: 

set datasource $DATA1 "foo.txt" paste "bar.txt"
plot $DATA1

This command would take the two files foo.txt and bar.txt and
concatenate them line by line, like the Unix "paste" command, and let
the plot command see the result of this combined file. 

Other combinations could be defined, for example "cat" which would just
concatenate the files, one after the other (like the similarly named
Unix command), or "transpose", which would act on just one file,
transposing its contents. The usual "using", "every" etc. modifiers
could be applied to the file names.

It is important that the "set datasource" command itself would not
perform these operations, it would just prepare them. The data files
would be read (and the specified transformations performed on them) only
when the plot command is executed.

(The alternative is that the operations are performed by the set command
itself and the results saved into a datablock. This would be simpler to
implement, but the resulting datablocks could take up a lot of space, or
the operation may not be possible at all if the input files are very
large -- this is a problem with the original merge proposal as well).


Note that I don't have any code to show yet, this RFC is just to poll
the public opinion to see if the concept makes sense at all.

Also note that I personally have some reservations about the whole
thing: it would potentially require rewriting / mucking up sensitive
"here be dragons" parts of the code, with unclear benefit. It would also
go against the Unix principle: we can run external commands and we have
adequate text processing utilities outside gnuplot, so there is little
need to make gnuplot into a text-processing-kitchen-sink-included
utility. 


Let me know what you think about all this.

Peter Juhasz