Re: gnuplot slow on BIG files

[Hans-Bernhard B. <br...@ph...>]

Dimitrios Apostolou wrote:
> I just notice that gnuplot (4.0) is extremely slow when dealing with big 
> files. In particular I execute the command:
> 
> splot 'matrix.asc' matrix every 500:500
> 
> where matrix.asc is an 130MB file containing a 6000x6000 matrix. 

That's exceptionally little data per point.  130MB/(6000x6000) leaves 
only 3 bytes/point, i.e. 2 decimal digits.

> What I don't like is that altough the points to plot are about 12x12
> the processing takes about half an hour.

So don't use gnuplot for it. 'using', 'every' are convenience features
for quick and easy on-the fly data selection and manipulation, not the
ultimate data processing tool. For that, use awk, perl, a spreadsheet,
or whatever floats your boat.

> If I don't specify "every 500:500" the gnuplot process uses more than 
> 1GB of memory (after much time) and gets killed by the OS. So a second 
> point is that it uses more memory than necessary. 

It's not *that* much more, actually.  A double-precision variable takes 
8 bytes, that's about three times as much as your ASCII data.  Add the 
implied x and y variables missing in your matrix file and you're at
6000*6000*3*8 Bytes = 864 MB of data.  gnuplot will use even more than 
that, and that's a problem.  But the real problem here is that a 
6000x6000 points data set is essentially unplottable --- no output 
device you're likely to be using has enough resolution to display all 
those points in a readable way.

> In both cases it is noteworthy that the hard disk is almost idle but the 
> CPU at 100% all the time. What I mean is that the reading of the file is 
> happening really slowly.

That's because gnuplot parses all data points, regardless of whether 
they'll be used or not --- and, like it or not, scanning ASCII 
representations of (presumably) floating-point numbers is *slow*.

The problem is with the datafile, so that's where the solution has to 
be.  Use external tools to reduce it to a manageable size.