Re: gnuplot slow on BIG files

[Dimitrios A. <ji...@gm...>]

Thank you all for your answers.

Hans-Bernhard Broeker wrote:
> Dimitrios Apostolou wrote:
> 
>> I just notice that gnuplot (4.0) is extremely slow when dealing with 
>> big files. In particular I execute the command:
>>
>> splot 'matrix.asc' matrix every 500:500
>>
>> where matrix.asc is an 130MB file containing a 6000x6000 matrix. 
> 
> 
> That's exceptionally little data per point.  130MB/(6000x6000) leaves 
> only 3 bytes/point, i.e. 2 decimal digits.

In case you care, what I try to do is a quick hack to plot SRTM data. If 
you want to reproduce my exact steps do the following:

- download a file from ftp://srtm.csi.cgiar.org/SRTM_Data_ArcAscii/ and 
unzip it
- sed -n '/^[0-9\-].*/p' thefile.asc | sed 's/-9999/0/g' > matrix.asc
- gnuplot
- splot 'matrix.asc' matrix every 500:500

>> What I don't like is that altough the points to plot are about 12x12
>> the processing takes about half an hour.
> 
> 
> So don't use gnuplot for it. 'using', 'every' are convenience features
> for quick and easy on-the fly data selection and manipulation, not the
> ultimate data processing tool. For that, use awk, perl, a spreadsheet,
> or whatever floats your boat.

I like gnuplot so I tried it. For this kind of data I really like the 
"map" capability of gnuplot. It would be interesting if the 
"convenience" features worked faster.

>> If I don't specify "every 500:500" the gnuplot process uses more than 
>> 1GB of memory (after much time) and gets killed by the OS. So a second 
>> point is that it uses more memory than necessary. 
> 
> 
> It's not *that* much more, actually.  A double-precision variable takes 
> 8 bytes, that's about three times as much as your ASCII data.  Add the 
> implied x and y variables missing in your matrix file and you're at
> 6000*6000*3*8 Bytes = 864 MB of data.  gnuplot will use even more than 

Is *3 necessary for this kind of data (matrix)?

> that, and that's a problem.  But the real problem here is that a 
> 6000x6000 points data set is essentially unplottable --- no output 
> device you're likely to be using has enough resolution to display all 
> those points in a readable way.

IMHO the more points we have the better looks the map or the surface 
mesh we plot. Of course I won't plot every point individually but as 
part of a surface.

>> In both cases it is noteworthy that the hard disk is almost idle but 
>> the CPU at 100% all the time. What I mean is that the reading of the 
>> file is happening really slowly.
> 
> 
> That's because gnuplot parses all data points, regardless of whether 

Is it really necessary? Why not parse only the needed points?

> they'll be used or not --- and, like it or not, scanning ASCII 
> representations of (presumably) floating-point numbers is *slow*.

I know of the overhead "parsing" implies, however I know that an 800 Mhz 
CPU ought to do it much faster. Don't you agree that gnuplot's 
implementation is highly inefficient on this?

Please don't be offended by my comments. I think gnuplot is a very nice 
program and I only try to make it better. Of course sending a patch to 
you would be better but I'm not at all familiar with its code.

> The problem is with the datafile, so that's where the solution has to 
> be.  Use external tools to reduce it to a manageable size.

I will do it, thanks. Or perhaps I will try to convert the datafile to 
binary format like someone else proposed.

Thank you all for your answers,
Dimitris