Re: gnuplot slow on BIG files

[Daniel J S. <dan...@ie...>]

Hans-Bernhard Broeker wrote:

>>
>> IMHO the more points we have the better looks the map or the surface 
>> mesh we plot. 
> 
> 
> That assumption is fatally flawed --- as soon as you have as many input 
> points than the output medium has pixels, adding more is guaranteed to 
> make the plot not better, but will actually render it increasingly 
> unreadable.  6000x6000 is well beyond that point: on a screen, you'll be 
> trying to display at least 36 data points in every pixel of your plot 
> --- that's not adding quality, that's adding confusion.

Hans is right, Dimitris.  There is also the very important issue of properly 
processing the data before discarding anything.  I'm not a fan of the "every" 
qualifier, but I put it into the binary input functionality because it already 
existed in the ascii input methods.

For images (or mesh plots, whatever) there is the concept of spatial frequency, 
i.e., how quickly intensity or color changes from pixel to pixel.  If one simply 
disregards that and tosses out every other pixel, or only keeps every 500th 
pixel, etc., there is the possibility of experiencing aliasing.  This means that 
some high spatial frequency part of the image could be aliased to a lower 
frequency, something that wasn't there previously.  The effect can be very bad.

The proper processing is to first low-pass filter the image with some form of 2D 
kernel, THEN toss out pixels.  In all likelihood, printers must do this step if 
you send it a 6000 x 6000 image.  Same would hold for a proper PostScript screen 
viewer.  However, if you know that the data in the file you are going to 
decimate is of sufficiently low frequency then sure, you can keep every Nth 
pixel without harm.  (But if that is the case then why store such a high 
resolution image?  Anyway...)  If not, you should process the thing in Matlab 
first and create a downsampled data set.

As someone in this discussion pointed out.  Once we have gnuplot doing all the 
low-pass filtering and so on suddenly gnuplot is more than just a plotting program.

> 
>>> That's because gnuplot parses all data points, regardless of whether 
> 
> 
>> Is it really necessary? Why not parse only the needed points?
> 
> 
> Necessity is not the issue --- convenience of maintaining the overall 
> structure of a very ancient code base is.  The code has always been 
> organized to read all data, then let the "every" filter decide which 
> points to actually use.  Changing that would be difficult, to say the 
> least.  Features like the fact that 'using' can be applied even to 
> 'matrix' data may well rely on such details.

Actually, from what I remember, in datafile.c the "everypoint" variable is used 
to toss out points.  Not completely sure on that, but that would mean that 
gnuplot does not internally store those points not used as a consequence of 
"every".  I believe "binary" works the same way with "every".  An hour does seem 
ridiculously long even for a slower machine.  If you are running out of memory 
linux will slow to a crawl because it is always swapping memory on and off the 
hard drive.

Dan