Re: gnuplot slow on BIG files

[Daniel J S. <dan...@ie...>]

Hans-Bernhard Broeker wrote:
> Daniel J Sebald wrote:
> 
>> Hans-Bernhard Broeker wrote:
> 
> 
>>> Necessity is not the issue --- convenience of maintaining the overall 
>>> structure of a very ancient code base is.  The code has always been 
>>> organized to read all data, then let the "every" filter decide which 
>>> points to actually use.
> 
> 
>> Actually, from what I remember, in datafile.c the "everypoint" 
>> variable is used to toss out points.  
> 
> 
> Yes.  But the question is: *when* does that happen: before sscanf()ing
> the values, or afterwards?  After a quick look into the current sources,
> it would seem that at least for 'matrix' files, it reads the entire 
> thing before applying 'every'.  That's probably the reason why 'every 
> 500:500' didn't achieve any speedup for Dimitrios.

Oh yeah, I see now... and my memory is coming back.  This is true in the case of 
binary data as well.  I can't recall if I originated the concept or carried it 
over from previous code, but the idea was to bring in all the data and create a 
data format similar to what the normal routines read, but in memory.  That could 
be changed I guess.  (But would have to think of the ramifications.)  The 
decimation could be done in the df_read_matrix() routine and then indicate to 
the main routine that "every" should be 1 instead of, say, 500.  I wouldn't call 
that an urgent change, however, seeing as I have a number of things to do right now.

Dan