Re: gnuplot slow on BIG files

[Dimitrios A. <ji...@gm...>]

Petr Mikulik wrote:
> Can you please update it for the current datafile.c from cvs on sourceforge?
> (There are minor rejects.)

Here you go. Warning: I haven't checked if it compiles and it still 
doesn't replace the old code. However, I think that the functions 
df_read_matrix_jimis() replaces are
df_read_matrix(), df_tokenise(), df_gets().

The reason I use fgetc() instead of fgets() is that one line might be 
very large to store it all at once. Speed shouldn't be reduced at all 
since stdio does buffered I/O.

Moreover, gnuplot is really memory inefficient IMHO. To read 36M numbers 
for example it needs:

(36M * sizeof(float)) + (36M * 3 * sizeof(float))

The first parenthesis is freed after creating the second one but the 
fact is that all this space is needed at copying time. Wouldn't it be 
interesting if we stored numbers from the file directly to gnuplot's 
format (second parenthesis)? That would result in 25% less memory 
requirements.

Finally, if you like the idea of rewriting the parser (me or somebody 
else), the matrix format has to be clarified. Which lines are considered 
comments: the ones starting immediately with "#"? are empty lines 
allowed? Fortran numbers? What is a valid delimiter (all whitespace)?

Thanks,
Dimitris