Re: gnuplot slow on BIG files

[Ethan M. <merritt@u.washington.edu>]

I have not been following this thread, so let me ask a few questions.

You say there is a speed-up of 10X.  
What were the actual times as reported by the "time" command?
Could you provide a profile analysis of the run time, 
so we could see where your CPU usage is being spent?

Speeding things up from 2 seconds to 0.2 seconds, for example,
is not very important. Or let me say that differently --
2 seconds to read a file may be prohibitively slow if you are
trying to use the mouse for interactive rotation, because currently
the file is re-read at each mouse increment.  But the proper fix
for this, IMHO, is not to fiddle with the file reading code. 
Instead we should modify the replot command so that it re-uses
the data previously read in if at all possible.


On Thursday 02 June 2005 03:53 pm, Dimitrios Apostolou wrote:
> 
> So I submit to you a patch (against the v. 4.0 gnuplot) for the file 
> src/datafile.c as a proof of concept

Could you please re-do the patch using
	diff -ur  <oldfile> <newfile>

> There are many things in my code that you 'll not probably like.

Your comments worry me.  For instance:

< 	/* malloc the maximum we may use, it's ok in an overcommiting OS like linux */
< 	tmp_arr = gp_alloc(filesize * sizeof(float), "df_matrix");

gnuplot core code must run on systems other than linux.
And even for linux your statement is not true.  Many people doing
serious number crunching will not run linux in overcommit mode, because
it is too painful to see a computation which has already run for 3 days
be killed by the OOM killer just because someone has opened a web browser,
or in this case because they try to run gnuplot.



-- 
Ethan A Merritt       merritt@u.washington.edu
Biomolecular Structure Center
Mailstop 357742
University of Washington, Seattle, WA 98195