gnuplot-beta

gnuplot slow on BIG files

[Dimitrios A. <ji...@gm...>]

Hello list,

I just notice that gnuplot (4.0) is extremely slow when dealing with big 
files. In particular I execute the command:

splot 'matrix.asc' matrix every 500:500

where matrix.asc is an 130MB file containing a 6000x6000 matrix. What I 
don't like is that altough the points to plot are about 12x12 the 
processing takes about half an hour.

If I don't specify "every 500:500" the gnuplot process uses more than 
1GB of memory (after much time) and gets killed by the OS. So a second 
point is that it uses more memory than necessary. Actually the memory 
consumed is enormous considering that the points are "only" 36.000.000.

In both cases it is noteworthy that the hard disk is almost idle but the 
CPU at 100% all the time. What I mean is that the reading of the file is 
happening really slowly.

Is this a bug? Or am I doing something wrong? Is there a workaround to 
speed things up?

Thanks in advance,
Dimitris

P.S. Please CC replies directly to me since I'm not subscribed to the list

Re: gnuplot slow on BIG files

[Petr M. <mi...@ph...>]

> I just notice that gnuplot (4.0) is extremely slow when dealing with big
> where matrix.asc is an 130MB file containing a 6000x6000 matrix. What I

I think writing such a huge file is very slow as well. Why don't you use
gnuplot binary format? See 'help binary'.

You can decreses memory consumption by setting in syscfg.h COORDVAL_FLOAT
from double to float.

Finally, consider using the development version 4.1. It supports images,
binary image files etc.

---
PM

Re: gnuplot slow on BIG files

[Hans-Bernhard B. <br...@ph...>]

Dimitrios Apostolou wrote:
> I just notice that gnuplot (4.0) is extremely slow when dealing with big 
> files. In particular I execute the command:
> 
> splot 'matrix.asc' matrix every 500:500
> 
> where matrix.asc is an 130MB file containing a 6000x6000 matrix. 

That's exceptionally little data per point.  130MB/(6000x6000) leaves 
only 3 bytes/point, i.e. 2 decimal digits.

> What I don't like is that altough the points to plot are about 12x12
> the processing takes about half an hour.

So don't use gnuplot for it. 'using', 'every' are convenience features
for quick and easy on-the fly data selection and manipulation, not the
ultimate data processing tool. For that, use awk, perl, a spreadsheet,
or whatever floats your boat.

> If I don't specify "every 500:500" the gnuplot process uses more than 
> 1GB of memory (after much time) and gets killed by the OS. So a second 
> point is that it uses more memory than necessary. 

It's not *that* much more, actually.  A double-precision variable takes 
8 bytes, that's about three times as much as your ASCII data.  Add the 
implied x and y variables missing in your matrix file and you're at
6000*6000*3*8 Bytes = 864 MB of data.  gnuplot will use even more than 
that, and that's a problem.  But the real problem here is that a 
6000x6000 points data set is essentially unplottable --- no output 
device you're likely to be using has enough resolution to display all 
those points in a readable way.

> In both cases it is noteworthy that the hard disk is almost idle but the 
> CPU at 100% all the time. What I mean is that the reading of the file is 
> happening really slowly.

That's because gnuplot parses all data points, regardless of whether 
they'll be used or not --- and, like it or not, scanning ASCII 
representations of (presumably) floating-point numbers is *slow*.

The problem is with the datafile, so that's where the solution has to 
be.  Use external tools to reduce it to a manageable size.

Re: gnuplot slow on BIG files

[Dimitrios A. <ji...@gm...>]

Thank you all for your answers.

Hans-Bernhard Broeker wrote:
> Dimitrios Apostolou wrote:
> 
>> I just notice that gnuplot (4.0) is extremely slow when dealing with 
>> big files. In particular I execute the command:
>>
>> splot 'matrix.asc' matrix every 500:500
>>
>> where matrix.asc is an 130MB file containing a 6000x6000 matrix. 
> 
> 
> That's exceptionally little data per point.  130MB/(6000x6000) leaves 
> only 3 bytes/point, i.e. 2 decimal digits.

In case you care, what I try to do is a quick hack to plot SRTM data. If 
you want to reproduce my exact steps do the following:

- download a file from ftp://srtm.csi.cgiar.org/SRTM_Data_ArcAscii/ and 
unzip it
- sed -n '/^[0-9\-].*/p' thefile.asc | sed 's/-9999/0/g' > matrix.asc
- gnuplot
- splot 'matrix.asc' matrix every 500:500

>> What I don't like is that altough the points to plot are about 12x12
>> the processing takes about half an hour.
> 
> 
> So don't use gnuplot for it. 'using', 'every' are convenience features
> for quick and easy on-the fly data selection and manipulation, not the
> ultimate data processing tool. For that, use awk, perl, a spreadsheet,
> or whatever floats your boat.

I like gnuplot so I tried it. For this kind of data I really like the 
"map" capability of gnuplot. It would be interesting if the 
"convenience" features worked faster.

>> If I don't specify "every 500:500" the gnuplot process uses more than 
>> 1GB of memory (after much time) and gets killed by the OS. So a second 
>> point is that it uses more memory than necessary. 
> 
> 
> It's not *that* much more, actually.  A double-precision variable takes 
> 8 bytes, that's about three times as much as your ASCII data.  Add the 
> implied x and y variables missing in your matrix file and you're at
> 6000*6000*3*8 Bytes = 864 MB of data.  gnuplot will use even more than 

Is *3 necessary for this kind of data (matrix)?

> that, and that's a problem.  But the real problem here is that a 
> 6000x6000 points data set is essentially unplottable --- no output 
> device you're likely to be using has enough resolution to display all 
> those points in a readable way.

IMHO the more points we have the better looks the map or the surface 
mesh we plot. Of course I won't plot every point individually but as 
part of a surface.

>> In both cases it is noteworthy that the hard disk is almost idle but 
>> the CPU at 100% all the time. What I mean is that the reading of the 
>> file is happening really slowly.
> 
> 
> That's because gnuplot parses all data points, regardless of whether 

Is it really necessary? Why not parse only the needed points?

> they'll be used or not --- and, like it or not, scanning ASCII 
> representations of (presumably) floating-point numbers is *slow*.

I know of the overhead "parsing" implies, however I know that an 800 Mhz 
CPU ought to do it much faster. Don't you agree that gnuplot's 
implementation is highly inefficient on this?

Please don't be offended by my comments. I think gnuplot is a very nice 
program and I only try to make it better. Of course sending a patch to 
you would be better but I'm not at all familiar with its code.

> The problem is with the datafile, so that's where the solution has to 
> be.  Use external tools to reduce it to a manageable size.

I will do it, thanks. Or perhaps I will try to convert the datafile to 
binary format like someone else proposed.

Thank you all for your answers,
Dimitris

Re: gnuplot slow on BIG files

[Hans-Bernhard B. <br...@ph...>]

Dimitrios Apostolou wrote:
> Hans-Bernhard Broeker wrote:
>> Dimitrios Apostolou wrote:

> I like gnuplot so I tried it. For this kind of data I really like the
> "map" capability of gnuplot. It would be interesting if the 
> "convenience" features worked faster.

Interesting, yes.  But we're already stretched somewhat thin on 
man-power as it is.  Concerning ourselves with the efficiency of side 
aspects that other tools are already a lot better at than gnuplot can 
possibly be, would be a waste of effort.

Note that I'm not trying to tell you not to use gnuplot at all --- I'm 
trying to convey the message that you should use more tools than only 
gnuplot.  gnuplot is good for plotting, but mediocre at mass data 
processing.  So use better tools for that part of the job, then come 
back to gnuplot with the actualy plottable data.

>> It's not *that* much more, actually.  A double-precision variable 
>> takes 8 bytes, that's about three times as much as your ASCII data.  
>> Add the implied x and y variables missing in your matrix file and 
>> you're at
>> 6000*6000*3*8 Bytes = 864 MB of data.  gnuplot will use even more than 
> 
> 
> Is *3 necessary for this kind of data (matrix)?

For reasons of program structure and internal efficiency, all the 
various kinds of input data have to end up in the *same* data structure, 
regardless of whether they were topologically and geometrically very 
limited matrix data, generic grid-topology data, or a point cloud 
without any kind of structure.

It's not strictly necessary to do it that way, but for most reasonable 
plots, this organization works well.

>> that, and that's a problem.  But the real problem here is that a 
>> 6000x6000 points data set is essentially unplottable --- no output 
>> device you're likely to be using has enough resolution to display all 
>> those points in a readable way.
> 
> 
> IMHO the more points we have the better looks the map or the surface 
> mesh we plot. 

That assumption is fatally flawed --- as soon as you have as many input 
points than the output medium has pixels, adding more is guaranteed to 
make the plot not better, but will actually render it increasingly 
unreadable.  6000x6000 is well beyond that point: on a screen, you'll be 
trying to display at least 36 data points in every pixel of your plot 
--- that's not adding quality, that's adding confusion.

>> That's because gnuplot parses all data points, regardless of whether 

> Is it really necessary? Why not parse only the needed points?

Necessity is not the issue --- convenience of maintaining the overall 
structure of a very ancient code base is.  The code has always been 
organized to read all data, then let the "every" filter decide which 
points to actually use.  Changing that would be difficult, to say the 
least.  Features like the fact that 'using' can be applied even to 
'matrix' data may well rely on such details.

>> they'll be used or not --- and, like it or not, scanning ASCII 
>> representations of (presumably) floating-point numbers is *slow*.

> I know of the overhead "parsing" implies, however I know that an 800 Mhz 
> CPU ought to do it much faster. 

Probably.  I just ran a little experiment, and found that 36000000 
double-precision numbers could be scanf()ed in about 80 seconds process 
CPU time, on a 650 MHz PIII.

It may be worthwile to profile your gnuplot in (a smaller version of) 
this case, to see where the time is actually spent.  Possibly, it's pure 
memory access time --- at this kind of size, memory bandwidth becomes a 
serious bottleneck, too.

 > Don't you agree that gnuplot's
> implementation is highly inefficient on this?

ASCII datafiles are inefficient by design, but at the same time, this 
inefficiency allows them to be understood by humans, and keeps them 100% 
portable across machine architectures.  Two sides of the same medal.

Re: gnuplot slow on BIG files

[Daniel J S. <dan...@ie...>]

Hans-Bernhard Broeker wrote:

>>
>> IMHO the more points we have the better looks the map or the surface 
>> mesh we plot. 
> 
> 
> That assumption is fatally flawed --- as soon as you have as many input 
> points than the output medium has pixels, adding more is guaranteed to 
> make the plot not better, but will actually render it increasingly 
> unreadable.  6000x6000 is well beyond that point: on a screen, you'll be 
> trying to display at least 36 data points in every pixel of your plot 
> --- that's not adding quality, that's adding confusion.

Hans is right, Dimitris.  There is also the very important issue of properly 
processing the data before discarding anything.  I'm not a fan of the "every" 
qualifier, but I put it into the binary input functionality because it already 
existed in the ascii input methods.

For images (or mesh plots, whatever) there is the concept of spatial frequency, 
i.e., how quickly intensity or color changes from pixel to pixel.  If one simply 
disregards that and tosses out every other pixel, or only keeps every 500th 
pixel, etc., there is the possibility of experiencing aliasing.  This means that 
some high spatial frequency part of the image could be aliased to a lower 
frequency, something that wasn't there previously.  The effect can be very bad.

The proper processing is to first low-pass filter the image with some form of 2D 
kernel, THEN toss out pixels.  In all likelihood, printers must do this step if 
you send it a 6000 x 6000 image.  Same would hold for a proper PostScript screen 
viewer.  However, if you know that the data in the file you are going to 
decimate is of sufficiently low frequency then sure, you can keep every Nth 
pixel without harm.  (But if that is the case then why store such a high 
resolution image?  Anyway...)  If not, you should process the thing in Matlab 
first and create a downsampled data set.

As someone in this discussion pointed out.  Once we have gnuplot doing all the 
low-pass filtering and so on suddenly gnuplot is more than just a plotting program.

> 
>>> That's because gnuplot parses all data points, regardless of whether 
> 
> 
>> Is it really necessary? Why not parse only the needed points?
> 
> 
> Necessity is not the issue --- convenience of maintaining the overall 
> structure of a very ancient code base is.  The code has always been 
> organized to read all data, then let the "every" filter decide which 
> points to actually use.  Changing that would be difficult, to say the 
> least.  Features like the fact that 'using' can be applied even to 
> 'matrix' data may well rely on such details.

Actually, from what I remember, in datafile.c the "everypoint" variable is used 
to toss out points.  Not completely sure on that, but that would mean that 
gnuplot does not internally store those points not used as a consequence of 
"every".  I believe "binary" works the same way with "every".  An hour does seem 
ridiculously long even for a slower machine.  If you are running out of memory 
linux will slow to a crawl because it is always swapping memory on and off the 
hard drive.

Dan

Re: gnuplot slow on BIG files

[Hans-Bernhard B. <br...@ph...>]

Daniel J Sebald wrote:
> Hans-Bernhard Broeker wrote:

>> Necessity is not the issue --- convenience of maintaining the overall 
>> structure of a very ancient code base is.  The code has always been 
>> organized to read all data, then let the "every" filter decide which 
>> points to actually use.

> Actually, from what I remember, in datafile.c the "everypoint" variable 
> is used to toss out points.  

Yes.  But the question is: *when* does that happen: before sscanf()ing
the values, or afterwards?  After a quick look into the current sources,
it would seem that at least for 'matrix' files, it reads the entire 
thing before applying 'every'.  That's probably the reason why 'every 
500:500' didn't achieve any speedup for Dimitrios.

Re: gnuplot slow on BIG files

[Daniel J S. <dan...@ie...>]

Hans-Bernhard Broeker wrote:
> Daniel J Sebald wrote:
> 
>> Hans-Bernhard Broeker wrote:
> 
> 
>>> Necessity is not the issue --- convenience of maintaining the overall 
>>> structure of a very ancient code base is.  The code has always been 
>>> organized to read all data, then let the "every" filter decide which 
>>> points to actually use.
> 
> 
>> Actually, from what I remember, in datafile.c the "everypoint" 
>> variable is used to toss out points.  
> 
> 
> Yes.  But the question is: *when* does that happen: before sscanf()ing
> the values, or afterwards?  After a quick look into the current sources,
> it would seem that at least for 'matrix' files, it reads the entire 
> thing before applying 'every'.  That's probably the reason why 'every 
> 500:500' didn't achieve any speedup for Dimitrios.

Oh yeah, I see now... and my memory is coming back.  This is true in the case of 
binary data as well.  I can't recall if I originated the concept or carried it 
over from previous code, but the idea was to bring in all the data and create a 
data format similar to what the normal routines read, but in memory.  That could 
be changed I guess.  (But would have to think of the ramifications.)  The 
decimation could be done in the df_read_matrix() routine and then indicate to 
the main routine that "every" should be 1 instead of, say, 500.  I wouldn't call 
that an urgent change, however, seeing as I have a number of things to do right now.

Dan

Re: gnuplot slow on BIG files

[V. <gae...@en...>]

  Hello,

  Have you tried using octave to preprocess the matrix before sending it
to Gnuplot via a buffer file or throught the Octave/Gnuplot interface.
Gnuplot is not a math program, it is a plotting program. If you want to
process huge amount of data use a math program (or for simple matrice
rechaping I found out awk is quite convenient).

--
  Ga=EBl

Re: gnuplot slow on BIG files

[Dimitrios A. <ji...@gm...>]

Hello.

It would be nice if gnuplot parsed only the numbers it needed, but I 
understand this is not a priority. This is indeed a problem that can 
(should?) be corrected in the datafile.

However, the second problem I noticed in gnuplot was the very slow 
parsing. I believe this is something that needs to be corrected.

I tried to improve the parsing speed but I couldn't understand *many* 
things in the existing code. I 'm sure those are used somewhere but 
since I couldn't I understand them, I rewrote all the parser, the 
simplest way possible.

So I submit to you a patch (against the v. 4.0 gnuplot) for the file 
src/datafile.c as a proof of concept, that the current parser is slow 
and can be improved. And you 'll see that this version is faster more 
than 10 times. Please forgive any silly programming mistakes, I'm not 
much experienced in C.

There are many things in my code that you 'll not probably like. They 
were added to bypass several problems that the old codebase created. 
After all I only wrote this code as a proof of concept. However, if you 
think that this can replace the existing parser, tell me so, because 
there are a few things that need to change. Of course you may improve it 
as you wish.

Dimitris

Re: gnuplot slow on BIG files

[Daniel J S. <dan...@ie...>]

Dimitrios Apostolou wrote:

> So I submit to you a patch (against the v. 4.0 gnuplot) for the file 
> src/datafile.c as a proof of concept, that the current parser is slow 
> and can be improved. And you 'll see that this version is faster more 
> than 10 times. Please forgive any silly programming mistakes, I'm not 
> much experienced in C.

Please *explain* why the patch is faster.  Those listening will understand. 
Also, when running diff be sure to use unified (-u) so that it indicates what 
file the hunks come from.

Dan

Re: gnuplot slow on BIG files

[Dimitrios A. <ji...@gm...>]

Daniel J Sebald wrote:
> Please *explain* why the patch is faster.  Those listening will 
> understand. Also, when running diff be sure to use unified (-u) so that 
> it indicates what file the hunks come from.

I don't know why it is faster. I just wrote a simple parser. I don't 
understand what more the old parser does. I just can see that it is much 
more complicated. My guess is that after so many years of development 
and after many additions that today we see but can't figure out, the 
code became a bit "bloated".

Do what you think is better: optimize the current parser, rewrite a new 
one, or use mine as a base for improvement. One thing I know for sure: 
it shouldn't stay as it is.

The patch I published is against the file:
gnuplot-4.0.0/src/datafile.c

Dimitris

Re: gnuplot slow on BIG files

[Daniel J S. <dan...@ie...>]

Dimitrios Apostolou wrote:
> Daniel J Sebald wrote:
> 
>> Please *explain* why the patch is faster.  Those listening will 
>> understand. Also, when running diff be sure to use unified (-u) so 
>> that it indicates what file the hunks come from.
> 
> 
> I don't know why it is faster. I just wrote a simple parser. I don't 
> understand what more the old parser does. I just can see that it is much 
> more complicated. My guess is that after so many years of development 
> and after many additions that today we see but can't figure out, the 
> code became a bit "bloated".

(OK, saw this file in a backlog of email.  Hans must be sending these through 
manually.)  It is a possibility.  That matrix code is sort of a tacked on thing. 
  But unless we understand what the problem is how can we know that there is 
extraneous, inefficient code?  However, if you believe what you've coded meets 
the definitions and format in the documentation and is faster, then consider 
redoing the patch with the old, unneeded code removed.  Otherwise, it will leave 
cruft floating about, just what you are attempting to improve upon.

Thanks,

Dan

Re: gnuplot slow on BIG files

[Dimitrios A. <ji...@gm...>]

> (OK, saw this file in a backlog of email.  Hans must be sending these 
> through manually.)  It is a possibility.  That matrix code is sort of a 
> tacked on thing.  But unless we understand what the problem is how can 
> we know that there is extraneous, inefficient code?  However, if you 
> believe what you've coded meets the definitions and format in the 
> documentation and is faster, then consider redoing the patch with the 
> old, unneeded code removed.  Otherwise, it will leave cruft floating 
> about, just what you are attempting to improve upon.

My patch certainly doesn't meet any definitions or format. I have no 
time right now to rewrite the patch correctly and according to the 
coding standards. If you wish that I send you another patch with the old 
code replaced, tell me so and I will. However it will be of the same 
(low) quality, which I think is not ready to replace the current code.

Did anyone actually tried it to see the speed improvement? I have done 
no benchmarks but what I described in an earlier email (how to plot big 
SRTM files) now works *much* faster (*10 or more speed improvement).

For now I will be happy to see an entry in the TODO file about 
optimizing the highly innefficient "matrix" parser. In the future, if 
you haven't found the time to fix it, perhaps I will submit a proper 
patch, compliant to the coding standards and good enough for you to use it.

Thanks,
Dimitris

Re: gnuplot slow on BIG files

[Daniel J S. <dan...@ie...>]

Dimitrios Apostolou wrote:

> For now I will be happy to see an entry in the TODO file about 
> optimizing the highly innefficient "matrix" parser. In the future, if 
> you haven't found the time to fix it, perhaps I will submit a proper 
> patch, compliant to the coding standards and good enough for you to use it.

You are familiar with the SourceForge "patch" site, aren't you?  What you are 
proposing is a slightly bigger fix and hence requires some consideration.  Often 
smaller patches submitted through discussion can go to CVS as a no-brainer. 
However, in the case of larger patches, SourceForge is nice for maintaining 
gradual development, both for other developers and yourself.  You can come back 
to it whenever you have the free time.  (Consider creating a SourceForge account.)

Dan

Re: gnuplot slow on BIG files

[Robert H. <en...@no...>]

On Wed, 2005-06-08 at 15:31 +0300, Dimitrios Apostolou wrote:

> Did anyone actually tried it to see the speed improvement? I have done 
> no benchmarks but what I described in an earlier email (how to plot big 
> SRTM files) now works *much* faster (*10 or more speed improvement).

I've had a look into this using n x n ascii matrix (generated from
perl's sin() function)

Existing code: 66 seconds

after adding:

#define NO_FORTRAN_NUMS 

to top of datafile.c: 5.9 seconds.

Rationale:

I used callgrind (and kcachegrind) to profile gnuplot whilst loading
this datafile. Turned out >95% of time was sping in the sscanf on
datafile.c:759

Questions: Where did this "NO_FORTRAN_NUMS" option come from, and why
isn't it enabled? Is Fortran number support useful? Could it be provided
in an alternative way (perhaps a run time option)?


Rob

Re: gnuplot slow on BIG files

[Ethan M. <merritt@u.washington.edu>]

On Wednesday 08 June 2005 01:50 pm, Robert Hart wrote:
> I used callgrind (and kcachegrind) to profile gnuplot whilst loading
> this datafile. Turned out >95% of time was sping in the sscanf on
> datafile.c:759

Thank you for taking the time to pin this down.


> Questions: Where did this "NO_FORTRAN_NUMS" option come from, and why
> isn't it enabled? Is Fortran number support useful? Could it be provided
> in an alternative way (perhaps a run time option)?

Excellent questions.

Here's a brief excerpt from a Fortran manual:
  A double precision constant has the same form as a scaled real constant 
  except that the E is replaced by D. Examples:
    6.1D2	is equivalent to 610.0
    +2.3D3	is equivalent to 2300.0
    -3.5D-1	is equivalent to -0.35
    +4D4	is equivalent to 40000

I have no idea how common this might be in real life data files
fed to gnuplot.

If you are willing, could you run one more check?
This entire section of code, with or without NO_FORTRAN_NUMS, is
inside a larger block which starts with the comment:

#ifdef OSK
            /* apparently %n does not work. This implementation
             * is just as good as the non-OSK one, but close
             * to a release (at last) we make it os-9 specific
             */
            int count;
            char *p = strpbrk(s, "dqDQ");
            if (p != NULL)
                *p = 'e';

            count = sscanf(s, "%lf", &df_column[df_no_cols].datum);
#else
   [Previously analysed code is here in the #else]

The question is whether this comment, which dates back at least to 1999,
is in fact correct.  Could you please compare the previous benchmarks
to the case where the code is prefixed by:   #define OSK 1   


I know this will break the checks for separators other than whitespace,
but we can sort that out afterwards. It would be nice to get a speed-up
of 10X by deleting 80+ lines of obfuscated code :-)




-- 
Ethan A Merritt       merritt@u.washington.edu
Biomolecular Structure Center
Mailstop 357742
University of Washington, Seattle, WA 98195

Re: gnuplot slow on BIG files

[Robert H. <en...@no...>]

On Wed, 8 Jun 2005, Ethan Merritt wrote:

> If you are willing, could you run one more check?
> This entire section of code, with or without NO_FORTRAN_NUMS, is
> inside a larger block which starts with the comment:
> 
> #ifdef OSK
>             /* apparently %n does not work. This implementation
>              * is just as good as the non-OSK one, but close
>              * to a release (at last) we make it os-9 specific
>              */
>             int count;
>             char *p = strpbrk(s, "dqDQ");
>             if (p != NULL)
>                 *p = 'e';
> 
>             count = sscanf(s, "%lf", &df_column[df_no_cols].datum);
> #else
>    [Previously analysed code is here in the #else]
> 
> The question is whether this comment, which dates back at least to 1999,
> is in fact correct.  Could you please compare the previous benchmarks
> to the case where the code is prefixed by:   #define OSK 1   

This is much much worse. (Taking nearly 5 minutes on my benchmark)

Here's why:

In the standard code path, sscanf is used to get the next float out of the
input. Then, if the *NEXT CHARACTER* is a d, D, q, or Q, that character is
replaced with an "e" and the sscanf is repeated.

In the NO_FORTRANS_NUMS code path, atof is used to get the next
float. This is much faster than sscanf.

In the OSK code path, strpbrk is used to scan *THE ENTIRE INPUT LINE* for
the first occurence of d, D, q, or Q. This is *REPEATED* for every value
on the line. sscanf is used to read the values which is slow.

Rob








This message has been checked for viruses but the contents of an attachment
may still contain software viruses, which could damage your computer system:
you are advised to perform your own checks. Email communications with the
University of Nottingham may be monitored as permitted by UK legislation.

Re: gnuplot slow on BIG files

[<br...@ph...>]

Robert Hart wrote:

> Questions: Where did this "NO_FORTRAN_NUMS" option come from, 

 From discussions a while back, which I now only dimly remember.
The Fortran numbers parsing was added because people insisted they 
needed it.  But it has a potential conflict with certain date/time
formats (those that have "Dec" immediately following a number, which was 
being overwritten to "eec" by the attempt to parse it as a Fortran 
number), and it makes things slow.

> and why isn't it enabled? 

Because that would break reading of datafiles that people care about.

 > Is Fortran number support useful?

Yes.

> Could it be provided in an alternative way (perhaps a run time
> option)?

Probably.

Re: gnuplot slow on BIG files

[Petr M. <mi...@ph...>]

> Do what you think is better: optimize the current parser, rewrite a new
> one, or use mine as a base for improvement. One thing I know for sure:
> it shouldn't stay as it is.
>
> The patch I published is against the file:
> gnuplot-4.0.0/src/datafile.c

Can you please update it for the current datafile.c from cvs on sourceforge?
(There are minor rejects.)

---
PM

Re: gnuplot slow on BIG files

[Dimitrios A. <ji...@gm...>]

Petr Mikulik wrote:
> Can you please update it for the current datafile.c from cvs on sourceforge?
> (There are minor rejects.)

Here you go. Warning: I haven't checked if it compiles and it still 
doesn't replace the old code. However, I think that the functions 
df_read_matrix_jimis() replaces are
df_read_matrix(), df_tokenise(), df_gets().

The reason I use fgetc() instead of fgets() is that one line might be 
very large to store it all at once. Speed shouldn't be reduced at all 
since stdio does buffered I/O.

Moreover, gnuplot is really memory inefficient IMHO. To read 36M numbers 
for example it needs:

(36M * sizeof(float)) + (36M * 3 * sizeof(float))

The first parenthesis is freed after creating the second one but the 
fact is that all this space is needed at copying time. Wouldn't it be 
interesting if we stored numbers from the file directly to gnuplot's 
format (second parenthesis)? That would result in 25% less memory 
requirements.

Finally, if you like the idea of rewriting the parser (me or somebody 
else), the matrix format has to be clarified. Which lines are considered 
comments: the ones starting immediately with "#"? are empty lines 
allowed? Fortran numbers? What is a valid delimiter (all whitespace)?

Thanks,
Dimitris

Re: gnuplot slow on BIG files

[Petr M. <mi...@ph...>]

> Moreover, gnuplot is really memory inefficient IMHO. To read 36M numbers
> for example it needs:
>
> (36M * sizeof(float)) + (36M * 3 * sizeof(float))

I have started a similar discussion some time ago, when I've concluded that
gnuplot needs 132 B per each point in 3D (data read + drawing cache in
terminal for X11 or Windows) ... it was for "set pm3d; splot ...".

Having maps organized in a different fashion would be definitely nice, but
too much code relies on the current model.

---
PM

Re: gnuplot slow on BIG files

[Dimitrios A. <ji...@gm...>]

Here is the unified patch. In the previous patch I also diffed the files 
in the wrong order.

Dimitris

Re: gnuplot slow on BIG files

[Ethan M. <merritt@u.washington.edu>]

I have not been following this thread, so let me ask a few questions.

You say there is a speed-up of 10X.  
What were the actual times as reported by the "time" command?
Could you provide a profile analysis of the run time, 
so we could see where your CPU usage is being spent?

Speeding things up from 2 seconds to 0.2 seconds, for example,
is not very important. Or let me say that differently --
2 seconds to read a file may be prohibitively slow if you are
trying to use the mouse for interactive rotation, because currently
the file is re-read at each mouse increment.  But the proper fix
for this, IMHO, is not to fiddle with the file reading code. 
Instead we should modify the replot command so that it re-uses
the data previously read in if at all possible.


On Thursday 02 June 2005 03:53 pm, Dimitrios Apostolou wrote:
> 
> So I submit to you a patch (against the v. 4.0 gnuplot) for the file 
> src/datafile.c as a proof of concept

Could you please re-do the patch using
	diff -ur  <oldfile> <newfile>

> There are many things in my code that you 'll not probably like.

Your comments worry me.  For instance:

< 	/* malloc the maximum we may use, it's ok in an overcommiting OS like linux */
< 	tmp_arr = gp_alloc(filesize * sizeof(float), "df_matrix");

gnuplot core code must run on systems other than linux.
And even for linux your statement is not true.  Many people doing
serious number crunching will not run linux in overcommit mode, because
it is too painful to see a computation which has already run for 3 days
be killed by the OOM killer just because someone has opened a web browser,
or in this case because they try to run gnuplot.



-- 
Ethan A Merritt       merritt@u.washington.edu
Biomolecular Structure Center
Mailstop 357742
University of Washington, Seattle, WA 98195

Re: gnuplot slow on BIG files

[Dimitrios A. <ji...@gm...>]

> On Thursday 02 June 2005 03:53 pm, Dimitrios Apostolou wrote:
>=20
>>So I submit to you a patch (against the v. 4.0 gnuplot) for the file=20
>>src/datafile.c as a proof of concept
>=20
>=20
> Could you please re-do the patch using
> 	diff -ur  <oldfile> <newfile>
>=20

I submit the patch for a third time, sorry for the spamming, but since=20
I=C2=B4m not subscribed to the list my emails wait for approval.

This time I used the format you suggested:
diff -ur  <oldfile> <newfile>

>>There are many things in my code that you 'll not probably like.
>=20
>=20
> Your comments worry me.  For instance:
>=20
> < 	/* malloc the maximum we may use, it's ok in an overcommiting OS lik=
e linux */
> < 	tmp_arr =3D gp_alloc(filesize * sizeof(float), "df_matrix");
>=20
> gnuplot core code must run on systems other than linux.
> And even for linux your statement is not true.  Many people doing
> serious number crunching will not run linux in overcommit mode, because
> it is too painful to see a computation which has already run for 3 days
> be killed by the OOM killer just because someone has opened a web brows=
er,
> or in this case because they try to run gnuplot.

As I said my code only serves as a proof of concept. In case you care to=20
use it this is one of the things that should change.

Dimitris