Re: gnuplot slow on BIG files

[<br...@ph...>]

Daniel J Sebald wrote:
> I've looked at the current code a bit.  It's quite involved, isn't it?

Yes.  But then, so is the variety of inputs gnuplot can handle.

> For Dimitris' sake, there seems to be some details he may have not 
> accounted for and would be difficult to add to the patch.  So, would it 
> pay for him to continue with the patch he's supplied?  Or does it look 
> like the current code will have to be the starting point?

The latter.  Its main value (and that of Robert's later works) is to 
prove that it might be possible to do it faster.  Next, we have to find out

a) if the faster code doesn't break any existing behaviour
b) if it does, if and how it's possible to have both the old code's
    flexibility and the new one's speed.

> Neither should it be a memory problem (something that would really slow 
> down Dimitris' machine perahps).  The block of memory it consumes will 
> not be much bigger than is eventually stored in memory.

For the case of plot "datafile" matrix every 500:500, it might...

> But, what could be a problem--in light of the discussion about FORTRAN 
> doubles--is that instead of "floats" maybe this routine should be 
> storing doubles.  

Yes, it should.  More to the point, it should store in type coordval.
That's what it's defined for.

 > Certainly a plotting program doesn't need the
> resolution of doubles. 

Oh, but it does --- if only because some people want to be able to zoom 
in somewhat deeply.  1/DBL_FLT is less than the number of pixels on the 
screen, and only a zoom factor of 1000 away from introducing aliasing 
artefacts.

> Unless one writes a more sophisticated parser.  Given how much testing 
> is already done here, that would almost be as good an option.  (But 
> perhaps not preferable.)

A parser wouldn't do --- we'ld need a dynamic parser *generator*, i.e. 
an engine that, given a set of 'using', 'every', 'index' and whatnot, 
will generate a parser on-the-fly and run that on the code.

> There are three tests here that can be done *before* reading data 

They are.

 > and combined into a single test within tokenise:

Not sure that that would achieve enough to be worth doing.

> (fast_columns == 0)

This one is essentially the Corey Satten optimization.

> (df_no_use_specs == 0)

> df_no_use_specs > 5

> and somehow my intuition tells me the following could be done in a 
> better way:
> 
>         || ((df_no_use_specs > 0)
>             && (use_spec[0].column == dfncp1
>             || (df_no_use_specs > 1
>                 && (use_spec[1].column == dfncp1
>                 || (df_no_use_specs > 2
>                     && (use_spec[2].column == dfncp1
>                     || (df_no_use_specs > 3
>                         && (use_spec[3].column == dfncp1
>                         || (df_no_use_specs > 4 && (use_spec[4].column 
> == dfncp1 || )

Not really.  This is a test for "the first <n> using specifiers are just 
1..<n>, with (<n> <= 5)".  I don't think there's much of a way of 
expressing that any more efficiently.  It might make sense to move this
test to a different place, though.

> NEXT ISSUE:
[...]
>         /* HBB 20001221: avoid breaking parsing of time/date
>          * strings like 01Dec2000 that would be caused by
>          * overwriting the 'D' with an 'e'... */

> Isn't this a bug?  

Not to my knowledge, no.

 > If so, why not?

Why should it be one?

 > Certainly the date string 01Dec2000
> should not be interpretted as a double.  

No.  But it has to survive the attempted interpretation as a double,
unchanged.  I.e. at this point of execution, the parsing engine simply 
doesn't know that 01Dec2000 isn't supposed to be a floating-point number.

 > But doesn't a value of 1.0
> result from scanning 01eec2000?

The FP number resulting from this is irrelevant --- if the user actually 
asked for this to be read as a time/date string, it'll be overwritten 
later by the FP number representing 01Dec2000 in gnuplot's internal time 
format.  If the user didn't specify time/date format for this column, he 
deserves whatever nonsense he may get.

> Now, I see there are some strings to contend with.  Aside from that, 
> what if one initially goes through the string, using string searching 
> routines to look for "dDqQ" and convert them.  

No.  This would kill both the scanf() format string allowed to be passed 
in, if that contained some random 'q' characters, and, perhaps more 
importantly, Ethan's datastrings stuff.