Re: gnuplot slow on BIG files

[Daniel J S. <dan...@ie...>]

I've looked at the current code a bit.  It's quite involved, isn't it?  F=
or=20
Dimitris' sake, there seems to be some details he may have not accounted =
for and=20
would be difficult to add to the patch.  So, would it pay for him to cont=
inue=20
with the patch he's supplied?  Or does it look like the current code will=
 have=20
to be the starting point?

Let me point out one thing here.  With the introduction of binary support=
, I=20
seem to have rewritten df_read_matrix() so that it stores the *whole* set=
 of=20
data as floats.  In that way, some other features could be reused.  Now, =
I don't=20
believe this is a problem as far as time.  Neither should it be a memory =
problem=20
(something that would really slow down Dimitris' machine perahps).  The b=
lock of=20
memory it consumes will not be much bigger than is eventually stored in m=
emory.

But, what could be a problem--in light of the discussion about FORTRAN=20
doubles--is that instead of "floats" maybe this routine should be storing=
=20
doubles.  Certainly a plotting program doesn't need the resolution of dou=
bles.=20
However, perhaps the data that is entered simply needs to have a very, ve=
ry=20
large magnitude exponent and that is why people requested it.  (Of course=
, there=20
are ways around that, but...)

So think that over folks.  Should it be

static double *
df_read_matrix(int *rows, int *cols)
{
<snip>
     double *linearized_matrix =3D NULL;

etc.?  Because, if someone uses doubles and, infact, has numbers outside =
the=20
dynamic range of a "float", it's a problem.  Do doubles make sense furthe=
r down=20
the line after df_read_matrix?


(more below)

Hans-Bernhard Br=F6ker wrote:

>> There is no error checking in the current code either, unless you
>> mean the count returned by sscanf.=20
>=20
>=20
> That's exactly the error checking I'm talking about.  Without it,=20
> handling missing or malformed data would be impossible.

Unless one writes a more sophisticated parser.  Given how much testing is=
=20
already done here, that would almost be as good an option.  (But perhaps =
not=20
preferable.)

>=20
>  > For true error checking we would
>=20
>> need strtod, unless I've overlooked some entirely different method.
>=20
>=20
> There's not a lot you could do with strtod() that sscanf() couldn't do=20
> just as well.
>=20
>>> sscanf() and its %n format are used for a reason.
>=20
>=20
>> But %n is non-portable, as documented by the OSK comment=20
>=20
>=20
> Just because some silly implementation is buggy doesn't exactly mean %n=
=20
> is unportable.  %n is as portable as you can ever hope to be: it's an=20
> ANSI requirement.
>=20
>> I'm also curious about that convoluted series of tests added by
>> Corey Satten and labelled "optimization".  I'll ask him if he
>> remembers what sort of problem case or test suite he was using.
>=20
>=20
> I dimly remember this being about not running sscanf() on all columns o=
f=20
> a multi-column data file if no extended using specs are in use.  If you
> have "using 1:2:3", you can simply ignore columns 4 to 1000.  If you=20
> have "using 1:2:(column(some_function($3)), datafile.c has to convert=20
> all 1000 of them.

I don't know how deep into those tests is the normal execution every time=
, but=20
too far and it's inefficient.

There are three tests here that can be done *before* reading data and com=
bined=20
into a single test within tokenise:

(fast_columns =3D=3D 0)
(df_no_use_specs =3D=3D 0)
df_no_use_specs > 5

and somehow my intuition tells me the following could be done in a better=
 way:

		|| ((df_no_use_specs > 0)
		    && (use_spec[0].column =3D=3D dfncp1
			|| (df_no_use_specs > 1
			    && (use_spec[1].column =3D=3D dfncp1
				|| (df_no_use_specs > 2
				    && (use_spec[2].column =3D=3D dfncp1
					|| (df_no_use_specs > 3
					    && (use_spec[3].column =3D=3D dfncp1
						|| (df_no_use_specs > 4 && (use_spec[4].column =3D=3D dfncp1 || )

Going from what Hans said, is there some logic that could be done beforeh=
and to=20
set up a more efficient chunk of code?

NEXT ISSUE:

There is this chunk of code:

	    if (count =3D=3D 1 &&
		(s[used] =3D=3D 'd' || s[used] =3D=3D 'D' ||
		 s[used] =3D=3D 'q' || s[used] =3D=3D 'Q')) {
		/* HBB 20001221: avoid breaking parsing of time/date
		 * strings like 01Dec2000 that would be caused by
		 * overwriting the 'D' with an 'e'... */
		char save_char =3D s[used];

		/* might be fortran double */
		s[used] =3D 'e';
		/* and try again */
		count =3D sscanf(s, "%lf", &df_column[df_no_cols].datum);
		s[used] =3D save_char;
	    }

Isn't this a bug?  If so, why not?  Certainly the date string 01Dec2000 s=
hould=20
not be interpretted as a double.  But doesn't a value of 1.0 result from=20
scanning 01eec2000?

I would think that one would have to test for a "+-0123456789" before and=
 after=20
a "dDqQ".  Then one could feel free to change that character to an 'e' an=
d not=20
worry about it.

Now, I see there are some strings to contend with.  Aside from that, what=
 if one=20
initially goes through the string, using string searching routines to loo=
k for=20
"dDqQ" and convert them.  (Need a simple method to make sure they are not=
 within=20
quotes, so one would probably have to include the double quote character =
in the=20
search somehow.)  Would it then be possible to use a much simpler routine=
?  Or=20
are we still left with using sscanf(), and that is the slow thing here?

Ethan, if

  	set datafile {no}fortran_floats

is implemented, perhaps a simple test for #{dDqQ}# or #{eE}# on the first=
 number=20
in the file would be useful for testing conflicts and giving a warning me=
ssage.=20
  (I'm assuming it isn't valid to mix FORTRAN and C format in one file.)

Dan