gnuplot-beta

Re: gnuplot slow on BIG files

[Ethan M. <merritt@u.washington.edu>]

On Wednesday 08 June 2005 03:53 pm, Robert Hart wrote:
> > The question is whether this comment, which dates back at least to 1999,
> > is in fact correct.  Could you please compare the previous benchmarks
> > to the case where the code is prefixed by:   #define OSK 1   
> 
> This is much much worse. (Taking nearly 5 minutes on my benchmark)

OK. No surprise, but I thought it worth testing.

I think I'll remove that OSK chunk altogether, and make the NO_FORTRAN_NUMS
a run-time option.  Probably
	set datafile {no}fortran_floats

> In the standard code path, sscanf is used to get the next float out of the
> input. Then, if the *NEXT CHARACTER* is a d, D, q, or Q, that character is
> replaced with an "e" and the sscanf is repeated.
> 
> In the NO_FORTRANS_NUMS code path, atof is used to get the next
> float. This is much faster than sscanf.

Are you saying that the speed gain could be achieved just by
replacing sscanf with atof in the standard code path, even though it still
takes the time to check for d/D/q/Q and rescan if found?

In that case, maybe we don't even need the run-time option.

	thanks,

		Ethan

-- 
Ethan A Merritt       merritt@u.washington.edu
Biomolecular Structure Center
Mailstop 357742
University of Washington, Seattle, WA 98195

Re: gnuplot slow on BIG files

[Ethan M. <merritt@u.washington.edu>]

On Thursday 09 June 2005 12:26 am, Hans-Bernhard Br=F6ker wrote:
> Ethan Merritt wrote:
>=20
> There's not a lot you could do with strtod() that sscanf() couldn't do=20
> just as well.

But slower, apparently.

> >>sscanf() and its %n format are used for a reason.
>=20
> > But %n is non-portable, as documented by the OSK comment=20
>=20
> Just because some silly implementation is buggy doesn't exactly mean %n=20
> is unportable.  %n is as portable as you can ever hope to be: it's an=20
> ANSI requirement.

Surely you know the quote about standards: "Standards are nice because
there are so many of them...".  =20
It turns out that different ANSI revisions describe
the behavior of %n differently, so it is not portable.  Here is an=20
excerpt from the current man page:
  "Probably  it  is  wise not  to  make any assumptions on the
   effect of %n conversions on the return value."

The current code assumes that=20
	count =3D sscanf(s, "%lf%n", ...)
will set count to 1 if successful, but on some systems it will instead
set count to 2.  I do not know if this is the specific problem with OSK.

strtod() is more portable than %n, and allows more complete=20
error reporting than sscanf().
=20
> I dimly remember this being about not running sscanf() on all columns of=
=20
> a multi-column data file if no extended using specs are in use.  If you
> have "using 1:2:3", you can simply ignore columns 4 to 1000.  If you=20
> have "using 1:2:(column(some_function($3)), datafile.c has to convert=20
> all 1000 of them.

That is an interesting point, and I now wonder if these optimization lines
conflict with the new column-based syntax elements like xticlabels().
I will have to experiment with this.


=2D-=20
Ethan A Merritt       merritt@u.washington.edu
Biomolecular Structure Center
Mailstop 357742
University of Washington, Seattle, WA 98195

Re: gnuplot slow on BIG files

[<br...@ph...>]

Ethan Merritt wrote:

> It turns out that different ANSI revisions describe
> the behavior of %n differently, so it is not portable.  

Even the (Linux) libc man page isn't sure of itself in this regars. 
They only say the TC "seems to say otherwise".  I think they're 
misguided.  The description of %n in C90 is very easy to misinterpret. 
I'm reasonably sure the update they mention was only meant to clarify, 
not change the standardized behaviour.  The tricky bit is that %n is not 
a "conversion", so it's not supposed to affect the return value.

 > Here is an
> excerpt from the current man page:
>   "Probably  it  is  wise not  to  make any assumptions on the
>    effect of %n conversions on the return value."

That comment doesn't necessarily concern the case of gnuplot.  We don't 
have to know whether our sscanf("%lf%n",...) returned 1 or 2, which is 
all this is talking about.  We only need to know if it's zero (because 
the %lf failed), EOF (empty input, shouldn't happen) or anything else 
(so the %lf succeeded).

> The current code assumes that 
> 	count = sscanf(s, "%lf%n", ...)
> will set count to 1 if successful, but on some systems it will instead
> set count to 2.  I do not know if this is the specific problem with OSK.

No. I dimly recall the problem with OS-9 (which #defines OSK) was that 
%n simply didn't work at all.  I.e. it didn't put anything into the 
'used' variable.

> strtod() is more portable than %n, 

Unless we have somebody verify to us that strtod() implements the same 
ANSI standard more correctly than sscanf(), on OS-9, we don't know that.

 > and allows more complete
> error reporting than sscanf().

No, it doesn't.  It tells us nothing that sscanf()'s return value, 
combined with the output of %n, wouldn't.

Re: gnuplot slow on BIG files

[Ethan M. <merritt@u.washington.edu>]

On Thursday 09 June 2005 07:34 pm, Hans-Bernhard Br=F6ker wrote:
> > strtod() is more portable than %n,
>
> Unless we have somebody verify to us that strtod() implements the same
> ANSI standard more correctly than sscanf(), on OS-9, we don't know that.

?? The point is that using strtod we don't *need* %n.
Where does "more correctly" come into it?

>  > and allows more complete
> > error reporting than sscanf().
>
> No, it doesn't.  It tells us nothing that sscanf()'s return value,
> combined with the output of %n, wouldn't.

strtod sets errno;  scanf does not.
Not that it's a big deal, since the code currently has no provision
for reported over- or under- flow.

=2D-=20
Ethan A Merritt
Biomolecular Structure Center
University of Washington 98195-7742

Re: gnuplot slow on BIG files

[Hans-Bernhard B. <br...@ph...>]

Ethan Merritt wrote:
> On Thursday 09 June 2005 07:34 pm, Hans-Bernhard Br=F6ker wrote:
>=20
>>>strtod() is more portable than %n,
>>
>>Unless we have somebody verify to us that strtod() implements the same
>>ANSI standard more correctly than sscanf(), on OS-9, we don't know that=
=2E
>=20
>=20
> ?? The point is that using strtod we don't *need* %n.
> Where does "more correctly" come into it?

We have two library functions doing essentially the same thing, both=20
specified precisely enough by C Standards for 15 years now.  We know of=20
one OS that got one of them wrong, at one point in time at least.  How=20
do we know there isn't also a platform that gets the other one just as=20
wrong? The answer is, we don't.

> Not that it's a big deal, since the code currently has no provision
> for reported over- or under- flow.

It has (weak) provision for inf and NaN, though, elsewhere.

Re: gnuplot slow on BIG files

[Robert H. <en...@no...>]

On Sat, 11 Jun 2005, Hans-Bernhard Broeker wrote:

> We have two library functions doing essentially the same thing, both 
> specified precisely enough by C Standards for 15 years now.  We know of 
> one OS that got one of them wrong, at one point in time at least.  How 
> do we know there isn't also a platform that gets the other one just as 
> wrong? The answer is, we don't.

You seem to be implying that we shouldn't be making any kinds of changes
to the core of gnuplot without being sure they will work on all of the
platforms that gnuplot has *historically* supported.

The fact is:

a) we've checked this on all the platforms we have access to.

b) we've attempted to contact people knowledgable in the known
"problem" platform.

c) anybody who later has difficulties compiling/running the new version,
will have access to comments in the source explaining the situation,
archives of the mailing list, and the CVS history of gnuplot itself.

So, I think unless somebody can show that either:

a) A significant number of people are running gnuplot on platforms we
aren't ourselves using,

or

b) strtod has known bugs in *current* implementations.

there's not a lot more we can do.

FYI: Known strtod bugs, I can find:

http://www.blackdown.org/cgi-bin/jdk/known-bugs?id=1001;page=4;user=guest

Bug in glibc 2.0.7: returned "0.0" on an input of "-0.0". I doubt this
would cause an issue for gnuplot anyway. Fixed in more recent versions.

http://savannah.nongnu.org/bugs/?func=detailitem&item_id=2924

bug in avr-libc: endptr is not set correctly. Fixed in current versions,
obscure platform anyway. Not sure gnuplot would be appropriate for it.

http://www.hmug.org/man/3/strtod.php

On mac OSX, "NaN" is not recognised.

http://www.opensource.apple.com/darwinsource/10.3.1/tcl-14/tcl/unix/tcl.m4

Under Solaris 2.4, strtod returns the wrong value for the
terminating character under some conditions.  Check for this
and if the problem exists use a substitute procedure
"fixstrtod" (provided by Tcl) that corrects the error.
Also, on Compaq's Tru64 Unix 5.0,
strtod(" ") returns 0.0 instead of a failure to convert.

int main() {
    char *infString="Inf", *nanString="NaN", *spaceString=" ";
    char *term;
    double value;
    value = strtod(infString, &term);
    if ((term != infString) && (term[-1] == 0)) {
	exit(1);
    }
    value = strtod(nanString, &term);A
    if ((term != nanString) && (term[-1] == 0)) {
	exit(1);
    }
    value = strtod(spaceString, &term);
    if (term == (spaceString+1)) {
	exit(1);
    }
    exit(0);
}

Re: gnuplot slow on BIG files

[Ethan M. <merritt@u.washington.edu>]

On Saturday 11 June 2005 03:53 am, Robert Hart wrote:
> You seem to be implying that we shouldn't be making any kinds of changes
> to the core of gnuplot without being sure they will work on all of the
> platforms that gnuplot has *historically* supported.

We dropped support of 16-bit DOS when we released version 4.0.
And that's an OS that everyone has heard of, and many of us probably
still have distro floppies for it gathering dust in a file drawer.
Furthermore, we knew how to fix the build problems but chose not to.

I have no problem with that; I think dropping it was the right thing
to do.  But then why go even one step out of our way to avoid maybe,
possibly, triggering a bug on a totally obscure OS that no-one has
heard of?  It's absurd!  If OSK or some other relic of an OS has a bug
in a system library, let someone who cares about that OS fix it.

Robert hit the nail on the head when he quoted the recent article
on "the burden of (extreme) minority architectures on open source
software".  I regret even wasting the time it took to track down
the fact that this bug probably disappeared when gcc/glibc was ported
to OSK *eleven years ago*! If someone is still using a pre-1994 buggy
libc on OSK, too bad for them.

Let's drop any special-casing for OSK right now.

Let's also issue a call to any Amiga users, and if none turn up
let's remove all the special casing for Amiga as well.

I'd be willing to bet that the Next/OpenStep special-case
PostScript drivers are no longer worth keeping either, in the sense
that they don't support the features added since version 3.7 so
it would be pointless for anyone using them to upgrade to a newer
gnuplot.

I propose that our guiding principle should be:

 Support for obsolete platforms will be frozen at the last
 gnuplot release for which there was evidence of an active user
 community. People running archaic hardware should not be surprised
 if they have to run less than state-of-the-art software on it.

> The fact is:
> a) we've checked this on all the platforms we have access to.

Err. That's not quite true.  I checked half a dozen platforms,
but I could double or triple that if it were strictly necessary.
And I did so before the 4.0 release, at least to the extent of
building from source and running "make check".

> FYI: Known strtod bugs, I can find:
>
> On mac OSX, "NaN" is not recognised.

We test for it elsewhere, so I don't think this would affect us.
But that is clearly an OSX bug.  Let them fix it.

> Under Solaris 2.4, strtod returns the wrong value for the
> terminating character under some conditions.

At worst that would fail to handle the Fortran D/Q format
case.  This also is clearly an OS bug, and not something we
should have to code around.

> Also, on Compaq's Tru64 Unix 5.0,
> strtod(" ") returns 0.0 instead of a failure to convert.

That could, indeed, have been a problem. On the other hand, the 
existing non-scanf code paths in datafile.c strip away leading whitespace
so it would not in fact have bitten us.  Anyhow, it seems to have been
fixed, or at least my lab machines running Tru64 do not behave that way.

-- 
Ethan A Merritt
Biomolecular Structure Center
University of Washington 98195-7742

Re: gnuplot slow on BIG files

[Hans-Bernhard B. <br...@ph...>]

Ethan Merritt wrote:
> On Saturday 11 June 2005 03:53 am, Robert Hart wrote:

>>You seem to be implying that we shouldn't be making any kinds of changes
>>to the core of gnuplot without being sure they will work on all of the
>>platforms that gnuplot has *historically* supported.

> We dropped support of 16-bit DOS when we released version 4.0.

Out of nothing but pure necessity combined with lack of user interest.
Since then, I've made progress in reviving it.  It takes some drastic 
measures to get there, though: not only will the postscript text blocks 
have to be moved to separate files, but I'd actually have to split 
graphics.c in half, because that module in itself is too large to be 
compiled on 16-bit platforms already.

> Furthermore, we knew how to fix the build problems but chose not to.

We knew how to fix some of them, not all.

> Let's also issue a call to any Amiga users, and if none turn up
> let's remove all the special casing for Amiga as well.

Careful what you ask for, there --- the chief proponent of Amiga has 
been Lars. ;-)

>  Support for obsolete platforms will be frozen at the last
>  gnuplot release for which there was evidence of an active user
>  community. People running archaic hardware should not be surprised
>  if they have to run less than state-of-the-art software on it.

"evidence of an active user community" would be a *very* strict 
criterion.  By that count, we'ld not be supporting anything but Linux, 
Windows and maybe MacOS X.  (Sorry, Petr, but one person doesn't make a 
community for OS/2).

> At worst that would fail to handle the Fortran D/Q format
> case.  This also is clearly an OS bug, and not something we
> should have to code around.

Actually, it's *exactly* the OS bugs we have to code around.  All other 
bugs can reasonably be avoided by asking people to upgrade some tool, 
library or whatnot, or maybe by replacing buggy OS-supplied tools by 
free replacements.  But neither we nor the typical user is in a position 
to upgrade an OS just to be able to run gnuplot on it.

Re: gnuplot slow on BIG files

[Petr M. <mi...@ph...>]

> (Sorry, Petr, but one person doesn't make a community for OS/2).

Because the gnuplot for OS/2 is working well :-) There are 3 active people.
Actually, I have some fresh patches to apply.

---
PM

Re: gnuplot slow on BIG files

[Ethan M. <merritt@u.washington.edu>]

On Monday 13 June 2005 01:14 am, Hans-Bernhard Broeker wrote:
> Ethan Merritt wrote:
> 
> Actually, it's *exactly* the OS bugs we have to code around.  All other 
> bugs can reasonably be avoided by asking people to upgrade some tool, 
> library or whatnot, or maybe by replacing buggy OS-supplied tools by 
> free replacements. 

There you have just described exactly the situation for OSK.
There was apparently a buggy, proprietary clib that gnuplot was
coded to work around. Since 1994, OSK has had the option of using
gcc/glibc intead.  So why should we continue to carry around a
work-around hack for a library that hasn't been needed since 1994?

> But neither we nor the typical user is in a position  
> to upgrade an OS just to be able to run gnuplot on it.

Several of the odd-ball machines I have in the lab were purchased
solely in order to support the software requirments of a particular
application or set of applications.  The world has changed; application
program requirements drive both hardware and OS purchasing decisions.

-- 
Ethan A Merritt       merritt@u.washington.edu
Biomolecular Structure Center
Mailstop 357742
University of Washington, Seattle, WA 98195

Re: gnuplot slow on BIG files

[Ethan M. <merritt@u.washington.edu>]

On Monday 13 June 2005 01:14 am, Hans-Bernhard Broeker wrote:
> Ethan Merritt wrote:
> 
> > Let's also issue a call to any Amiga users, and if none turn up
> > let's remove all the special casing for Amiga as well.
> 
> Careful what you ask for, there --- the chief proponent of Amiga has 
> been Lars. ;-)

Google does not seem to find any evidence for an Amiga build 
newer than version 3.7.1 and most of the "AMINET" mirrors are no
longer responding.

Lars - was version 4.0 ever built and tested on Amiga?


-- 
Ethan A Merritt       merritt@u.washington.edu
Biomolecular Structure Center
Mailstop 357742
University of Washington, Seattle, WA 98195

Re: gnuplot slow on BIG files

[Lars H. <lhe...@us...>]

 
> > Careful what you ask for, there --- the chief proponent of Amiga has 
> > been Lars. ;-)
> 
> Google does not seem to find any evidence for an Amiga build 
> newer than version 3.7.1 and most of the "AMINET" mirrors are no
> longer responding.
 
 No, aminet is alive and kicking. But the last version I uploaded is
 3.7.1.

> Lars - was version 4.0 ever built and tested on Amiga?
 
 Not sure, I'd need to check my machine at home.

 Getting the sources over is a tad difficult. My Amiga is not networked,
 and I only have a 33k modem. Hhm, I'll have to check whether we have a
 Sun QIC tape drive here at work, that would help a lot.

 That said, gnuplot 4 will probably build with the Amiga version of gcc.
 I will check it, give me a few days.

Re: gnuplot slow on BIG files

[Robert H. <en...@no...>]

> I propose that our guiding principle should be:
> 
>  Support for obsolete platforms will be frozen at the last
>  gnuplot release for which there was evidence of an active user
>  community. People running archaic hardware should not be surprised
>  if they have to run less than state-of-the-art software on it.
> 

s/evidence of an active user community/a developer/ ?

I think that every additional platform adds a "cost" to the maintenance
and development of gnuplot. For many platforms that cost is low -
perhaps a few tweaks to Makefiles etc., or a platform specific terminal
driver that otherwise doesn't get in anybody's way.

How many of the #ifdef'ed features are "optional" because of lack of
support on various platforms, and how many because they are considered
"experimental"?

egrep -h "#if[n]{0,1}def" `find . -name "*.[ch]"` | sed -e
"s/#ifn*def//" -e "s/\/\*.*//" | sort | uniq -c | sort -n

I imagine there are quite a few cases which do not see any testing...

Rob

-- 
Robert Hart <en...@no...>
University of Nottingham


This message has been checked for viruses but the contents of an attachment
may still contain software viruses, which could damage your computer system:
you are advised to perform your own checks. Email communications with the
University of Nottingham may be monitored as permitted by UK legislation.

Re: gnuplot slow on BIG files

[Robert H. <en...@no...>]

On Wed, 8 Jun 2005, Ethan Merritt wrote:

> Are you saying that the speed gain could be achieved just by
> replacing sscanf with atof in the standard code path, even though it still
> takes the time to check for d/D/q/Q and rescan if found?

Not quite. atof is definitely faster than sscanf, however, atof doesn't
tell you how many characters it actually read, so that is why (I presume)
that sscanf is used instead. However, in both cases, we manually scan for
the next seperator one character at a time, so it wouldn't be hard to
'notice' is we pass a dDqQ on the way.

Another option I haven't tried/benchmarked is to use strtof. If this is as
fast as atof, then we should use it, because it'll give us a pointer to
where it finished.

Anyway, bed time here, but I think any code that can be culled from this
function has got to be a win. 

Good luck


Rob





This message has been checked for viruses but the contents of an attachment
may still contain software viruses, which could damage your computer system:
you are advised to perform your own checks. Email communications with the
University of Nottingham may be monitored as permitted by UK legislation.

Re: gnuplot slow on BIG files

[<br...@ph...>]

Ethan Merritt wrote:
> On Wednesday 08 June 2005 03:53 pm, Robert Hart wrote:

> I think I'll remove that OSK chunk altogether, 

Please don't.  It's quite harmless as it is --- it'll only be used on 
one platform, and you would risk breaking that platform completely by 
doing this.

> and make the NO_FORTRAN_NUMS a run-time option.  Probably
> 	set datafile {no}fortran_floats

OK.

>>In the NO_FORTRANS_NUMS code path, atof is used to get the next
>>float. This is much faster than sscanf.

Is it?  Why would that be?  More to the point, atof has *no* error 
handling capabilities, which I don't believe to be a good idea in this case.

> Are you saying that the speed gain could be achieved just by
> replacing sscanf with atof in the standard code path, 

That wouldn't work at all --- atof leaves you with no way of knowing the 
end of that number, no way to position to the next one, and no way to 
find the 'D' in floating point number.  sscanf() and its %n format are 
used for a reason.

Re: gnuplot slow on BIG files

[Ethan M. <merritt@u.washington.edu>]

On Wednesday 08 June 2005 09:47 pm, HBB wrote:

I assume Robert benchmarked atof because that's what is currently
in the source code as a compile-time alternative.  But for a real
change-over in the default code I think we would need to go with
strtod instead.

> Ethan Merritt wrote:
> >
> > I think I'll remove that OSK chunk altogether,
>
> Please don't.  It's quite harmless as it is --- it'll only be used on
> one platform, and you would risk breaking that platform completely by
> doing this.

I don't think it's likely to break anything. The problem is in the %n 
format, which we can do without by using strtod instead.
But yeah, it's only a few lines of code.

> More to the point, atof has *no* error handling capabilities,
> which I don't believe to be a good idea in this case.

There is no error checking in the current code either, unless you
mean the count returned by sscanf. For true error checking we would
need strtod, unless I've overlooked some entirely different method.

> sscanf() and its %n format are used for a reason.

But %n is non-portable, as documented by the OSK comment and also
by the linux man pages.  So if Robert can benchmark the performance of
strtod as no worse than the current default code, then replacing both
paths with a strtod call may solve all these issues at one stroke.

I'm also curious about that convoluted series of tests added by
Corey Satten and labelled "optimization".  I'll ask him if he
remembers what sort of problem case or test suite he was using.
We have many more plot options now than when this optimization was
done, and it could well be either that it has become pointless or
that it is still benificial but has become incomplete.
Either way I'd like to know what the original rationale was.


-- 
Ethan A Merritt
Biomolecular Structure Center
University of Washington 98195-7742

Re: gnuplot slow on BIG files

[<br...@ph...>]

Ethan Merritt wrote:
> On Wednesday 08 June 2005 09:47 pm, HBB wrote:

>>More to the point, atof has *no* error handling capabilities,
>>which I don't believe to be a good idea in this case.

> There is no error checking in the current code either, unless you
> mean the count returned by sscanf. 

That's exactly the error checking I'm talking about.  Without it, 
handling missing or malformed data would be impossible.

 > For true error checking we would
> need strtod, unless I've overlooked some entirely different method.

There's not a lot you could do with strtod() that sscanf() couldn't do 
just as well.

>>sscanf() and its %n format are used for a reason.

> But %n is non-portable, as documented by the OSK comment 

Just because some silly implementation is buggy doesn't exactly mean %n 
is unportable.  %n is as portable as you can ever hope to be: it's an 
ANSI requirement.

> I'm also curious about that convoluted series of tests added by
> Corey Satten and labelled "optimization".  I'll ask him if he
> remembers what sort of problem case or test suite he was using.

I dimly remember this being about not running sscanf() on all columns of 
a multi-column data file if no extended using specs are in use.  If you
have "using 1:2:3", you can simply ignore columns 4 to 1000.  If you 
have "using 1:2:(column(some_function($3)), datafile.c has to convert 
all 1000 of them.

Re: gnuplot slow on BIG files

[Daniel J S. <dan...@ie...>]

I've looked at the current code a bit.  It's quite involved, isn't it?  F=
or=20
Dimitris' sake, there seems to be some details he may have not accounted =
for and=20
would be difficult to add to the patch.  So, would it pay for him to cont=
inue=20
with the patch he's supplied?  Or does it look like the current code will=
 have=20
to be the starting point?

Let me point out one thing here.  With the introduction of binary support=
, I=20
seem to have rewritten df_read_matrix() so that it stores the *whole* set=
 of=20
data as floats.  In that way, some other features could be reused.  Now, =
I don't=20
believe this is a problem as far as time.  Neither should it be a memory =
problem=20
(something that would really slow down Dimitris' machine perahps).  The b=
lock of=20
memory it consumes will not be much bigger than is eventually stored in m=
emory.

But, what could be a problem--in light of the discussion about FORTRAN=20
doubles--is that instead of "floats" maybe this routine should be storing=
=20
doubles.  Certainly a plotting program doesn't need the resolution of dou=
bles.=20
However, perhaps the data that is entered simply needs to have a very, ve=
ry=20
large magnitude exponent and that is why people requested it.  (Of course=
, there=20
are ways around that, but...)

So think that over folks.  Should it be

static double *
df_read_matrix(int *rows, int *cols)
{
<snip>
     double *linearized_matrix =3D NULL;

etc.?  Because, if someone uses doubles and, infact, has numbers outside =
the=20
dynamic range of a "float", it's a problem.  Do doubles make sense furthe=
r down=20
the line after df_read_matrix?


(more below)

Hans-Bernhard Br=F6ker wrote:

>> There is no error checking in the current code either, unless you
>> mean the count returned by sscanf.=20
>=20
>=20
> That's exactly the error checking I'm talking about.  Without it,=20
> handling missing or malformed data would be impossible.

Unless one writes a more sophisticated parser.  Given how much testing is=
=20
already done here, that would almost be as good an option.  (But perhaps =
not=20
preferable.)

>=20
>  > For true error checking we would
>=20
>> need strtod, unless I've overlooked some entirely different method.
>=20
>=20
> There's not a lot you could do with strtod() that sscanf() couldn't do=20
> just as well.
>=20
>>> sscanf() and its %n format are used for a reason.
>=20
>=20
>> But %n is non-portable, as documented by the OSK comment=20
>=20
>=20
> Just because some silly implementation is buggy doesn't exactly mean %n=
=20
> is unportable.  %n is as portable as you can ever hope to be: it's an=20
> ANSI requirement.
>=20
>> I'm also curious about that convoluted series of tests added by
>> Corey Satten and labelled "optimization".  I'll ask him if he
>> remembers what sort of problem case or test suite he was using.
>=20
>=20
> I dimly remember this being about not running sscanf() on all columns o=
f=20
> a multi-column data file if no extended using specs are in use.  If you
> have "using 1:2:3", you can simply ignore columns 4 to 1000.  If you=20
> have "using 1:2:(column(some_function($3)), datafile.c has to convert=20
> all 1000 of them.

I don't know how deep into those tests is the normal execution every time=
, but=20
too far and it's inefficient.

There are three tests here that can be done *before* reading data and com=
bined=20
into a single test within tokenise:

(fast_columns =3D=3D 0)
(df_no_use_specs =3D=3D 0)
df_no_use_specs > 5

and somehow my intuition tells me the following could be done in a better=
 way:

		|| ((df_no_use_specs > 0)
		    && (use_spec[0].column =3D=3D dfncp1
			|| (df_no_use_specs > 1
			    && (use_spec[1].column =3D=3D dfncp1
				|| (df_no_use_specs > 2
				    && (use_spec[2].column =3D=3D dfncp1
					|| (df_no_use_specs > 3
					    && (use_spec[3].column =3D=3D dfncp1
						|| (df_no_use_specs > 4 && (use_spec[4].column =3D=3D dfncp1 || )

Going from what Hans said, is there some logic that could be done beforeh=
and to=20
set up a more efficient chunk of code?

NEXT ISSUE:

There is this chunk of code:

	    if (count =3D=3D 1 &&
		(s[used] =3D=3D 'd' || s[used] =3D=3D 'D' ||
		 s[used] =3D=3D 'q' || s[used] =3D=3D 'Q')) {
		/* HBB 20001221: avoid breaking parsing of time/date
		 * strings like 01Dec2000 that would be caused by
		 * overwriting the 'D' with an 'e'... */
		char save_char =3D s[used];

		/* might be fortran double */
		s[used] =3D 'e';
		/* and try again */
		count =3D sscanf(s, "%lf", &df_column[df_no_cols].datum);
		s[used] =3D save_char;
	    }

Isn't this a bug?  If so, why not?  Certainly the date string 01Dec2000 s=
hould=20
not be interpretted as a double.  But doesn't a value of 1.0 result from=20
scanning 01eec2000?

I would think that one would have to test for a "+-0123456789" before and=
 after=20
a "dDqQ".  Then one could feel free to change that character to an 'e' an=
d not=20
worry about it.

Now, I see there are some strings to contend with.  Aside from that, what=
 if one=20
initially goes through the string, using string searching routines to loo=
k for=20
"dDqQ" and convert them.  (Need a simple method to make sure they are not=
 within=20
quotes, so one would probably have to include the double quote character =
in the=20
search somehow.)  Would it then be possible to use a much simpler routine=
?  Or=20
are we still left with using sscanf(), and that is the slow thing here?

Ethan, if

  	set datafile {no}fortran_floats

is implemented, perhaps a simple test for #{dDqQ}# or #{eE}# on the first=
 number=20
in the file would be useful for testing conflicts and giving a warning me=
ssage.=20
  (I'm assuming it isn't valid to mix FORTRAN and C format in one file.)

Dan

Re: gnuplot slow on BIG files

[<br...@ph...>]

Daniel J Sebald wrote:
> I've looked at the current code a bit.  It's quite involved, isn't it?

Yes.  But then, so is the variety of inputs gnuplot can handle.

> For Dimitris' sake, there seems to be some details he may have not 
> accounted for and would be difficult to add to the patch.  So, would it 
> pay for him to continue with the patch he's supplied?  Or does it look 
> like the current code will have to be the starting point?

The latter.  Its main value (and that of Robert's later works) is to 
prove that it might be possible to do it faster.  Next, we have to find out

a) if the faster code doesn't break any existing behaviour
b) if it does, if and how it's possible to have both the old code's
    flexibility and the new one's speed.

> Neither should it be a memory problem (something that would really slow 
> down Dimitris' machine perahps).  The block of memory it consumes will 
> not be much bigger than is eventually stored in memory.

For the case of plot "datafile" matrix every 500:500, it might...

> But, what could be a problem--in light of the discussion about FORTRAN 
> doubles--is that instead of "floats" maybe this routine should be 
> storing doubles.  

Yes, it should.  More to the point, it should store in type coordval.
That's what it's defined for.

 > Certainly a plotting program doesn't need the
> resolution of doubles. 

Oh, but it does --- if only because some people want to be able to zoom 
in somewhat deeply.  1/DBL_FLT is less than the number of pixels on the 
screen, and only a zoom factor of 1000 away from introducing aliasing 
artefacts.

> Unless one writes a more sophisticated parser.  Given how much testing 
> is already done here, that would almost be as good an option.  (But 
> perhaps not preferable.)

A parser wouldn't do --- we'ld need a dynamic parser *generator*, i.e. 
an engine that, given a set of 'using', 'every', 'index' and whatnot, 
will generate a parser on-the-fly and run that on the code.

> There are three tests here that can be done *before* reading data 

They are.

 > and combined into a single test within tokenise:

Not sure that that would achieve enough to be worth doing.

> (fast_columns == 0)

This one is essentially the Corey Satten optimization.

> (df_no_use_specs == 0)

> df_no_use_specs > 5

> and somehow my intuition tells me the following could be done in a 
> better way:
> 
>         || ((df_no_use_specs > 0)
>             && (use_spec[0].column == dfncp1
>             || (df_no_use_specs > 1
>                 && (use_spec[1].column == dfncp1
>                 || (df_no_use_specs > 2
>                     && (use_spec[2].column == dfncp1
>                     || (df_no_use_specs > 3
>                         && (use_spec[3].column == dfncp1
>                         || (df_no_use_specs > 4 && (use_spec[4].column 
> == dfncp1 || )

Not really.  This is a test for "the first <n> using specifiers are just 
1..<n>, with (<n> <= 5)".  I don't think there's much of a way of 
expressing that any more efficiently.  It might make sense to move this
test to a different place, though.

> NEXT ISSUE:
[...]
>         /* HBB 20001221: avoid breaking parsing of time/date
>          * strings like 01Dec2000 that would be caused by
>          * overwriting the 'D' with an 'e'... */

> Isn't this a bug?  

Not to my knowledge, no.

 > If so, why not?

Why should it be one?

 > Certainly the date string 01Dec2000
> should not be interpretted as a double.  

No.  But it has to survive the attempted interpretation as a double,
unchanged.  I.e. at this point of execution, the parsing engine simply 
doesn't know that 01Dec2000 isn't supposed to be a floating-point number.

 > But doesn't a value of 1.0
> result from scanning 01eec2000?

The FP number resulting from this is irrelevant --- if the user actually 
asked for this to be read as a time/date string, it'll be overwritten 
later by the FP number representing 01Dec2000 in gnuplot's internal time 
format.  If the user didn't specify time/date format for this column, he 
deserves whatever nonsense he may get.

> Now, I see there are some strings to contend with.  Aside from that, 
> what if one initially goes through the string, using string searching 
> routines to look for "dDqQ" and convert them.  

No.  This would kill both the scanf() format string allowed to be passed 
in, if that contained some random 'q' characters, and, perhaps more 
importantly, Ethan's datastrings stuff.

Re: gnuplot slow on BIG files

[Daniel J S. <dan...@ie...>]

Hans-Bernhard Br=F6ker wrote:
>> Neither should it be a memory problem (something that would really=20
>> slow down Dimitris' machine perahps).  The block of memory it consumes=
=20
>> will not be much bigger than is eventually stored in memory.
>=20
>=20
> For the case of plot "datafile" matrix every 500:500, it might...

OK.  Point taken.  That could probably be fixed by doing the decimation f=
irst=20
and then changing the "every" internally to 1.

>=20
>> But, what could be a problem--in light of the discussion about FORTRAN=
=20
>> doubles--is that instead of "floats" maybe this routine should be=20
>> storing doubles. =20
>=20
>=20
> Yes, it should.  More to the point, it should store in type coordval.
> That's what it's defined for.

OK.  Who want's to change that?  Or should we hold off until everything e=
lse is=20
worked out?

>=20
>  > Certainly a plotting program doesn't need the
>=20
>> resolution of doubles.=20
>=20
>=20
> Oh, but it does --- if only because some people want to be able to zoom=
=20
> in somewhat deeply.  1/DBL_FLT is less than the number of pixels on the=
=20
> screen, and only a zoom factor of 1000 away from introducing aliasing=20
> artefacts.

You're right.  I forgot about zooming.

>=20
>> Unless one writes a more sophisticated parser.  Given how much testing=
=20
>> is already done here, that would almost be as good an option.  (But=20
>> perhaps not preferable.)
>=20
>=20
> A parser wouldn't do --- we'ld need a dynamic parser *generator*, i.e.=20
> an engine that, given a set of 'using', 'every', 'index' and whatnot,=20
> will generate a parser on-the-fly and run that on the code.
[...]
> Not really.  This is a test for "the first <n> using specifiers are jus=
t=20
> 1..<n>, with (<n> <=3D 5)".  I don't think there's much of a way of=20
> expressing that any more efficiently.  It might make sense to move this
> test to a different place, though.

I think you are saying the same thing here, and sort of what I was gettin=
g at.=20
Couldn't these tests be done before looping through the code to figure ou=
t <n>=20
and then test the loop counter against <n>.  Not sure...



>=20
>> NEXT ISSUE:
>=20
> [...]
>=20
>>         /* HBB 20001221: avoid breaking parsing of time/date
>>          * strings like 01Dec2000 that would be caused by
>>          * overwriting the 'D' with an 'e'... */
>=20
>=20
>> Isn't this a bug? =20
[...]
> No.  This would kill both the scanf() format string allowed to be passe=
d=20
> in, if that contained some random 'q' characters, and, perhaps more=20
> importantly, Ethan's datastrings stuff.

Oy.  Guess that is so.

Dan

Re: gnuplot slow on BIG files

[Robert H. <en...@no...>]

On Thu, 2005-06-09 at 15:26 +0800, Hans-Bernhard Br=F6ker wrote:

> That's exactly the error checking I'm talking about.  Without it,=20
> handling missing or malformed data would be impossible.
>
> There's not a lot you could do with strtod() that sscanf() couldn't do=20
> just as well.

strtod() is indeed significantly faster than sscanf.=20

I am testing using the following:
#ifdef USE_STRTOD
	      char *fin;
	      df_column[df_no_cols].datum=3Dstrtod(s,&fin);
	      used=3Ds-fin;
	      count=3Dused?1:0;
#else
	      count =3D sscanf(s, "%lf%n", &df_column[df_no_cols].datum, &used);
#endif

Can anybody come up with any cases where these two methods give
different results/error handling? strtod is ANSI C, so should be
portable.

I've attached a "non-intrusive" patch, however if this approach is
acceptable, I think the OSK path should be removed, and the
NO_FORTRAN_NUMS code should be removed or simplified. Are there any
cases when somebody wouldn't want a fortran float interpreting properly?

Rob
=20

--=20
Robert Hart <en...@no...>
University of Nottingham

This message has been checked for viruses but the contents of an attachment
may still contain software viruses, which could damage your computer system:
you are advised to perform your own checks. Email communications with the
University of Nottingham may be monitored as permitted by UK legislation.

Re: gnuplot slow on BIG files

[<br...@ph...>]

Robert Hart wrote:

> strtod() is indeed significantly faster than sscanf. 

And I still don't see why that should be the case...

> Can anybody come up with any cases where these two methods give
> different results/error handling?

Well, here's an ancient comment right taken directly from datafile.c:

  /* cannot trust strtod - eg strtod("-",&p) */

 > strtod is ANSI C, so should be
> portable.

So should sscanf().

Re: gnuplot slow on BIG files

[Robert H. <en...@no...>]

On Thu, 2005-06-09 at 21:18 +0800, Hans-Bernhard Br=F6ker wrote:
> Robert Hart wrote:
>=20
> > strtod() is indeed significantly faster than sscanf.=20
>
> And I still don't see why that should be the case...

sscanf takes an arbitrary format string as an argument, which it must
parse every time it is called, it is also very generic so, I imagine,
ends up doing a lot of extra work to support this. If you wanted to you
could look through the source for libc and figure it out.

> > Can anybody come up with any cases where these two methods give
> > different results/error handling?
>=20
> Well, here's an ancient comment right taken directly from datafile.c:
>=20
>   /* cannot trust strtod - eg strtod("-",&p) */

I've seen this comment, and tried it out. In my tests this isn't an
issue. The man page says:

       If  no  conversion is performed, zero is returned and the value of n=
ptr
       is stored in the location referenced by endptr.

So if(nptr!=3Dendptr) you know it failed.

Am I missing something? Are there other libc's that behave differently?

I have attached my test program. I can't find any input on my system
that gives different results between sscanf and strtod.

>  > strtod is ANSI C, so should be
> > portable.
>=20
> So should sscanf().

Well comments in the code say "%n" doesn't work on OSK. I don't even
know what OSK is, but if strtod works where sscanf doesn't then that's a
win.

Rob

--=20
Robert Hart <en...@no...>
University of Nottingham

This message has been checked for viruses but the contents of an attachment
may still contain software viruses, which could damage your computer system:
you are advised to perform your own checks. Email communications with the
University of Nottingham may be monitored as permitted by UK legislation.

Re: gnuplot slow on BIG files

[Ethan M. <merritt@u.washington.edu>]

On Thursday 09 June 2005 09:12 am, Robert Hart wrote:
> 
> I have attached my test program. I can't find any input on my system
> that gives different results between sscanf and strtod.

Very nice. Thank you.
So strtod is faster on all platforms I have tried, with the largest
difference on DU/alpha and the smallest difference on SunOS.

Not the factor of 10X that Dimitrios Apostolou was hoping for,
but up to a factor of 5X depending on the platform.


linux/x86
---------
./scan_timing 100000 "3.5D 1E07"
Test atod: 0.059186
Test sscanf: 0.150265
Test strtod: 0.060353

DU/alpha
---------
./scan_timing 100000 "3.5D 1E07"
Test atod: 0.100689
Test sscanf: 0.588027
Test strtod: 0.124872

linux/AMD64
-----------
 ./scan_timing 100000 "3.5D 1E07"
Test atod: 0.033294
Test sscanf: 0.076438
Test strtod: 0.037122

irix/MIPS R4400
----------------
./scan_timing  100000 "3.5D 1E07"
Test atod: 0.352085
Test sscanf: 1.012914
Test strtod: 0.357065

SunOS 4.1.4 1 sun4m
---------------------------
./scan_timing 100000 "3.5D 1E07"
Test atod: 2.245737
Test sscanf: 4.014354
Test strtod: 2.706037


-- 
Ethan A Merritt       merritt@u.washington.edu
Biomolecular Structure Center
Mailstop 357742
University of Washington, Seattle, WA 98195

Re: gnuplot slow on BIG files

[Daniel J S. <dan...@ie...>]

Interesting.  Of course, I think the same situation exists with scanf() in the 
non-matrix ascii input.  That is what brought about the "binary" option in the 
first place.  Sending images from octave to gnuplot was sloooooow...

Dan


Ethan Merritt wrote:

> Not the factor of 10X that Dimitrios Apostolou was hoping for,
> but up to a factor of 5X depending on the platform.
> 
> 
> linux/x86
> ---------
> ./scan_timing 100000 "3.5D 1E07"
> Test atod: 0.059186
> Test sscanf: 0.150265
> Test strtod: 0.060353
> 
> DU/alpha
> ---------
> ./scan_timing 100000 "3.5D 1E07"
> Test atod: 0.100689
> Test sscanf: 0.588027
> Test strtod: 0.124872
> 
> linux/AMD64
> -----------
>  ./scan_timing 100000 "3.5D 1E07"
> Test atod: 0.033294
> Test sscanf: 0.076438
> Test strtod: 0.037122
> 
> irix/MIPS R4400
> ----------------
> ./scan_timing  100000 "3.5D 1E07"
> Test atod: 0.352085
> Test sscanf: 1.012914
> Test strtod: 0.357065
> 
> SunOS 4.1.4 1 sun4m
> ---------------------------
> ./scan_timing 100000 "3.5D 1E07"
> Test atod: 2.245737
> Test sscanf: 4.014354
> Test strtod: 2.706037