Re: gnuplot slow on BIG files

[Hans-Bernhard B. <br...@ph...>]

Daniel J Sebald wrote:
> Hans-Bernhard Br=F6ker wrote:
>> Robert Hart wrote:

>>> sscanf takes an arbitrary format string as an argument, which it must=

>>> parse every time it is called, it is also very generic so, I imagine,=

>>> ends up doing a lot of extra work to support this.=20

>> None of that extra work can possibly be "a lot", compared to the heavy=
=20
>> task of actually doing a floating-point conversion. =20

[... Dan found a quote from myself...:]

> BTW: if I change the call from sscanf() to strtod(), the run time=20
> becomes O(n) in all platforms I cared to test.=20

Well, the processing that caused that example to be O(N^2) wasn't=20
related to the format string at all --- it was triggered by handling of=20
input string.  So my answer to Robert quoted above still holds.

Apparently the culprit implementations in that case (glibc, DJGPP,=20
possibly others) run strlen() on the input string as part of the job of=20
constructing a memory-based pseudo-file, so they can essentially=20
implement sscanf() on top of fscanf().  That's really quite a stupid way =

of doing this.

But yes, indeed, a QoI issue like this could explain the speed=20
difference we're seeing here.  The enormous length of the OP's matrix=20
datafile's lines would trigger this copying overhead on platforms=20
working that way.  The crucial difference in that case turned out the=20
length of the "tail" after the scanned number, e.g.

   sscanf("3.14 ", "%lf", &mydouble);

would complete much faster than

   sscanf("3.14 [10000 other characters here...]", "%lf", &mydouble);

> Now, here's the interesting part.  :)  The date of this contribution is=
=20
> May 22, 2002.  And the contributor of this was... Hans-Bernhard Broeker=
=20
> :)

Let's just say I know full well why I never claim to have perfect memory =
;-)