[TCLCORE] Re: Hintermayer's string allocation patch

[Dan K. <ku...@aj...>]

> You use less memory, but make more reallocations.  We can argue about
> tradeoffs.  But I still agree with John.  When one is asking for more
> than 1/2 virtual memory, you probably have exhausted physical memory
> and are now benchmarking the swapper.
> 
> I've never run into this problem and at some point it becomes special
> purpose enough to require a custom solution (e.g. allocate your own
> buffer).  I don't know if this a practical or theoretical problem?
> Why does anyone need a 1 Gig string?

I don't want to drag this out longer than it needs to.
As several people have tried to state earlier, on real
UNIX systems (solaris and linux) you don't run out of memory
at 1 GB.  Currently in tcl the string append growth algorithm
will fail when you have a buffer that is 1/2 as big as the
amount of available memory.

In real terms, if you have 384 MB of memory and swap space
(not unrealistic for a modern desktop computer) and you
have netscape and some other things running you probably
are using roughly 120 MB.  So for the tcl interpreter
there is about 264 MB of memory available.  When a tcl
variable is using more than half of what remains you
will be unable to append to that string without the
interpreter panic'ing.

Eric sent a test program to tclcore (as did Jim Ingham)
that can be run on a UNIX system to determine in real
terms what this limit is.

If you no longer have the messages this is web archived
at:

http://dev.scriptics.com/lists/tclcore/

> But if one still wants to handle this within Tcl, I have a slightly
> different tact.  When realloc fails, pull back the size.  It can fail
> when you can't allocate even the extra space needed.

Right.  If you look at Jeff's posts from yesterday
he mentions this option.  I initially proposed this
several weeks ago.  The current tcl C memory allocation
APIs will panic when the allocation fails.  I proposed
that a new API be added for growth algorithms where
you could specify two quantities.  One would be the
bare minimum amount of space needed to continue (the
size of the initial string + the size of the appended
string).  The second size would be the desired amount
of space to allocate (this would use a growth algorithm
similar to the one in the core today).  Then the
allocation function would (if the allocation failed)
gradually reduce the quantity of space asked for by
some algorithm/factor until it was at the minimum.
When the minimum was reached, if the allocation failed
it would panic.

The only issue with this was that it was probably a
little more work to implement.  I think that there
is room for both solutions.  One makes the core a
little more memory efficient and the other makes
the core a little more flexible in its memory
allocation.

Alternatively, if the TCT really thinks this is
too specialized the 'large string' size could
be included at a *really* high setting, and
only those who need it could tune the core
to use it.

--Dan

--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.