[TCLCORE] Re: Hintermayer's string allocation patch

[George H. <ga...@si...>]

In message <39A...@in...>, Gerhard Hintermayer writes:

 : The original problem was the "Maximum string length" thread discussed so
 : time ago in c.l.t. Actually I never had a problem running out of memory
 : when using strings, but somebody else had, and this was an interesting
 : challence. I thought there had to be changed something. The doubling
 : algorithm might be good under normal circumstances, but not when
 : reaching really large strings.

 : Suppose you give somebody 100 $, but at the same moment you tell him,
 : "be carefull, you cannot spend more than 50 $". Thats not logical. If
 : some Tcl-programmers need a string longer than half of his system
 : memory, for whatever reason, why not giving him the chance to do so ?
 : You could tell him, you _can_ handle that large strings, but at a cost
 : of performance.


 : As I've already stated in c.l.t., the speed improvement is only because
 : I replaced the call of malloc with a call to realloc and by not having
 : to copy the string data if the allocated memory block is simply
 : extended.
 : If you don't like changing the doubling algorithm, well you don't have
 : to. It is independent of the replacement of the malloc call. If you
 : don't use it, you will still have the speed improvements.

Thank you for the more complete description.  It helps a lot.

It does seem worthwhile to change malloc/free to realloc and test the
performance differences.  The test should include reallocations at
different rates: 200 bytes, 2K, 2M, 200M, variable sizes, etc.

The current algorithm is a bit greedy.  Usually one keeps doubling the
old size until it's at least the new required size.
    
    newSize = oldSize + appendSize;
    while (oldSize < newSize) {
	oldSize += oldSize;
    }

You use less memory, but make more reallocations.  We can argue about
tradeoffs.  But I still agree with John.  When one is asking for more
than 1/2 virtual memory, you probably have exhausted physical memory
and are now benchmarking the swapper.

I've never run into this problem and at some point it becomes special
purpose enough to require a custom solution (e.g. allocate your own
buffer).  I don't know if this a practical or theoretical problem?
Why does anyone need a 1 Gig string?

But if one still wants to handle this within Tcl, I have a slightly
different tact.  When realloc fails, pull back the size.  It can fail
when you can't allocate even the extra space needed.

    growth = oldSize + appendSize;
    do {
	newSize = oldSize + appendSize + growth;
	newMem = realloc(oldMem, newSize);
	growth /= 2;
	if (growth < appendSize) {
	     break;
        }
    } while (newMem == NULL);

At least this way the penalty is all on the special case, not general
usage.   

--gah

--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.