tcl-core

[TCLCORE] Re: Hintermayer's string allocation patch

[Scott S. <st...@aj...>]

George Howlett said:

> But if one still wants to handle this within Tcl, I have a slightly
> different tact.  When realloc fails, pull back the size.  It can fail
> when you can't allocate even the extra space needed.

Jeffrey Hobbs said:
 >  The other suggestion in the thread was a fall-back allocation 
mechanism,
 >  whereby one still goes for *2 all the time, until malloc says, "No".
 >  Then we would try for just what we need (or a bit more), and see if 
that
 >  works.  However, that requires changing around the ckalloc routines,
 >  because they panic by default when malloc returns NULL.  You'd have to
 >  create some sort of alternative like:
 > 
 > 	bytes = (char *) ckallocEx(howMuchIWant, howMuchINeed);

I think we should really consider implementing this approach.  So far, 
this sounds like the best approach anyone has mentioned yet.  It only 
costs you when you really run into the large allocation failure case.  
Arguably, the dynamic string mechanism should be in cahoots with the 
memory allocator for optimal performance anyway.  It wouldn't be that hard 
to add an alternate entry point that doesn't panic on malloc/realloc 
failures and then build the double realloc behavior on top.  This seems 
like a simple way to get the best performance while still allowing very 
large allocations.

--Scott


--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

[TCLCORE] Re: Hintermayer's string allocation patch

[Jeffrey H. <jef...@aj...>]

> From: Scott Stanton [mailto:st...@aj...]
	...
> George Howlett said:
>
> > But if one still wants to handle this within Tcl, I have a slightly
> > different tact.  When realloc fails, pull back the size.  It can fail
> > when you can't allocate even the extra space needed.
>
> Jeffrey Hobbs said:
>  >  The other suggestion in the thread was a fall-back allocation
> mechanism,
>  >  whereby one still goes for *2 all the time, until malloc says, "No".
>  >  Then we would try for just what we need (or a bit more), and see if
> that
>  >  works.  However, that requires changing around the ckalloc routines,
>  >  because they panic by default when malloc returns NULL.  You'd have to
>  >  create some sort of alternative like:
>  >
>  > 	bytes = (char *) ckallocEx(howMuchIWant, howMuchINeed);
>
> I think we should really consider implementing this approach.  So far,
> this sounds like the best approach anyone has mentioned yet.  It only
> costs you when you really run into the large allocation failure case.
> Arguably, the dynamic string mechanism should be in cahoots with the
> memory allocator for optimal performance anyway.  It wouldn't be that hard
> to add an alternate entry point that doesn't panic on malloc/realloc
> failures and then build the double realloc behavior on top.  This seems
> like a simple way to get the best performance while still allowing very
> large allocations.

I think it would be valuable to have a cross-platform malloc wrapper that
didn't panic.  Upon that you could build better fall-back algorithmns.
The example above is just a limited one-case fall-back, and if we want to
get more elaborate, we're still SOL.

At the same time, the new APIs could be done using the proper casts (size_t),
in preparation for 64-bit support.

In any case, we still need to allow the Tcl string allocator a way to not
be so aggressive in large string situations.

Jeffrey Hobbs                     Tcl Ambassador
ho...@Aj...          Ajuba Solutions (née Scriptics)

--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

[TCLCORE] Re: Hintermayer's string allocation patch

[George H. <ga...@si...>]

In message <NDB...@aj...>, "Je
ffrey Hobbs" writes:

 :  However, that requires changing around the ckalloc routines,
 :  because they panic by default when malloc returns NULL.  

In 8.3, "ckalloc" (TclpAlloc) panics at memory exhaustion only when
you have compiled with -DTCL_DEBUG and -DUSE_TCLALLOC=1.  If it does
otherwise, then TclpAlloc is broken.  The implication from the manual
page is that Tcl_Alloc and Tcl_Free are replacements for malloc and
free.  Therefore Tcl_Alloc should ape malloc's behavior.

     These procedures provide a platform and compiler independent
     interface  for  memory  allocation.   Programs  that need to
     transfer ownership of memory blocks between  Tcl  and  other
     modules  should  use  these  routines rather than the native
     malloc() and free() routines  provided  by  the  C  run-time
     library.

And from Sun's manual on malloc(3X)...

     If  there  is  no  available  memory,  malloc(),  realloc(),
     memalign(),  valloc(),  and  calloc() return a null pointer.
     When realloc() returns NULL, the block pointed to by ptr  is
     left  intact.   If  size,  nelem,  or  elsize is 0, a unique
     pointer to the arena is returned.

 : bytes = (char *) ckallocEx(howMuchIWant, howMuchINeed);

I disagree with this.  We don't need another API function/macro to do
the right thing.  Is there some unexplained reason why ckalloc should
panic when out of memory?  Other than the fact the calling routine
sometimes doesn't test the result...

--gah

--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

[TCLCORE] Re: Hintermayer's string allocation patch

[John O. <ou...@aj...>]

At 01:45 PM 8/31/2000 -0700, Jeffrey Hobbs wrote:
> > From: George Howlett [mailto:ga...@si...]
>         ...
> >  :  However, that requires changing around the ckalloc routines,
> >  :  because they panic by default when malloc returns NULL.
> >
> > In 8.3, "ckalloc" (TclpAlloc) panics at memory exhaustion only when
> > you have compiled with -DTCL_DEBUG and -DUSE_TCLALLOC=1.  If it does
> > otherwise, then TclpAlloc is broken.  The implication from the manual
>
>Well, that's just the way it is.  Tcl_Alloc/ckalloc *will* panic in
>Tcl (since 8.1) if you get a NULL from malloc.  This has been hashed
>out in numerous newsgroup threads.  The basic reasoning that I've heard
>(I wasn't around for this decision), is that if you are running out of
>memory, you're screwed.  It's better to panic with the "Unable to alloc"
>message than it is to follow on and fail because nowhere does the core
>actually handle a NULL return from ckalloc.

This is one of those issues that keeps coming up over and over again
when people learn about the behavior for the first time.  There's a
much better reason than "that's just the way it is".

The reason for the behavior is that recovering from memory outages
just isn't possible (IMHO).  There are way too many places where
memory is allocated, so it would add a lot of complexity to Tcl to
try and behave reasonably in every instance.  Furthermore, it is
difficult to figure out how to test memory exhaustion in all of these
places reliably, so in practice recovery wouldn't work ("if it hasn't
been used, it doesn't work").  Lastly, the whole idea of recovery
assumes that Tcl could do something useful even if it ran out of
memory, and that isn't true.  Dynamic memory allocation is used
*everywhere*, even in the guts of the interpreter.  So, if you run
out of memory, you can't necessarily even execute scripts to free
up memory.  Bottom line: better to panic cleanly with a nice simple
error message.

-John-
________________________________________________________________________
John Ousterhout                            650-210-0102 tel
Chairman and Chief Technology Officer      650-230-4070 fax
Ajuba Solutions                            ou...@aj...
http://www.ajubasolutions.com

--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

[TCLCORE] Re: Hintermayer's string allocation patch

[Mark H. <ma...@us...>]

John Ousterhout <ou...@aj...> wrote:
> So, if you run
> out of memory, you can't necessarily even execute scripts to free
> up memory.  Bottom line: better to panic cleanly with a nice simple
> error message.

At the VHLL conference, Steve Johnson mentioned that this behavior
(dying on memory allocation failure) was a reason that he would never
consider use Tcl in a mission-critical system.

After his talk, we chatted a bit, and I explained that a memory
error was just one of the failure modes we deal with when writing
this kind of system.  The mechanisms that are routinely put into
place for dealing with process death and node failure handle this
situation.

Mark.

--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

[TCLCORE] Re: Hintermayer's string allocation patch

[George H. <ga...@si...>]

In message <200...@po...>, Dan Kuchler writes:
 : > But if one still wants to handle this within Tcl, I have a slightly
 : > different tact.  When realloc fails, pull back the size.  It can fail
 : > when you can't allocate even the extra space needed.
 : 
 : Right.  If you look at Jeff's posts from yesterday
 : he mentions this option.  I initially proposed this
 : several weeks ago.  The current tcl C memory allocation
 : APIs will panic when the allocation fails.  I proposed
 : that a new API be added for growth algorithms where
 : you could specify two quantities.  One would be the
 : bare minimum amount of space needed to continue (the
 : size of the initial string + the size of the appended
 : string).  The second size would be the desired amount
 : of space to allocate (this would use a growth algorithm
 : similar to the one in the core today).  Then the
 : allocation function would (if the allocation failed)
 : gradually reduce the quantity of space asked for by
 : some algorithm/factor until it was at the minimum.
 : When the minimum was reached, if the allocation failed
 : it would panic.
 : 
 : The only issue with this was that it was probably a
 : little more work to implement.  I think that there
 : is room for both solutions.  One makes the core a
 : little more memory efficient and the other makes
 : the core a little more flexible in its memory
 : allocation.
 : 
 : Alternatively, if the TCT really thinks this is
 : too specialized the 'large string' size could
 : be included at a *really* high setting, and
 : only those who need it could tune the core
 : to use it.

This is a solution in need of a problem. 

While I think Gerhard's realloc mods sound very good, I reiterate my
question.  Where do you need a a 1 Gigabyte string?  (BTW, that's 1/2
of a 32-bit address space).

Please don't fix Tcl's growth scheme until it's broken.  First, make
the case for all the applications that this change fixes.  Otherwise
this is another Tcl pseudo-improvement.

--gah



--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

[TCLCORE] Re: Hintermayer's string allocation patch

[Brent W. <we...@aj...>]

>>>"George Howlett" said:

 > While I think Gerhard's realloc mods sound very good, I reiterate my
 > question.  Where do you need a a 1 Gigabyte string?  (BTW, that's 1/2
 > of a 32-bit address space).
 > 
 > Please don't fix Tcl's growth scheme until it's broken.  First, make
 > the case for all the applications that this change fixes.  Otherwise
 > this is another Tcl pseudo-improvement.

>>> Brent Welch said:
> Apparently MacOS is more advanced than UNIX in this area.  In UNIX you
> have to be able to allocate the swap space to cover the virtual
> memory.  So, in practice, if you have 300 Meg of swap space, 192 Meg
> of real memory, and a 160 Meg string, you cannot append one byte to it
> in Tcl. 

So, it is broken.  It is not that we cannot allocate a 1 or 2 Gig string,
it is that we may not even be able to grow a Tcl string that will
fit into physical memory/swap space.

Also, if you look at the patch, it replaces like 8 lines with 16 lines,
and by tuning the "large size" limit and the growth chunk size, we
can make the performance hit for things over the "large size" acceptable.

Can we be done with this now?

--	Brent Welch	<we...@aj...>
	http://www.ajubasolutions.com
	Scriptics changes to Ajuba Solutions
	scriptics.com => ajubasolutions.com


--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

[TCLCORE] Re: Hintermayer's string allocation patch

[Jeffrey H. <jef...@aj...>]

> From: George Howlett [mailto:ga...@si...]
	...
>  :  However, that requires changing around the ckalloc routines,
>  :  because they panic by default when malloc returns NULL.
>
> In 8.3, "ckalloc" (TclpAlloc) panics at memory exhaustion only when
> you have compiled with -DTCL_DEBUG and -DUSE_TCLALLOC=1.  If it does
> otherwise, then TclpAlloc is broken.  The implication from the manual

Well, that's just the way it is.  Tcl_Alloc/ckalloc *will* panic in
Tcl (since 8.1) if you get a NULL from malloc.  This has been hashed
out in numerous newsgroup threads.  The basic reasoning that I've heard
(I wasn't around for this decision), is that if you are running out of
memory, you're screwed.  It's better to panic with the "Unable to alloc"
message than it is to follow on and fail because nowhere does the core
actually handle a NULL return from ckalloc.

BTW, the docs for Tcl_Alloc, et al don't say anywhere that they are
drop-in replacements for malloc, et al, just that they should instead
be used when fiddling with memory shared by the core routines.

>  : bytes = (char *) ckallocEx(howMuchIWant, howMuchINeed);
>
> I disagree with this.  We don't need another API function/macro to do
> the right thing.  Is there some unexplained reason why ckalloc should
> panic when out of memory?  Other than the fact the calling routine
> sometimes doesn't test the result...

"sometimes" is actually never for the core, and I suppose it was easier
to make the check in the alloc function.  Anyway, creating an alternative
ckalloc would solve these problems.

In any case for core "improvements", lets not argue about what is the
"right thing" (this goes with other running threads).  Accept what the
core does now, for whatever reason, and then lets question whether there
is a more intuitive/appropriate way to do things.

Jeffrey Hobbs                     Tcl Ambassador
ho...@Aj...          Ajuba Solutions (née Scriptics)

--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

[TCLCORE] Re: Hintermayer's string allocation patch

[Andreas K. <a.k...@we...>]

Hi, my two cents or so in this.

I go with George and John here, i.e. allocate until you the wall and
then retract from the boundary to fill the rest in smaller
increments.

What I like about this algorithm is that it not only preserves maximum
performance for as long as possible while still being able to fill the
memory to the max. but also that it is "self-tuning".

With this I mean that there is no need to adjust (tinker / fiddle
with) some parameters like with the current patch proposed as by
Gerhard to adapt the core to the host it is running on to get good
performance on average. We get this automatically.

This makes it also easier for distributors of binaries. No need to
guess some values for the parameters which will give good performance
over a __range_of_machines__.



Gerhard, are you willing to try and write a second patch which uses
this algorithm to solve the problem so that we may compare hard
numbers instead of running in argumentative circles without hard data ?


-- 
Sincerely,
	Andreas Kupries <a.k...@we...>
		<http://www.purl.org/NET/akupries/>
-------------------------------------------------------------------------------

--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

Re: [TCLCORE] Re: Hintermayer's string allocation patch

[Mark H. <ma...@us...>]

Andreas Kupries <a.k...@we...> wrote:
> I go with George and John here, i.e. allocate until you the wall and
> then retract from the boundary to fill the rest in smaller
> increments.

I also concur with this...

Joe English <jen...@fl...> wrote:
> If it has the potential to
> adversely affect the performance of Tcl programs
> by a quadratic factor (which I believe it does),
> it shouldn't be applied.

...and this.

Eric Melski <er...@aj...> wrote:
> Again, I urge everybody not to get hung up on the performance issue and
> consider the memory issue:  without _some_ kind of change, Tcl simply
> cannot use all available memory.

I disagree with this.  It is not a good tradeoff.  In addition, if an
application has a requirement to deal with data this size, there are
probably better solutions than shuffling strings around in swap space.

Amusing anecdote:  DSC Communications had exactly one customer problem
report related to Tcl in five years.  A programmer had attempted to
use a text widget to watch log data, but didn't get the code write
to delete the oldest test lines, so the text data kept growing.  After
about 3 days, the machine ran out of swap, and the other processes in
the system started to die -- but the text widget process kept going with
no problem.  It's the only trouble I've ever seen complaining that a
program was too robust.

Mark.

--
Mark Harrison                          AsiaInfo Holdings, Inc.
ma...@us...                Beijing/Santa Clara

Re: [TCLCORE] Re: Hintermayer's string allocation patch

[Gerhard H. <g.h...@in...>]

Andreas Kupries wrote:
> 
> Hi, my two cents or so in this.
> 
> I go with George and John here, i.e. allocate until you the wall and
> then retract from the boundary to fill the rest in smaller
> increments.
> 
> What I like about this algorithm is that it not only preserves maximum
> performance for as long as possible while still being able to fill the
> memory to the max. but also that it is "self-tuning".
> 
> With this I mean that there is no need to adjust (tinker / fiddle
> with) some parameters like with the current patch proposed as by
> Gerhard to adapt the core to the host it is running on to get good
> performance on average. We get this automatically.
> 
> This makes it also easier for distributors of binaries. No need to
> guess some values for the parameters which will give good performance
> over a __range_of_machines__.
> 
> Gerhard, are you willing to try and write a second patch which uses
> this algorithm to solve the problem so that we may compare hard
> numbers instead of running in argumentative circles without hard data ?
> 

Well, I'm sorry, no. I do not have that much spare time and besides the
discussion I started by submitting this patch was a bit frustrating.
Ok, the patch is not the best solution, but I think it was worth
spending the time, I looked for a quick solution and didn't want to
rewrite the whole memory handling. Of course there are better
implementations, but that would require more time and maybe expertise,
which I don't have.
Maybe someone from the core team can do this.
It is a pity, that this discussion consumed more of my time than
actually writing the patch.

Gerhard
-- 
Gerhard Hintermayer
http://www.inode.at/g.hintermayer

[TCLCORE] Re: Hintermayer's string allocation patch

[Jeffrey H. <jef...@aj...>]

> -----Original Message-----
> From: ak...@bl... [mailto:ak...@bl...]On
	...
> ... allocate until you the wall and
> then retract from the boundary to fill the rest in smaller
> increments.
>
> What I like about this algorithm is that it not only preserves maximum
> performance for as long as possible while still being able to fill the
> memory to the max. but also that it is "self-tuning".
>
> With this I mean that there is no need to adjust (tinker / fiddle
> with) some parameters like with the current patch proposed as by
> Gerhard to adapt the core to the host it is running on to get good
> performance on average. We get this automatically.

While this is a good approach, it does require a more extensive patch
to the core - adding new alloc stubs routines (which is probably a
good idea anyway).

Also, it still leaves behind the remnants of poor mem allocation in
Tcl if you use multiple large objects.  For the worst case, each string
in Tcl takes up twice as much space as it needs to.  Thus if we are
working with multiple large objects, we can have 120MB of actual data
taking up 240MB of memory, and we still hit a wall because a new large
object may not get any more memory, although 120MB is wasted.

Of course, the same can happen for lots of smaller objects if you have
the min algorithm switchover high, so there's no 100% win.  Thus the
point is, do we want to accept the fact that Tcl can't effectively use
more than half the available memory?  I find that hard to swallow, which
is why a trade-off should be considered, one that kicks in at a hard-coded
level (where excessive 2* waste is a lot more painful).

Jeffrey Hobbs                     Tcl Ambassador
ho...@Aj...          Ajuba Solutions (née Scriptics)

--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

[TCLCORE] Re: Hintermayer's string allocation patch

[George H. <ga...@si...>]

In message <200...@po...>, Brent Welch writes:
 : So, it is broken.  It is not that we cannot allocate a 1 or 2 Gig string,
 : it is that we may not even be able to grow a Tcl string that will
 : fit into physical memory/swap space.

We can agree that replacing free/malloc with realloc is Ok.  It may
also be worth looking into tuning the grow rate of strings.  If may be
that realloc sufficiently diminishes the cost of allocations such that
a less aggressive growth rate is attractive.

But I don't agree that it's at all necessary that Tcl be able to grow
a string that exceeds physical memory.  Please make the case for any
application that requires such capabilities.

I especially don't like the "large size" memory parameter that doesn't
scale with memory sizes.  It's an simple matter to write a version of
TclpAlloc (and thereby Tcl_Realloc) that doesn't panic when it runs
out of memory.

 : Also, if you look at the patch, it replaces like 8 lines with 16 lines,
 : and by tuning the "large size" limit and the growth chunk size, we
 : can make the performance hit for things over the "large size" acceptable.
 : 
 : Can we be done with this now?

Please forgive me if I'm a little slow in seeing how the TCT operates.
BTW. Was this how the panic-ing version of TclpAlloc was included?

--gah

--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

[TCLCORE] Re: Hintermayer's string allocation patch

[John O. <ou...@aj...>]

I think this discussion has gone on too long now.  We're at the
point where the same people are repeating the same arguments
over and over again.  This means that there isn't going to be
total agreement.  So, it's time for someone to summarize the
arguments and then for people to reach a decision.

-John-
________________________________________________________________________
John Ousterhout                            650-210-0102 tel
Chairman and Chief Technology Officer      650-230-4070 fax
Ajuba Solutions                            ou...@aj...
http://www.ajubasolutions.com

--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

[TCLCORE] Re: Hintermayer's string allocation patch

[Brent W. <we...@aj...>]

>>>"George Howlett" said:
 >  : Brent said:
 >  : Also, if you look at the patch, it replaces like 8 lines with 16 lines,
 >  : and by tuning the "large size" limit and the growth chunk size, we
 >  : can make the performance hit for things over the "large size" acceptable.
 >  : 
 >  : Can we be done with this now?
 > 
 > Please forgive me if I'm a little slow in seeing how the TCT operates.
 > BTW. Was this how the panic-ing version of TclpAlloc was included?

My "can we be done now" comment was premature and not appropriate.
I think we are all novices in the way TCT operates, or should operate.

I would like to bring this issue to closure, if possible.  So, let me
start by stating two different proposals, one for realloc, which I
hope is easy to resolve, and the other is for the large string append
problem where there are two different camps.
I'll start new threads for it.

--	Brent Welch	<we...@aj...>
	http://www.ajubasolutions.com
	Scriptics changes to Ajuba Solutions
	scriptics.com => ajubasolutions.com

--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

[TCLCORE] Re: Hintermayer's string allocation patch

[George H. <ga...@bl...>]

In message <4.2...@ma...>, John Ouste
rhout writes:

Thank you for taking the time to explain this.

 : The reason for the behavior is that recovering from memory outages
 : just isn't possible (IMHO).  There are way too many places where
 : memory is allocated, so it would add a lot of complexity to Tcl to
 : try and behave reasonably in every instance.  Furthermore, it is
 : difficult to figure out how to test memory exhaustion in all of these
 : places reliably, so in practice recovery wouldn't work ("if it hasn't
 : been used, it doesn't work").  Lastly, the whole idea of recovery
 : assumes that Tcl could do something useful even if it ran out of
 : memory, and that isn't true.  Dynamic memory allocation is used
 : *everywhere*, even in the guts of the interpreter.  So, if you run
 : out of memory, you can't necessarily even execute scripts to free
 : up memory.  Bottom line: better to panic cleanly with a nice simple
 : error message.

I agree with you on this last point.  In my own code do the same
thing.  When allocating memory for structures, strings, etc., there's
no practical way of recovering from memory exhaustion.  Panic-ing is a
reasonable solution.

But there's another class of allocations where I do want to know if I
can't allocate the requested amount of memory.  When I allocate a
large block of storage for for a vector, the contents of a file, etc.
I want to tell the user that his/her request failed.

    Tcl_GetIntFromObj(interp, objv[2], &requestedSize);
    arrayPtr = (double *)Tcl_Alloc(sizeof(double) * requestedSize);
    if (arrayPtr == NULL) {
        Tcl_AppendResult(interp, "can't allocate vector of ",
  	    Tcl_GetString(objv[2]), " bytes", (char *)NULL);
	return TCL_ERROR;
    }

I have no argument with panic-ing.  But after looking at TclpAlloc, I
don't see where it will panic unless TCL_DEBUG is defined.  And this
is only if you also turn on USE_TCL_ALLOC (-DUSE_TCLALLOC=1).  So for
non-Windows users, they will get a segfault instead of an error
message.  

We can make this consistent for all platforms by removing the
panic-ing from the TclpAlloc and putting it into (let's say) the
ckalloc macro.  As a consequence, I can check against memory
exhaustion for the rare cases where I can recover gracefully from it.

--gah

--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

[TCLCORE] Re: Hintermayer's string allocation patch

[Jeffrey H. <jef...@aj...>]

> From: Mark Harrison [mailto:ma...@us...]
	...
> John Ousterhout <ou...@aj...> wrote:
> > So, if you run
> > out of memory, you can't necessarily even execute scripts to free
> > up memory.  Bottom line: better to panic cleanly with a nice simple
> > error message.
>
> At the VHLL conference, Steve Johnson mentioned that this behavior
> (dying on memory allocation failure) was a reason that he would never
> consider use Tcl in a mission-critical system.
>
> After his talk, we chatted a bit, and I explained that a memory
> error was just one of the failure modes we deal with when writing
> this kind of system.  The mechanisms that are routinely put into
> place for dealing with process death and node failure handle this
> situation.

At the same time it should also be said that people can override the
panic proc (with Tcl_SetPanicProc) and create a system that will
capture out-of-mem situations and allow some element of recovery.
I've outlined in detail how this can be done on the newsgroup, if
someone needs to pass it on.

Jeffrey Hobbs                     Tcl Ambassador
ho...@Aj...          Ajuba Solutions (née Scriptics)

--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

[TCLCORE] Re: Hintermayer's string allocation patch

[Jeffrey H. <jef...@aj...>]

> From: George Howlett [mailto:gah@blueberry]
	...
> I agree with you on this last point.  In my own code do the same
> thing.  When allocating memory for structures, strings, etc., there's
> no practical way of recovering from memory exhaustion.  Panic-ing is a
> reasonable solution.
>
> But there's another class of allocations where I do want to know if I
> can't allocate the requested amount of memory.  When I allocate a
> large block of storage for for a vector, the contents of a file, etc.
> I want to tell the user that his/her request failed.

For that reason I think the need for a non-panic'ing alloc still exists.
Tcl_SafeAlloc perhaps?

	...
> I have no argument with panic-ing.  But after looking at TclpAlloc, I
> don't see where it will panic unless TCL_DEBUG is defined.  And this
> is only if you also turn on USE_TCL_ALLOC (-DUSE_TCLALLOC=1).  So for
> non-Windows users, they will get a segfault instead of an error
> message.

The default builds on both Unix and Windows (I don't know about the
Mac) will hit the panic code.  The twisty trail of following #defines
in tcl.h/tclCkalloc.c can be confusing, but it happens.  I assume the
same for the Mac.

BTW, just to make it clear for others, Tcl still does use the Tcl mem
allocator by default on Windows (-DUSE_TCLALLOC=1), but not on Unix.
We were considering changing this when moving to default compiling with
VC++6, as we've been told the default mem allocator finally has some
brains on Windows.

Jeffrey Hobbs                     Tcl Ambassador
ho...@Aj...          Ajuba Solutions (née Scriptics)

--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

[TCLCORE] Re: Hintermayer's string allocation patch

[D. R. H. <dr...@hw...>]

Jeffrey Hobbs wrote:
> 
> For that reason I think the need for a non-panic'ing alloc still exists.
> Tcl_SafeAlloc perhaps?
> 

I want to take issue with the name.  When I hear
"SafeAlloc()" I think of a malloc() that never
returns NULL.  In other words, it is "safe" to use
without checking the return type.  Your usage 
completely inverts that meaning.

Perhaps Tcl_TryToAlloc() or Tcl_AttemptAlloc()
instead.  Here the idea is "try to get me some
memory but its OK if you can't, just let me know."
This will be (I think) a seldom used function, 
so a longer name is not a handicap.
-- 
D. Richard Hipp -- dr...@hw... -- http://www.hwaci.com/drh/

--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

[TCLCORE] Re: Hintermayer's string allocation patch

[Brent W. <we...@aj...>]

>>>"Jeffrey Hobbs" said:
 
 > The default builds on both Unix and Windows (I don't know about the
 > Mac) will hit the panic code.  The twisty trail of following #defines
 > in tcl.h/tclCkalloc.c can be confusing, but it happens.  I assume the
 > same for the Mac.

The panics are in the generic code

char *
Tcl_Alloc (size)
    unsigned int size;
{
    char *result;

    result = TclpAlloc(size);
    /*
     * Most systems will not alloc(0), instead bumping it to one so
     * that NULL isn't returned.  Some systems (AIX, Tru64) will alloc(0)
     * by returning NULL, so we have to check that the NULL we get is
     * not in response to alloc(0).
     *
     * The ANSI spec actually says that systems either return NULL *or*
     * a special pointer on failure, but we only check for NULL
     */
    if ((result == NULL) && size) {
        panic("unable to alloc %d bytes", size);
    }
    return result;
}

char *
Tcl_DbCkalloc(size, file, line)
    unsigned int size;
    char        *file;
    int          line;
{
    struct mem_header *result;

    if (validate_memory)
        Tcl_ValidateAllMemory (file, line);

    result = (struct mem_header *) TclpAlloc((unsigned)size +
                              sizeof(struct mem_header) + HIGH_GUARD_SIZE);
    if (result == NULL) {
        fflush(stdout);
        TclDumpMemoryInfo(stderr);
        panic("unable to alloc %d bytes, %s line %d", size, file, line);
    }


--	Brent Welch	<we...@aj...>
	http://www.ajubasolutions.com
	Scriptics changes to Ajuba Solutions
	scriptics.com => ajubasolutions.com


--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

[TCLCORE] Re: Hintermayer's string allocation patch

[Scott S. <st...@aj...>]

"D. Richard Hipp" said:
 > Perhaps Tcl_TryToAlloc() or Tcl_AttemptAlloc()
 > instead.  Here the idea is "try to get me some
 > memory but its OK if you can't, just let me know."
 > This will be (I think) a seldom used function, 
 > so a longer name is not a handicap.

The existing TclpAlloc could be the right way to handle this.  It is 
usually just a wrapper around malloc() on Unix, so if we fixed the Tcl 
allocator to avoid panics (as George suggested), then on all platforms it 
would return NULL on allocation failures.  Then the outer Tcl_Alloc call 
can implement the guarantee never to return NULL.

--Scott


--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

[TCLCORE] Re: Hintermayer's string allocation patch

[Jim I. <ji...@ap...>]

On Friday, September 1, 2000, at 11:45 AM, Scott Stanton wrote:

>  
> "D. Richard Hipp" said: 
>  > Perhaps Tcl_TryToAlloc() or Tcl_AttemptAlloc() 
>  > instead.  Here the idea is "try to get me some 
>  > memory but its OK if you can't, just let me know." 
>  > This will be (I think) a seldom used function,  
>  > so a longer name is not a handicap. 
>  
> The existing TclpAlloc could be the right way to handle this.  It is  
> usually just a wrapper around malloc() on Unix, so if we fixed the Tcl  
> allocator to avoid panics (as George suggested), then on all platforms it  
> would return NULL on allocation failures.  Then the outer Tcl_Alloc call  
> can implement the guarantee never to return NULL. 
>  

The only problem with this is that we would then be making a "p" routine part of the public Tcl API's.  I am not sure what the  thinking on this is.

Jim

> --Scott 
>  
>  
> -- 
> The TclCore mailing list is sponsored by Ajuba Solutions 
> To unsubscribe:  email tcl...@aj... with the  
>                  word UNSUBSCRIBE as the subject. 
>  
>  

--
Jim Ingham                                   ji...@ap...
Developer Tools - gdb
Apple Computer

--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.