tcl-core — Tcl Core Team discussions about Tcl development

[TCLCORE] Re: Procedure/Namespace Behaviour

[Donald G P. <dg...@ca...>]

From: Brent Welch <we...@aj...>
> OK, I was confused about which side of the argument you were on, ...

It doesn't help matters that I keep switching sides... :)

> As for the "history" of this issue, I know that when namespaces were
> first being implemented in Tcl 8.0 that rename did not change the behavior of
> a procedure inside a namespace.  

Armed with that clue, I dove into my limited archives of Tcl sources,
and sure enough in Tcl 8.0b2, [rename] behaves the way Donal wishes it
did.  This was changed between release 8.0b2 and release 8.0p0.  The
ChangeLog doesn't go back that far, but the changes file contains an
entry:

7/30/97 (bug fix) If a procedure is renamed into a different namespace, it
now executes in the context of that namespace. (BL)

So...  Who is BL, and why did they think this was a bug in need of
fixing?

This might also be an opportunity to make observations on the wisdom
of feature changes between the last beta release and the first
official release....

DGP

--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

[TCLCORE] Re: Hintermayer's string allocation patch

[Jim I. <ji...@ap...>]

Eric,

On Thursday, August 31, 2000, at 11:15 AM, Eric Melski wrote:

> The same way that we pick any values like these:  take something that 
> seems reasonable and run with it.  If in 5 years it isn't working, adjust 
> it.  As I consider these factors again, values of 4MB/4KB (respectively) 
> seem more reasonable to me. 

This seems more reasonable.

>  
> Again, I urge everybody not to get hung up on the performance issue and 
> consider the memory issue:  without _some_ kind of change, Tcl simply 
> cannot use all available memory.  This modification is simple, it works, 
> and it is already implemented.  I have not yet seen any other proposal 
> that can say the same. 
>  

Yes BUT...  I can see myself writing and reading 1-4 Meg of data in realistic circumstances.  Not often, but I can see this.  I can't right now see a use for reading 128Meg of data into Tcl.  So if you make operations on the 1-4 Meg area slow, just on the off chance someone will do the 128Meg allocation, you are not making a good trade-off.

So you should make sure the cutoff where you switch from one algorithm to the other is well above the size most people will use, then it should be ok.  I would argue for erring on the side of high rather than low, however, since if you can't get the very last 4-8 Meg of a 128 meg string allocation, for example, that is not so bad.  It will almost never be an issue, and the solution is to just add some more swap space (this is not hard to do on most modern systems) and try again...

Jim

--
Jim Ingham                                   ji...@ap...
Developer Tools - gdb
Apple Computer

--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

[TCLCORE] Re: Memory Allocators (wasHintermayer's string allocation patch)

[Jim I. <ji...@ap...>]

Brent,

Yeah...  It has a dynamic swap file, rather than a separate swap partition.  So the swap size is bounded above by the size of your disk not the size of the swap partition.  It is bounded below by some default "smallest swap size".  I couldn't find where this is set, but somewhere in the /etc forest or in the NetInfo database, one or the other. 

It also seems (I haven't looked at the code, this is just from watching as I filled up my disk) that it will stop to compress unused swap as it goes along.  It is a little too agressive, I think, so that when I had filled up my entire disk, and then tried to launch gdb on gdb, it actually tried to do it, but went into a seemingly endless cycle of crunching the disk.  However, I could reboot the machine and it came up with no disk errors, so the failure mode is not that bad, and the upside is pretty nice...  

Appropriate for a desktop system, but maybe not for a server, where anything that might compromise the system's running should be killed immediately.  You can of course do all the classic Unix admin stuff, and limit process sizes, # of processes, etc for any/all users, if you want to make a MacOS X system secure from external attacks or malicious users.  Needless to say, I don't do this for myself.  If I hang the machine in this way, it is usually my fault (and I haven't been able to for a while without trying pretty hard - which is nice).

Jim  

On Wednesday, August 30, 2000, at 10:06 PM, Brent Welch wrote:

> Apparently MacOS is more advanced than UNIX in this area.  In UNIX you have 
> to be able to allocate the swap space to cover the virtual memory.  So, in practice, 
> if you have 300 Meg of swap space, 192 Meg of real memory, and a 160 Meg string, 
> you cannot append one byte to it in Tcl. 
>  
>  

--
Jim Ingham                                   ji...@ap...
Developer Tools - gdb
Apple Computer

--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

[TCLCORE] Re: Procedure/Namespace Behaviour

[Donald G P. <dg...@ca...>]

From: Brent Welch <we...@aj...>
> By the way, isn't the big picture here that we need an
> 
> onexit callback
> 
> command so that Tcl-level applications can take action on exit?

Couldn't this just be an application of Vince Darley's proposed
[trace execute] command?

DGP

--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

[TCLCORE] Re: Hintermayer's string allocation patch

[Eric M. <er...@aj...>]

On Thu, 31 Aug 2000, Donald G Porter wrote:

> It's worth considering the asymptotics, too.  As others have
> observed, the O(N**2) behavior of the new algorithm will bite when
> the "1000" in these benchmarks grows to 10000, 100000, 1000000, ...

Yes, the performance will be bad, but is it better to have a badly
performing system that can allocate all of memory, or a really fast system
that cannot?  This seems like a no-brainer to me:  I can write a program
to give you the wrong answer to any question, and it can be really,
really fast... but it still gives you the wrong answer.

> Another concern is the platform-specific tuning necessary to choose
> good values for TCL_GROWTH_LARGE_STRING and TCL_GROWTH_MIN_ALLOC.
> How will good values for these be determined over all the platforms
> Tcl runs on and will run on over the next, say, 5 years?

The same way that we pick any values like these:  take something that
seems reasonable and run with it.  If in 5 years it isn't working, adjust
it.  As I consider these factors again, values of 4MB/4KB (respectively)
seem more reasonable to me.

Again, I urge everybody not to get hung up on the performance issue and
consider the memory issue:  without _some_ kind of change, Tcl simply
cannot use all available memory.  This modification is simple, it works,
and it is already implemented.  I have not yet seen any other proposal
that can say the same.

   Eric Melski                            The Other Tcl Guy
   ericm at ajubasolutions.com            Ajuba Solutions

--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

[TCLCORE] Re: Hintermayer's string allocation patch

[Donald G P. <dg...@ca...>]

> 2.  Tcl is faster in all append cases.  

All the cases you tested.

Add this to string.bench and check the results:

bench -desc "STR append (1MB + 1KB * 1000)" \
	-body {str-append $base $kilobyte 1000}

Seems to me the gradual growth of a large string to a much larger
string in lots of little pieces is common enough the benchmark
suite ought to test that too.

DGP

--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

[TCLCORE] Re: Hintermayer's string allocation patch

[Donald G P. <dg...@ca...>]

> From: "George Howlett" <ga...@si...>
> It would help to see the patch, 

cvs diff -c -r 1.17 -r HEAD generic/tclStringObj.c

> I'd really like to see the original problem.  I looked at the thread
> and I couldn't find it.  I want to know what problem the Tcl code (not
> C code) is attempting to solve and can't (because of the current
> allocation scheme). 

Seconded!

> It would help to see the benchmarks too. 

cvs checkout tclbench
cat tclbench/tcl/string.bench

It's reasonable to assume that members of TCLCORE have a CVS working
directory copy of Tcl's source code, right?

DGP

--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

Re: [TCLCORE] Re: Procedure/Namespace Behaviour

[Andreas K. <a.k...@we...>]

--------; charset=us-ascii

> From: Brent Welch <we...@aj...>
> > OK, I was confused about which side of the argument you were on, ...
> 
> It doesn't help matters that I keep switching sides... :)
> 
> > As for the "history" of this issue, I know that when namespaces were
> > first being implemented in Tcl 8.0 that rename did not change the behavior of
> > a procedure inside a namespace.  
 
> Armed with that clue, I dove into my limited archives of Tcl
> sources, and sure enough in Tcl 8.0b2, [rename] behaves the way
> Donal wishes it did.  This was changed between release 8.0b2 and
> release 8.0p0.  The ChangeLog doesn't go back that far, but the
> changes file contains an entry:
 
> 7/30/97 (bug fix) If a procedure is renamed into a different
> namespace, it now executes in the context of that namespace. (BL)
 
> So...  Who is BL,

IIRC BL = Brian Lewis	(bytecode compiler)

-- 
So long,
	Andreas Kupries <a.k...@we...>
			<http://www.purl.org/NET/akupries/>
-------------------------------------------------------------------------------

[TCLCORE] Re: Hintermayer's string allocation patch

[Jeffrey H. <jef...@aj...>]

Your welcome to add to the tclbench suite (should anyone want write
access to that module), but here is a repitition adding in DGP's
suggested bad case:

nerja [~/cvs/tclbench] 179 % ./runbench.tcl -verbose -min 8.4 -match "STR
append*" tcl/string.bench
TCL_INTERP: 1:8.4a1 2:8.4a2
STARTED Thu Aug 31 10:13:24 PDT 2000
Benchmark 1:8.4a1 /export/hobbs/build/install/solaris-x86/bin/tclsh8.4
Benchmark 2:8.4a2 /home/hobbs/install/solaris-x86/bin/tclsh8.4
000 VERSIONS:                               1:8.4a1 2:8.4a2
001 STR append                                   64      63
002 STR append (1KB + 1KB)                       53      56
003 STR append (10KB + 1KB)                     146      98
004 STR append (1MB + 2b * 1000)              20933   13693
005 STR append (1MB + 1KB)                    15195    7716
006 STR append (1MB + 1KB * 20)               15373    7955
007 STR append (1MB + 1KB * 1000)             25950   19695
008 STR append (1MB + 1MB * 3)                37702   30938
009 STR append (1MB + 1MB * 5)                80795   38287
010 STR append (1MB + (1b + 1K + 1b) * 100)   16251    9297
END VERSIONS:                               1:8.4a1 2:8.4a2
FINISHED Thu Aug 31 10:18:49 PDT 2000

Note that I have modified 'string repeat', which builds the base strings,
to build exact count strings.  This is another performance improvement for
'string repeat' which I'll elaborate on later (and isn't yet checked in).

Jeffrey Hobbs                     Tcl Ambassador
ho...@Aj...          Ajuba Solutions (née Scriptics)

> -----Original Message-----
> From: Donald G Porter [mailto:dg...@ca...]
> Sent: Thursday, August 31, 2000 8:14 AM
> To: tc...@po...
> Subject: [TCLCORE] Re: Hintermayer's string allocation patch
>
>
>
> > 2.  Tcl is faster in all append cases.
>
> All the cases you tested.
>
> Add this to string.bench and check the results:
>
> bench -desc "STR append (1MB + 1KB * 1000)" \
> 	-body {str-append $base $kilobyte 1000}
>
> Seems to me the gradual growth of a large string to a much larger
> string in lots of little pieces is common enough the benchmark
> suite ought to test that too.

--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

[TCLCORE] Re: Hintermayer's string allocation patch

[George H. <ga...@si...>]

 > This proposed change makes me nervous, because it results in
 > O(N**2) behavior for a long series of appends.  Suppose you add
 > a large number of additional blocks of data, each 3*TCL_GROWTH_MIN_ALLOC
 > bytes long, to an existing large string.  Each append will cause
 > the string to be reallocated and the entire existing contents to be
 > copied.  This should produce very slow behavior when the string is
 > large.  The reason for doubling the string size each time it's
 > reallocated is that it guarantees that on average each byte is only
 > copied about twice, no matter how large the string grows.
 > 
 > I'm confused by the performance measurements; either there's something
 > I don't understand, or the new approach should work badly in the
 > case of large strings being appended to gradually.  Also, I don't see
 > why appends to short strings should improve in performance with the
 > new algorithm; has something else changed in the new version besides
 > what's described in your message?
 > 
 > Lastly, I don't understand the comment about memory running out;
 > virtually all machines where Tcl runs have virtual memory, no?  This means
 > you can exceed the physical memory size in your allocations.  By the time
 > you get near to exceeding the swapping space you have already started
 > thrashing so badly that the system is unusable anyway.
 > 
 > I think there may be a much better way to fix this problem.  Why not
 > take advantage of the Tcl object system?  When appending to a string
 > object, don't keep reallocating a single large buffer.  Instead, create
 > a chain of fixed-size buffers as a special internal representation.
 > When the value is referenced as a string, then you can convert it into
 > a single large buffer.  This approach should be much faster in the
 > common case where a string is appended to frequently without ever
 > being referenced.  It may be slightly slower if appends and references
 > are intermixed, but I bet it won't be much slower even in this extreme
 > case.

This is hopefully the kind of problem/solution that should benefit
from having more eyes to examine it.   

It would help to see the patch, but I agree with John here.  Slowing
the allocation rate just makes amortized cost of each insert more
expensive.  You still have to copy the data from the old memory into
the new.  And with smaller allocations, you will simply do that more
often.

I'd really like to see the original problem.  I looked at the thread
and I couldn't find it.  I want to know what problem the Tcl code (not
C code) is attempting to solve and can't (because of the current
allocation scheme).  Where and why is he allocating a string so large
that it exceeds his virtual memory?  Can this string be broken into
multiple parts?

It would help to see the benchmarks too.  I don't know what they are
really measuring.  I'm suspicious of the different versions too.

I wonder if you are fixing a theoretical case at the expense of real
performance.  In practice, you can't run near the virtual memory limit
because of the places in the code where is no check is made for memory
exhaustion.  

There may still be good things in this patch (outside slowing the
growth rate).  But I can't tell from either the description of the
algorithm or by the timing numbers.

--gah



--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

[TCLCORE] [trace execute] for [onexit callback]

[Eric M. <er...@aj...>]

On Thu, 31 Aug 2000, Donald G Porter wrote:

> From: Brent Welch <we...@aj...>
> > By the way, isn't the big picture here that we need an
> > 
> > onexit callback
> > 
> > command so that Tcl-level applications can take action on exit?
> 
> Couldn't this just be an application of Vince Darley's proposed
> [trace execute] command?

This depends on which vision of [trace execution] you are referring to.
With Vince's original patch, no; with the latest stuff he sent to me,
maybe.  Vince can probably answer better than I.

   Eric Melski                            The Other Tcl Guy
   ericm at ajubasolutions.com            Ajuba Solutions

--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

[TCLCORE] Re: Hintermayer's string allocation patch

[Eric M. <er...@aj...>]

On Thu, 31 Aug 2000, Donald G Porter wrote:

> > 2.  Tcl is faster in all append cases.  
> 
> All the cases you tested.
> 
> Add this to string.bench and check the results:
> 
> bench -desc "STR append (1MB + 1KB * 1000)" \
> 	-body {str-append $base $kilobyte 1000}

Does this case produce significantly different results from:

bench -desc "STR append (1MB + 3KB * 1000)" \
        -body {str-append $base $kilo3 1000}

The results of which I included in one of my posts here?  I have yet
committed this particular change to CVS, but you can see the results of
running this benchmark in my other post.

I believe that what Dan means by "Tcl is faster in all append cases" is
that Tcl, using ckrealloc and the new algorithm for large strings, is
faster than Tcl, using alloc/memcpy/free and the old algorithm.  That is,
the speed benefits from using ckrealloc outweigh the losses from using the
new growth algorithm for large strings, so you still have a net win.  I
think everybody here understands that the new algorithm is slower than the
old algorithm in a head-to-head comparison.  I don't really think that is
relevant.  Right now, Tcl cannot effectively use all of the available
memory.  In my opinion, this is, bluntly, stupid.  There's no reason why
we shouldn't be able to use all available memory.  Particularly when we
claim that you can (at least in newsgroup posts and in our own Tcl
training materials).

Sure, it may be that there are very few people who need to use very large
strings.  If this is your argument, then you have just made my point too:
this patch will not affect most people (if they are not using large
strings, they are using small strings, and will thus still be using the
old algorithm), but those people that are affected will be significantly,
positively impacted.  It may be that there are other ways for the user to
write their code so that they don't have to have a single big string, but
does anybody here honestly believe that at the Tcl level, the programmer
should be worrying about memory allocation schemes?

   Eric Melski                            The Other Tcl Guy
   ericm at ajubasolutions.com            Ajuba Solutions


--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

[TCLCORE] [Fwd: Maximum variable length in TCL]

[Dan K. <ku...@aj...>]

Here is Gerhards initial patch.  I think Eric made some
changes while integrating the patch, but I think they
mostly involved cleaning up the constants, etc.

--Dan

[TCLCORE] Re: Hintermayer's string allocation patch

[Dan K. <ku...@aj...>]

The initial problem from the newsgroup (from what I remember) was
that a user was planning on reading a large amount of data (from
a channel) into memory and wanted to know how much of his memory
tcl could use.  The response he got was 'amount of memory minus
some overhead' but when he wrote a test script using append he
found he could only use 128 K of his 256 K of memory.

This was because the tcl allocator was trying to double its
allocation whan a byte was appended to a 128K string and the
malloc would fail.  Currently when malloc fails in ckalloc
in tcl (the memory allocation wrapper) tcl will panic, because
the thought was that this means you have used all of your memory
and you can't really recover from the failure in a sane manor.

With the current allocation scheme this isn't true.  The greedy
allocation scheme works really well with small memory blocks
(it is more efficient to have less allocations), but as memory
grows large, you continue to double the space allocated.  This
means that if you are reading a lot of relatively large files
into memory using a fileevent and append, you are probably going
to waste a good chunk of memory with the current algorithm.

The intent of the patch was to do two things.  Make tcl more
memory efficient when the data in a variable is considered to
be 'big'.  This is a tunable that we need to find the right
size for.  Currently the old memory allocation schme is used
up to 1 MB but after that the new allocation algorithm is used.
The second change was to improve the performance of the
allocation by changing the allocation from a ckalloc to a 
ckrealloc.

In performance testing Eric found that simply using ckrealloc
instead of using ckalloc gave performance speedup across the
board.  Then when the new allocation algorithm was added
into the code, the times were the same for small allocations
and appends (as would be expected) and slower for doing
a series of large appends (as would be expected since in
this case, the new algorithm would start being used, and
the allocation would need to happend more frequently).

The net result of the two changes (the change to use ckrealloc
instead of ckalloc) and using the new memory allocation
algorithm was:

1.  Tcl is able to use a larger percentage of total memory
in *all* cases where strings over 1MB are built up using
append.  This is because the (new) allocation algorithm 
kicks in which is less greedy and more memory efficient.

2.  Tcl is faster in all append cases.  This is because the
change to using ckrealloc causes the code to be faster
than when ckalloc was used. There is speed up every where
append is done (even when it is done internal to the core).

It may make more sense to change the 'large string' size
from 1MB up to 4MB or so, but I am not sure.  I would
like it to be some size that is less than what the average
PC that is currently in use has.  I think that is probably
16 or 32MB (I am talking about common people, not corporate). 

I hope that helps.

--Dan

--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

[TCLCORE] Re: Hintermayer's string allocation patch

[Brent W. <we...@aj...>]

>>>Jim Ingham said:
 > 
 > On Wednesday, August 30, 2000, at 04:49 PM, Jeffrey Hobbs wrote:
 > 
 > > Yes and no.  Sure, the memory is just reserved, but the mem allocator 
 > > will still bomb if you ask for more space than it could promise.  The 
 > > point of the mem efficiency part is that we don't get too greedy about 
 > > needing so much more space than we need.  That way we can push the 
 > > boundaries of how much mem we will actually be able to use without 
 > > the mem allocator crapping out because we're overly greedy. 
 > >  
 > 
 > Yes, but the point of the last note was that what "more" is is pretty huge.  For instance, I th
     ink in general the allocator doesn't check whether a given allocation fits in the available s
     wap space but rather fills it as you actually write out pages to swap.  This is the right thi
     ng to do, since that way you can for instance allocate a large matrix, and if you only fill i
     t sparsely, you aren't limited by the size of the matrix, only by the pages you use.  MacOS X
     , does it this way, at least.  I imagine other Unixes work in much the same way...

Apparently MacOS is more advanced than UNIX in this area.  In UNIX you have
to be able to allocate the swap space to cover the virtual memory.  So, in practice,
if you have 300 Meg of swap space, 192 Meg of real memory, and a 160 Meg string,
you cannot append one byte to it in Tcl.

I also note that using realloc accounts for essentially all the performance speedup.
Changing the doubling behaviour after a certain limit makes
good sense, and it looks like using realloc may allow the system to optimize
the reallocation by doing VM page remapping and avoiding the big copy that John O
is worrying about.

--	Brent Welch	<we...@aj...>
	http://www.ajubasolutions.com
	Scriptics changes to Ajuba Solutions
	scriptics.com => ajubasolutions.com

--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

[TCLCORE] Re: Procedure/Namespace Behaviour

[Brent W. <we...@aj...>]

>>>Donald G Porter said:
 > So we have a command named [::B::foo] which has a body that evaluates
 > in namespace ::A.  [namespace import] makes that possible.  So, I 
 > reject the hypothesis that [rename] was designed to work the way it
 > does in service of a general principle that commands with names matching
 > [::B::*] always evaluate in namespace comtext ::B.  There is no such
 > principle, so that can't explain the behavior of [rename].  What does
 > explain it?
 > 
 > > But, I happen to know that the original implementation of rename didn't
 > > think hard about namespaces and so you got the behavior you request -
 > > the namespace scope was not changed by the rename. 
 > 
 > OK, now I think you're the one being unclear.  Are you saying that
 > [rename] used to behave differently with namespaces than it does in
 > Tcl 8.3.2 ?  What is the history then?

OK, I was confused about which side of the argument you were on, and the
namespace import threw me off.

I urge everyone to consider the following test program that sets a namespace
variable.  If the procedure is renamed to a different namespace, should it
modify the namespace variable in the new namespace, or continue to modify
the variable it was originally written to modify?

# cut here
namespace eval space1 {
    variable x
    proc setx_in_1 {value} {
	variable x
	set x $value
    }
}
namespace eval space2 {
    variable x
    proc setx_in_2 {value} {
	variable x
	set x $value
    }
}
space1::setx_in_1 x1
space2::setx_in_2 x2

puts [list space1::x $space1::x]
puts [list space2::x $space2::x]

puts rename
rename space2::setx_in_2 space1::setx_in_2
space1::setx_in_2 5

puts [list space1::x $space1::x]
puts [list space2::x $space2::x]
# end

I wrote this up, confidently executed it, and was blown away when 
the space1::x variable was modified!  As noted, this means you have
to know way to much about the internals of the namespaces to
safely rename procedures between namespaces.

As for the "history" of this issue, I know that when namespaces were
first being implemented in Tcl 8.0 that rename did not change the behavior of
a procedure inside a namespace.  Frankly, these are the only semantics that
make sense to me.  With the current semantics you cannot safely rename
proceudres between namespaces without knowing everything about the
internals of those namespaces.

--	Brent Welch	<we...@aj...>
	http://www.ajubasolutions.com
	Scriptics changes to Ajuba Solutions
	scriptics.com => ajubasolutions.com

--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

[TCLCORE] Re: Hintermayer's string allocation patch

[Jim I. <ji...@ap...>]

Eric,

On Wednesday, August 30, 2000, at 06:21 PM, Eric Melski wrote:

>  
> On Wed, 30 Aug 2000, Jim Ingham wrote: 
>  
> > As a test, try this little program: 
> >  
> > int 
> > main (int argc, char **argv) 
> > { 
> >   int bigsize = 0x5fffffff; 
> >   char * foo = (char *) malloc(bigsize); 
> >   if (!foo) { 
> >     printf("Oops, could not allocate %d bytes\n", bigsize); 
> >     return 1; 
> >   } else { 
> >     printf("I allocated %ud bytes\n", bigsize); 
> >   } 
> >  
> >   foo[500] = 'c'; 
> >   foo[0xffffff] = 'd'; 
> >   while (1) ; 
> >   return 0; 
> > } 
>  
> Again,  on Linux and Solaris, malloc returns 0 when the size requested 
> exceeds the available memory (physical memory + swap space).  When I 
> compile and run your program on Solaris I get: 
>  
> Oops, could not allocate 1610612735 bytes 
>  
> And on Linux: 
>  
> Oops, could not allocate 1610612735 bytes 
>  

Well, I am a bit happy about this.  Gdb (and thus yours truely) suffers a lot from having to cope with all the complexities of Mach + BSD, it is good to know that it really is good at some things :-)  Actually, the memory subsystem is pretty cool, you can do all sorts of wild things like allocate memory in another process's address space (provided you have the appropriate permissions) stuff junk there, and then tell the other process about it...  And since it can both deal with sparse memory regions, and dynamic swap sizes, it is good at handling process that need a lot of address space and/or memory.

>  
>  
> The performance penalty for using the new algorithm seems to be fairly 
> minimal, as the numbers that I posted earlier showed.  In addition, we can 
> tune TCL_GROWTH_LARGE_STRING up to a larger value if necessary, retaining  
> the original doubling algorithm for a longer period but still allowing 
> for growth of very large string.  I'm not dead set on using one megabyte 
> for that threshhold.  Perhaps 2 MB or 8 MB or 10 MB would be better. 
>  

Yeah, seems like you are getting into pathological cases anyway, even on Windows, so stepping the size up so you only hit the limit in pathological cases is probably not a bad idea.

Jim
--
Jim Ingham                                   ji...@ap...
Developer Tools - gdb
Apple Computer

--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

[TCLCORE] Thread Tutorials

[Andreas K. <a.k...@we...>]

http://www-4.ibm.com/software/developer/library/posix1.html
http://www-4.ibm.com/software/developer/library/posix2.html

Something we may want to link to from the tcl thread documentation.
It mentions Tcl too, in the resource section. Pthreads-Tcl actually.
We should tell the author about the thread-safety we have since 8.3.

-- 
So long,
	Andreas Kupries <a.k...@we...>
			<http://www.purl.org/NET/akupries/>
-------------------------------------------------------------------------------

--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

[TCLCORE] Re: Hintermayer's string allocation patch

[Eric M. <er...@aj...>]

On Wed, 30 Aug 2000, Jim Ingham wrote:

> As a test, try this little program:
> 
> int
> main (int argc, char **argv)
> {
>   int bigsize = 0x5fffffff;
>   char * foo = (char *) malloc(bigsize);
>   if (!foo) {
>     printf("Oops, could not allocate %d bytes\n", bigsize);
>     return 1;
>   } else {
>     printf("I allocated %ud bytes\n", bigsize);
>   }
> 
>   foo[500] = 'c';
>   foo[0xffffff] = 'd';
>   while (1) ;
>   return 0;
> }

Again,  on Linux and Solaris, malloc returns 0 when the size requested
exceeds the available memory (physical memory + swap space).  When I
compile and run your program on Solaris I get:

Oops, could not allocate 1610612735 bytes

And on Linux:

Oops, could not allocate 1610612735 bytes



The performance penalty for using the new algorithm seems to be fairly
minimal, as the numbers that I posted earlier showed.  In addition, we can
tune TCL_GROWTH_LARGE_STRING up to a larger value if necessary, retaining 
the original doubling algorithm for a longer period but still allowing
for growth of very large string.  I'm not dead set on using one megabyte
for that threshhold.  Perhaps 2 MB or 8 MB or 10 MB would be better.

   Eric Melski                            The Other Tcl Guy
   ericm at ajubasolutions.com            Ajuba Solutions

--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

[TCLCORE] Re: Hintermayer's string allocation patch

[Jeffrey H. <jef...@aj...>]

> From: Jim Ingham [mailto:ji...@ap...]
	...
> On Wednesday, August 30, 2000, at 04:49 PM, Jeffrey Hobbs wrote:
> 
> > Yes and no.  Sure, the memory is just reserved, but the mem allocator 
> > will still bomb if you ask for more space than it could promise.  The 
> > point of the mem efficiency part is that we don't get too greedy about 
> > needing so much more space than we need.  That way we can push the 
> > boundaries of how much mem we will actually be able to use without 
> > the mem allocator crapping out because we're overly greedy. 
> >  
> 
> Yes, but the point of the last note was that what "more" is is pretty 
> huge.  For instance, I think in general the allocator doesn't check 
> whether a given allocation fits in the available swap space but 
> rather fills it as you actually write out pages to swap.  This is the 
> right thing to do, since that way you can for instance allocate a 
> large matrix, and if you only fill it sparsely, you aren't limited by 
> the size of the matrix, only by the pages you use.  MacOS X, does it 
> this way, at least.  I imagine other Unixes work in much the same way...

It's always a bummer about those theories, because Unix really does limit
you to known available mem space (note that 1610612735 == 0x5fffffff):

nerja [~/tmp/mem] 183 % gcc -o testmem mem.c
nerja [~/tmp/mem] 184 % testmem 1610612735
Oops, could not allocate 1610612735 bytes
nerja [~/tmp/mem] 185 % testmem 810612735
Oops, could not allocate 810612735 bytes
nerja [~/tmp/mem] 186 % testmem 410612735
Oops, could not allocate 410612735 bytes
nerja [~/tmp/mem] 187 % testmem 210612735
I allocated 210612735d bytes
^C
nerja [~/tmp/mem] 188 % cat mem.c
int
main (int argc, char **argv)
{
  int bigsize = 0x5fffffff;
  char * foo;
  if (argc == 2) {
      bigsize = atoi(argv[1]);
  }
  foo = (char *) malloc(bigsize);
  if (!foo) {
    printf("Oops, could not allocate %d bytes\n", bigsize);
    return 1;
  } else {
    printf("I allocated %ud bytes\n", bigsize);
  }

  foo[500] = 'c';
  foo[0xffffff] = 'd';
  while (1) ;
  return 0;
}

With the last, you get 480K resident on a 200+MB size program from top.
This is a Solaris/x86 machine.  You could jack up your swap space quite
a bit, but that'll get you into trouble for other reasons if someone
actually got around to using it.

Jeffrey Hobbs                     Tcl Ambassador
ho...@Aj...          Ajuba Solutions (nee Scriptics)

--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

[TCLCORE] Re: Hintermayer's string allocation patch

[Jim I. <ji...@ap...>]

On Wednesday, August 30, 2000, at 04:49 PM, Jeffrey Hobbs wrote:

> Yes and no.  Sure, the memory is just reserved, but the mem allocator 
> will still bomb if you ask for more space than it could promise.  The 
> point of the mem efficiency part is that we don't get too greedy about 
> needing so much more space than we need.  That way we can push the 
> boundaries of how much mem we will actually be able to use without 
> the mem allocator crapping out because we're overly greedy. 
>  

Yes, but the point of the last note was that what "more" is is pretty huge.  For instance, I think in general the allocator doesn't check whether a given allocation fits in the available swap space but rather fills it as you actually write out pages to swap.  This is the right thing to do, since that way you can for instance allocate a large matrix, and if you only fill it sparsely, you aren't limited by the size of the matrix, only by the pages you use.  MacOS X, does it this way, at least.  I imagine other Unixes work in much the same way...

So again, you are limited by the size of the address space, which means that on a 32 bit machine, you won't have problems until you have already allocated 1 Gig, and are going to write a little more beyond that.  I think you will be having lots of other problems on most machines long before you get to this point...

I don't see paying any performance cost to allow filling a 1 Gig string, and again, if you really need to do this, get a 64 bit machine...  With the caveat that WinNT or Win95 may do this in some more brain-dead fashion, and we may need to accommodate them...

As a test, try this little program:

int
main (int argc, char **argv)
{
  int bigsize = 0x5fffffff;
  char * foo = (char *) malloc(bigsize);
  if (!foo) {
    printf("Oops, could not allocate %d bytes\n", bigsize);
    return 1;
  } else {
    printf("I allocated %ud bytes\n", bigsize);
  }

  foo[500] = 'c';
  foo[0xffffff] = 'd';
  while (1) ;
  return 0;
}

I ran it, and it runs fine, then I run top, and get a virtual size of 1.5 Gig (I have 128 Meg of RAM, MacOS X doesn't use a swap partition, but rather uses space in /var/vm, so you have up to the size of the partition var is mounted on, but this is ~800Meg for me...).  As you would imagine, the resident size is 504 K...  I got NULL back when I ask for ~1.8Gig, so the estimate I gave in the last note is a little high, but still not worth worrying about.

Jim

--
Jim Ingham                                   ji...@ap...
Developer Tools - gdb
Apple Computer

--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

[TCLCORE] Re: Hintermayer's string allocation patch

[Eric M. <er...@aj...>]

On Wed, 30 Aug 2000, Jim Ingham wrote:

> 
> Most virtual memory systems give an address space to the application
> which is governed, not by the available memory on the machine, but by
> the size of the address space.  Actually, the available space is usually
> less than the absolute max, since the stack is up at the top, and most
> systems don't start it at the tip-top of the address space.  For
> instance MacOS X on a 32 bit processor loads the stack starting around
> 0xc000000 for some reason.  Still this is two Gig less the code space
> and stack space for the app for a 32 bit machine.  So the
> "totalMemorySize" in your above equation is at least 2Gig or so big on a
> 32 bit machine.  I don't think we need to worry too much about that; a
> Tcl programmer who reads more than a Gig into a single string is
> probably doing something wrong anyway, and if they really need to do
> that they should go out and buy themselves a 64 bit machine...  And we
> probably won't be running Tcl on any more 16 bit processors in the near
> future, most of the hand-held devices like Palm, etc use 32 bit
> processors.

This may be true on MacOS X, but a little experiment on my Linux box shows
that the memory allocation functions more the way that I expect than the
way that you expect.  This is a 32-bit machine, so yes, I can address 2 GB
of memory.  However, I have 192MB of physical memory plus 256 MB of swap
space; when I try to allocate more than this amount of memory, malloc
fails.  It doesn't matter that I can address 2 GB of memory; my system
just doesn't have the memory to give me.

With the original code, if I had a 230MB string (unusual, perhaps, but not
completely unrealistic), I would be unable to append even a single byte to
it, simply because Tcl would try to allocate 460 MB, and on my system,
that would fail.  With the new code, I can happily append all the way up
to the entirety of my available memory.

The code I used to test my hypothesis:

#include <stdio.h>

int main() {
        char *c;
        c = (char *)malloc(/* 460 Megabytes */ 482344960);
        printf("return from malloc: %X\n", c);
        return 0;
}

Compiled and executed like so:

gcc -o foo foo.c
./foo

produces this result:

return from malloc: 0

   Eric Melski                            The Other Tcl Guy
   ericm at ajubasolutions.com            Ajuba Solutions

--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

[TCLCORE] Re: Procedure/Namespace Behaviour

[Donald G P. <dg...@ca...>]

> >>>Donald G Porter said:
>  > Certainly there's no problem with command named [::B::foo] evaluating in
>  > the namespace context ::A, since that's what [namespace import] does.
> 
> Nope - I don't think so.  namespace import creates a short alias for the 
> command, it certainly does not affect the namespace scope in which it
> executes.

Right.  So here's Tcl code to express clearly what I tried to say:

% namespace eval ::A {namespace export foo}
% proc ::A::foo {} {puts "foo in [namespace current]"}
% namespace eval ::B {namespace import ::A::foo}
% ::B::foo
foo in ::A

So we have a command named [::B::foo] which has a body that evaluates
in namespace ::A.  [namespace import] makes that possible.  So, I 
reject the hypothesis that [rename] was designed to work the way it
does in service of a general principle that commands with names matching
[::B::*] always evaluate in namespace comtext ::B.  There is no such
principle, so that can't explain the behavior of [rename].  What does
explain it?

> But, I happen to know that the original implementation of rename didn't
> think hard about namespaces and so you got the behavior you request -
> the namespace scope was not changed by the rename. 

OK, now I think you're the one being unclear.  Are you saying that
[rename] used to behave differently with namespaces than it does in
Tcl 8.3.2 ?  What is the history then?

> By the way, isn't the big picture here that we need an
> 
> onexit callback
> 
> command so that Tcl-level applications can take action on exit?

While that might be nice to have (although I would make it a low
priority), Tcl has namespaces and Tcl has [rename].  Figuring out
the right way they ought to work together is "big picture" enough
for me.

DGP

--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

[TCLCORE] Re: Hintermayer's string allocation patch

[Jeffrey H. <jef...@aj...>]

> From: Jim Ingham [mailto:ji...@ap...]
	...
> Moreover, when you allocate a block of memory, all that happens is 
> that that chunk of your address space is reserved, no  real memory or 
> swap is set aside for it until you actually touch one of the pages.  
> So again, the act of doubling a 100 Meg string, for instance, doesn't 
> require that you have 200 Meg of swap or memory.  Only the act of 
> actually writing to all 200 Meg of it will do that.  And no algorithm 
	...

Yes and no.  Sure, the memory is just reserved, but the mem allocator
will still bomb if you ask for more space than it could promise.  The
point of the mem efficiency part is that we don't get too greedy about
needing so much more space than we need.  That way we can push the
boundaries of how much mem we will actually be able to use without
the mem allocator crapping out because we're overly greedy.

The other suggestion in the thread was a fall-back allocation mechanism,
whereby one still goes for *2 all the time, until malloc says, "No".
Then we would try for just what we need (or a bit more), and see if that
works.  However, that requires changing around the ckalloc routines,
because they panic by default when malloc returns NULL.  You'd have to
create some sort of alternative like:

	bytes = (char *) ckallocEx(howMuchIWant, howMuchINeed);

Jeffrey Hobbs                     Tcl Ambassador
ho...@Aj...          Ajuba Solutions (nee Scriptics)

--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.

[TCLCORE] Re: Hintermayer's string allocation patch

[Jim I. <ji...@ap...>]

On Wednesday, August 30, 2000, at 03:06 PM, Eric Melski wrote:

> John: with respect to memory running out, yes, all machines that Tcl runs 
> on will likely have virtual memory.  There is no argument that once you 
> start allocating large blocks of virtual memory, your application will 
> become basically useless.  This does not alter the fact that with the 
> original algorithm, you could not allocate, in a single Tcl obj, the 
> entirety of available memory, unless you were very very lucky and 
> happened to start with an object that was  
> (totalMemorySize/2)-(aFewBytes*2) and you appended aFewBytes.  Also, when 
> you had a very large data object, you also would likely have a very large 
> chunk of essentially wasted memory.  This patch enables you to more easily 
> use more available memory, and wastes far less memory. 
>  
>  

I guess I still don't understand the force of your argument.  

Most virtual memory systems give an address space to the application which is governed, not by the available memory on the machine, but by the size of the address space.  Actually, the available space is usually less than the absolute max, since the stack is up at the top, and most systems don't start it at the tip-top of the address space.  For instance MacOS X on a 32 bit processor loads the stack starting around 0xc000000 for some reason.  Still this is two Gig less the code space and stack space for the app for a 32 bit machine.  So the "totalMemorySize" in your above equation is at least 2Gig or so big on a 32 bit machine.  I don't think we need to worry too much about that; a Tcl programmer who reads more than a Gig into a single string is probably doing something wrong anyway, and if they really need to do that they should go out and buy themselves a 64 bit machine...  And we probably won't be running Tcl on any more 16 bit processors in the near future, most of the han!
d-held devices like Palm, etc use 32 bit processors.

Moreover, when you allocate a block of memory, all that happens is that that chunk of your address space is reserved, no  real memory or swap is set aside for it until you actually touch one of the pages.  So again, the act of doubling a 100 Meg string, for instance, doesn't require that you have 200 Meg of swap or memory.  Only the act of actually writing to all 200 Meg of it will do that.  And no algorithm will get around having to have swap space (at least) for the all the pages you actually touch.

This sort of fix would be great for classic MacOS, where you have an admittedly pretty lame memory allocation scheme.  But this doesn't strike me as a practical concern for Unix, and I doubt that it is for NT either.  I have no idea how good the Win9x Virtual memory scheme is - I can imagine there are all sorts of little DOS-bred goblins hiding in the corners, but one can always hope...

Jim
--
Jim Ingham                                   ji...@ap...
Developer Tools - gdb
Apple Computer

--
The TclCore mailing list is sponsored by Ajuba Solutions
To unsubscribe:  email tcl...@aj... with the 
                 word UNSUBSCRIBE as the subject.