tcl-soc2008 — General mailing list for Google SOC 2008 Tcl discussion

Re: [Tcl-soc2008] Possibly another bug in Tcl?

[Daniel A. S. <da...@us...>]

On 13/07/2008, at 1:12, Remigiusz Jan Andrzej Modrzejewski wrote:

> I'm too tired to be sure about this, but it seems I've found two  
> things:
>
> 1) The hash table actually gets automagically deleted.

probably that's only the case during process exit though? but not when  
a slave interp is destroyed for instance? i.e. you would only need to  
delete the hashtable manually in the AssocData destructor, not in the  
exit handler...

> 2) No matter what I do one leak just is there:
>
> #v+
> -bash-3.2# cat helo.tcl
> puts helo
> -bash-3.2# bcheck -q /opt/tcldebug/bin/tclsh8.5 helo.tcl
> helo
> -bash-3.2# cat tclsh8.5.errs
>
> Actual leaks report    (actual leaks:            1  total size: 1072  
> bytes)
>
> <rtc> Memory Leak (mel):
> Found leaked block of size 1072 bytes at address 0x8078d60
> At time of allocation, the call stack was:
>        [1] _nss_XbyY_buf_alloc() at 0xe7d64235
>        [2] get_pwbuf() at 0xe7d50a2f
>        [3] _getpwuid() at 0xe7d50a58
>        [4] TclpGetPwUid() at 0xee1e2086
>        [5] TclpSetVariables() at 0xee1e1cce
>        [6] Tcl_CreateInterp() at 0xee12ec2c
>        [7] Tcl_Main() at 0xee1a2aa7
>        [8] main() at line 87 in "tclAppInit.c"
>
>
>
> Possible leaks report  (possible leaks:          0  total size:  0  
> bytes)
> #v-
>
> A bug in current Tcl?


looks like an unavoidable leak in the OS getpwuid() implementation,  
I'm guessing it internally allocates storage for the passwd struct and  
returns a pointer to that and there is no way to free that storage  
(but it gets reused in the next call to getpwuid()). In the Mac OS X  
getpwuid manpage that behaviour is even cited in BUGS.

in short, nothing to worry about...

Cheers,

Daniel

-- 
** Daniel A. Steffen                   **
** <mailto:da...@us...>  **

Re: [Tcl-soc2008] Coding... So refreshing :)

[Kevin K. <kk...@ny...>]

Daniel A. Steffen wrote:
> Hi Remigiusz,
> 
> On 12/07/2008, at 11:35, Remigiusz Jan Andrzej Modrzejewski wrote:
> 
>> Daniel A. Steffen wrote:
>>> yes, I wondered about that, I don't understand where your  
>>> Tcl_CreateNamespace linkage is coming from at all in fact, is there  
>>> some hidden linking to -ltcl going on ? what unknown symbols does  
>>> 'nm' report? certainly libtclstub8.4 does not export  
>>> Tcl_CreateNamespace ...
>> It's coming from the binary loading dtrace.so. In fact I've just  
>> trimmed the cc line to:
>> #v+
>> cc -Kpic -G -g -z text  -o dtrace.so dtrace.c -L/usr/lib -ldtrace
>> #v-
>> Still works flawlessly...
> 
> that's because you're not using '-z defs', the recommended linking  
> mode for shared libs (c.f. sun ld manpage), which does not allow  
> undefined symbols.
> If you add that flag, you'll see that you also need to add -lc and - 
> ltcl8.4 to your compile line, i.e. the following works for me
>      /opt/SunStudioExpress/bin/cc -G -z defs -z text -Kpic -g - 
> Isolaris -o dtrace.so dtrace.c -lc -ltcl8.4 -ldtrace
> and for gcc, I need the following
>      /usr/gnu/bin/gcc -shared -fpic -g -std=gnu99 -Isolaris -o  
> dtrace.so dtrace.c -lc -ltcl8.4 -ldtrace

That might work, but certainly some linkers require libraries
to appear in dependency order.  -ltcl8.4 -ldrtrace -lc would
be a more prudent sequence.

> ah, yes, I remember now, Tcl_Main calls [exit] at the end of the  
> program...
> try again with [exit] redefined to nothing and the AssocData  
> destructor should be called (Tcl_Main calls Tcl_DeleteInterp() if  
> [exit] returns)

And if tracing memory leaks is of interest, also compile
Tcl with -DPURIFY (or its custom small-block allocator will
get in the way), and without -DTCL_MEM_DEBUG (because of internal
inconsistencies that we'll get sorted out one of these days).

-- 
73 de ke9tv/2, Kevin

Re: [Tcl-soc2008] Coding... So refreshing :)

[Daniel A. S. <da...@us...>]

Hi Remigiusz,

On 12/07/2008, at 11:35, Remigiusz Jan Andrzej Modrzejewski wrote:

> Daniel A. Steffen wrote:
>> yes, I wondered about that, I don't understand where your  
>> Tcl_CreateNamespace linkage is coming from at all in fact, is there  
>> some hidden linking to -ltcl going on ? what unknown symbols does  
>> 'nm' report? certainly libtclstub8.4 does not export  
>> Tcl_CreateNamespace ...
>
> It's coming from the binary loading dtrace.so. In fact I've just  
> trimmed the cc line to:
> #v+
> cc -Kpic -G -g -z text  -o dtrace.so dtrace.c -L/usr/lib -ldtrace
> #v-
> Still works flawlessly...

that's because you're not using '-z defs', the recommended linking  
mode for shared libs (c.f. sun ld manpage), which does not allow  
undefined symbols.
If you add that flag, you'll see that you also need to add -lc and - 
ltcl8.4 to your compile line, i.e. the following works for me
     /opt/SunStudioExpress/bin/cc -G -z defs -z text -Kpic -g - 
Isolaris -o dtrace.so dtrace.c -lc -ltcl8.4 -ldtrace
and for gcc, I need the following
     /usr/gnu/bin/gcc -shared -fpic -g -std=gnu99 -Isolaris -o  
dtrace.so dtrace.c -lc -ltcl8.4 -ldtrace

BTW, I think it might be better to avoid depending on C99 syntax, not  
sure if we can assume it's available everywhere we might want to  
target...

the -Isolaris is needed because 2008.05 does not come with libproc.h,  
so I had to add that locally (similarly to the darwin headers we'll  
need on Mac OS X)

>> my Solaris (opensolaris 2008.05) comes with all the gnu tools, gcc  
>> is the default, and it has gmake etc. I know this is not the case  
>> on older solaris, what release are you using?
>
> #v+
> -bash-3.2$ cat /etc/release
>                   Solaris Express Community Edition snv_86 X86
>           Copyright 2008 Sun Microsystems, Inc.  All Rights Reserved.
>                        Use is subject to license terms.
>                             Assembled 27 March 2008
> #v-

ok, 2008.05 is slightly newer, snv_86_rc3, assembled 26 april

> But that's not the point. There is the GNU userland available. But  
> the Sun toolchain is popularly perceived superior.

not so sure that is true anymore, AFAICT the whole of opensolaris  
seems to move gradually toward using gcc...

in any case, I agree that we want to be compatible with the sun as  
well as the gcc toolchain.

> In fact after using it a bit I can say, that Sun CC has better  
> warnings and bcheck does it's work without being as slow as  
> valgrind. And sampleextension buildsystem actually chooses the Sun  
> toys ;)

not on my 2008.05:
	$ which cc
	/usr/gnu/bin/cc
	$ cc --version
	cc (GCC) 3.4.3 (csl-sol210-3_4-20050802)
I have to manually force sun cc use by setting CC=/opt/ 
SunStudioExpress/bin/cc

>> probably depends on how you're exiting from the script, as  
>> mentioned, via [exit], the interp is not destroyed.
>
> Actually that was by EOF ;)


ah, yes, I remember now, Tcl_Main calls [exit] at the end of the  
program...
try again with [exit] redefined to nothing and the AssocData  
destructor should be called (Tcl_Main calls Tcl_DeleteInterp() if  
[exit] returns)

even given this, I think we will still want to have both methods of  
cleanup, TclDTrace may have been loaded into a slave interp (or an  
interp on a separate thread), so memory should be freed and  
dtrace_close() called when those interps are destroyed (indeed loading  
into a slave interp and destroying would be another easy way to test  
that the AssocData destructor is called).
It occurred to me that in addition to SIGABRT we may also need to  
handle SIGINT and maybe other signals, to ensure that grabbed  
processes are released in all cases...

Cheers,

Daniel

-- 
** Daniel A. Steffen                   **
** <mailto:da...@us...>  **

[Tcl-soc2008] Fwd: Coding... So refreshing :)

[Daniel A. S. <da...@us...>]

oops, forgot to CC the list on these:

Begin forwarded message:

> From: Remigiusz Jan Andrzej Modrzejewski <lr...@go...>
> Date: 12 July 2008 1:41:41
> To: "Daniel A. Steffen" <da...@us...>
> Subject: Re: Coding... So refreshing :)
>
> Daniel A. Steffen wrote:
>> the stubs system avoids having to link your extension to libtcl.so  
>> (which is where Tcl_CreateNamespace is presumably currently coming  
>> from now?).
>
> Hmmm, I have to disable -ltclstub8.4? Whole compiling looks like:
> #v+
> cc -Kpic -O -G -g -z text  -o dtrace.so dtrace.c -L/usr/lib - 
> ltclstub8.4 -lproc -lctf -lelf -ldtrace -DUSE_TCL_STUBS
> #v-
>
>> As Donal mentioned, it is worth considering whether we want to  
>> maintain 8.4 compatibility at all
>
> I feel that we have to. It's the version shipped with Solaris... And  
> this system seems to have a glacial speed of updates in the wild.  
> Dropping 8.4 support would be more or less dropping a really  
> considerable part of the audience.
>
>> the point would be to have a custom Tcl_Obj type for our dtrace  
>> handles, this would help efficiency, because you could cache the  
>> handle_data pointer directly in the Tcl_Obj's otherValuePtr and  
>> thus avoid the hash table lookup (except in the uncommon case where  
>> the internal representation has been lost). This is only an  
>> optimization however and so can be left till later if you prefer.
>
> As I already got that into a separate function, switching to the  
> full fledged Tcl_Obj alternative will be a drop-in replacement. So  
> I'll prefer to do that later, to get any usefulness earlier.
>
>>> For now I've just switched to int handles, which work, but  
>>> probably aren't that safe.
>> yes, I think the previous format of having "dtrace%d" handles was  
>> nicer, you could just use those strings as hash table keys instead  
>> of the ints you use now.
>
> It doesn't matter this much. Eventually I'm going to swicht to  
> Tcl_Obj specific to TclDtrace handles and I'd be most happy to get  
> any users to scream on me about that in the meantime ;)
>
>>> I've changed the code a bit to be more close to your suggestions  
>>> and the Tcl/TK Engineering Manual
>> do you mind if I fix up any remaining formatting nits as I read  
>> through the code in detail (tomorrow) and commit the changes (if  
>> any)?
>
> I'm afraid I do mind. I already put a bit of strain on my eyes  
> getting so far from my way of doing things. A patch would be  
> preferable, for me to consider all the proposed changes.
>
>> I will also have a look tomorrow at getting the current state of  
>> things to build&run on Mac OS X, which will make it easier for me  
>> to run the code on an ongoing basis without having to start up the  
>> Solaris VM...
>
> That probably will involve some happy coding. Especially around the  
> uber-simple Makefile I did. BTW, you know that Solaris has it's own  
> extremely basic make command? It like doesn't support anything...
>
>> there are several options, one is indeed to use an exit handler as  
>> you have done now, but the better option IMO would be to do the  
>> cleanup when the interp where the handles are relevant is  
>> destroyed, i.e. when the assoc data setup via Tcl_SetAssocData() is  
>> freed (c.f. delProcPtr param to Tcl_SetAssocData()).
>
> I tried that, but it somehow didn't work. The function just didn't  
> fire when running the script (tried an explicit printf() at the  
> beginning).
>
>> BTW, you probably also want to free the hash table in DeInit (via  
>> Tcl_DeleteHashTable()).
>
> Probably, yes. I've been wondering whether this gets done  
> automagically. Or is it a part of the 33 'possible leaks'. I guess  
> that the custom built Tcl you proposed will help a lot.


Begin forwarded message:

> From: "Daniel A. Steffen" <da...@us...>
> Date: 12 July 2008 2:12:22
> To: Remigiusz Jan Andrzej Modrzejewski <lr...@go...>
> Subject: Re: Coding... So refreshing :)
>
>
> On 12/07/2008, at 1:41, Remigiusz Jan Andrzej Modrzejewski wrote:
>
>> Daniel A. Steffen wrote:
>>> the stubs system avoids having to link your extension to libtcl.so  
>>> (which is where Tcl_CreateNamespace is presumably currently coming  
>>> from now?).
>>
>> Hmmm, I have to disable -ltclstub8.4? Whole compiling looks like:
>> #v+
>> cc -Kpic -O -G -g -z text  -o dtrace.so dtrace.c -L/usr/lib - 
>> ltclstub8.4 -lproc -lctf -lelf -ldtrace -DUSE_TCL_STUBS
>> #v-
>
> yes, I wondered about that, I don't understand where your  
> Tcl_CreateNamespace linkage is coming from at all in fact, is there  
> some hidden linking to -ltcl going on ? what unknown symbols does  
> 'nm' report? certainly libtclstub8.4 does not export  
> Tcl_CreateNamespace ...
>
> maybe easier to figure out if you use your own tcl built from  
> scratch as mentioned...
>
>>> As Donal mentioned, it is worth considering whether we want to  
>>> maintain 8.4 compatibility at all
>>
>> I feel that we have to. It's the version shipped with Solaris...  
>> And this system seems to have a glacial speed of updates in the  
>> wild. Dropping 8.4 support would be more or less dropping a really  
>> considerable part of the audience.
>
> yes, that's a good point to consider. OTOH they don't have a tcl  
> with dtrace enabled in solaris yet either (and nor does Mac OS X  
> Leopard) so a lot of the potential users may want to get a more  
> recent tcl anyway...
>
>>> the point would be to have a custom Tcl_Obj type for our dtrace  
>>> handles, this would help efficiency, because you could cache the  
>>> handle_data pointer directly in the Tcl_Obj's otherValuePtr and  
>>> thus avoid the hash table lookup (except in the uncommon case  
>>> where the internal representation has been lost). This is only an  
>>> optimization however and so can be left till later if you prefer.
>>
>> As I already got that into a separate function, switching to the  
>> full fledged Tcl_Obj alternative will be a drop-in replacement. So  
>> I'll prefer to do that later, to get any usefulness earlier.
>
> that's fine
>
>>>> For now I've just switched to int handles, which work, but  
>>>> probably aren't that safe.
>>> yes, I think the previous format of having "dtrace%d" handles was  
>>> nicer, you could just use those strings as hash table keys instead  
>>> of the ints you use now.
>>
>> It doesn't matter this much. Eventually I'm going to swicht to  
>> Tcl_Obj specific to TclDtrace handles and I'd be most happy to get  
>> any users to scream on me about that in the meantime ;)
>
> switching to custom Tcl_Obj type will not involve changing the  
> format of the handles actually (i.e. the string representation will  
> be unchanged, only the internal representation will change), but  
> you're right that this can be changed later also
>
>>>> I've changed the code a bit to be more close to your suggestions  
>>>> and the Tcl/TK Engineering Manual
>>> do you mind if I fix up any remaining formatting nits as I read  
>>> through the code in detail (tomorrow) and commit the changes (if  
>>> any)?
>>
>> I'm afraid I do mind. I already put a bit of strain on my eyes  
>> getting so far from my way of doing things. A patch would be  
>> preferable, for me to consider all the proposed changes.
>
> sure, a patch is fine with me as well of course
>
>>> I will also have a look tomorrow at getting the current state of  
>>> things to build&run on Mac OS X, which will make it easier for me  
>>> to run the code on an ongoing basis without having to start up the  
>>> Solaris VM...
>>
>> That probably will involve some happy coding. Especially around the  
>> uber-simple Makefile I did. BTW, you know that Solaris has it's own  
>> extremely basic make command? It like doesn't support anything...
>
> my Solaris (opensolaris 2008.05) comes with all the gnu tools, gcc  
> is the default, and it has gmake etc. I know this is not the case on  
> older solaris, what release are you using?
>
>>> there are several options, one is indeed to use an exit handler as  
>>> you have done now, but the better option IMO would be to do the  
>>> cleanup when the interp where the handles are relevant is  
>>> destroyed, i.e. when the assoc data setup via Tcl_SetAssocData()  
>>> is freed (c.f. delProcPtr param to Tcl_SetAssocData()).
>>
>> I tried that, but it somehow didn't work. The function just didn't  
>> fire when running the script (tried an explicit printf() at the  
>> beginning).
>
> probably depends on how you're exiting from the script, as  
> mentioned, via [exit], the interp is not destroyed.
>
>>> BTW, you probably also want to free the hash table in DeInit (via  
>>> Tcl_DeleteHashTable()).
>>
>> Probably, yes. I've been wondering whether this gets done  
>> automagically. Or is it a part of the 33 'possible leaks'. I guess  
>> that the custom built Tcl you proposed will help a lot.
>
> for leak checking, I also find the patch below to be useful (waits  
> forever just before calling exit(), so the leaks tools can continue  
> to examine memory with symbols available etc).
> You also may need to redefine [exit] do do nothing if it is used in  
> your test scripts, i.e.
>    proc exit {} {}
>
> Cheers,
>
> Daniel
>
> -- 
> ** Daniel A. Steffen                   **
> ** <mailto:da...@us...>  **
>
> Index: generic/tclEvent.c
> ===================================================================
> RCS file: /cvsroot/tcl/tcl/generic/tclEvent.c,v
> retrieving revision 1.82
> diff -u -p -r1.82 tclEvent.c
> --- generic/tclEvent.c	13 Jun 2008 05:45:10 -0000	1.82
> +++ generic/tclEvent.c	12 Jul 2008 00:06:52 -0000
> @@ -107,6 +107,10 @@ typedef struct {
> static Tcl_ThreadCreateType NewThreadProc(ClientData clientData);
> #endif /* TCL_THREADS */
>
> +#ifdef PURIFY
> +int tclSleepAtExit = 0;
> +#endif
> +
> /*
>  * Prototypes for functions referenced only in this file:
>  */
> @@ -826,6 +830,16 @@ Tcl_Exit(
> 	 */
>
> 	Tcl_Finalize();
> +#ifdef PURIFY
> +	if (tclSleepAtExit) {
> +	    int t = 100000;
> +	
> +	    do {
> +		fprintf(stderr, "Exited, sleeping forever...\n");
> +		t = sleep(t);
> +	    } while (t);
> +	}
> +#endif
> 	TclpExit(status);
> 	Tcl_Panic("OS exit failed!");
>     }
> @@ -902,6 +916,12 @@ TclInitSubsystems(void)
> 	    TclInitEncodingSubsystem();	/* Process wide encoding init. */
> 	    TclpSetInterfaces();
>     	    TclInitNamespaceSubsystem();/* Register ns obj type  
> (mutexed). */
> +#ifdef PURIFY
> +	    if (getenv("TCL_SLEEP_AT_EXIT")) {
> +		tclSleepAtExit = 1;
> +		unsetenv("TCL_SLEEP_AT_EXIT");
> +	    }
> +#endif
> 	}
> 	TclpInitUnlock();
>     }
>
>

Re: [Tcl-soc2008] Coding... So refreshing :)

[Daniel A. S. <st...@ma...>]

On 12/07/2008, at 0:57, Daniel A. Steffen wrote:

> re DeInit() and the question you asked in review 1 about how to  
> close handles on tcl exit:
>
> there are several options, one is indeed to use an exit handler as  
> you have done now, but the better option IMO would be to do the  
> cleanup when the interp where the handles are relevant is destroyed,  
> i.e. when the assoc data setup via Tcl_SetAssocData() is freed (c.f.  
> delProcPtr param to Tcl_SetAssocData()).

just saw that you switched from doing exactly this to an exit handler  
in r28, I'm guessing the reason was to make sure that that  
dtrace_close() is called even if Tcl_DeleteInterp() is not (e.g. when  
[exit] is called) ?
if so, I'd recommend doing both, and as it appears important that  
dtrace_close() is _always_ called, maybe also install an atexit()  
handler and a SIGABRT signal handler (note that Tcl_Panic calls  
abort()...)

Cheers,

Daniel

-- 
** Daniel A. Steffen                   **
** <mailto:da...@us...>  **

Re: [Tcl-soc2008] Coding... So refreshing :)

[Daniel A. S. <da...@us...>]

Hi Remigiusz,

responding to several of your mails at once, sorry I was a bit busy  
yesterday and today, I hope the following covers everything you asked  
(that has not yet been answered by others):

the changes made in r21-r28 look very good BTW!

On 10/07/2008, at 0:28, Remigiusz Jan Andrzej Modrzejewski wrote:

>> - switch to compiling with stubs enabled (i.e. with - 
>> DUSE_TCL_STUBS); as mentioned earlier, I'd recommend switching to  
>> the TEA sampleextension buildsystem as soon as possible, that takes  
>> care of all that and more for you.
>
> What are the exact benefits of that? Code already works under both  
> 8.4 and 8.5. And I'd prefer to postpone that to the next milestone,  
> unless it solves some problem I will stumble upon now.

> -DUSE_TCL_STUBS didn't change anything observable.  
> Tcl_CreateNamespace() keeps on working with the extern declaration I  
> have in the header. That probably doesn't change the fact that I  
> should change to eval, but I'd like to understand the reasons first.

the stubs system avoids having to link your extension to libtcl.so  
(which is where Tcl_CreateNamespace is presumably currently coming  
from now?). This allows the extension to be loaded in circumstances  
where libtcl.so is not available, in particular into an executable  
that statically links with tcl. This is a very common case because  
tclkit and related single-file deployment technologies all statically  
link with tcl.
The other advantage is that you can load the same extension binary  
into different minor versions of tcl, the stubs system enables  
backwards and forwards compatibility, whereas linking with the shared  
library does not allow that (because major.minor version numbers are  
part of the shared library name).

As Donal mentioned, it is worth considering whether we want to  
maintain 8.4 compatibility at all, giving it up would take care of the  
Tcl_CreateNamespace() issue and have the big added advantage that we  
could use the Tcl Dict API (c.f. DictObj.3) to create/manipulate the  
various dicts that we'll need...

>> - don't use global state (i.e your handles and options arrays),  
>> state should be stored per-interp, I'd recommend in a ckalloc'd  
>> structure that you store in a tcl hash table keyed by the handle  
>> name, see e.g. Tcl_GetChannel() in tclIO.c
>
> OK with everything I cut out, but I somehow feel bad about using  
> hashes here. A voice from back of my head screams, that hashing  
> autoindex identifiers is wasteful ;)

with the Tcl_Obj optimization for handles (c.f. below), having to look  
things up in the hashtable will be uncommon, so the cost of that  
should not be a big concern, and the Tcl hash API gives you an easy  
premade way to stash away all the handle_data without having to manage  
storage manually (e.g. with the array you had, you might need to worry  
about extending it when space runs out, and how to deal with the  
unused space in the array when handles are freed etc)

>> - don't repeatedly convert from strings to handles and back, use  
>> the Tcl_Obj mechanism, see e.g. the way channel names are handled  
>> in the core: TclGetChannelFromObj(), SetChannelFromAny() in tclIO.c
>
> That was a hard read, and had to google for them first.

I would definitely recommend you get yourself a local copy of the tcl  
codebase from CVS if you have not already done so
     cvs -d:pserver:ano...@tc...:/cvsroot/tcl co - 
r core-8-5-branch -P tcl

this gives you all the latest manpages, and allows grepping through  
the code & docs easily (usually the best examples of how to use the  
Tcl C API are in the core itself...).

As far as tclIO.c goes, there are probably easier examples of custom  
Tcl_Obj use than channel names (they just seemed most similar to our  
situation), e.g. ISTR good pages on this on the wiki.

Once you have the sources, I would also recommend developing against a  
static mem-debug build of tcl (i.e. configured with --disable-shared -- 
enable-symbols=mem --enable-dtrace), I find having a fully  
symbolicated tcl makes debugging much easier esp if you get problems  
inside tcl e.g. due to Tcl C API misuse.

If you do leaks testing, it is also best to switch off the custom tcl  
allocator by compiling for purify (i.e. with 'export CPPFLAGS=- 
DPURIFY'), otherwise you might not see any leaks involving the Tcl C  
API...

> You mean toying with Tcl_Obj structure directly? It certainly can be  
> done.

the point would be to have a custom Tcl_Obj type for our dtrace  
handles, this would help efficiency, because you could cache the  
handle_data pointer directly in the Tcl_Obj's otherValuePtr and thus  
avoid the hash table lookup (except in the uncommon case where the  
internal representation has been lost). This is only an optimization  
however and so can be left till later if you prefer.

>  For now I've just switched to int handles, which work, but probably  
> aren't that safe.


yes, I think the previous format of having "dtrace%d" handles was  
nicer, you could just use those strings as hash table keys instead of  
the ints you use now.

On 11/07/2008, at 19:00, Remigiusz Jan Andrzej Modrzejewski wrote:

> I've changed the code a bit to be more close to your suggestions and  
> the Tcl/TK Engineering Manual

do you mind if I fix up any remaining formatting nits as I read  
through the code in detail (tomorrow) and commit the changes (if any)?

I will also have a look tomorrow at getting the current state of  
things to build&run on Mac OS X, which will make it easier for me to  
run the code on an ongoing basis without having to start up the  
Solaris VM...

> Now I'll be putting the comments in there (and the DeInit function),  
> so you're free to comment on the code you see.


re DeInit() and the question you asked in review 1 about how to close  
handles on tcl exit:

there are several options, one is indeed to use an exit handler as you  
have done now, but the better option IMO would be to do the cleanup  
when the interp where the handles are relevant is destroyed, i.e. when  
the assoc data setup via Tcl_SetAssocData() is freed (c.f. delProcPtr  
param to Tcl_SetAssocData()).
BTW, you probably also want to free the hash table in DeInit (via  
Tcl_DeleteHashTable()).

Cheers,

Daniel

-- 
** Daniel A. Steffen                   **
** <mailto:da...@us...>  **

[Tcl-soc2008] Mid-term Evals

[Matthew M. B. <mm...@gw...>]

All,

I won't shame anybody in public (YET), but as of my re-loading the web 
page 2 seconds ago, we are still missing 6 of 9 mentor evaluations and 5 
of 9 student evaluations.  The evals are due this coming Monday at noon 
Google time.

Students *WILL NOT* get paid w/out these surveys.

Mentors---if you do not think you can get yours done, please let me 
know.  I can do them for you---but you'll need to email me details of 
how you've been interacting with your student, how frequently, whether 
they are meeting your expectations, if they're on schedule, etc., etc.  
Hmm,  after all that, it would be quicker for you to just do them 
yourself! :)


Best,

Matt

Re: [Tcl-soc2008] The elegant way to include memleak checking into test suite?

[Jeff H. <je...@ac...>]

Remigiusz Jan Andrzej Modrzejewski wrote:
> Solaris has that neat tool for finding memory leaks called /bcheck/.
> What is the elegant way of using external tools like that in a test 
> suite? In shell I can write that as:
> #v+
> bcheck -o /dev/null 2>&1 /opt/ActiveTcl-8.5/bin/tclsh test.tcl | grep 
> 'actual leaks:' | grep '[1-9]'
> #v-
> It there is any output, there are leaks.

Check out the valgrind targets in the Makefile.  They should be a good 
indicator of how you would make a 'make bchecktest' target.

Jeff

[Tcl-soc2008] The elegant way to include memleak checking into test suite?

[Remigiusz J. A. M. <lr...@go...>]

Hi,

Solaris has that neat tool for finding memory leaks called /bcheck/.
What is the elegant way of using external tools like that in a test 
suite? In shell I can write that as:
#v+
bcheck -o /dev/null 2>&1 /opt/ActiveTcl-8.5/bin/tclsh test.tcl | grep 
'actual leaks:' | grep '[1-9]'
#v-
It there is any output, there are leaks.

Kind regards,
Remigiusz Modrzejewski

Re: [Tcl-soc2008] Quick reminder - this is the week ofMIDTERM EVALUATIONS

[Andreas K. <and...@ac...>]

> On Tue, Jul 08, 2008 at 04:10:09PM +0200, Remigiusz Jan Andrzej
> Modrzejewski wrote:
> > Hi,
> >
> > This is the week of midterm evaluations - these are required to proceed
> > with the project so don't forget it. Remember: it works both ways -
> > mentor has to evaluate his student, and student has to evaluate
> his mentor.
> >
> > Kind regards,
> > Remigiusz Modrzejewski
>
> Just in case I'm not the only idiot who doesn't remember where to do
> the midterm evaluation, could someone remind/tell us what the URL is or
> where to send the email?


Your mentor page at GSoC should have direct links to the the surveys, in
red, at the top.

--
	Andreas Kupries <andreask@ActiveState.com>
	Developer @ http://www.ActiveState.com
	Tel: +1 778-786-1122

Re: [Tcl-soc2008] Coding... So refreshing :)

[Donal K. F. <don...@ma...>]

Daniel A. Steffen wrote:
> - re. Tcl_CreateNamespace, declaring it yourself is not going to work  
> with stubs (you'd need the declaration as in tclIntDecls.h involving  
> tclIntStubsPtr), if we want 8.4 compatibility without relying on  
> internal headers, eval "namespace create" instead.

Also decide whether you really want 8.4 compat. If not (and it is a bit 
like declaring that you're targeting Debian "stable" these days), the 
declaration is "in" tcl.h[*] already.

(Otherwise, great advice from Dan.)

Donal.
[* Actually in a file included from tcl.h, but that shouldn't be 
directly included. ]

[Tcl-soc2008] Midterm Evaluations

[Matthew M. B. <mm...@gw...>]

Greetings from your not-particularly-effective-Admin:

As has been pointed out by Remigiusz and others, midterm evaluations are 
due by the 14th.  Both students and mentors with students are required 
to fill these out.  Mentors without students are asked to complete the 
eval as well.  Details can be found at

http://groups.google.com/group/google-summer-of-code-announce/web/midterm-survey-information


And feel free to email me with questions (which I probably cannot 
answer, but I'll try and hunt down the answers).  If you are a mentor 
and you think you will not be able to get your eval done, please let me 
know As Soon As Possible!  And I can do it for you.  But I'd prefer not to.

As an aside, this summer has snowballed for me into a mountain of work I 
wasn't anticipating---some as a result of being the least senior member 
of my department and being stuck with a bunch of things, some because 
I'm a push-over and can't say no, and some that I actually wanted to do 
(like this).  In particular, I am half-way through a week long CS for 
High School students camp that I got suckered into teaching and the 
conference I am organizing is taking place this Monday and Tuesday.

So, DON'T ADD TO MY STRESS----GET YOUR EVALS DONE!!!  But, seriously, do 
contact me if you need help.  Next week when camp, conference, and a few 
other things are done, I intend to be more pro-active with regards to GSoC.


Matt

Re: [Tcl-soc2008] Quick reminder - this is the week of MIDTERM EVALUATIONS

[Clif F. <cl...@cf...>]

On Tue, Jul 08, 2008 at 04:10:09PM +0200, Remigiusz Jan Andrzej Modrzejewski wrote:
> Hi,
> 
> This is the week of midterm evaluations - these are required to proceed 
> with the project so don't forget it. Remember: it works both ways - 
> mentor has to evaluate his student, and student has to evaluate his mentor.
> 
> Kind regards,
> Remigiusz Modrzejewski

Just in case I'm not the only idiot who doesn't remember where to do
the midterm evaluation, could someone remind/tell us what the URL is or
where to send the email?

  Thanks,
  Clif

-- 
.... Clif Flynt ... http://www.cflynt.com ... cl...@cf... ...
.. Tcl/Tk: A Developer's Guide (2nd edition) - Morgan Kauffman ..
. 15'th Annual Tcl/Tk Conference:  Oct 20-25 2008,  Manassas VA .
.............  http://www.tcl.tk/community/tcl2008/  ............






-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

Re: [Tcl-soc2008] Coding... So refreshing :)

[Andreas K. <and...@ac...>]

> For ease of review, it would also be helpful to adopt the coding
> conventions in the Tcl/Tk engineering manual:
> 	http://tip.tcl.tk/247
> 	http://www.tcl.tk/doc/engManual.pdf
> this is what is used throughout the Tcl core source code and most
> extensions, and will make it easier for others to review (and maybe
> later modify) your source code.
>
> In particular indenting should be 4 spaces (with 8 spaces == 1 tab),
> and the bracketing style should be
>      if () {
>
>      } else {
>
>      }
>
> c.f. the links above for the full detail

Note that many files in Tcl's code base have the following comment at the
bottom

/*
 * Local Variables:
 * mode: c
 * c-basic-offset: 4
 * fill-column: 78
 * End:
 */

If you are using Emacs this comment sets up most of the necessary stuff
(c-mode and indentation) for the code to fit in.

--
	Andreas Kupries <andreask@ActiveState.com>
	Developer @ http://www.ActiveState.com
	Tel: +1 778-786-1122

Re: [Tcl-soc2008] Coding... So refreshing :)

[Daniel A. S. <da...@us...>]

Hi Remigiusz,

On 09/07/2008, at 22:22, Remigiusz Jan Andrzej Modrzejewski wrote:

> Today I've finally started coding:
> #v+
> # wc -l dtrace.c dtrace.h
>     293 dtrace.c
>      64 dtrace.h
>     357 total
> #v-
> This implements:
> ::dtrace::open
> ::dtrace::close
> ::dtrace::configure
> And was a really refreshing thing to do :)
> The code works well under both Tcl 8.4 and 8.5 and does produce  
> warnings nor leak memory... I must say bcheck really rocks :)
> Progress bar for implementation[1] moved up to 27% already.

great to see some code from you, looks like a good start!

as a general comment I'd like to see more comments, esp for data  
structures declared in the header file and used throughout...

For ease of review, it would also be helpful to adopt the coding  
conventions in the Tcl/Tk engineering manual:
	http://tip.tcl.tk/247
	http://www.tcl.tk/doc/engManual.pdf
this is what is used throughout the Tcl core source code and most  
extensions, and will make it easier for others to review (and maybe  
later modify) your source code.

In particular indenting should be 4 spaces (with 8 spaces == 1 tab),  
and the bracketing style should be
     if () {

     } else {

     }

c.f. the links above for the full detail

> I've also added a question to the Peer Review system[2].

I do not see this, when logged in I see no reviews in peerReviewMain.

Actually, it would be good if you created a review for the whole of  
dtrace.c and assigned to me, there are a few things that I think could  
be done better (mainly by using more Tcl API), some quick notes below  
for now.

> This unfortunately does not go into the Timeline (as well as Ticket  
> comments) :/

I looks like ticket comments can be made to appear in the timeline  
(ticket_show_details config option), that would very helpful for  
tracking tickets via RSS in fact (esp since email notification of  
tracker changes does not work...)
     http://dev.lrem.net/tcldtrace/wiki/TracIni#timeline

> Finally: tomorrow I'll be submitting the midterm evaluations. Hope  
> you've already arranged time for that... Because without it the  
> project will be automatically dropped form GSoC, at least if I read  
> it correctly.

yes, that's correct, the deadline is on Monday evening, so I was  
planning to do it on Sunday and Monday morning to give us the most  
time to talk about actual committed code in the evaluation.

> [1] - http://dev.lrem.net/tcldtrace/roadmap
> [2] - http://dev.lrem.net/tcldtrace/peerReviewMain


here brief comments from reading through the code committed so far  
(r20):

- switch to compiling with stubs enabled (i.e. with -DUSE_TCL_STUBS);  
as mentioned earlier, I'd recommend switching to the TEA  
sampleextension buildsystem as soon as possible, that takes care of  
all that and more for you.
- re. Tcl_CreateNamespace, declaring it yourself is not going to work  
with stubs (you'd need the declaration as in tclIntDecls.h involving  
tclIntStubsPtr), if we want 8.4 compatibility without relying on  
internal headers, eval "namespace create" instead.
- don't roll your own options processing, use Tcl_GetIndexFromObj(),  
it has error message formatting, performance optimizations etc all  
built in (as mentioned earlier, tclsample.c in the sampleextension has  
an example of that, or it is used throughout the Tcl/Tk core code).
- use Tcl_WrongNumArgs() for usage error messages
- don't use string operations to retrieve argument values if they're  
booleans, integers etc, use Tcl_GetBooleanFromObj(),  
Tcl_GetLongFromObj() etc
- don't repeatedly convert from strings to handles and back, use the  
Tcl_Obj mechanism, see e.g. the way channel names are handled in the  
core: TclGetChannelFromObj(), SetChannelFromAny() in tclIO.c
- don't use global state (i.e your handles and options arrays), state  
should be stored per-interp, I'd recommend in a ckalloc'd structure  
that you store in a tcl hash table keyed by the handle name, see e.g.  
Tcl_GetChannel() in tclIO.c
- start writing the testsuite and test your new commands as you  
implement them, much easier than doing it all later, and much more  
helpful in terms of turning up bugs.

Cheers,

Daniel

-- 
** Daniel A. Steffen                   **
** <mailto:da...@us...>  **

[Tcl-soc2008] Quick reminder - this is the week of MIDTERM EVALUATIONS

[Remigiusz J. A. M. <lr...@go...>]

Hi,

This is the week of midterm evaluations - these are required to proceed 
with the project so don't forget it. Remember: it works both ways - 
mentor has to evaluate his student, and student has to evaluate his mentor.

Kind regards,
Remigiusz Modrzejewski

[Tcl-soc2008] TclDtrace API release candidate finished

[Remigiusz J. A. M. <lr...@go...>]

Hi,

Version 45 seems to be (hopefully) final, at least for initial 
implementation. This is what is now going to be shown to general public:
http://dev.lrem.net/tcldtrace/wiki/CommandsList
Hope you'll like it. Any comments here, or preferably on the wiki, are 
welcome.

Kind regards,
Remigiusz Modrzejewski

Re: [Tcl-soc2008] TclDtrace - important question, new API draft

[Daniel A. S. <da...@us...>]

Hi Remigiusz,

On 07/07/2008, at 1:25, Remigiusz Jan Andrzej Modrzejewski wrote:

> Daniel A. Steffen wrote:
>> I also don't think it makes sense for dtrace::open to be special by  
>> not throwing any errors, can you explain your reasoning behind  
>> that? if the dtrace::open call fails, the rest of your dtrace::  
>> commands cannot execute anyway, so it makes sense to throw a  
>> (catchable) error that breaks out of the program, no?
>
> It's not valid now as I decided to accept the suggestion to throw  
> everything as an error. But it wouldn't hurt much either - if you  
> didn't check for this by yourself, you would get the error on the  
> next call.

I see, ok, good point, I hadn't considered that; I do like what you're  
going to do now better though (throw as soon as the error occurs).

> I'll integrate all the suggestions tomorrow. That should be more or  
> less close to final, so I'll immediately post a call for comments to  
> broader public.

sounds good! posting to comp.lang.tcl ?

It might also be interesting to post on dtrace-discuss, I'm not sure  
how much the folks there know about tcl, but they certainly do know a  
lot about dtrace (including the implementation), so we may get a  
different perspective...

> I'll also probably finally start coding the whole thing.

great! I'd suggest starting with the TEA sampleextension from SF tcl  
CVS:
	cvs -d:pserver:ano...@tc...:/cvsroot/tcl co -P - 
d tcldtrace sampleextension

the generic/sample.* files are specific to the sha1 example and can be  
dropped (along with the ChangeLog, README, tea* and doc/* files), for  
the others, I usually start with a global search&replace of 'sample',  
'Sample' and 'SAMPLE' in all files by the new extension name (i.e.  
'dtrace' in our case)...
generic/tclsample.c has the extension initialisation and command  
creation, the sample sha1 command implementation there also has an  
example of option handling...
There are not a lot of docs on the TEA buildsystem so best to ask if  
it is unclear how something works (essentially, the goal is for all  
customization to be done in configure.in, but sometimes minor changes  
to Makefile.in are necessary, e.g. to the test target).

> After all doing the 1:1 C-Tcl callback should be eventually easier  
> to do, what I like. Going the event loop way will just require an  
> additional thread, what I don't like, but it's just me.


as mentioned, I wouldn't worry about the event loop integration now,  
we can look into that later if there is time left... as discussed, it  
may not be necessary to use a separate thread (I'm all for trying to  
avoid that), if we can replicate the timeout calculation from  
dtrace_sleep(), we may be able to use an event source that  
periodically calls dtrace_work() with that timeout (best to wait to  
experiment with this until everything else works...)

Cheers,

Daniel

-- 
** Daniel A. Steffen                   **
** <mailto:da...@us...>  **

Re: [Tcl-soc2008] TclDtrace - important question, new API draft

[Tomasz K. <tk...@da...>]

If we would like to lower overhead of Tcl callback we could do one Tcl
callback for N dtrace C callbacks and accumulate only N tuples in
memory. But how long shall we wait to collect those N dtrace C
callback - we will probably need some configurable timeout. N should
be also configurable.

These were my ideas before reading your suggestions, but I think that
they are preoptimizations and should be for now avoided.

Cheers,
Tomasz Kosiak


On Sun, Jul 6, 2008 at 11:22 PM, Daniel A. Steffen
<da...@us...> wrote:
> Hi Remigiusz,
>
> On 04/07/2008, at 17:13, Remigiusz Jan Andrzej Modrzejewski wrote:
>
>> However I have some doubts about the most important thing: changing
>> the API to callback style.
>>
>> I'm not sure whether this will be good for performance - this might
>> mean a few thousands calls every second.
>
> A priori, I don't think that'll be a problem, there are tcl based
> servers that handle thousands of connections per second via callbacks.
>
> However, it is always best to measure for these types of questions:
> invoking a command prefix based callback is very similar to calling an
> alias (c.f. AliasObjCmd() in tclInterp.c), so we can use alias call
> performance as an indication of what kind of numbers we can expect. It
> all depends on how work much the callback does of course, with three
> examples of decreasing complexity I get:
>
>        % set f [open "/tmp/out.txt" w]
>        % proc callback {hdl probe cpu id {type none} {data {}}} {puts $::f
> "$probe $cpu $id $type $data"}
>        % interp alias {} invoke {} callback hdl
>        % time {invoke dtrace:::BEGIN 1 prinf out} 60000
>        16.603524466666666 microseconds per iteration
>        % proc callback {handle probe cpu id {type none} {data {}}}
> {incr ::count}
>        % time {invoke dtrace:::BEGIN 1 prinf out} 500000
>        2.078615204 microseconds per iteration
>        % proc callback {handle probe cpu id {type none} {data {}}} {}
>        % time {invoke dtrace:::BEGIN 1 prinf out} 700000
>        1.5304863228571428 microseconds per iteration
>
> i.e. for a noop callback ~700000 calls/sec and for a more expensive
> callback (writing to a file) ~60000 calls/sec.
>
> Of course it will be possible to write callbacks that take too much
> time, but the only negative effects of that will be drops, i.e. it
> will be up to the user to deal with that by making the callback more
> efficient or increasing the buffer size etc. That's similar to the
> existing situation with dtrace(1), it is already very possible to
> write D scripts that result in drops...
>
> That brings to mind that we don't have an API for drop notification
> yet (i.e. drophandler() in dtrace.c), I would suggest a second
> callback passed to dtrace::go that gets called for drops
> (dtrace_handle_drop()) as well as errors (dtrace_handle_err()) and
> proc changes (dtrace_handle_proc()).
>
>> On the other hand we have constructing a list containing two lists
>> containing few thousands records each, which may be even worse and
>> memory consuming too.
>
> yes, that was my concern, the memory allocation cost alone could well
> be higher than the callback cost. If the user wants to keep tracing
> results around, the callback can always store them in a global list,
> but the API should avoid relying internally on such list construction
> I think.
>
>> If it turns out that we don't like big lists, then maybe it's better
>> to make ::dtrace::aggregations and ::dtrace::list also callback-
>> based? They'd both be generated after an explicit call, so it's only
>> performance oriented change.
>
> for dtrace::list a callback interface may make sense, esp. since it
> can generate very large lists ('dtrace -l' is at least 25000 lines and
> up to 200000 lines on my OSX system), it would also avoid having to
> generate many copies of identical dict keys as in the current
> proposal, so yes, making dtrace::list callback-based sounds like a
> good idea.
>
> for dtrace::aggregations, I'm not so sure, the most common output
> operations on aggregations involve sorting of some kind, so you
> usually need the whole aggregation all in memory anyway, I would
> suggest leaving that one as is (in general, aggregations tend to be
> smaller as well, but of course you _can_ generate huge aggregations if
> you try...)
>
>> I also feel that error should be thrown only in situations when
>> proceeding is actually impossible. I'd stick to returning empty
>> strings for the minor faults. But I'm not pressing on this - it's
>> just that I'd personally prefer using /if/ rather than /catch/ for
>> checking, whether something compiles well. But I'm a Tcl noob ;)
>
>
> as a general principle, tcl interfaces report errors via the TCL_ERROR
> mechanism, I think deviating from that is likely to confuse the user
> and should be avoided unless there is a very good reason. Also, having
> more than one way to report errors is likely to be confusing and not a
> good idea I think.
>
> For the case of failed compilation, IMO it depends on the
> circumstances whether that should be a hard error or not, e.g. in an
> interactive DTrace script editor, it should certainly be a soft error,
> but e.g. in a program wrapping a fixed D script, it should be a hard
> error as it'll make it impossible for the program to proceed.
> Given that [catch] will work just fine for the first case, using
> TCL_ERROR for a compilation failure seems like the right thing to do
> and corresponds to what is usually done in tcl (e.g. [eval] throws
> errors when passed invalid scripts, [info complete] can be used to
> check for some errors beforehand, but not all...)
>
> It might be be good to detail in the spec the errors that can occur,
> and which ones you'd consider minor, a concrete list of minor faults
> might clarify why you feel there should be two classes of errors.
>
> I also don't think it makes sense for dtrace::open to be special by
> not throwing any errors, can you explain your reasoning behind that?
> if the dtrace::open call fails, the rest of your dtrace:: commands
> cannot execute anyway, so it makes sense to throw a (catchable) error
> that breaks out of the program, no?
>
> I have added all these suggestions as well as the following (some of
> which I already mentioned earlier) to the wiki <http://dev.lrem.net/tcldtrace/wiki/CommandsListDraftComments
>  > for easier tracking/checking off:
>
> - dtrace::work: switch to callback model, empty return value, explain
> that calling this will become optional once event loop integration is
> done.
>
> - dtrace::go: add a list argument specifying the callback for tracing
> data processing.
>
> - dtrace::go: add a list argument specifying the callback for drop,
> error and proc notification (i.e. via the handlers set by
> dtrace_handle_err(), dtrace_handle_drop(), and dtrace_handle_proc()).
>
> - dtrace::sleep: drop, instead add a boolean argument to dtrace::work
> indicating whether it should sleep before calling dtrace_work().
>
> - dtrace::work: specify all possible strings for 'output type'
> explicitly, as well as the format for every type (e.g. for stacks a
> dict or list, for aggregations a dict etc).
>
> - dtrace::work: rename to dtrace::process.
>
> - dtrace::list: make callback based.
>
> - dtrace::configure: drop the cpp related options, better to run cpp
> from tcl directly.
>
> - dtrace::configure: detail _all_ legal options in the api spec, it's
> hard to know from the LibDTrace doc page which options are relevant
> for TclDTrace, there are also options missing in the LibDTrace doc
> which are clearly relevant (but are not dtrace(1) options), c.f.
> dt_options.c.
>
> - dtrace::configure: relate option names to #pragmas, explain that
> compiling programs with #pragmas can change configuration.
>
> - dtrace::configure: add an option specifying a pid to be grabbed for
> tracing, with 0 indicating no pid grabbed (like 'dtrace -p', can also
> be used to replicate 'dtrace -c' (via [exec]) from tcl).
>
> - dtrace::configure: clarify language about return value in the three
> different call modes (see e.g. language in fconfigure.n man page).
>
> - dtrace::compile: the '[[[provider:] module:] function:] name
> [[predicate] action]' spec is incorrect (square brackets are nested
> incorrectly): you can give a predicate without an action (in which
> case the default action, i.e. print probe identifier, is used);
> similarly, you can give a provider without a module and a module
> without a function (e.g. dtrace:::BEGIN)
>
> - dtrace::info: wrap dtrace_program_info() and return the same dict as
> for dtrace::exec (the user may not want to execute a program, only to
> examine it, e.g. as in 'dtrace -l').
>
> - dtrace::status: wrap dtrace_status() (no way to determine if tracing
> has stopped due to external events otherwise).
>
> - dtrace::errno, dtrace::errstr: drop, report all errors by throwing a
> tcl error, drop all language about empty string returns in error case
> from the spec.
>
> - dtrace::open: throw errors.
>
> - dtrace::close: drop "OK" string return.
>
>
> Cheers,
>
> Daniel
>
> --
> ** Daniel A. Steffen                   **
> ** <mailto:da...@us...>  **
>
>
>
> -------------------------------------------------------------------------
> Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW!
> Studies have shown that voting for your favorite open source project,
> along with a healthy diet, reduces your potential for chronic lameness
> and boredom. Vote Now at http://www.sourceforge.net/community/cca08
> _______________________________________________
> Tcl-soc2008 mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-soc2008
>

Re: [Tcl-soc2008] TclDtrace - important question, new API draft

[Daniel A. S. <da...@us...>]

Hi Remigiusz,

On 04/07/2008, at 17:13, Remigiusz Jan Andrzej Modrzejewski wrote:

> However I have some doubts about the most important thing: changing  
> the API to callback style.
>
> I'm not sure whether this will be good for performance - this might  
> mean a few thousands calls every second.

A priori, I don't think that'll be a problem, there are tcl based  
servers that handle thousands of connections per second via callbacks.

However, it is always best to measure for these types of questions:
invoking a command prefix based callback is very similar to calling an  
alias (c.f. AliasObjCmd() in tclInterp.c), so we can use alias call  
performance as an indication of what kind of numbers we can expect. It  
all depends on how work much the callback does of course, with three  
examples of decreasing complexity I get:

	% set f [open "/tmp/out.txt" w]
	% proc callback {hdl probe cpu id {type none} {data {}}} {puts $::f  
"$probe $cpu $id $type $data"}
	% interp alias {} invoke {} callback hdl
	% time {invoke dtrace:::BEGIN 1 prinf out} 60000
	16.603524466666666 microseconds per iteration
	% proc callback {handle probe cpu id {type none} {data {}}}  
{incr ::count}
	% time {invoke dtrace:::BEGIN 1 prinf out} 500000
	2.078615204 microseconds per iteration
	% proc callback {handle probe cpu id {type none} {data {}}} {}
	% time {invoke dtrace:::BEGIN 1 prinf out} 700000
	1.5304863228571428 microseconds per iteration

i.e. for a noop callback ~700000 calls/sec and for a more expensive  
callback (writing to a file) ~60000 calls/sec.

Of course it will be possible to write callbacks that take too much  
time, but the only negative effects of that will be drops, i.e. it  
will be up to the user to deal with that by making the callback more  
efficient or increasing the buffer size etc. That's similar to the  
existing situation with dtrace(1), it is already very possible to  
write D scripts that result in drops...

That brings to mind that we don't have an API for drop notification  
yet (i.e. drophandler() in dtrace.c), I would suggest a second  
callback passed to dtrace::go that gets called for drops  
(dtrace_handle_drop()) as well as errors (dtrace_handle_err()) and  
proc changes (dtrace_handle_proc()).

> On the other hand we have constructing a list containing two lists  
> containing few thousands records each, which may be even worse and  
> memory consuming too.

yes, that was my concern, the memory allocation cost alone could well  
be higher than the callback cost. If the user wants to keep tracing  
results around, the callback can always store them in a global list,  
but the API should avoid relying internally on such list construction  
I think.

> If it turns out that we don't like big lists, then maybe it's better  
> to make ::dtrace::aggregations and ::dtrace::list also callback- 
> based? They'd both be generated after an explicit call, so it's only  
> performance oriented change.

for dtrace::list a callback interface may make sense, esp. since it  
can generate very large lists ('dtrace -l' is at least 25000 lines and  
up to 200000 lines on my OSX system), it would also avoid having to  
generate many copies of identical dict keys as in the current  
proposal, so yes, making dtrace::list callback-based sounds like a  
good idea.

for dtrace::aggregations, I'm not so sure, the most common output  
operations on aggregations involve sorting of some kind, so you  
usually need the whole aggregation all in memory anyway, I would  
suggest leaving that one as is (in general, aggregations tend to be  
smaller as well, but of course you _can_ generate huge aggregations if  
you try...)

> I also feel that error should be thrown only in situations when  
> proceeding is actually impossible. I'd stick to returning empty  
> strings for the minor faults. But I'm not pressing on this - it's  
> just that I'd personally prefer using /if/ rather than /catch/ for  
> checking, whether something compiles well. But I'm a Tcl noob ;)


as a general principle, tcl interfaces report errors via the TCL_ERROR  
mechanism, I think deviating from that is likely to confuse the user  
and should be avoided unless there is a very good reason. Also, having  
more than one way to report errors is likely to be confusing and not a  
good idea I think.

For the case of failed compilation, IMO it depends on the  
circumstances whether that should be a hard error or not, e.g. in an  
interactive DTrace script editor, it should certainly be a soft error,  
but e.g. in a program wrapping a fixed D script, it should be a hard  
error as it'll make it impossible for the program to proceed.
Given that [catch] will work just fine for the first case, using  
TCL_ERROR for a compilation failure seems like the right thing to do  
and corresponds to what is usually done in tcl (e.g. [eval] throws  
errors when passed invalid scripts, [info complete] can be used to  
check for some errors beforehand, but not all...)

It might be be good to detail in the spec the errors that can occur,  
and which ones you'd consider minor, a concrete list of minor faults  
might clarify why you feel there should be two classes of errors.

I also don't think it makes sense for dtrace::open to be special by  
not throwing any errors, can you explain your reasoning behind that?  
if the dtrace::open call fails, the rest of your dtrace:: commands  
cannot execute anyway, so it makes sense to throw a (catchable) error  
that breaks out of the program, no?

I have added all these suggestions as well as the following (some of  
which I already mentioned earlier) to the wiki <http://dev.lrem.net/tcldtrace/wiki/CommandsListDraftComments 
 > for easier tracking/checking off:

- dtrace::work: switch to callback model, empty return value, explain  
that calling this will become optional once event loop integration is  
done.

- dtrace::go: add a list argument specifying the callback for tracing  
data processing.

- dtrace::go: add a list argument specifying the callback for drop,  
error and proc notification (i.e. via the handlers set by  
dtrace_handle_err(), dtrace_handle_drop(), and dtrace_handle_proc()).

- dtrace::sleep: drop, instead add a boolean argument to dtrace::work  
indicating whether it should sleep before calling dtrace_work().

- dtrace::work: specify all possible strings for 'output type'  
explicitly, as well as the format for every type (e.g. for stacks a  
dict or list, for aggregations a dict etc).

- dtrace::work: rename to dtrace::process.

- dtrace::list: make callback based.

- dtrace::configure: drop the cpp related options, better to run cpp  
from tcl directly.

- dtrace::configure: detail _all_ legal options in the api spec, it's  
hard to know from the LibDTrace doc page which options are relevant  
for TclDTrace, there are also options missing in the LibDTrace doc  
which are clearly relevant (but are not dtrace(1) options), c.f.  
dt_options.c.

- dtrace::configure: relate option names to #pragmas, explain that  
compiling programs with #pragmas can change configuration.

- dtrace::configure: add an option specifying a pid to be grabbed for  
tracing, with 0 indicating no pid grabbed (like 'dtrace -p', can also  
be used to replicate 'dtrace -c' (via [exec]) from tcl).

- dtrace::configure: clarify language about return value in the three  
different call modes (see e.g. language in fconfigure.n man page).

- dtrace::compile: the '[[[provider:] module:] function:] name  
[[predicate] action]' spec is incorrect (square brackets are nested  
incorrectly): you can give a predicate without an action (in which  
case the default action, i.e. print probe identifier, is used);  
similarly, you can give a provider without a module and a module  
without a function (e.g. dtrace:::BEGIN)

- dtrace::info: wrap dtrace_program_info() and return the same dict as  
for dtrace::exec (the user may not want to execute a program, only to  
examine it, e.g. as in 'dtrace -l').

- dtrace::status: wrap dtrace_status() (no way to determine if tracing  
has stopped due to external events otherwise).

- dtrace::errno, dtrace::errstr: drop, report all errors by throwing a  
tcl error, drop all language about empty string returns in error case  
from the spec.

- dtrace::open: throw errors.

- dtrace::close: drop "OK" string return.


Cheers,

Daniel

-- 
** Daniel A. Steffen                   **
** <mailto:da...@us...>  **

[Tcl-soc2008] Updating Tktest suite - GSoC project

[Ania P. <ani...@gm...>]

Hello,

I'm Ania Pawelczyk, a student from Poland who participates in Google Summer
of Code and works on updating Tk test suite.

I'd like to ask for comments on my work, get to know whether I took right
assumptions. I'd be also thankful for hints and suggestions on the other
aspects that I omitted but I should concentrate on.
I'd very welcome if people involved in upgrading test suite (like dgp,
patthoyts, dkf, das...) would steer me on the right direction.

As for now I took the following assumptions and concentrated on those
aspects:

1. Failing tests
I work by comparing results on different environments and my goal is to let
tktests be passed on them. Currently I work on Slackware, Ubuntu and XP (on
2 machines and tomorrow I'll get a third one which, I think, is the minimum
to do simultaneous testing). WinXP and Ubuntu pass rather well (Ubuntu has
problems generally with tests where font constraint is checked whereas Win
and Slack don't pass the font constraint at all). I'd install on the third
one Vista and p.e. SuSE.

I usually work by assuming that the test that passes on one machine but not
on the others has probably "good" - I mean required - settings and the
failing ones lack something or take wrong assumptions. I compare
configurations and try to find what p.e.
- wasn't set, but matters in the result and may have different default
values
- was somewhere wrongly assumed (like in
http://www.assembla.com/flows/show/brq6hCsrar3BB4ab7jnrAJ)
Is this alright?


2. Constraints.tcl
I also wonder whether I could mess up with constraints.tcl to try them to be
passed more likely (like fonts; i.e.
http://www.assembla.com/flows/show/dM_zSCsdWr3zNyab7jnrAJ ). Or rather
Should I only try to configure different desktop settings? Or maybe should I
leave tests with skipped constraints as they are?


3. Individual test's structure
Furthermore, should I also restructure tests? So that they'd follow the
pattern that is presented in man tcltest page?
P.e. message.test. I was advised to work on. As it passes all the tests on
my current OSes and my friends ones, what should be done with it? This kind?

    # Current look

    test message-3.6 {MessageWidgetObjCmd procedure, "configure"} {
            message .m
            set result [list [catch {.m configure -foo} msg] $msg]
            destroy .m
            set result
    } {1 {unknown option "-foo"}}

    # I thought about
    test message-3.0 {MessageWidgetObjCmd procedure, "cget"} -setup {
            message .m
    } -body {
            .m cget
    } -cleanup {
            destroy .m
    } -returnCodes error -result {wrong # args: should be ".m cget option"}

There're also are quite often widgets defined and set outside the test case
and test case that uses the previous settings;
Sometimes a few tests take advantage of that settings. Should I take care
for this and turn it into (the same) individual test case setups?


4. Wiki page
I added my project's wiki page where I put my ideas (marked(i)) and
questions (marked(?)).
http://www.assembla.com/flows/flow/a_KwcurkGr3ztnab7jnrAJ

For the moment there're rfc for, briefly, my following problems:

> Should I try to make the tktest constraints in a way that they'd be more
likely to pass? {(?i) constraints.tcl - testConstraint fonts}

> How to figure out in tcl if a definition was made during the compilation?
{(?) frame.test frame-2.5}

> Strange ubutnu's behaviour - wm geometry x and x coordination {(?) wm.test
- wm-geometry-2.1 in Ubuntu }

> RFC scrollbar.test scrollbar-3.42

I'd very welcome any comments for that or new threads with suggestions.
I'm also available via email ani...@gm...

Thanks in advance,
Regards,

Ania.

[Tcl-soc2008] TclDtrace - important question, new API draft

[Remigiusz J. A. M. <lr...@go...>]

Hi,

I've just changed parts of the API to suit some of the common needs I've 
heard. However I have some doubts about the most important thing: 
changing the API to callback style.

I'm not sure whether this will be good for performance - this might mean 
a few thousands calls every second. On the other hand we have 
constructing a list containing two lists containing few thousands 
records each, which may be even worse and memory consuming too.

If it turns out that we don't like big lists, then maybe it's better to 
make ::dtrace::aggregations and ::dtrace::list also callback-based? 
They'd both be generated after an explicit call, so it's only 
performance oriented change.

I also had concerns about the program flow, but those seem invalid. Just 
for reassuring: there is nothing wrong with Tcl level callbacks being 
called en masse from libdtrace's dtrace_work?

I also feel that error should be thrown only in situations when 
proceeding is actually impossible. I'd stick to returning empty strings 
for the minor faults. But I'm not pressing on this - it's just that I'd 
personally prefer using /if/ rather than /catch/ for checking, whether 
something compiles well. But I'm a Tcl noob ;)

Once again thanks for all the input and I hope for more.

Kind regards,
Remigiusz Modrzejewski

Re: [Tcl-soc2008] My request for comments pretty much ignored

[Daniel A. S. <da...@us...>]

On 04/07/2008, at 3:20, Daniel A. Steffen wrote:

> - dtrace::exec: I see no need for the compiled_program handle,

oops, meant to say program_info handle of course

> In the chew function passed to dtrace_work(), you would then call this
> callback once for every probe result, this also has the advantage that
> it is easier & less memory intensive than building up a tcl list of
> results in memory and passing it all back at once as a result.

clarifying: easier & less memory intensive than building up in memory  
a tcl list of tracing results and passing it all back as a single tcl  
result.

Cheers,

Daniel

-- 
** Daniel A. Steffen                   **
** <mailto:da...@us...>  **

Re: [Tcl-soc2008] My request for comments pretty much ignored

[Daniel A. S. <da...@us...>]

Hi Remigiusz,

sorry the delay, it took a few hours longer than expected to write up  
my comments...

first off, let me say that the first draft looks good overall, I have  
a number of suggestions on how to improve it but none that will change  
anything really fundamental.

As a general comment, my feeling is that the draft is a bit closer to  
the C api than is really necessary, I think we can clarify & simplify  
things a bit for the tcl API user without loosing functionality. I  
also don't think it is necessary to maintain the close association  
with the command names from libdtrace, some of them are less than self- 
explanatory IMO and can be replaced by something clearer.

Replies/details on yesterday's discussion below, first my list of  
briefer suggestions (in imperative style, don't take that to mean that  
I think these things are not up for discussion, they very much are,  
just quicker to write this way):

BTW, when I refer to dicts below, I mean tcl 8.5 dicts, which (as  
data) are equivalent to an even-length list of key-value pairs. These  
are the natural datastructure to use IMO (esp for aggregations), they  
can be dealt with efficiently from tcl via [foreach k v $dict] or the  
new 8.5 [dict] API.

- dtrace::option: rename to dtrace::configure for consistency with  
tcl, and have it behave like other tcl configuration commands: i.e. if  
no arguments given, return a dict of all -options and values, if an - 
option is given, return its value, otherwise take a sequence of - 
options and values. Do away with the set/unset business, use 0/1 for  
boolean flags like everywhere else in tcl. Specify the names of the  
options that can be used in the spec in detail, don't refer to the  
libdtrace doc.
(Re. implementation, the Tcl C API has efficient functions (that use  
caching etc) to help implement such configuration commands, they're  
used all over tcl).

- dtrace::open: have this take a sequence of -options & values same as  
dtrace::configure, so that initial configuration can be done without  
calling a second command (makes things simpler to setup and is more  
consistent with other tcl commands).

- dtrace::close: explain why calling this is important (in particular  
it frees the memory for all the compiled programs and their handles,  
there is no other libdtrace api for freeing resources).

- dtrace::open, dtrace::close: I'm not so happy with these command  
names, 'create' and 'destroy' might be clearer but are not great  
either, other ideas?

- dtrace::compile vs dtrace::fcompile: reading the implementation,  
there appears to be no difference between compiling a string and  
compiling a file, not sure where you got the impression that string  
compilation does not support whole programs (grep for pcb_fileptr and  
pcb_string in dt_cc.c, dt_lex.l and dt_grammar.y) so I think we can do  
away with dtrace::fcompile and let tcl handle the IO (which has many  
advantages). If wanted, a tcl level implementation of fcompile could  
be added that wraps tcl [open]/[read] and dtrace::compile.  
Preprocessing via cpp could also be added from the tcl level (later).

- dtrace::exec: this does not need to take a handle, a  
compiled_program is specific to a handle anyway, so the handle info  
can be stored inside the compiled_program info.

- dtrace::exec: clarify here that several programs can be executed (by  
calling this repeatedly) and that all will be run when dtrace::go is  
called. Don't say 'running program' here it is confusing (as the  
program is in fact only running ionce dtrace::go is called).

- dtrace::exec: I see no need for the compiled_program handle, return  
the progam info directly here as a dict (I agree with Tomasz that a  
dict would be clearer than the list you proposed and the extra cost to  
setup the dict does not matter here).

- dtrace::info: not needed with the previous point, so instead make  
this wrap dtrace_program_info() and return the same dict as above (the  
user may not want to execute a program, only to examine it, e.g. as in  
'dtrace -l').

- dtrace::list: add an API that wraps dtrace_stmt_iter() as in  
list_prog() in dtrace.c (important for replicating 'dtrace -l').

- dtrace::status: add an API that wraps dtrace_status() (no way to  
determine if tracing has stopped due to external events otherwise).

- dtrace::aggregations: make this return a dict of dicts, i.e. the  
outer dict's keys are the aggregation name and its values are dicts of  
tuples and values.

- comments on the remaining dtrace:: commands below:

On 03/07/2008, at 14:53, Remigiusz Jan Andrzej Modrzejewski wrote:

> Daniel A. Steffen wrote:
> > I have a number of suggestions for improvement, I'll write a  
> detailed
>> response tomorrow, for now the most important suggestion that I'd  
>> have is to look into integrating dtrace processing with the tcl  
>> event loop (i.e. do away with ::dtrace::sleep, and instead  
>> of ::dtrace::work have a callback like for tcl file events).
>
> I've investigated this for some hours now and... I don't see a  
> reason for changing anything in the API. Some facts I missed in the  
> docs:
> 1) dtrace_sleep is not really needed - in most cases any delay will  
> be ok.
> 2) There are no bad side-effects of sleeping to short, other than  
> normal reasons to avoid active waiting.
> 3) If you sleep to long, DTrace will abort

> I'll update the wiki and leave API as is - if someone wants to use  
> TclDtrace inside of (Tk?) event loop, it'll be his duty to schedule  
> frequent ::dtrace::work. I'll write a SampleProgram for that later  
> today.

Most larger tcl programs (and of course all tk programs) as well as  
programs embedding tcl rely on the tcl event loop and have it running,  
(especially server programs which might be particularly interested in  
running DTrace on themselves), so I think it is important to support a  
way to process dtrace results that does not require calling  
dtrace::work periodically.

The other reason I don't like the idea of polling via frequent calls  
from dtrace::work from tcl code is that I think that approach has the  
potential to burn a lot of cycles uselessly. If we need to poll, it  
would be better to do it directly from C in the tcl event loop via an  
event source (with a configurable sleep time), i.e. this would  
periodically call dtrace_work() from inside the tcl event loop and  
pass any results back to tcl via callback.
Having looked at the dtrace_sleep implementation, it does look safe &  
feasible to call dtrace_sleep in a separate thread (with some locking  
around dtrace_hdl_t access) and then wake up the tcl event loop via  
Tcl_AlertNotifier(), so that'd be an alternative not requiring polling.

In any case, we do not need to worry about the details of this now,  
implementing this and how exactly to do it can be deferred to later  
(or after the GSoC even), but we need to come up with an API that can  
support such a model in addition to the polling model (which may have  
it's uses also).

I think the main change required for this is that the passing back of  
dtrace_work() results to tcl needs to be callback based, i.e. instead  
of returning the results from dtrace::work, pass them as arguments to  
a specified command prefix. The natural place to specify that prefix  
would be dtrace::go, i.e. make this take a list argument (to which  
'handle' and the dtrace_work() results are appended on callback)

In the chew function passed to dtrace_work(), you would then call this  
callback once for every probe result, this also has the advantage that  
it is easier & less memory intensive than building up a tcl list of  
results in memory and passing it all back at once as a result.

With this API, dtrace_work() can then be called explicitly from tcl  
via dtrace::work (which will return nothing), and in the future also  
from an event loop source (that TclDTrace sets up).

I also see no use case for making dtrace::sleep a separate command, it  
will never be called on its own without dtrace::work being called  
afterwards, so to simplify things, drop this command and add a boolean  
argument to dtrace::work indicating whether it should sleep before  
calling dtrace_work().

IMO dtrace::work is not a very good/descriptive name, dtrace::process  
might be better ?

W.r.t the specification of the tracing results, the possible strings  
for 'output type' need to be explicitly detailed in the spec, as well  
as the format for every type (e.g. for stacks a dict or list, for  
aggregations a dict etc).

With the callback solution, there is also no longer a need for probe  
data & probe output lists, everything can just be passed as arguments  
to the callback command prefix, i.e the first three arguments are  
probe data and there are 5 more optional arguments if there is probe  
output.

>> Another change I would suggest would be w.r.t return 'status' from  
>> the dtrace API, I think this is only used to return error  
>> conditions right? if so, the usual tcl way to approach this is by  
>> throwing an error (i.e. return TCL_ERROR) and putting the error  
>> description into the interp result. This would avoid having to  
>> return lists of two items for the API calls that need to return  
>> things other than 'status'.
>
> Done.

I probably was not very clear before. I did not mean to suggest to  
return empty strings in the error case and having to call separate  
procs dtrace::errstr and dtrace::errno to get the error message &  
code, the standard tcl approach to this is much simpler:

If a problem occurs, the tcl command throws an error (i.e. in C the  
command function returns TCL_ERROR instead of TCL_OK, which stops  
execution of the tcl program unless caught by [catch]) and the command  
returns the error message as the result and the error code in  
$errorCode.
This also obviates the need for a bogus return "OK" value for API like  
dtrace::close that does not return anything, the absence of an error  
having been thrown is the indication that everything went well...

This is easy to implement with the Tcl C API, essentially in the error  
case you'd just do
	int err = dtrace_errno(dth); const char *msg = dtrace_errmsg(err);
	Tcl_AppendResult(interp, "DTrace error: \"", msg, "\"", NULL);
	Tcl_SetErrorCode(interp, "DTRACE", err, msg, NULL);
	return TCL_ERROR;

the tcl core source has many more examples of this and the manpages  
document tcl error handling thoroughly (c.f. SetResult.3, AddErrInfo. 
3, error.n, catch.n and return.n).

Cheers,

Daniel

-- 
** Daniel A. Steffen                   **
** <mailto:da...@us...>  **

Re: [Tcl-soc2008] [Fwd: Re: My request for comments pretty much ignored]

[Remigiusz J. A. M. <lr...@go...>]

Tomasz Kosiak wrote:
> My quick comment are:
> 
> 000. you may try to use syntax from tcl manual to describe you api

OK, I'll do it.

> 00. In sample program you use dtrace as en ensamble (command with
> subcommands like in string) but in API you describe commands in
> namespace dtrace:: - I suspect that you plan to use [namespace
> ensamble] to implement dtrace ensamble - am I right?

Indeed. Is there something wrong with it?

> 0. I don't understand what you mean by instrumenst in [dtrace::open
> instrument] 

If you use it with instrument==0, then you get a handle that's unable to 
do the instrumentation. This can be useful if you want just to validate 
some scripts. Put a word on that on wiki. This might get really useful 
once I implement elf files (probe compiling and dumping/loading from 
disk), which I don't even try now.

> Is it worth/possible to open dtrace
> lib several times from one process (what about threads - is dtrace
> thread safe?).

It seems possible to do so, at least an additional handle opened in 
mintrace doesn't break anything. This might be quite useful if you have 
a probe that's possibly going to get flooded - you put it into separate 
handle and you'll get dropping only in this one.

> 01. in C dtrace API you've got dtrace_hdl_t *handle, dtrace_prog_t
> *program,  dtrace_proginfo_t *program_info - how to free  program or
> program_info from tcl level?

OK, I didn't even think about that. What would feel good?

> 02. if dtrace_close is so important we should maybe arrange that
> opened handles are automatically closed on program exit - like we do
> with open tcl channels

OK, put that in the wiki. But explicit close call is good for programs, 
that do not want to end at the time of closing dtrace.

> 1. don't oblige user to check errno if some errors occur, but rather
> throw exception (tcl error) - applies to all situation you mentioned
> that error may be checked with dtrace::errstr. It is yet worth
> considering what should be handled by throwing error and what by
> returning empty string - for example problems in program compilation
> could be signaled better with an empty string - some guidance could be
> gain from how we handle i/o in Tcl

In fact error always may be checked with dtrace::errstr. If your script 
fails to compile, it's nice to know what's wrong with it. But you're 
right - some bad things would better interrupt by default. I'd make that:
::dtrace::open
::dtrace::work
::dtrace::aggregations
Or maybe leave ::dtrace::open alone and make everything error on invalid 
handle?
Other suggestions?

> 2. in Tcl option handling style is exemplified by fconfigure
> http://www.tcl.tk/man/tcl8.5/TclCmd/fconfigure.htm

Didn't get that.

> 3. dtrace::info is expandable if you return key-value list, ex.
> {aggregations 5 record 15 matched 4 speculations 3} - maybe use the
> same convention as in fconfigure for reading single item.

No, at the moment I return just the value list. And it is expandable by 
just adding new things (if they ever come) at the end. If it feels 
better, I might change to your suggestion. But for me, it doesn't look 
like worth the extra strings passed.
The convention is nice. But taking into consideration that these are 
just 4 integers, is it worth it? Getting it there is no problem, but 
only if you find it useful.

> 3.5 rather than {{probedata0 probedata1 ...} {output0 output1 ...}} I
> would propose {probedata0 output0 probedata1 output1} that could
> easily be iterated with foreach {probedata output} [dtrace::work]

Um, you misunderstood. These are not coupled. I've just changed that on 
the wiki to don't look that misleading.

> 4. is it worth compiling from files - maybe we should only compile
> from strings and if needed do: set program [read [open $file]] and
> only provide dtrace::compile from strings - but I don't know if dtrace
> programs could contain includes or could be compiled be chunks

Compiling from string supports only one-liners. It might be much more 
convenient to the end user to write his D scripts in the traditional way.
That's libdtrace imposed limit, not mine.

> 5. where dtrace uses FILE* it would be nice to be able to use tcl channels

I use NULL for the FILE*, catch it as buffered output and give to the 
user structured data. If he wanted to just throw it to a file, he'd 
probably use dtrace(1) instead.

> 5. in your example program for signal handling I recommend signal_ext
> (http://www.nyx.net/~mschwart/signal_ext.html) rather than tclx - it
> integrates nicely with tcl event loop - you can take a look into
> extention C source to see how it uses tcl event loop and tcl async api

That's a nice suggestion. I'll investigate into it.

> 6. I feel that it is crucial to integreate dtrace api with tcl event
> loop esspecialy if dtrace lib already has callback interface. If you
> provide only blocking api for TclDtrace it will be hard to use it from
> real Tcl and esspecially Tk programs - every user will have to invent
> yourself how to do it - so it seems reasonable to do it right at the
> Tcl bindings level
> 
> 7. dtrace::work, dtrace::sleep, dtrace::go are useful blocking api,
> but I think we should also think on buffered dtrace api which seems
> suitable to be used from tcl event loop

This is the second time I read that, so I'll elaborate a bit more on this.
As it's described on the wiki - dtrace::sleep is not mandatory. In fact 
it's an ordinary sleep unless we grabbed some processes (which we don't 
at this shape of the API). We can live with whatever source of ticking 
we want.
There are no such things as buffered and blocking APIs. The dtrace_go 
call is not blocking. And (the interesting) callbacks are never called 
without a call to dtrace_work. Data is accumulated, then you call 
dtrace_work, then dtrace_work calls appropriate callbacks for the data 
it has. It will always be your duty to call dtrace_work periodically. 
Therefore *I* do not see a much better way to do this. If you have 
better ideas, go ahead, I'm here to listen.

> Enough. I feel that Daniel will give you more precise advise as he is
> more familiar with dtrace usage - I've only studied your description
> of C dtrace API.

Thank for your input. It's been quite helpful :)

Kind regards,
Remigiusz Modrzejewski