Menu
▾
▴
naviserver-devel
|
From: Zoran V. <zv...@ar...> - 2007-08-06 09:11:41
|
Hi!
After spenging a weekend over this, I know less than
before...
The question is how we are going to improve the Tcl
initialization so people could benefit from that better
than now?
As of today, what we have is:
a.
introspective script based on the "old" way as 4.0 AS
did it (init the interp by loading all Tcl files then
run a script to pull-out namespaces, variables etc and
create a script that get saved with [ns_ictl save]
and run in each new interp by [ns_ictl update])
b.
the ttrace way of setting Tcl traces on selected commands
and have the "things" setup by those commands record in
shared nsv arrays. Then use Tcl [unknown] to pull out the
"things" on as-needed basis from shared arrays
Well, none of those is really good. The a. is limited in
what it "introspects". The b. is pretty complex and in some
situations it does not work as expected (although there is
no design problem there, more probably just a bug or lack a
feature).
Now, we do have [ns_ictl trace] where one can register things
to execute at interp creation, allocation etc.
So, I tried to do a simple thing: for each script executed
during the startup of the server I create
[=10ns_ictl trace [list source $file]]
create-trace.
This "mostly" works, but has two negative effects:
1. all scripts needs to be re-adjusted as they are going to
be sourced more than once (for each new interp)
2. in case 1000's of files are needed for the startup this may
pose significant delay in interp creation
I think that old AS approach is the worst as it is really limited.
The Tcl-trace based approach is better but is known to have some
problems (I do not know which ones), yet, it is pretty powerful
and has some other nice side-effects (less memory footprint).
Lastly, the [ns_ictl trace] is simplest to understand and deploy
(this is also very important) but requires rewriting of existing
scripts and most probably needs somekind of caching to speedup
the load process when 1000's of startup files are involved.
For me, the Tcl-trace or ns_ictl trace are the way to go.
Looking at those two, the latter is more appealing since of the
simplicity, but it may hit us hard speedwise. OTOH, the former
is faster and uses far less resources but is pretty complex in
implementation (this might be done simpler, yes, yet it is always
going to be more complex than ns_ictl trace).
Now, what you all think? Should we do something there or should
we just stay with the old AS introspective script?
Cheers
Zoran
|
|
From: Neophytos D. <k2...@ph...> - 2007-08-06 09:52:48
|
Hi Zoran, I've spend the weekend compiling and re-compiling the kernel and different versions of gcc/glib on two machines (with different CFLAGS/CHOST settings). NS "make test" worked only once on one of the two machines (with CHOST set to "i386-pc-linux-gnu" - GCC 4.2 no longer supports that). The rest of the time it was just hanging on the first test (adp.test). > For me, the Tcl-trace or ns_ictl trace are the way to go. > Looking at those two, the latter is more appealing since of the > simplicity, but it may hit us hard speedwise. OTOH, the former > is faster and uses far less resources but is pretty complex in > implementation (this might be done simpler, yes, yet it is always > going to be more complex than ns_ictl trace). > > Now, what you all think? Should we do something there or should > we just stay with the old AS introspective script? I'm all for Tcl-trace and ns_ictl trace but I gather it would take a considerable amount of time to prepare and test those scripts. Please feel free to let me know what I can do to help --- I'm a bit short on time this month but things should get better after September 8 - getting married :). IMHO, I would prefer mostly a quick migration path from AS (i.e. the old AS introspective script). I have already ported/adapted my code to play nicely with most important changes introduced recently in NS but I'm not quite confident to work with the modified init.tcl from 4.99.1. This is due to the fact that scheduled threads are no longer triggered after init (they seem to be queued successfully but no -sched- thread is created). Best wishes, Neophytos |
|
From: Neophytos D. <k2...@ph...> - 2007-08-06 09:52:48
|
Hi Zoran, I've spend the weekend compiling and re-compiling the kernel and different versions of gcc/glib on two machines (with different CFLAGS/CHOST settings). NS "make test" worked only once on one of the two machines (with CHOST set to "i386-pc-linux-gnu" - GCC 4.2 no longer supports that). The rest of the time it was just hanging on the first test (adp.test). > For me, the Tcl-trace or ns_ictl trace are the way to go. > Looking at those two, the latter is more appealing since of the > simplicity, but it may hit us hard speedwise. OTOH, the former > is faster and uses far less resources but is pretty complex in > implementation (this might be done simpler, yes, yet it is always > going to be more complex than ns_ictl trace). > > Now, what you all think? Should we do something there or should > we just stay with the old AS introspective script? I'm all for Tcl-trace and ns_ictl trace but I gather it would take a considerable amount of time to prepare and test those scripts. Please feel free to let me know what I can do to help --- I'm a bit short on time this month but things should get better after September 8 - getting married :). IMHO, I would prefer mostly a quick migration path from AS (i.e. the old AS introspective script). I have already ported/adapted my code to play nicely with most important changes introduced recently in NS but I'm not quite confident to work with the modified init.tcl from 4.99.1. This is due to the fact that scheduled threads are no longer triggered after init (they seem to be queued successfully but no -sched- thread is created). Best wishes, Neophytos |
|
From: Gustaf N. <ne...@wu...> - 2007-08-06 12:10:22
|
Zoran Vasiljevic schrieb:
> I think that old AS approach is the worst as it is really limited.
>
true. and not scalable
> The Tcl-trace based approach is better but is known to have some
> problems (I do not know which ones), yet, it is pretty powerful
> and has some other nice side-effects (less memory footprint).
> Lastly, the [ns_ictl trace] is simplest to understand and deploy
> (this is also very important) but requires rewriting of existing
> scripts and most probably needs somekind of caching to speedup
> the load process when 1000's of startup files are involved.
>
just for the record: a monster like openacs consists of 2200 files which
might
be merged into the blueprint. openacs has its own package management,
containing
currently 309 packages (from cvs head). Certainly, not many people activate
all packages.
> For me, the Tcl-trace or ns_ictl trace are the way to go.
> Looking at those two, the latter is more appealing since of the
> simplicity, but it may hit us hard speedwise. OTOH, the former
> is faster and uses far less resources but is pretty complex in
> implementation (this might be done simpler, yes, yet it is always
> going to be more complex than ns_ictl trace).
>
i would envision some mixture as a good solution, which might not be so
bloated as the AS version.
- having a kernel of functions, which are available for all connection
and schedules threads.
- having a mechanisms like a naviserver version of "package require"
which can load a "package" on demand. A package can consist
of multiple files, and might require other "packages"
i would see a "package" and the kernel in a simple version
as cat-ed files cached in nsvs.
every connection says during the start of the connection that it
requires certain packages and evals it as required. i would expect
this to reduce the memory requirements, since only needed packages
are loaded into ther interps. For loading the kernel, ns_ictl traces
might an option, these won't be so many files.
however, concerning getting openacs to work on naviserver,
i would prefer having support for some time the good old blueprint way.
there are certainly many consequences on applications using
ns_eval to update the blueprint...
-gustaf
|
|
From: Zoran V. <zv...@ar...> - 2007-08-07 18:09:34
|
Am 06.08.2007 um 14:10 schrieb Gustaf Neumann: > however, concerning getting openacs to work on naviserver, > i would prefer having support for some time the good old blueprint > way. I understand. So the idea is to: a. keep the introspective loader b. design some "package loader" for AS similar to Tcl [package require] For the b. this works even now. All you need to do is to say "package require XYZ" in some library script and this will do the right thing. Provided the extension is pure Tcl OR it is thread-safe. So we need not design anything new. Just install Tcl package and use it from NS as you would from Tcl shell. Well, almost... The exception is Xotcl (and perhaps some other extensions that I'm not aware of) that do tricks with namespace client-data and some other nitty-gritty Tcl internals that Tcl itself does not know how to "clone". The problem is how to "persuade" a generic introspective loader that attempts to clone the "xotcl" namespace not to do that? Or, how to tell it to introscpect something that it does not know about? It "knows" about packages, procs, variables etc (well-known Tcl "things") but it does not "know" about any other data structures provided by extensions. Now, speaking of Xotcl... The older versions of init.tcl script contained a hack for it. Being considered a hack, I just left it out, with the hope to ditch that altogether in favour of a ns_ictl trace or Trace (lazyloader) as I'm still convinced that those are better (faster, simpler, more generic). Now, that popular demand is gearing towards "keep the a." we must think of how to parametrize the introspective loader so it can be "told" what to do and what not to do in certain cases. Right? I believe this will eventually lead to the same result as Ttrace, as I already followed that path some time ago, but I will let all you bright people think that over. Perhaps there is an elegant way I'm not aware of ;-) Cheers, Zoran |
|
From: Vlad S. <vl...@cr...> - 2007-08-06 16:44:53
|
I would keep old initialization if i need speed, if performance is going to suffer with Tcl/ns_ictl approach, then for fast/busy sites we need to be able to use old(fast) way. Otherwise, the server will be limited for non-realtime or personal web sites usage only. Zoran Vasiljevic wrote: > Hi! > > After spenging a weekend over this, I know less than > before... > > The question is how we are going to improve the Tcl > initialization so people could benefit from that better > than now? > > As of today, what we have is: > > a. > introspective script based on the "old" way as 4.0 AS > did it (init the interp by loading all Tcl files then > run a script to pull-out namespaces, variables etc and > create a script that get saved with [ns_ictl save] > and run in each new interp by [ns_ictl update]) > > b. > the ttrace way of setting Tcl traces on selected commands > and have the "things" setup by those commands record in > shared nsv arrays. Then use Tcl [unknown] to pull out the > "things" on as-needed basis from shared arrays > > Well, none of those is really good. The a. is limited in > what it "introspects". The b. is pretty complex and in some > situations it does not work as expected (although there is > no design problem there, more probably just a bug or lack a > feature). > > Now, we do have [ns_ictl trace] where one can register things > to execute at interp creation, allocation etc. > So, I tried to do a simple thing: for each script executed > during the startup of the server I create > [ns_ictl trace [list source $file]] > create-trace. > > This "mostly" works, but has two negative effects: > 1. all scripts needs to be re-adjusted as they are going to > be sourced more than once (for each new interp) > 2. in case 1000's of files are needed for the startup this may > pose significant delay in interp creation > > I think that old AS approach is the worst as it is really limited. > The Tcl-trace based approach is better but is known to have some > problems (I do not know which ones), yet, it is pretty powerful > and has some other nice side-effects (less memory footprint). > Lastly, the [ns_ictl trace] is simplest to understand and deploy > (this is also very important) but requires rewriting of existing > scripts and most probably needs somekind of caching to speedup > the load process when 1000's of startup files are involved. > > For me, the Tcl-trace or ns_ictl trace are the way to go. > Looking at those two, the latter is more appealing since of the > simplicity, but it may hit us hard speedwise. OTOH, the former > is faster and uses far less resources but is pretty complex in > implementation (this might be done simpler, yes, yet it is always > going to be more complex than ns_ictl trace). > > Now, what you all think? Should we do something there or should > we just stay with the old AS introspective script? > > Cheers > Zoran > > > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. > Still grepping through log files to find problems? Stop. > Now Search log events and configuration files using AJAX and a browser. > Download your FREE copy of Splunk now >> http://get.splunk.com/ > _______________________________________________ > naviserver-devel mailing list > nav...@li... > https://lists.sourceforge.net/lists/listinfo/naviserver-devel > |
|
From: Zoran V. <zv...@ar...> - 2007-08-06 16:54:47
|
Am 06.08.2007 um 18:47 schrieb Vlad Seryakov: > I would keep old initialization if i need speed, if performance is > going > to suffer with Tcl/ns_ictl approach, then for fast/busy sites we > need to > be able to use old(fast) way. Otherwise, the server will be limited > for > non-realtime or personal web sites usage only. What you haven't seen perhaps is... I modified the "old way" NOT to use introspective script, rather to use Tcl command traces :-) So Tcl command traces work allright and also much more "logical" then the old ugly and error-prone code. The only difference to ttrace (lazyloader=true) is that I do not use [unknown] to reload commands on the fly, rather all "things" are synthetized in a large script that loads everything that is registered, at once. That's why I told about wanting to go to ns_ictl trace or tcl-cmd-traces (nstrace.tcl). Cheers, Zoran |
|
From: Zoran V. <zv...@ar...> - 2007-08-06 17:08:12
|
Am 06.08.2007 um 18:54 schrieb Zoran Vasiljevic: > > I modified the "old way" NOT to use introspective > script, rather to use Tcl command traces :-) OK, not really *everything*. Tcl namespace serialization (variables/procedures) is still done "the old way". The problem with that that a namespace is a difficult beast to handle. Some OO extensions (Xotcl) create their own namespaces which you just cannot serialize the Tcl-way... |
|
From: Stephen D. <sd...@gm...> - 2007-08-06 17:01:48
|
On 8/6/07, Zoran Vasiljevic <zv...@ar...> wrote: > Hi! > > After spenging a weekend over this, I know less than > before... > > The question is how we are going to improve the Tcl > initialization so people could benefit from that better > than now? > > As of today, what we have is: > > a. > introspective script based on the "old" way as 4.0 AS > did it (init the interp by loading all Tcl files then > run a script to pull-out namespaces, variables etc and > create a script that get saved with [ns_ictl save] > and run in each new interp by [ns_ictl update]) > > b. > the ttrace way of setting Tcl traces on selected commands > and have the "things" setup by those commands record in > shared nsv arrays. Then use Tcl [unknown] to pull out the > "things" on as-needed basis from shared arrays > > Well, none of those is really good. The a. is limited in > what it "introspects". The b. is pretty complex and in some > situations it does not work as expected (although there is > no design problem there, more probably just a bug or lack a > feature). > > Now, we do have [ns_ictl trace] where one can register things > to execute at interp creation, allocation etc. > So, I tried to do a simple thing: for each script executed > during the startup of the server I create > [ ns_ictl trace [list source $file]] > create-trace. > > This "mostly" works, but has two negative effects: > 1. all scripts needs to be re-adjusted as they are going to > be sourced more than once (for each new interp) > 2. in case 1000's of files are needed for the startup this may > pose significant delay in interp creation > > I think that old AS approach is the worst as it is really limited. > The Tcl-trace based approach is better but is known to have some > problems (I do not know which ones), yet, it is pretty powerful > and has some other nice side-effects (less memory footprint). > Lastly, the [ns_ictl trace] is simplest to understand and deploy > (this is also very important) but requires rewriting of existing > scripts and most probably needs somekind of caching to speedup > the load process when 1000's of startup files are involved. > > For me, the Tcl-trace or ns_ictl trace are the way to go. > Looking at those two, the latter is more appealing since of the > simplicity, but it may hit us hard speedwise. OTOH, the former > is faster and uses far less resources but is pretty complex in > implementation (this might be done simpler, yes, yet it is always > going to be more complex than ns_ictl trace). > > Now, what you all think? Should we do something there or should > we just stay with the old AS introspective script? > What about making naviserver modules Tcl packages? That is, instead of saying: rewrite your scripts to survive being loaded multiple times via ns_ictl callbacks; we say: write Tcl packages. I think we want Tcl packages for a variety of reasons, but interestingly they also have a lazy loading mechanism via pkgIndex.tcl scripts. What do you think? I think we need to maintain backwards compatibility with the existing scheme (however it's actually implemented) regardless of what else we do, right? |
|
From: Stephen D. <sd...@gm...> - 2007-08-06 17:24:31
|
On 8/6/07, Zoran Vasiljevic <zv...@ar...> wrote:
>
> As of today, what we have is:
>
> a.
> introspective script based on the "old" way as 4.0 AS
> did it (init the interp by loading all Tcl files then
> run a script to pull-out namespaces, variables etc
"variables etc." -- do we support variables?
We have a pretty explicit (and simple) end-user programming model: the
Tcl code that runs for each connection begins in a known environment,
which includes all the procs/commands you need, plus a clean slate as
far as global variables go. Connections are independent.
#
# ns_cleanupvars --
#
# Destroy global variables. Namespaced variables
# are left and could introduce unwanted state.
#
IIRC namespaced variables aren't touched because it would be too expensive.
Do they get cloned at startup though? This test:
test init-1.4 {namespace variables do not get cloned} -body {
ns_job wait $qid [ns_job queue $qid { info exists
testnamespace::testvariable2 }]
} -result 0
..fails when lazyloader=false, but passes when lazyloader=true. What's
going on here?
|
|
From: Zoran V. <zv...@ar...> - 2007-08-06 19:38:07
|
Am 06.08.2007 um 19:24 schrieb Stephen Deasey:
> On 8/6/07, Zoran Vasiljevic <zv...@ar...> wrote:
>>
>> As of today, what we have is:
>>
>> a.
>> introspective script based on the "old" way as 4.0 AS
>> did it (init the interp by loading all Tcl files then
>> run a script to pull-out namespaces, variables etc
>
> "variables etc." -- do we support variables?
If somebody in any Tcl library file says:
variable dummy
set dummy 1
the "dummy" gets initiated with the value 1 every
time somebody creates a new fresh interp.
>
> We have a pretty explicit (and simple) end-user programming model: the
> Tcl code that runs for each connection begins in a known environment,
> which includes all the procs/commands you need, plus a clean slate as
> far as global variables go. Connections are independent.
>
> #
> # ns_cleanupvars --
> #
> # Destroy global variables. Namespaced variables
> # are left and could introduce unwanted state.
> #
>
> IIRC namespaced variables aren't touched because it would be too
> expensive.
That is right (and wrong). Wrong because of the hidden state
introduced by that. But, again, it is too expensive to catch
those things. It might be simpler to ditch the interp and
create it again (with lazyloader=true this would be a snap).
>
> Do they get cloned at startup though? This test:
>
> test init-1.4 {namespace variables do not get cloned} -body {
> ns_job wait $qid [ns_job queue $qid { info exists
> testnamespace::testvariable2 }]
> } -result 0
>
> ..fails when lazyloader=false, but passes when lazyloader=true. What's
> going on here?
Eh, no idea but I'm going to find out... Stay tuned.
Cheers
Zoran
>
> ----------------------------------------------------------------------
> ---
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems? Stop.
> Now Search log events and configuration files using AJAX and a
> browser.
> Download your FREE copy of Splunk now >> http://get.splunk.com/
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
|
|
From: Zoran V. <zv...@ar...> - 2007-08-07 17:51:26
|
Am 06.08.2007 um 19:24 schrieb Stephen Deasey:
> IIRC namespaced variables aren't touched because it would be too
> expensive.
>
> Do they get cloned at startup though? This test:
>
> test init-1.4 {namespace variables do not get cloned} -body {
> ns_job wait $qid [ns_job queue $qid { info exists
> testnamespace::testvariable2 }]
> } -result 0
>
> ..fails when lazyloader=false, but passes when lazyloader=true. What's
> going on here?
Explanation....
Normally the namespace'd variable are copied by the
introspective loader based solely on their existence.
The lazyloader cannot really know about variable
(it is far too expensive to trace [set] command
hence I left that out (but is strictly speaking
possible and would work). So, it needs to be told
with the "variable" command that testvariable2 is
a namespaced variable that needs to be copied:
namespace eval testnamespace {
variable testvariable2
set testvariable2 dummy
}
This would get recognized by the lazy loader and it
would properly copy/clone the variable.
We MUST copy the namespace variables as some packages
uses them to remember some state. So it is absolutely
crucial. The introspective loader does not care much
(it is dumb); if the variable is there (info vars)
it gets copied, regardles how the var got there.
In a sense, the ttrace (lazyloader) is more appropriate
for such things as you can really tell him
(with [variable]) that it needs to be copied since it
is a namespace variable whereas other (possibly just junk)
vars need not.
Hope this makes sense...
Cheers
Zoran
|
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X