naviserver-devel — Developer discussion

Re: [naviserver-devel] Proposal: just an idea about config syntax

[Zoran V. <zv...@ar...>]

Am 11.08.2007 um 15:26 schrieb Zoran Vasiljevic:

>  This would implicitly
> mean that you should not have config option named
> after any built-in Tcl command, yes.

One can create a virgin interp and then scratch
all commands from it and just define the [unknown].
I've never done that but in theory, this could work. 

Re: [naviserver-devel] Proposal: just an idea about config syntax

[Zoran V. <zv...@ar...>]

Am 11.08.2007 um 15:15 schrieb Stephen Deasey:

> One problem with this is you can't add comments:
>
> section fastpath {
>     cache false    ;# Turn off caching
>     ...
> }

Well, you could be overriding the [unknown] and
eval'in the whole block. This would implicitly
mean that you should not have config option named
after any built-in Tcl command, yes.

Altogether, I do not know if this all is worth
the trouble. It looks nicer, true. But is it
worth it?

Cheers
Zoran

Re: [naviserver-devel] Proposal: just an idea about config syntax

[Zoran V. <zv...@ar...>]

Hi!

If you are going to that extent, why just not:

   ns_section section_name ?script?

so you are aotomatically backwards compatible.
Either you write:

    ns_section ns/fastpath {
         cache            false
         cachemaxsize     512000
    }

or omit the block and just default to
using ns_param.

Also, I believe that sections should nest:

   ns_section ns/some/section {
      param value
      ns_section my/section {
         param1 value1
      }
   }

where param and value are assigned to
  ns/some/section
and param1 and value1 are assigned to
  ns/some/section/my/section

I would't use namespaces for config storage
It be independent of Tcl interpreter as it
is used throughout the code and, the startup
interp gets trashed after startup anyway.

Cheers
Zoran

> For example instead of this (or it could be used at the same time,  
> it is
> not a replacement)
>
> ns_section      "ns/fastpath"
> ns_param        cache                   false
> ns_param        cachemaxsize            5120000
> ns_param        cachemaxentry           512000
> ns_param        mmap                    false
>
> it may look like this (still Tcl)
>
> section "fastpath" {
>      cache                   false
>      cachemaxsize            5120000
>      cachemaxentry           512000
>      mmap                    false
> }
>
> section "server/${servername}" {
>      connsperthread          0
>      flushcontent            false
>      maxconnections          100
>      maxthreads              10
>      minthreads              0
>      threadtimeout           120
> }
>
>
> As you can see this is Tcl command section
>
> proc section { name args } {
>
>     ns_section ns/$name
>     foreach { key val } $args {
>        ns_param $key $val
>     }
> }
>
> But config looks like more 21 century and more structured and  
> easier to
> read. Very similar to lighttpd by the way.
>
>
> -- 
> Vlad Seryakov
> vl...@cr...
> http://www.crystalballinc.com/vlad/
>
> ---------------------------------------------------------------------- 
> ---
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems?  Stop.
> Now Search log events and configuration files using AJAX and a  
> browser.
> Download your FREE copy of Splunk now >>  http://get.splunk.com/
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] Proposal: just an idea about config syntax

[Stephen D. <sd...@gm...>]

On 8/10/07, Vlad Seryakov <vl...@cr...> wrote:
>
> section "fastpath" {
>      cache                   false
>      cachemaxsize            5120000
>      cachemaxentry           512000
>      mmap                    false
> }


One problem with this is you can't add comments:

section fastpath {
    cache false    ;# Turn off caching
    ...
}

With the recursive version, you can't add full line comments to
sections.  You could switch to a real parser, but then you loose the
simplicity of using pure Tcl, which is kind of the appeal.


On 8/10/07, Brett Schwarz <bre...@ya...> wrote:
>
> namespace eval fastpath {
>       set cache                    false
>       set cachemaxsize        5120000
>       set cachemaxentry       512000
>       set mmap                    false
> }


This solves the problem of comments by making the body eval'able and
using the set command.  But it breaks duplicate values:

    ns_section  "ns/server/${servername}/adp"
    ns_param   map    "/*.adp"
    ns_param   map    "/*.u_adp"
    ns_param   map    "/*.gb_adp"
    ns_param   map    "/*.sjis_adp"


On 8/10/07, Jeff Rogers <dv...@di...> wrote:
>
> I like the general look and that it can be implemented in pure tcl
> (heck, even by just sourcing the implementation at the top of your
> current config).


I think this nails it. The benefit of the Tcl config file is that you
make it do whatever you want. In this case, with 5 lines of code.
Isn't this validation that the current scheme is working well and this
is all that's needed?

I've started a new wiki page to gather config file tricks 'n tips:

  http://naviserver.sourceforge.net/wiki/index.php/Tcl_Configuration_File



(btw. I updated the wiki software and SF updated their servers. Seems
pretty responsive.)

Re: [naviserver-devel] Proposal: just an idea about config syntax

[Vlad S. <vl...@cr...>]

By implementing it in C and using Tcl_Eval ascripts will be executed at 
global level and all variables will be visible.

It is just may result in supporting 2 diferent config styles which may 
add maintanence and confusion but overall i personally like new style better

Jeff Rogers wrote:
> Vlad Seryakov wrote:
> 
> I like the general look and that it can be implemented in pure tcl 
> (heck, even by just sourcing the implementation at the top of your 
> current config).
> 
>> it may look like this (still Tcl)
>>
>> section "fastpath" {
>>      cache                   false
>>      cachemaxsize            5120000
>>      cachemaxentry           512000
>>      mmap                    false
>> }
>>
>> section "server/${servername}" {
>>      connsperthread          0
>>      flushcontent            false
>>      maxconnections          100
>>      maxthreads              10
>>      minthreads              0
>>      threadtimeout           120
>> }
> 
> Alot of the section names are heirarchical too so could this concept be 
> taken further?  e.g.,
> 
> server "server1" {
>      directoryfile index.html
>      pageroot      /root
>      section "tcl" {
>          library   /root/server1/lib/tcl
>      }
>      section "modules" {
>          nslog  ${bindir}/nslog.so
>      }
> }
> 
> 
> 
>> As you can see this is Tcl command section
>>
>> proc section { name args } {
>>
>>     ns_section ns/$name
>>     foreach { key val } $args {
>>        ns_param $key $val
>>     }
>> }
> 
> It would need to be uplevel-ed if variable expansion is to take place 
> but that's pretty minor.
> 
> -J
> 
>> But config looks like more 21 century and more structured and easier to 
>> read. Very similar to lighttpd by the way.
>>
>>
> 
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems?  Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >>  http://get.splunk.com/
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

Re: [naviserver-devel] Proposal: just an idea about config syntax

[Jeff R. <dv...@di...>]

Vlad Seryakov wrote:

I like the general look and that it can be implemented in pure tcl 
(heck, even by just sourcing the implementation at the top of your 
current config).

> it may look like this (still Tcl)
> 
> section "fastpath" {
>      cache                   false
>      cachemaxsize            5120000
>      cachemaxentry           512000
>      mmap                    false
> }
> 
> section "server/${servername}" {
>      connsperthread          0
>      flushcontent            false
>      maxconnections          100
>      maxthreads              10
>      minthreads              0
>      threadtimeout           120
> }

Alot of the section names are heirarchical too so could this concept be 
taken further?  e.g.,

server "server1" {
     directoryfile index.html
     pageroot      /root
     section "tcl" {
         library   /root/server1/lib/tcl
     }
     section "modules" {
         nslog  ${bindir}/nslog.so
     }
}



> As you can see this is Tcl command section
> 
> proc section { name args } {
> 
>     ns_section ns/$name
>     foreach { key val } $args {
>        ns_param $key $val
>     }
> }

It would need to be uplevel-ed if variable expansion is to take place 
but that's pretty minor.

-J

> 
> But config looks like more 21 century and more structured and easier to 
> read. Very similar to lighttpd by the way.
> 
> 

Re: [naviserver-devel] Proposal: just an idea about config syntax

[Brett S. <bre...@ya...>]

I always wondered if Tcl's namespace mechanism could have been used for the=
 config file, and by your example below, it is tending towards that. Again,=
 the idea would be to utilize, where possible, existing Tcl mechanisms. So,=
 something like:=0A=0Anamespace eval fastpath {=0A      set cache          =
          false=0A      set cachemaxsize        5120000=0A      set cachema=
xentry       512000=0A      set mmap                    false=0A}=0A=0A .=
=0A .=0A .=0A=0Anamespace eval server::${servername} {=0A     set connspert=
hread          0=0A     set flushcontent            false=0A     set maxcon=
nections          100=0A     set maxthreads              10=0A     set mint=
hreads              0=0A     set threadtimeout           120=0A}=0A=0AWhat'=
s good about this as well, is if you needed to access that variable later o=
n in the config file, then you could just use the FQ name, i.e.=0A=0A  set =
some_val [$fastpath::cache]=0A=0A=0AHaven't really thought this all the way=
 through, so there might be some pitfalls.=0A=0AJust an idea,=0A=0A    --br=
ett=0A=0A----- Original Message ----=0AFrom: Vlad Seryakov <vlad@crystalbal=
linc.com>=0ATo: nav...@li...=0ASent: Friday, Augu=
st 10, 2007 3:37:07 PM=0ASubject: Re: [naviserver-devel] Proposal: just an =
idea about config syntax=0A=0AAlso if section command be made recursive-awa=
re, so it will track =0Acurrent position then the syntax can be event more =
eaiser=0A=0A=0Asection "server/$servername" {=0A     connsperthread        =
  0=0A     flushcontent            false=0A     maxconnections          100=
=0A     maxthreads              10=0A     minthreads              0=0A     =
threadtimeout           120=0A=0A     section "modules" {=0A         nsdb  =
   nsdb.so=0A         nssock   nssock.so=0A     }=0A=0A     section "adp" {=
=0A        map               "/*.adp"=0A        enableexpire      false=0A =
    }=0A=0A     section "tcl" {=0A        nsvbuckets              16=0A    =
    library                 ${homedir}/modules/tcl=0A        lazyloader    =
          false=0A     }=0A=0A     section "module/nssock" {=0A         por=
t                    $port=0A         address                 $address=0A  =
   }=0A}=0A=0A=0AVlad Seryakov wrote:=0A> For example instead of this (or i=
t could be used at the same time, it is =0A> not a replacement)=0A> =0A> ns=
_section      "ns/fastpath"=0A> ns_param        cache                   fal=
se=0A> ns_param        cachemaxsize            5120000=0A> ns_param        =
cachemaxentry           512000=0A> ns_param        mmap                    =
false=0A> =0A> it may look like this (still Tcl)=0A> =0A> section "fastpath=
" {=0A>      cache                   false=0A>      cachemaxsize           =
 5120000=0A>      cachemaxentry           512000=0A>      mmap             =
       false=0A> }=0A> =0A> section "server/${servername}" {=0A>      conns=
perthread          0=0A>      flushcontent            false=0A>      maxcon=
nections          100=0A>      maxthreads              10=0A>      minthrea=
ds              0=0A>      threadtimeout           120=0A> }=0A> =0A> =0A> =
As you can see this is Tcl command section=0A> =0A> proc section { name arg=
s } {=0A> =0A>     ns_section ns/$name=0A>     foreach { key val } $args {=
=0A>        ns_param $key $val=0A>     }=0A> }=0A> =0A> But config looks li=
ke more 21 century and more structured and easier to =0A> read. Very simila=
r to lighttpd by the way.=0A> =0A> =0A=0A-- =0AVlad Seryakov=0Avlad@crystal=
ballinc.com=0Ahttp://www.crystalballinc.com/vlad/=0A=0A--------------------=
-----------------------------------------------------=0AThis SF.net email i=
s sponsored by: Splunk Inc.=0AStill grepping through log files to find prob=
lems?  Stop.=0ANow Search log events and configuration files using AJAX and=
 a browser.=0ADownload your FREE copy of Splunk now >>  http://get.splunk.c=
om/=0A_______________________________________________=0Anaviserver-devel ma=
iling list=0An...@li...=0Ahttps://lists.sourcef=
orge.net/lists/listinfo/naviserver-devel=0A=0A=0A=0A=0A=0A       =0A_______=
___________________________________________________________________________=
__=0ANeed a vacation? Get great deals=0Ato amazing places on Yahoo! Travel.=
=0Ahttp://travel.yahoo.com/

Re: [naviserver-devel] Proposal: just an idea about config syntax

[Vlad S. <vl...@cr...>]

Also if section command be made recursive-aware, so it will track 
current position then the syntax can be event more eaiser


section "server/$servername" {
     connsperthread          0
     flushcontent            false
     maxconnections          100
     maxthreads              10
     minthreads              0
     threadtimeout           120

     section "modules" {
         nsdb     nsdb.so
         nssock   nssock.so
     }

     section "adp" {
        map               "/*.adp"
        enableexpire      false
     }

     section "tcl" {
        nsvbuckets              16
        library                 ${homedir}/modules/tcl
        lazyloader              false
     }

     section "module/nssock" {
         port                    $port
         address                 $address
     }
}


Vlad Seryakov wrote:
> For example instead of this (or it could be used at the same time, it is 
> not a replacement)
> 
> ns_section      "ns/fastpath"
> ns_param        cache                   false
> ns_param        cachemaxsize            5120000
> ns_param        cachemaxentry           512000
> ns_param        mmap                    false
> 
> it may look like this (still Tcl)
> 
> section "fastpath" {
>      cache                   false
>      cachemaxsize            5120000
>      cachemaxentry           512000
>      mmap                    false
> }
> 
> section "server/${servername}" {
>      connsperthread          0
>      flushcontent            false
>      maxconnections          100
>      maxthreads              10
>      minthreads              0
>      threadtimeout           120
> }
> 
> 
> As you can see this is Tcl command section
> 
> proc section { name args } {
> 
>     ns_section ns/$name
>     foreach { key val } $args {
>        ns_param $key $val
>     }
> }
> 
> But config looks like more 21 century and more structured and easier to 
> read. Very similar to lighttpd by the way.
> 
> 

-- 
Vlad Seryakov
vl...@cr...
http://www.crystalballinc.com/vlad/

[naviserver-devel] Proposal: just an idea about config syntax

[Vlad S. <vl...@cr...>]

For example instead of this (or it could be used at the same time, it is 
not a replacement)

ns_section      "ns/fastpath"
ns_param        cache                   false
ns_param        cachemaxsize            5120000
ns_param        cachemaxentry           512000
ns_param        mmap                    false

it may look like this (still Tcl)

section "fastpath" {
     cache                   false
     cachemaxsize            5120000
     cachemaxentry           512000
     mmap                    false
}

section "server/${servername}" {
     connsperthread          0
     flushcontent            false
     maxconnections          100
     maxthreads              10
     minthreads              0
     threadtimeout           120
}


As you can see this is Tcl command section

proc section { name args } {

    ns_section ns/$name
    foreach { key val } $args {
       ns_param $key $val
    }
}

But config looks like more 21 century and more structured and easier to 
read. Very similar to lighttpd by the way.


-- 
Vlad Seryakov
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] aolserver incompatibilities

[Gustaf N. <ne...@wu...>]

Stephen Deasey schrieb:
>> .
>>
>> Do you have details?  Most of these sound more like bugs or accidental
>> changes. Don't work around bugs... :-)
>>     
from my referenced posting:

>  The argument passing for filters is different in naviserver, in 
addition, the useless conn arguments are removed.

The details for the filters are in the diff.

one example for the removed dummy connection argument is ns_headers.
Grep for connid in aolserver nsd/tclresp. most are optional, but the 
question
is, how serious one is about compatibility.

are you saying, that ns_cache is now fully compatible in naviserver with
the version in aolserver 4.5?
 
-gustaf

Re: [naviserver-devel] aolserver incompatibilities

[Stephen D. <sd...@gm...>]

On 8/10/07, Gustaf Neumann <ne...@wu...> wrote:
> Zoran Vasiljevic schrieb:
> > Yet, so far I do not know anybody who tried to replace
> > the AS with our code. Whatever, we would help you get
> > it up and running if you like. Perhaps this is a good chance
> > to start something like a compatibility-list.
> >
> For naviserver 4.99.1, the following changes were needed to run openacs
> on top of it.
>
> http://openacs.org/forums/message-view?message_id=1148743
>
> maybe a first start for the list...


Do you have details?  Most of these sound more like bugs or accidental
changes. Don't work around bugs... :-)

Re: [naviserver-devel] aolserver incompatibilities

[Stephen D. <sd...@gm...>]

On 8/9/07, Jeff Rogers <dv...@di...> wrote:
> Is there a simple, concise, and comprehensive[*] list of specific
> incompatibilities between the current versions of ns and aolserver?
> I'm considering moving my aolserver code to naviserver, I want to know
> what pitfalls I can expect.
>
> Thanks,
> -J
>
> [*] - irony intentional :)


There shouldn't be anything huge. I don't recall us taking any
genuinely useful or not-a-bug feature and deliberately breaking it.

But there's been some changes...

You will of course have to recompile any C modules you're using.

We have ns_cache_* commands. But there's nothing stopping you from
also loading the old ns_cache module. Vald imported a version of that
into our CVS -- shout out if it doesn't work.

We dropped the optional connid argument from many of the Tcl commands
such as ns_register_proc and ns_return*, but this has been optional
and deprecated in AOLserver since the last century so consider this an
opportunity to update your old code :-)

Mainly, the thing you're going to notice is that quite a lot has moved
around in the config file. There's a sample-config.tcl but it could be
better. One new feature we have here is logging of config values as
they're accessed. Turn on dev level debugging and you can see exactly
what the code is expecting to find as the server starts up.

Re: [naviserver-devel] aolserver incompatibilities

[Gustaf N. <ne...@wu...>]

Zoran Vasiljevic schrieb:
> Yet, so far I do not know anybody who tried to replace
> the AS with our code. Whatever, we would help you get
> it up and running if you like. Perhaps this is a good chance
> to start something like a compatibility-list.
>   
For naviserver 4.99.1, the following changes were needed to run openacs
on top of it.

http://openacs.org/forums/message-view?message_id=1148743

maybe a first start for the list...

-gustaf

Re: [naviserver-devel] aolserver incompatibilities

[Zoran V. <zv...@ar...>]

Am 09.08.2007 um 23:04 schrieb Jeff Rogers:

> Is there a simple, concise, and comprehensive[*] list of specific
> incompatibilities between the current versions of ns and aolserver?
> I'm considering moving my aolserver code to naviserver, I want to know
> what pitfalls I can expect.

There is no simple, concise and comprehensive list of
specific incompatibilities unfortunately. The best thing
is to fetch the cvs head, compile and give it a try.
We wanted to tag the current cvs with 5.0 release but we
are still lagging behind (my fault mostly).

Command-wise (ns_*) it should be backwards compatible
to 4.0 although we ported much of the interesting things
from 4.5. We also improved quite a few of server
internals but from the Tcl-API you should be more/less OK.
C-API is in some places considerably changed but for most
of the things we left older compatibility wrappers.

Yet, so far I do not know anybody who tried to replace
the AS with our code. Whatever, we would help you get
it up and running if you like. Perhaps this is a good chance
to start something like a compatibility-list.

Cheers
Zoran

[naviserver-devel] aolserver incompatibilities

[Jeff R. <dv...@di...>]

Is there a simple, concise, and comprehensive[*] list of specific 
incompatibilities between the current versions of ns and aolserver?
I'm considering moving my aolserver code to naviserver, I want to know 
what pitfalls I can expect.

Thanks,
-J

[*] - irony intentional :)

Re: [naviserver-devel] Tcl inits

[Zoran V. <zv...@ar...>]

Am 06.08.2007 um 14:10 schrieb Gustaf Neumann:

> however, concerning getting openacs to work on naviserver,
> i would prefer having support for some time the good old blueprint  
> way.

I understand. So the idea is to:

   a. keep the introspective loader
   b. design some "package loader" for AS similar to Tcl [package  
require]


For the b. this works even now.  All you need to do is to
say "package require XYZ" in some library script and this
will do the right thing. Provided the extension is pure
Tcl OR it is thread-safe. So we need not design anything
new. Just install Tcl package and use it from NS as you
would from Tcl shell.

Well, almost...

The exception is Xotcl (and perhaps some other extensions
that I'm not aware of) that do tricks with namespace
client-data and some other nitty-gritty Tcl internals
that Tcl itself does not know how to "clone".

The problem is how to "persuade" a generic introspective
loader that attempts to clone the "xotcl" namespace
not to do that? Or, how to tell it to introscpect something
that it does not know about? It "knows" about packages, procs,
variables etc (well-known Tcl "things") but it does not
"know" about any other data structures provided by extensions.

Now, speaking of Xotcl...
The older versions of init.tcl script contained a hack for
it. Being considered a hack, I just left it out, with the
hope to ditch that altogether in favour of a ns_ictl trace
or Trace (lazyloader) as I'm still convinced that those are
better (faster, simpler, more generic).

Now, that popular demand is gearing towards "keep the a."
we must think of how to parametrize the introspective loader
so it can be "told" what to do and what not to do in certain
cases. Right? I believe this will eventually lead to the same
result as Ttrace, as I already followed that path some time
ago, but I will let all you bright people think that over.
Perhaps there is an elegant way I'm not aware of ;-)

Cheers,
Zoran

Re: [naviserver-devel] Tcl inits

[Zoran V. <zv...@ar...>]

Am 06.08.2007 um 19:24 schrieb Stephen Deasey:

> IIRC namespaced variables aren't touched because it would be too  
> expensive.
>
> Do they get cloned at startup though?  This test:
>
> test init-1.4 {namespace variables do not get cloned} -body {
>     ns_job wait $qid [ns_job queue $qid { info exists
> testnamespace::testvariable2 }]
> } -result 0
>
> ..fails when lazyloader=false, but passes when lazyloader=true. What's
> going on here?

Explanation....

Normally the namespace'd variable are copied by the
introspective loader based solely on their existence.

The lazyloader cannot really know about variable
(it is far too expensive to trace [set] command
hence I left that out (but is strictly speaking
possible and would work). So, it needs to be told
with the "variable" command that testvariable2 is
a namespaced variable that needs to be copied:

   namespace eval testnamespace {
       variable testvariable2
       set testvariable2 dummy
   }

This would get recognized by the lazy loader and it
would properly copy/clone the variable.

We MUST copy the namespace variables as some packages
uses them to remember some state. So it is absolutely
crucial. The introspective loader does not care much
(it is dumb); if the variable is there (info vars)
it gets copied, regardles how the var got there.
In a sense, the ttrace (lazyloader) is more appropriate
for such things as you can really tell him
(with [variable]) that it needs to be copied since it
is a namespace variable whereas other (possibly just junk)
vars need not.

Hope this makes sense...

Cheers
Zoran

Re: [naviserver-devel] Tcl inits

[Zoran V. <zv...@ar...>]

Am 06.08.2007 um 19:24 schrieb Stephen Deasey:

> On 8/6/07, Zoran Vasiljevic <zv...@ar...> wrote:
>>
>> As of today, what we have is:
>>
>> a.
>>     introspective script based on the "old" way as 4.0 AS
>>     did it (init the interp by loading all Tcl files then
>>     run a script to pull-out namespaces, variables etc
>
> "variables etc."  -- do we support variables?

If somebody in any Tcl library file says:

   variable dummy
   set dummy 1

the "dummy" gets initiated with the value 1 every
time somebody creates a new fresh interp.

>
> We have a pretty explicit (and simple) end-user programming model: the
> Tcl code that runs for each connection begins in a known environment,
> which includes all the procs/commands you need, plus a clean slate as
> far as global variables go. Connections are independent.
>
> #
> # ns_cleanupvars --
> #
> #     Destroy global variables. Namespaced variables
> #     are left and could introduce unwanted state.
> #
>
> IIRC namespaced variables aren't touched because it would be too  
> expensive.

That is right (and wrong). Wrong because of the hidden state
introduced by that. But, again, it is too expensive to catch
those things. It might be simpler to ditch the interp and
create it again (with lazyloader=true this would be a snap).

>
> Do they get cloned at startup though?  This test:
>
> test init-1.4 {namespace variables do not get cloned} -body {
>     ns_job wait $qid [ns_job queue $qid { info exists
> testnamespace::testvariable2 }]
> } -result 0
>
> ..fails when lazyloader=false, but passes when lazyloader=true. What's
> going on here?

Eh, no idea but I'm going to find out... Stay tuned.

Cheers
Zoran

>
> ---------------------------------------------------------------------- 
> ---
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems?  Stop.
> Now Search log events and configuration files using AJAX and a  
> browser.
> Download your FREE copy of Splunk now >>  http://get.splunk.com/
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] Tcl inits

[Stephen D. <sd...@gm...>]

On 8/6/07, Zoran Vasiljevic <zv...@ar...> wrote:
>
> As of today, what we have is:
>
> a.
>     introspective script based on the "old" way as 4.0 AS
>     did it (init the interp by loading all Tcl files then
>     run a script to pull-out namespaces, variables etc

"variables etc."  -- do we support variables?

We have a pretty explicit (and simple) end-user programming model: the
Tcl code that runs for each connection begins in a known environment,
which includes all the procs/commands you need, plus a clean slate as
far as global variables go. Connections are independent.

#
# ns_cleanupvars --
#
#     Destroy global variables. Namespaced variables
#     are left and could introduce unwanted state.
#

IIRC namespaced variables aren't touched because it would be too expensive.

Do they get cloned at startup though?  This test:

test init-1.4 {namespace variables do not get cloned} -body {
    ns_job wait $qid [ns_job queue $qid { info exists
testnamespace::testvariable2 }]
} -result 0

..fails when lazyloader=false, but passes when lazyloader=true. What's
going on here?

Re: [naviserver-devel] Tcl inits

[Zoran V. <zv...@ar...>]

Am 06.08.2007 um 18:54 schrieb Zoran Vasiljevic:

>
> I modified the "old way" NOT to use introspective
> script, rather to use Tcl command traces :-)

OK, not really *everything*. Tcl namespace serialization
(variables/procedures) is still done "the old way".

The problem with that that a namespace is a difficult
beast to handle. Some OO extensions (Xotcl) create
their own namespaces which you just cannot serialize
the Tcl-way...

Re: [naviserver-devel] Tcl inits

[Stephen D. <sd...@gm...>]

On 8/6/07, Zoran Vasiljevic <zv...@ar...> wrote:
> Hi!
>
> After spenging a weekend over this, I know less than
> before...
>
> The question is how we are going to improve the Tcl
> initialization so people could benefit from that better
> than now?
>
> As of today, what we have is:
>
> a.
>     introspective script based on the "old" way as 4.0 AS
>     did it (init the interp by loading all Tcl files then
>     run a script to pull-out namespaces, variables etc and
>     create a script that get saved with [ns_ictl save]
>     and run in each new interp by [ns_ictl update])
>
> b.
>     the ttrace way of setting Tcl traces on selected commands
>     and have the "things" setup by those commands record in
>     shared nsv arrays. Then use Tcl [unknown] to pull out the
>     "things" on as-needed basis from shared arrays
>
> Well, none of those is really good. The a. is limited in
> what it "introspects". The b. is pretty complex and in some
> situations it does not work as expected (although there is
> no design problem there, more probably just a bug or lack a
> feature).
>
> Now, we do have [ns_ictl trace] where one can register things
> to execute at interp creation, allocation etc.
> So, I tried to do a simple thing: for each script executed
> during the startup of the server I create
>     [ ns_ictl trace [list source $file]]
> create-trace.
>
> This "mostly" works, but has two negative effects:
>    1. all scripts needs to be re-adjusted as they are going to
>       be sourced more than once (for each new interp)
>    2. in case 1000's of files are needed for the startup this may
>       pose significant delay in interp creation
>
> I think that old AS approach is the worst as it is really limited.
> The Tcl-trace based approach is better but is known to have some
> problems (I do not know which ones), yet, it is pretty powerful
> and has some other nice side-effects (less memory footprint).
> Lastly, the [ns_ictl trace] is simplest to understand and deploy
> (this is also very important) but requires rewriting of existing
> scripts and most probably needs somekind of caching to speedup
> the load process when 1000's of startup files are involved.
>
> For me, the Tcl-trace or ns_ictl trace are the way to go.
> Looking at those two, the latter is more appealing since of the
> simplicity, but it may hit us hard speedwise. OTOH, the former
> is faster and uses far less resources but is pretty complex in
> implementation (this might be done simpler, yes, yet it is always
> going to be more complex than ns_ictl trace).
>
> Now, what you all think? Should we do something there or should
> we just stay with the old AS introspective script?
>


What about making naviserver modules Tcl packages? That is, instead of
saying: rewrite your scripts to survive being loaded multiple times
via ns_ictl callbacks; we say: write Tcl packages.

I think we want Tcl packages for a variety of reasons, but
interestingly they also have a lazy loading mechanism via pkgIndex.tcl
scripts.

What do you think?


I think we need to maintain backwards compatibility with the existing
scheme (however it's actually implemented) regardless of what else we
do, right?

Re: [naviserver-devel] Tcl inits

[Zoran V. <zv...@ar...>]

Am 06.08.2007 um 18:47 schrieb Vlad Seryakov:

> I would keep old initialization if i need speed, if performance is  
> going
> to suffer with Tcl/ns_ictl approach, then for fast/busy sites we  
> need to
> be able to use old(fast) way. Otherwise, the server will be limited  
> for
> non-realtime or personal web sites usage only.

What you haven't seen perhaps is...

I modified the "old way" NOT to use introspective
script, rather to use Tcl command traces :-)
So Tcl command traces work allright and also much more
"logical" then the old ugly and error-prone code.
The only difference to ttrace (lazyloader=true) is that
I do not use [unknown] to reload commands on the fly,
rather all "things" are synthetized in a large script
that loads everything that is registered, at once.

That's why I told about wanting to go to ns_ictl trace
or tcl-cmd-traces (nstrace.tcl).

Cheers,
Zoran

Re: [naviserver-devel] Tcl inits

[Vlad S. <vl...@cr...>]

I would keep old initialization if i need speed, if performance is going 
to suffer with Tcl/ns_ictl approach, then for fast/busy sites we need to 
be able to use old(fast) way. Otherwise, the server will be limited for 
non-realtime or personal web sites usage only.

Zoran Vasiljevic wrote:
> Hi!
> 
> After spenging a weekend over this, I know less than
> before...
> 
> The question is how we are going to improve the Tcl
> initialization so people could benefit from that better
> than now?
> 
> As of today, what we have is:
> 
> a.
>     introspective script based on the "old" way as 4.0 AS
>     did it (init the interp by loading all Tcl files then
>     run a script to pull-out namespaces, variables etc and
>     create a script that get saved with [ns_ictl save]
>     and run in each new interp by [ns_ictl update])
> 
> b.
>     the ttrace way of setting Tcl traces on selected commands
>     and have the "things" setup by those commands record in
>     shared nsv arrays. Then use Tcl [unknown] to pull out the
>     "things" on as-needed basis from shared arrays
> 
> Well, none of those is really good. The a. is limited in
> what it "introspects". The b. is pretty complex and in some
> situations it does not work as expected (although there is
> no design problem there, more probably just a bug or lack a
> feature).
> 
> Now, we do have [ns_ictl trace] where one can register things
> to execute at interp creation, allocation etc.
> So, I tried to do a simple thing: for each script executed
> during the startup of the server I create
>     [ns_ictl trace [list source $file]]
> create-trace.
> 
> This "mostly" works, but has two negative effects:
>    1. all scripts needs to be re-adjusted as they are going to
>       be sourced more than once (for each new interp)
>    2. in case 1000's of files are needed for the startup this may
>       pose significant delay in interp creation
> 
> I think that old AS approach is the worst as it is really limited.
> The Tcl-trace based approach is better but is known to have some
> problems (I do not know which ones), yet, it is pretty powerful
> and has some other nice side-effects (less memory footprint).
> Lastly, the [ns_ictl trace] is simplest to understand and deploy
> (this is also very important) but requires rewriting of existing
> scripts and most probably needs somekind of caching to speedup
> the load process when 1000's of startup files are involved.
> 
> For me, the Tcl-trace or ns_ictl trace are the way to go.
> Looking at those two, the latter is more appealing since of the
> simplicity, but it may hit us hard speedwise. OTOH, the former
> is faster and uses far less resources but is pretty complex in
> implementation (this might be done simpler, yes, yet it is always
> going to be more complex than ns_ictl trace).
> 
> Now, what you all think? Should we do something there or should
> we just stay with the old AS introspective script?
> 
> Cheers
> Zoran
> 
> 
> 
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems?  Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >>  http://get.splunk.com/
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

Re: [naviserver-devel] Tcl inits

[Gustaf N. <ne...@wu...>]

Zoran Vasiljevic schrieb:
> I think that old AS approach is the worst as it is really limited.
>   
true. and not scalable
> The Tcl-trace based approach is better but is known to have some
> problems (I do not know which ones), yet, it is pretty powerful
> and has some other nice side-effects (less memory footprint).
> Lastly, the [ns_ictl trace] is simplest to understand and deploy
> (this is also very important) but requires rewriting of existing
> scripts and most probably needs somekind of caching to speedup
> the load process when 1000's of startup files are involved.
>   
just for the record: a monster like openacs consists of 2200 files which 
might
be merged into the blueprint. openacs has its own package management, 
containing
currently 309 packages (from cvs head). Certainly, not many people activate
all packages.
> For me, the Tcl-trace or ns_ictl trace are the way to go.
> Looking at those two, the latter is more appealing since of the
> simplicity, but it may hit us hard speedwise. OTOH, the former
> is faster and uses far less resources but is pretty complex in
> implementation (this might be done simpler, yes, yet it is always
> going to be more complex than ns_ictl trace).
>   
i would envision some mixture as a good solution, which might not be so
bloated as the AS version.
  - having a kernel of functions, which are available for all connection
    and schedules threads.
  - having a mechanisms like a naviserver version of "package require"
    which can load a "package" on demand. A package can consist
    of multiple files, and might require other "packages"

i would see a "package" and the kernel in a simple version
as cat-ed files cached in nsvs.

every connection says during the start of the connection that it
requires certain packages and evals it as required. i would expect
this to reduce the memory requirements, since only needed packages
are loaded into ther interps. For loading the kernel, ns_ictl traces
might an option, these won't be so many files.

however, concerning getting openacs to work on naviserver,
i would prefer having support for some time the good old blueprint way.

there are certainly many consequences on applications using
ns_eval to update the blueprint...

-gustaf

Re: [naviserver-devel] Tcl inits

[Neophytos D. <k2...@ph...>]

Hi Zoran,

I've spend the weekend compiling and re-compiling the kernel and
different versions of gcc/glib on two machines (with different
CFLAGS/CHOST settings).

NS "make test" worked only once on one of the two machines (with CHOST
set to "i386-pc-linux-gnu" - GCC 4.2 no longer supports that). The rest
of the time it was just hanging on the first test (adp.test).

> For me, the Tcl-trace or ns_ictl trace are the way to go.
> Looking at those two, the latter is more appealing since of the
> simplicity, but it may hit us hard speedwise. OTOH, the former
> is faster and uses far less resources but is pretty complex in
> implementation (this might be done simpler, yes, yet it is always
> going to be more complex than ns_ictl trace).
> 
> Now, what you all think? Should we do something there or should
> we just stay with the old AS introspective script?

I'm all for Tcl-trace and ns_ictl trace but I gather it would take a
considerable amount of time to prepare and test those scripts. Please
feel free to let me know what I can do to help --- I'm a bit short on
time this month but things should get better after September 8 - getting
married :).

IMHO, I would prefer mostly a quick migration path from AS (i.e. the old
AS introspective script). I have already ported/adapted my code to play
nicely with most important changes introduced recently in NS but I'm not
quite confident to work with the modified init.tcl from 4.99.1. This is
due to the fact that scheduled threads are no longer triggered after
init (they seem to be queued successfully but no -sched- thread is created).


Best wishes,
Neophytos