naviserver-devel — Developer discussion

Re: [naviserver-devel] Documentation & website question

[Bernd E. <b.e...@ki...>]

> NO. MORE burden!
> Please lets forget XML before we write yet another line on that.
>
> To my opinion, the tcl doctools format is the simplest most natural
> to understand and write format I've come accross when writing short(er)
> man pages. And this is the bulk of that we need....

Relax! I won't delete the src directory and replace it with XSLT scripts :-)
I felt into the format-discussion-trap again, sorry.

When using doctools we have various output formats available, nroff, html, 
tmml, latex, wiki (see src/README.doc). We can put stuff below docroot, into 
man/, on the wiki. The technical stuff and format is already there.

The general problem is: We don't _want_ to write documentation. Not for man 
pages and not for the Wiki.
If we make it a priority we can surely provide appropriate templates that 
everyone feels comfortable with. We just have to cut the Gordian knot.

Bernd.

Re: [naviserver-devel] Documentation & website question

[Zoran V. <zv...@ar...>]

On 05.09.2006, at 14:57, Zoran Vasiljevic wrote:

> But, it is the former where problems start. As it may be in
> several usable formats (nroff, perhaps man, perhaps pdf).

I ment really:

   (nroff, perhaps html, perhaps pdf).

Re: [naviserver-devel] Documentation & website question

[Zoran V. <zv...@ar...>]

On 05.09.2006, at 09:34, Bernd Eidenschink wrote:
> How do Apache folks do it? Is the base of documentation HTML?
> I would suggest some simple XML at least, so we are able to use the  
> same tool
> (e.g. tdom) and mechanism (e.g. XSLT) for both man pages and the
> documentation Vlad suggested to put under docroot. Even the NEWS  
> file could
> be compiled from the XML base, to not double work.

AHHHHHH! XML again!? I wen't there and I came back!
The first doc in threading extension was TMML (based on XML).
It was a HORROR to maintain and I switched to tcllib instead.
Had a look at POD (perl doc format pendant)? Well this is basically
tcllib format. Easy to write and maintain. Sometimes when a page
becomes pretty large, you can get lost, but in XML I get lost
about 10 lines below. Same as HTML. You need a damn good tool to
use it productively, especially when making large changes.

>
> Btw: Right now we use (ok, don't use) doctools and dtplite from  
> tcllib. Would
> some readable XML feel more natural or would it lower the mental  
> burden to
> start documenting? Using XML/XSLT would still allow to also create  
> doctools
> format and then creating usual xyz-roff format.

NO. MORE burden!
Please lets forget XML before we write yet another line on that.

To my opinion, the tcl doctools format is the simplest most natural
to understand and write format I've come accross when writing short(er)
man pages. And this is the bulk of that we need....

Cheers
Zoran

Re: [naviserver-devel] Documentation & website question

[Zoran V. <zv...@ar...>]

On 04.09.2006, at 22:51, Vlad Seryakov wrote:

>
>> I see the docs in two parts, the man pages, which are a technical
>> reference, and the HOWTO stuff.  Seems to me like the technical ref
>> should be kept uptodate with the code, and the HOWTO stuff on the
>> wiki, where people can, hopefully, update it.
>
> Even technical pages, having them quickly on the same server you are
> working under /docs or /_docs or /whatever with search could be  
> easier,
> i like man pages but cross referencing and quick access is not  
> their best

Allright: cross-referencing is of course miserable.
But as I'm most of the time on command-line, a proper usage of
'apropos', 'man' and 'grep' solves 95 % of my needs. I never
saw ANY other Tcl documentation (apart from very first book
from JO).

>
>> Pointing people in the right direction is a good idea though.  How
>> about a default start page for the installed server which focusses
>> more on what to do next, rather than advertising?  Could point to the
>> installed html version of the docs with a file:/// reference, and the
>> online wiki for HOWTO stuff, talk about the reference config file,  
>> and
>> the the stats module can be enabled for introspection stuff.
>
> I do not see where to point now, too many pieces around, so i think
> about one repository for all docs, better under CVS so changes can be
> tracked and all docs easily retrievd by one cvs command.

This also speaks for centralized reference docs to which I agree.
But the HOWTO's are best done under Wiki where others may add more
salt.

Zoran

Re: [naviserver-devel] Documentation & website question

[Zoran V. <zv...@ar...>]

On 04.09.2006, at 22:43, Stephen Deasey wrote:

> I was looking at the docs too.  I installed fedoras tcllib and there's
> a doctools lib, but no standalone tool to convert things.  Should
> there be? Or where does this live?

zvpb:~/meta/usr/local/aw/log zoran$ dtplite
/usr/local/bin/dtplite wrong#args, expected: -o outputpath ?-merge? ?- 
ext ext? ?-style file? ?-header file? ?-footer file? ?-nav label  
url?... format inputpath

This gets done if you "make install" the tcllib.

>
> Also, how are folks converting the templates to man and html pages? I
> can't see any Makefile rules, in the Tcl thread package for example.
> Are people just converting by hand and putting the result in CVS?

The "folks" in the Tcl threading extension (i.e. myself) are not
doing it from the makefile. I have a Tcl script doing the conversion
and I hit that script once per hand before doing the distribution.
A million$ question: why not over makefile?
Answer: perhaps I'm lazy to write a makefile rule for that?

>
> I dunno about the below -- docs live in
> /usr/shar/doc/naviserver-x.x...  I'd like to make things more
> standard, not less.
>
> I see the docs in two parts, the man pages, which are a technical
> reference, and the HOWTO stuff.  Seems to me like the technical ref
> should be kept uptodate with the code, and the HOWTO stuff on the
> wiki, where people can, hopefully, update it.

Agree 100%. The technical reference should be preferably
in form of man pages (I never look ANY Tcl docs except man-pages)
and optionally in HTML for those bright young people arround
(not including me).
The HOWTO should be definitely a Wiki.

But, it is the former where problems start. As it may be in
several usable formats (nroff, perhaps man, perhaps pdf).

>
> Pointing people in the right direction is a good idea though.  How
> about a default start page for the installed server which focusses
> more on what to do next, rather than advertising?  Could point to the
> installed html version of the docs with a file:/// reference, and the
> online wiki for HOWTO stuff, talk about the reference config file, and
> the the stats module can be enabled for introspection stuff.

Nothing to object to.

>
> The current index.adp in contrib doesn't mention any docs at all...
> Or how t osign up to the mailing list, or how to report a bug, or how
> to find modules to install...

Ditto...

I believe the best way to do that is get somebody new use the code
and tell him to document all that he/she finds odd or missing :-)
We are (I am) far too much into it over the years and it looks all
so "known" to me that I really have to motivate myself to write that
"obvious" things.

Zoran

Re: [naviserver-devel] Documentation & website question

[Bernd E. <eid...@we...>]

> Maybe instead of solving documentation issue technically using
> conversion tools we can just include html pages with all we have docs
> into distributon, like Apache does, once installed we will copy manuals
> under pages so every new install will have access to docs even without
> writing any script. Those docs can have working examples, simple ones or
> complex does not matter but they will perform testing and educational
> tasks at the same time.
> I went through AOlserver site, wiki and our wiki and pieces of docs, it
> is spread and having this simply as html docs in one place may be easier.

How do Apache folks do it? Is the base of documentation HTML? 
I would suggest some simple XML at least, so we are able to use the same tool 
(e.g. tdom) and mechanism (e.g. XSLT) for both man pages and the 
documentation Vlad suggested to put under docroot. Even the NEWS file could 
be compiled from the XML base, to not double work. 

Btw: Right now we use (ok, don't use) doctools and dtplite from tcllib. Would 
some readable XML feel more natural or would it lower the mental burden to 
start documenting? Using XML/XSLT would still allow to also create doctools 
format and then creating usual xyz-roff format.

Most of the content on the Wiki is static. I would not vote against replacing 
it with static HTML content (about, license, news; links to SF) that mainly 
includes the transformed parts of the documentation base. Nothing is lost, 
the current activity is mainly removing hidden div-layers with SPAM links :-|

Bernd.

Re: [naviserver-devel] Documentation & website question

[Vlad S. <vl...@cr...>]

> I see the docs in two parts, the man pages, which are a technical
> reference, and the HOWTO stuff.  Seems to me like the technical ref
> should be kept uptodate with the code, and the HOWTO stuff on the
> wiki, where people can, hopefully, update it.

Even technical pages, having them quickly on the same server you are 
working under /docs or /_docs or /whatever with search could be easier, 
i like man pages but cross referencing and quick access is not their best

> Pointing people in the right direction is a good idea though.  How
> about a default start page for the installed server which focusses
> more on what to do next, rather than advertising?  Could point to the
> installed html version of the docs with a file:/// reference, and the
> online wiki for HOWTO stuff, talk about the reference config file, and
> the the stats module can be enabled for introspection stuff.

I do not see where to point now, too many pieces around, so i think 
about one repository for all docs, better under CVS so changes can be 
tracked and all docs easily retrievd by one cvs command.

> The current index.adp in contrib doesn't mention any docs at all...
> Or how t osign up to the mailing list, or how to report a bug, or how
> to find modules to install...

Agree, why stats module is password protected, we can change it to make 
no password for locahost at least.

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] Documentation & website question

[Stephen D. <sd...@gm...>]

On 9/4/06, Vlad Seryakov <vl...@cr...> wrote:
>
> Just a thought inspired by thread on AOLserver mailing list:-))


It must be that time of year again...  :-)

Re: [naviserver-devel] Documentation & website question

[Stephen D. <sd...@gm...>]

I was looking at the docs too.  I installed fedoras tcllib and there's
a doctools lib, but no standalone tool to convert things.  Should
there be? Or where does this live?

Also, how are folks converting the templates to man and html pages? I
can't see any Makefile rules, in the Tcl thread package for example.
Are people just converting by hand and putting the result in CVS?

I dunno about the below -- docs live in
/usr/shar/doc/naviserver-x.x...  I'd like to make things more
standard, not less.

I see the docs in two parts, the man pages, which are a technical
reference, and the HOWTO stuff.  Seems to me like the technical ref
should be kept uptodate with the code, and the HOWTO stuff on the
wiki, where people can, hopefully, update it.

Pointing people in the right direction is a good idea though.  How
about a default start page for the installed server which focusses
more on what to do next, rather than advertising?  Could point to the
installed html version of the docs with a file:/// reference, and the
online wiki for HOWTO stuff, talk about the reference config file, and
the the stats module can be enabled for introspection stuff.

The current index.adp in contrib doesn't mention any docs at all...
Or how t osign up to the mailing list, or how to report a bug, or how
to find modules to install...




On 9/4/06, Vlad Seryakov <vl...@cr...> wrote:
>
> Maybe instead of solving documentation issue technically using
> conversion tools we can just include html pages with all we have docs
> into distributon, like Apache does, once installed we will copy manuals
> under pages so every new install will have access to docs even without
> writing any script. Those docs can have working examples, simple ones or
> complex does not matter but they will perform testing and educational
> tasks at the same time.
> I went through AOlserver site, wiki and our wiki and pieces of docs, it
> is spread and having this simply as html docs in one place may be easier.
>
> Just a thought inspired by thread on AOLserver mailing list:-))
>
> --
> Vlad Seryakov
> 571 262-8608 office
> vl...@cr...
> http://www.crystalballinc.com/vlad/
>
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>

[naviserver-devel] Documentation & website question

[Vlad S. <vl...@cr...>]

Maybe instead of solving documentation issue technically using 
conversion tools we can just include html pages with all we have docs 
into distributon, like Apache does, once installed we will copy manuals 
under pages so every new install will have access to docs even without 
writing any script. Those docs can have working examples, simple ones or 
complex does not matter but they will perform testing and educational 
tasks at the same time.
I went through AOlserver site, wiki and our wiki and pieces of docs, it 
is spread and having this simply as html docs in one place may be easier.

Just a thought inspired by thread on AOLserver mailing list:-))

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] read/write ns_config

[Zoran V. <zv...@ar...>]

On 31.08.2006, at 17:03, Vlad Seryakov wrote:

> Have you run any performance benchmark similar to Tcl array/ns_sets =20=

> tests?

Well, hard to do that. I mean: against WHAT should it do it?
Comparing [ns_config=10] to [ns_conf get] isn't fair as those
are entirely different things.

But if you will:

server1:nscp 3>  time {ns_config section key 100} 1000
4.211 microseconds per iteration
server1:nscp 5> time {ns_conf get section key 100} 1000
5.002 microseconds per iteration

Not bad, taking in account that ns_config does almost
nothing here, whereas ns_conf does quite a few things
under the hood.

But, I believe, and you can see that by experimenting
yourself, that ns_conf will be as fast as ns_config is,
so it can *theoretically* replace the ns_config speedwise.
Theoretically, because it is case-sensitive per-desing which
limits its snap-in replacement caps, of course.

You are most welcome to glance at the code and
tell what you think about it.

Cheers,
Zoran=

Re: [naviserver-devel] read/write ns_config

[Vlad S. <vl...@cr...>]

Have you run any performance benchmark similar to Tcl array/ns_sets tests?

Zoran Vasiljevic wrote:
> On 22.08.2006, at 22:13, Stephen Deasey wrote:
> 
>> I don't see how you will handle the locking...  :-(
> 
> If you go and get the:
> 
> https://sourceforge.net/tracker/? 
> func=detail&atid=719009&aid=1549952&group_id=130646
> 
> you can inspect the code ans see how I'm doing the locking.
> 
> Essentially (in a nutshell) every thread has its own
> private copy of all config parameters it requested so far.
> There is also a shared configuration, holding ALL known
> config parameters.
> 
> Under normal circumstances, a reader will go to the shared
> config for a parameter only once: first time it fetches the
> param. All other accesses to the same param are done in
> the local (per-thread) cache. The cache is locked in order
> to synchtonize with the writer thread.
> 
> A writer thread will update the shared config and then walk
> private configs of all theads (which registered themselves
> for a given section - see above) and update them one by one,
> locking each as it goes.
> 
> Since I expect writers to be *far* less active, the reader
> will almost always succeed in locking the private cache.
> Hence, almost no locking contention!
> 
> A bad scenario can happen if you have numerous threads
> reading the same parameter at the same time. Then
> you will get contention on the shared config mutex. But
> this is normally not going to happen in real life, and if
> it happens it will be a very short interval as after that
> all will be cached per-thread privately.
> 
> I believe this is pretty good. It uses some more memory
> but avoids locking contention at maximum.
> 
> What do you think?
> 
> Cheers
> Zoran
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] read/write ns_config

[Zoran V. <zv...@ar...>]

On 22.08.2006, at 22:13, Stephen Deasey wrote:

>
> I don't see how you will handle the locking...  :-(

If you go and get the:

https://sourceforge.net/tracker/? 
func=detail&atid=719009&aid=1549952&group_id=130646

you can inspect the code ans see how I'm doing the locking.

Essentially (in a nutshell) every thread has its own
private copy of all config parameters it requested so far.
There is also a shared configuration, holding ALL known
config parameters.

Under normal circumstances, a reader will go to the shared
config for a parameter only once: first time it fetches the
param. All other accesses to the same param are done in
the local (per-thread) cache. The cache is locked in order
to synchtonize with the writer thread.

A writer thread will update the shared config and then walk
private configs of all theads (which registered themselves
for a given section - see above) and update them one by one,
locking each as it goes.

Since I expect writers to be *far* less active, the reader
will almost always succeed in locking the private cache.
Hence, almost no locking contention!

A bad scenario can happen if you have numerous threads
reading the same parameter at the same time. Then
you will get contention on the shared config mutex. But
this is normally not going to happen in real life, and if
it happens it will be a very short interval as after that
all will be cached per-thread privately.

I believe this is pretty good. It uses some more memory
but avoids locking contention at maximum.

What do you think?

Cheers
Zoran

Re: [naviserver-devel] read/write ns_config

[Zoran V. <zv...@ar...>]

On 29.08.2006, at 22:29, Stephen Deasey wrote:

> It would be more flexible to create ns_*ctl commands and put them
> *lower* in the stack, and then build a dynamic configuration tool on
> top of this, than to bake a dynamic configuration mechanism into the
> server, and have code depend on that.

Yet, if module A changes value B, how would that look like
if I'd need to have it arround the next boot time?

IOW, how would you handle persistence with that?

Re: [naviserver-devel] read/write ns_config

[Stephen D. <sd...@gm...>]

On 8/25/06, Zoran Vasiljevic <zv...@ar...> wrote:
>
> On 24.08.2006, at 21:36, Stephen Deasey wrote:
>
> >
> > So I think you can't get away from specialisation, unfortunately.
> >
>
> What I still cannot (easily) accept is that we can't
> do a config module as repository for the server and
> other module's configuration options.
> Think of it as the Windows registry (not that I'm
> a fan of Microsoft, I'm not, but this "centralized"
> approach also has benefits).
>
> What I could imagine is:
>
> We can have the config file as is now. But it would
> form just a part of the configuration. Over the runtime
> of the server, new configuration could be added or
> existing changed. This is non-volatile, means at the
> next restart, the changes are still there.
> At the startup, one can decide wether the config file
> (as now) is the applied first or after loading the,
> lets call it registry for now, file. This way you can
> decide wether the changes to the values from the config
> file are overriden by the registry or not. New stuff
> which is only in the registry is of course loaded.
> But during the startup you can even discard them completely.
> We can have a web-page which handles registry maintenance
> in the server. So with one management interface you can
> tune various knobs mostly w/o restarting the server.
>
> The persistency can be implemented by a simple
> dbm hash-file (ndbm, gdbm).
>
> I mean, nobody can tell me that this is impossible
> or not practicable: there are at lest 60.000.000 proofs
> of concept out there...
>
> I would need to hear 1 good reason against this.
> As you see, I'm not letting loose...
>
> As the matter of fact, I=10 will write this for us internally
> as a module. Lets call it "nsregistry" module. It will have
> a library to link with so other modules can use it from the
> C-level. It will have a Tcl and C API similar to what we have
> now, with a small difference: values are obtained by *copy*
> and not by reference. It will be properly locked so it can
> be used from many threads at the same time. =10I will of course
> not repeat the silly case-insensitivity junk: the thing will
> be case sensitive. It will have the same section/parameter
> approach as the current config. The idea of this module would
> be to provide centralized and persistent storage for all kinds
> of various configuration parameters one can imagine, for the
> server itself and for the modules. From Tcl and from C-modules.
> Actually, that what the current config does but w/o aiming to
> replace it because of all possible concerns about locking of
> speed (or whatever).  And it will be persistent.
>


It would be more flexible to create ns_*ctl commands and put them
*lower* in the stack, and then build a dynamic configuration tool on
top of this, than to bake a dynamic configuration mechanism into the
server, and have code depend on that.

For example, if you went with the callback approach that Rick
mentioned (seems easiest), the callback would run the ns_*ctl command
when a config value is updated.

You keep your options open.

If you go the other way, you have to start thinking about: do we
create an interface that allows people to plug in different
persistence mechanism for the back end, etc...?

Re: [naviserver-devel] read/write ns_config

[Zoran V. <zv...@ar...>]

On 26.08.2006, at 12:30, Bernd Eidenschink wrote:

>  Would it then be a lot of work to provide a
> transformation command that makes it human readable?

Piece of cake. As the matter of fact I may invent
a command like

   ns_confctl save ?-decode? channel
   ns_confctl load ?-encode? channel

and this will read/write the config to
some channel optionally encoding/decoding
to/from human readble form.

This is all opened for later polish. The
first thing for us now is to have a good
proof of concept. And the concept is to
provide reasonably fast read/write
configuration storage module with C and Tcl
API with minimal locking requirements.

Lets see...

Cheers
Zoran

Re: [naviserver-devel] read/write ns_config

[Bernd E. <eid...@we...>]

> This is where I'm still not sure what to do. I can easily attach
> a (x)dbm hash underneath which would be perfectly OK. But with a
> drawback of not being human-readable. I will have to think more
> here...

You could use something as storage you are most familiar with and what
fits the purpose, be it (x)dbm. Would it then be a lot of work to provide a
transformation command that makes it human readable? Maybe this is
sufficient.

Bernd.

Re: [naviserver-devel] read/write ns_config

[Zoran V. <zv...@ar...>]

On 25.08.2006, at 19:09, Rick Cobb wrote:

> Here's what we ran into when we did something similar; just cautionary
> information, not trying to shoot anything down here.

Actually this gave me another idea...

(BTW...
The reason why I'm still not letting loose on this point:
althogh I like the idea of tool-based config, it has one
strong drawback. How am I going to make this persistent?
Per module? This is just plain unpracticable as it would
result in a "config-file" per module adding to complexity
if I would like to pull to config data from some other
source(s)).

>
> Our approach was to create a separate repository / heirarchy of
> configuration variables that is *not* module oriented,

Which is also what I'm after...


> but
> administrative task oriented. So, for example, we have a section  
> called
> "ServiceNetwork" with a variable called "HTTPInterface".  That  
> variable
> gets one or more IP addresses.  Only these variables are dynamic.  To
> get a dynamic update for one of these variables, any piece of C/C++  
> code
> registers a callback based on the variable name.

.. but what I'd do is make this "callback" entirely transparent to the
caller. Instead, I'd split the config repository in 2 contexts:
shared and thread-private. The shared gets loaded, updated and the
thread-private is a partial copy of the shared, build-up during the
config lookup, dynamically. Each "reader" of the config will first go
ask its private context and then shared context for a value.
If found in private: ok. If found in shared: ok. Otherwise it updtes
its both private and shared context with the default value.
The locking would be minimized so that during the normal operation
only two threads (the reader and writer) would content for a mutex,
which is, given the majority of reads and tiny fractions of writes,
neglectable.  Now you'd ask: where is the callback?
Simple: the reader therad, not founding the value in its private context
will set it (under lock) in the shared and register itself automatically
for having "interest" in this section. Any writer thereafter, which  
updates
any of the keys of that section will walk the list of interested threads
and update their own *private* context after updating the shared one.
This way, the locking is minimized to absolute minimum. You will see  
that
when I'm ready with the code. The task of the writer will not be cheap
as it will neeed to update not one but N places (depeding on howmany
threads expressed "interest" in particular section) but this way the
readers are cheap and fast as they will mostly find what they're
looking for in their own private cache.

>
> This repository is read at startup, before the normal
> ns_section/ns_param file.

OK. But I'd opt to give the startup option to read it before
or after the ns_section/ns_param file. This way there is
more flexibility what overrides what.

>
> The declarations for these variables state whether changes to the
> variable require a server restart to take effect, or not.
>

This is a good idea. Although I will not enforce a declaration
of the variable (it will be as dynamic as the current config)
I will add one qualifier: scope. This will be either startup
or runtime depicting wether the thing is read/set only at
startup only or not. The scope can be selected during the
lookup-falure followed by the shared config update automatically.

> Our management console works entirely in terms of this repository;

YES. This is the other benefit of having it centralized as you
need NOT know in advance what modules you have out there. All is
in one place and can be saved, restored, uniformly controlled etc.

>
> The nsd.tcl-equivalent, which is still written in terms of
> ns_section/ns_param, then has access to those variables as TCL global
> variables (just a convenience, not necessarily my best design  
> idea).  If
> more than one module needs access to a configuration variable, they  
> just
> use it in this section. That lets us use normal AOLServer modules
> out-of-the-box, yet coordinate their tuning by writing complicated  
> TCL.

This is where I'm still not sure what to do. I can easily attach
a (x)dbm hash underneath which would be perfectly OK. But with a
drawback of not being human-readable. I will have to think more
here...

>
> The positive implications have been:
> * People configuring our system rarely have to copy the same parameter
> to different sections.
> * People configuring our system never have to write TCL.
> * No changes to existing modules are required
> * Our modules, or our own edits to AOLServer modules, get dynamic
> capabilities
> * All configuration changes can be made over HTTP
> * All of our modules have dynamic debug capability; in fact, for  
> almost
> all of them, it's possible to adjust their debug settings over  
> HTTP, so
> you can almost get to the point where you can debug a specific HTTP
> request (of course, with lots of traffic, that doesn't work).
>
> The negative implications have been:
> * Our nsd.tcl-equivalent is *very* long and complicated (~3000 lines),
> and is gradually becoming dominated by TCL control statements
> (if/for/etc), not sections & params.
> * If a variable requires calculation to become an internal  
> configuration
> state, the calculation is often repeated in both TCL (nsd.tcl) and C.
> * Startup time is slowed by all this processing, but it's not a big  
> deal
> because we also recover a database during our startups, which  
> dominates
> the startup time.
> * It is very hard to see all the coupling for a specific variable.
> * Any module that uses a dynamic variable has to code for it three
> times, in three different areas: once for its use for
> ns_section/ns_param, once for the global configuration area, and  
> once in
> the callback.
>
> We could mitigate some of these with better generators or code
> conventions, but haven't.

Reading the above, I see no mention about how you do the locking?
If a writer chages a variable, he'd have to propagate this change
to readers, or? Now, readers, accessing the same variable must
access it under lock as some writer can call their registered callback
which updates the value at the same time, right?

In my (still) "vapourware", this is/must-be so, but the locking is
actually very cheap while most of the time reader will just lock
its private context, lookup and unlock. There will be no lock contenton
on that mutex. Only when one of the writers (which is a rare event) goes
and updates reader's private context (this being a very short operation
anyway) a short contention may happen. I believe this can be entirely
neglected. The cost of locking a mutex is low, if there is nobody
else I have to fight with. Actually, I'm sure this will work excellent.

Anyway, I will have to write this one in the next few days as
we already have plans for expading our product. We cannot
tolerate shutdowns of the server just to flip a bit any more.
This is now pressing us for some time and we've come to a point
where this must be done.

Thanks for the valuable information. I will make (our) nsconf or
nsregistry module public and we can then all see if this is usable
for others or not.

Cheers,
Zoran

Re: [naviserver-devel] read/write ns_config

[Rick C. <rc...@Kn...>]

Here's what we ran into when we did something similar; just cautionary
information, not trying to shoot anything down here.

Our approach was to create a separate repository / heirarchy of
configuration variables that is *not* module oriented, but
administrative task oriented. So, for example, we have a section called
"ServiceNetwork" with a variable called "HTTPInterface".  That variable
gets one or more IP addresses.  Only these variables are dynamic.  To
get a dynamic update for one of these variables, any piece of C/C++ code
registers a callback based on the variable name.

This repository is read at startup, before the normal
ns_section/ns_param file.

The declarations for these variables state whether changes to the
variable require a server restart to take effect, or not.

Our management console works entirely in terms of this repository; the
ns_section/ns_param variables are never mentioned anywhere in our
product documentation or configuration interfaces.  If you change a
value that is can't be managed dynamically, an indicator comes on saying
you need to restart. This comes on for every management console, so if I
change a variable and you also have the console up, you see it the
indicator.

The nsd.tcl-equivalent, which is still written in terms of
ns_section/ns_param, then has access to those variables as TCL global
variables (just a convenience, not necessarily my best design idea).  If
more than one module needs access to a configuration variable, they just
use it in this section. That lets us use normal AOLServer modules
out-of-the-box, yet coordinate their tuning by writing complicated TCL.

For example, we configure both the number of Conn threads and some of
the resources for our publish-subscribe system based on variables called
"Workload/NumberOfSubscribers" and "Workload/NumberOfPublishers".  The
publish-subscribe system can register a callback to adjust its tuning on
the fly; the conn threads can't, but aren't quite as sensitive to it.

The positive implications have been:
* People configuring our system rarely have to copy the same parameter
to different sections.
* People configuring our system never have to write TCL.
* No changes to existing modules are required
* Our modules, or our own edits to AOLServer modules, get dynamic
capabilities
* All configuration changes can be made over HTTP
* All of our modules have dynamic debug capability; in fact, for almost
all of them, it's possible to adjust their debug settings over HTTP, so
you can almost get to the point where you can debug a specific HTTP
request (of course, with lots of traffic, that doesn't work).

The negative implications have been:
* Our nsd.tcl-equivalent is *very* long and complicated (~3000 lines),
and is gradually becoming dominated by TCL control statements
(if/for/etc), not sections & params.
* If a variable requires calculation to become an internal configuration
state, the calculation is often repeated in both TCL (nsd.tcl) and C.
* Startup time is slowed by all this processing, but it's not a big deal
because we also recover a database during our startups, which dominates
the startup time.
* It is very hard to see all the coupling for a specific variable.
* Any module that uses a dynamic variable has to code for it three
times, in three different areas: once for its use for
ns_section/ns_param, once for the global configuration area, and once in
the callback.

We could mitigate some of these with better generators or code
conventions, but haven't.

-- ReC

-----Original Message-----
From: nav...@li...
[mailto:nav...@li...] On Behalf Of
Zoran Vasiljevic
Sent: Friday, August 25, 2006 3:20 AM
To: nav...@li...
Subject: Re: [naviserver-devel] read/write ns_config


On 24.08.2006, at 21:36, Stephen Deasey wrote:

>
> So I think you can't get away from specialisation, unfortunately.
>

What I still cannot (easily) accept is that we can't
do a config module as repository for the server and
other module's configuration options.
Think of it as the Windows registry (not that I'm
a fan of Microsoft, I'm not, but this "centralized"
approach also has benefits).

What I could imagine is:

We can have the config file as is now. But it would
form just a part of the configuration. Over the runtime
of the server, new configuration could be added or
existing changed. This is non-volatile, means at the
next restart, the changes are still there.
At the startup, one can decide wether the config file
(as now) is the applied first or after loading the,
lets call it registry for now, file. This way you can
decide wether the changes to the values from the config
file are overriden by the registry or not. New stuff
which is only in the registry is of course loaded.
But during the startup you can even discard them completely.
We can have a web-page which handles registry maintenance
in the server. So with one management interface you can
tune various knobs mostly w/o restarting the server.

The persistency can be implemented by a simple
dbm hash-file (ndbm, gdbm).

I mean, nobody can tell me that this is impossible
or not practicable: there are at lest 60.000.000 proofs
of concept out there...

I would need to hear 1 good reason against this.
As you see, I'm not letting loose...

As the matter of fact, I=10 will write this for us internally
as a module. Lets call it "nsregistry" module. It will have
a library to link with so other modules can use it from the
C-level. It will have a Tcl and C API similar to what we have
now, with a small difference: values are obtained by *copy*
and not by reference. It will be properly locked so it can
be used from many threads at the same time. =10I will of course
not repeat the silly case-insensitivity junk: the thing will
be case sensitive. It will have the same section/parameter
approach as the current config. The idea of this module would
be to provide centralized and persistent storage for all kinds
of various configuration parameters one can imagine, for the
server itself and for the modules. From Tcl and from C-modules.
Actually, that what the current config does but w/o aiming to
replace it because of all possible concerns about locking of
speed (or whatever).  And it will be persistent.


Cheers
Zoran
------------------------------------------------------------------------
-
Using Tomcat but need to do more? Need to support web services,
security?
Get stuff done quickly with pre-integrated technology to make your job
easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache
Geronimo
http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=3D120709&bid=3D263057&dat=3D=
121642
_______________________________________________
naviserver-devel mailing list
nav...@li...
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] read/write ns_config

[Zoran V. <zv...@ar...>]

On 24.08.2006, at 21:36, Stephen Deasey wrote:

>
> So I think you can't get away from specialisation, unfortunately.
>

What I still cannot (easily) accept is that we can't
do a config module as repository for the server and
other module's configuration options.
Think of it as the Windows registry (not that I'm
a fan of Microsoft, I'm not, but this "centralized"
approach also has benefits).

What I could imagine is:

We can have the config file as is now. But it would
form just a part of the configuration. Over the runtime
of the server, new configuration could be added or
existing changed. This is non-volatile, means at the
next restart, the changes are still there.
At the startup, one can decide wether the config file
(as now) is the applied first or after loading the,
lets call it registry for now, file. This way you can
decide wether the changes to the values from the config
file are overriden by the registry or not. New stuff
which is only in the registry is of course loaded.
But during the startup you can even discard them completely.
We can have a web-page which handles registry maintenance
in the server. So with one management interface you can
tune various knobs mostly w/o restarting the server.

The persistency can be implemented by a simple
dbm hash-file (ndbm, gdbm).

I mean, nobody can tell me that this is impossible
or not practicable: there are at lest 60.000.000 proofs
of concept out there...

I would need to hear 1 good reason against this.
As you see, I'm not letting loose...

As the matter of fact, I=10 will write this for us internally
as a module. Lets call it "nsregistry" module. It will have
a library to link with so other modules can use it from the
C-level. It will have a Tcl and C API similar to what we have
now, with a small difference: values are obtained by *copy*
and not by reference. It will be properly locked so it can
be used from many threads at the same time. =10I will of course
not repeat the silly case-insensitivity junk: the thing will
be case sensitive. It will have the same section/parameter
approach as the current config. The idea of this module would
be to provide centralized and persistent storage for all kinds
of various configuration parameters one can imagine, for the
server itself and for the modules. =46rom Tcl and from C-modules.
Actually, that what the current config does but w/o aiming to
replace it because of all possible concerns about locking of
speed (or whatever).  And it will be persistent.


Cheers
Zoran=

Re: [naviserver-devel] whole lotta errors

[Zoran V. <zv...@ar...>]

On 24.08.2006, at 22:08, Stephen Deasey wrote:

>>
>> Hm... how would you (generally) check for a byte-array?
>> You can get a byte-array from object but you can't
>> (without looking into the object type which is really not
>> something that is portable) say which type of object you
>> are dealing with.
>
>
> Sure you can:
>
>   byteArrayTypePtr = Tcl_GetObjType("bytearray");
>
>   if (objPtr->typePtr == byteArrayTypePtr) {
>       /* It's a bute array... */
>   }

Yup. I know that. I'm just not sure if you
are "allowed" to peek at the type from the
outside. Normally, Tcl would provide you
with such an API, like Tcl_IsByteArrayObj(objPtr)
or such. The fact they don't obviously means
something, I believe.


>
>> I believe the main source of problem here is somebody
>> slurping the whole file and wanting to return that file
>> as-is i.e. w/o any translations. In that case, he/she
>> cound use [ns_conn encoding] to set the encoding of the
>> connection before calling ns_return. This way we can
>> strip away all the extra options from the content-returning
>> commands and request the writer to use ns_conn to set
>> the correct encoding OR to skip encoding altogether
>> (for example: ns_conn encoding binary).
>> Wouldn't that make sense?
>
>
> I was thinking more of the case where you dynamically create a binary
> object, like a 'captcha' image.  You want to ns_return it and have the
> server do the right thing without having to fiddle with a -binary
> switch.
>
> Another place where checking for a byte array might be good is the
> caching code.  when you cache an object, it first gets converted to a
> valid utf-8 rep.  When you get the object out of the cache, if you
> treat it as a byte array, evrything still works -- the conversion from
> byte array to utf-8 string and back again is non lossy. It's just non
> optimal.

That's true.

>
> As we're starting to see things like the -binary switch spread, I was
> wondering if it was, in general, a good idea to check for byte arrays
> and have things work transparently. Are there any gotchas?


I will think about that. I would like to see less of those
"-binary" things all arround as we all see that it just
confuses people. I just have a bad feeling to rely on the
object type as it can be changed (dropped) by Tcl easily
because of the "everything is a string" paradigm that Tcl
still enforces. There must be some better way to do that,
although if you ask me how, I can't give an answer to that.

Cheers
Zoran

Re: [naviserver-devel] read/write ns_config

[Stephen D. <sd...@gm...>]

On 8/24/06, Zoran Vasiljevic <zv...@ar...> wrote:
>
> On 24.08.2006, at 21:53, Stephen Deasey wrote:
>
> >
> > Well, it's an idea...
>
> Allright :-) That's what I ment.
>
> Seriously, I kind of like it. I guess
> we can move to this gradually.
> As I see it, this relieves the server
> to provide such service and puts the
> control into the module-writer hands.
> This seems OK, as you sooner/later get
> problems trying to find a shoe that
> fits all feet.
>
> OK. Let see where this will lead us.
> I will put this glases from today on
> and see what the colors of the world
> will look like...
>
> This reminds me of the SNMP protocol
> and MIB thing. Wouldn't that be a
> way to go? You know what I mean?
>


SNMP config would be great.  As would self-tuning sub systems. I think
we could do a lot better at making things "just work" without all the
knob tuning.

SNMP would also be great for getting runtime stats and so on.

Would suck to force that into the core though...

Re: [naviserver-devel] whole lotta errors

[Stephen D. <sd...@gm...>]

On 8/20/06, Zoran Vasiljevic <zv...@ar...> wrote:
>
> On 15.08.2006, at 21:09, Stephen Deasey wrote:
> > * is checking for bytearray type a good way to handle binary Tcl
> > objects?
>
> Hm... how would you (generally) check for a byte-array?
> You can get a byte-array from object but you can't
> (without looking into the object type which is really not
> something that is portable) say which type of object you
> are dealing with.


Sure you can:

  byteArrayTypePtr = Tcl_GetObjType("bytearray");

  if (objPtr->typePtr == byteArrayTypePtr) {
      /* It's a bute array... */
  }


> I believe the main source of problem here is somebody
> slurping the whole file and wanting to return that file
> as-is i.e. w/o any translations. In that case, he/she
> cound use [ns_conn encoding] to set the encoding of the
> connection before calling ns_return. This way we can
> strip away all the extra options from the content-returning
> commands and request the writer to use ns_conn to set
> the correct encoding OR to skip encoding altogether
> (for example: ns_conn encoding binary).
> Wouldn't that make sense?


I was thinking more of the case where you dynamically create a binary
object, like a 'captcha' image.  You want to ns_return it and have the
server do the right thing without having to fiddle with a -binary
switch.

Another place where checking for a byte array might be good is the
caching code.  when you cache an object, it first gets converted to a
valid utf-8 rep.  When you get the object out of the cache, if you
treat it as a byte array, evrything still works -- the conversion from
byte array to utf-8 string and back again is non lossy. It's just non
optimal.

As we're starting to see things like the -binary switch spread, I was
wondering if it was, in general, a good idea to check for byte arrays
and have things work transparently. Are there any gotchas?

Re: [naviserver-devel] read/write ns_config

[Zoran V. <zv...@ar...>]

On 24.08.2006, at 21:53, Stephen Deasey wrote:

>
> Well, it's an idea...

Allright :-) That's what I ment.

Seriously, I kind of like it. I guess
we can move to this gradually.
As I see it, this relieves the server
to provide such service and puts the
control into the module-writer hands.
This seems OK, as you sooner/later get
problems trying to find a shoe that
fits all feet.

OK. Let see where this will lead us.
I will put this glases from today on
and see what the colors of the world
will look like...

This reminds me of the SNMP protocol
and MIB thing. Wouldn't that be a
way to go? You know what I mean?

Cheers
Zoran

Re: [naviserver-devel] read/write ns_config

[Stephen D. <sd...@gm...>]

On 8/24/06, Zoran Vasiljevic <zv...@ar...> wrote:
>
> On 24.08.2006, at 21:36, Stephen Deasey wrote:
>
> > So I think you can't get away from specialisation, unfortunately.
>
> IOW, move to ns_logctl kind of commands?
> Identify a module, then provide a control/setup
> konbs via Tcl command?
> Is this the new "mantra" we should follow?


Well, it's an idea...