naviserver-devel — Developer discussion

Re: [naviserver-devel] read/write ns_config

[Zoran V. <zv...@ar...>]

On 23.08.2006, at 13:01, Zoran Vasiljevic wrote:

>
> The master tree structure will have a single global
> mutex and associated epoch counter.

To reduce contention on that mutex, one can compute
a simple checksum of the section name and use this to
address fixed number of buckets, each holding one mutex
and the epoch counter. The number of buckets can be
tuned to fight the contention, if needed. This technique
is already used in nsv implementation.

Cheers
Zoran

Re: [naviserver-devel] read/write ns_config

[Zoran V. <zv...@ar...>]

On 22.08.2006, at 22:13, Stephen Deasey wrote:

>
> I don't see how you will handle the locking...  :-(

I imagine the entire configuration tree being implemented
as a global master (in-memory) tree, close to the DOM.
Actually, it can be really a DOM structure (see below for
the alternative).

I'm assuming that reads (queries into the) config tree will
be a majority of operations whereas writes will be occasional.
I expect this to be far more than 80/20 ratio in favour
of readers vs. writers.

Every thread will have a copy (in TSD) of the master tree
structure and will use it to read config parameters from.

The master tree structure will have a single global
mutex and associated epoch counter.

A thread wanting to read some config param will have
to lock the mutex, examine the epoch and compare it with
the epoch of the copy of the tree it's holding in the TSD.
If there is a match, the mutex is released.
If there is no match, the entire private copy is trashed,
new copy is made and the mutex is released.
In any case, the result is returned from the *local* copy.

A thread wanting to change a parameter will lock the mutex,
bump the epoch, change the parameter in the master copy,
then copy the entire master DOM to its private storage and
release the mutex.

Having the configuration storage in tree means that we can
easily serialize it to XML and store it as a plain file
on the filesystem. This can be done either:
    never
    at server shutdown (or some signal perhaps)
    periodically over some scheduled procedure

Alternatively, one can attach some kind of permanent
persistence (a (g)dbm file for example) to the tree
so every modification is immediately saved.

Due to the assumption that readers will be far more
frequent than writers, so the tree copying between
the master and private copy will be very rare,  all we
have to "accept" would be a short lock on a global mutex
in order to examine the epoch counter and decide wether
our local copy is up-to-date or not.

Alternatively, the DOM tree can be replaced by a combination
of hash-tables and C-structures. Each ns_section will
effectively be a hash table of parameters with a parameter
being a simple C-struct. Also, there will be one global hashtable
needed to resolve ns_section names to hash-tables holding parameters.
I doubt however that this would be any faster then a good DOM
implementation.

This is not a "piece of cake" project hence I would like
to hear some comments before proceeding.

Cheers
Zoran 

Re: [naviserver-devel] read/write ns_config

[Zoran V. <zv...@ar...>]

On 22.08.2006, at 22:13, Stephen Deasey wrote:

>
> I don't see how you will handle the locking...  :-(

Me neiter. I haven't looked at the beast in detail yet.
What I cannot imagine, though, is that there is no way
to have a propoerly locked config depository.

Let me have a look there and see how this could be
done the best.

Cheers
Zoran

Re: [naviserver-devel] read/write ns_config

[Stephen D. <sd...@gm...>]

On 8/22/06, Zoran Vasiljevic <zv...@ar...> wrote:
>
> You know, I have this "requirement" is sitting for about 4 years in my
> todo list. I was not very keen on getting it out as I know that it
> might bear some security (more important) and locking (less important)
> issues. But we've come to a point where we either have to abandon
> the whole ns_config stuff and write everything ourselves OR to
> get this modification in the core server. Needles to say that I
> hate reinventing the wheel over and over again. Hence the decision
> to try to get it done with existing ns_config code.
>
> Basically,  most important information is: from the two (are there
> more?)
> issues (locking and security) which of those two is more
> important/sensitive/whatever to everybody. So I know where I will
> have to make tradeoffs.
>
> Basically, all this comes down to:
>
> a.
>   should I make everything compile-time option so only people who need
>   it would compile-in that code, everybody else being not affected
>
> or
>
> b.
>   should I make it runtime selectable, because people would like to
>   have both, depending on the site/application
>
> or
>
> c.
>   should I make it read/write entirely, basically ignoring both
>   locking and security issues.
>
> The simplest is of course c. The a. is next,  with b. being more/most
> complex. I would of course not want to invest much work with that and
> both security/locking are no problem for me, so the logical choice
> for me would be c. But I do not know if everybody is ok with that.
> Hence all this email ping-pong.
>


I think the C code does the right thing: create a struct to hold the
config values and use the Ns_Config* routines to populate it at
startup. Use the custom struct directly in code.

The Tcl level ns_config is kinda broken. *Every* time you call it the
code does a linear, case insensitive search for the right set; then it
does a linear case insensitive search for the correct key in that set;
then it parses the string value and converts it into an int or bool or
whatever.

ArsDigita figured out this was hurting them back in '99 and created a
wrapper which on first access put the config value in an nsv array.
And *that* is kinda broken as now you suffer locking overhead on
hundreds of config values which are essentially static.

So instead you call ns_config at startup and put the result in a name
spaced Tcl variable. And *that* is kinda broken as the only reason it
works is an accident of implementation -- no one could figure out how
to clean out name spaced variables between requests as efficiently as
global ones.

Anyway, I don't see how you're going to do the locking. Do you have
one global lock around the whole config? A lock per section?  Do you
free updated config values/sets, or do you leak them?

The idea is good -- don't reboot the server when config values change.
Looks hard to me as there are different locking requirements for
different pieces of code.

For example, the ns_log code can now be updated at runtime!  You turn
on/off debug logging, maybe to isolate a specific problem. It's easy,
as updating the boolean value is atomic, and even if it's not it
doesn't matter as it's just a boolean value.

But even there you call ns_logctl -- the underlying config isn't
changed. If every time Ns_Log() was called it in turn called Ns_Conf*
to figure out whether debug logging was on or off, performance would
be unacceptably slow.

Yes, the Tcl code at the moment *does* scan the sets every time
(unless it doesn't because you hacked around it), but it probably
shouldn't. It should also be fast.

How do you handle notifying the code which has parsed and stashed away
a copy of the config value that the config has been re-written?

I don't see how you will handle the locking...  :-(

Re: [naviserver-devel] read/write ns_config

[Zoran V. <zv...@ar...>]

On 22.08.2006, at 18:31, Rick Cobb wrote:

>
> Now, looking @ Bernd's idea, I see, perhaps, a bit where he's  
> going: if,
> for example, the ns_param dictionary had a bit saying "I've been  
> read by
> a module" (no matter which language the module was in), then at server
> *shutdown*, you could print a report of unused ns_params and
> ns_sections.  But really, you can't tell before then (as you've  
> pointed
> out, Zoran).

This is true. If we had a counter on an parameter, we could dump
the whole config tree where counter == 0. This would probably mean
that those have been set by somebody, but never used.
Although I can understand this, somebody has to show me any
practical use of it. You can catch typos with that, allright but
this is at the wrong side of the street! I would expect the server
to refuse booting in such case. Or, emit a big fat error log during
startup. Not at shutdown.

>
> I'll note that in the configuration system for LiveServer, we used our
> publish/subscribe facilities (the stuff we sell) to handle the "write
> back" issues you're solving.  We allow coders to register a (C- 
> only, at
> present) callback for their variables, so if they change, they're
> informed and can immediately change behavior.  We also actually write
> the physical configuration file; if no callback was provided, we  
> inform
> the administrators (presumably one of whom is making the change) that
> the server needs to be restarted before their change will take effect.
>
> To do this, we use a completely separate layer of TCL variables,  
> procs,
> and sourced TCL files to construct ns_section/ns_param calls.  It's  
> not
> elegant, and doesn't fit the real NaviServer/AOLServer world very  
> well,
> but we've found it quite useful.

You know, I have this "requirement" is sitting for about 4 years in my
todo list. I was not very keen on getting it out as I know that it
might bear some security (more important) and locking (less important)
issues. But we've come to a point where we either have to abandon
the whole ns_config stuff and write everything ourselves OR to
get this modification in the core server. Needles to say that I
hate reinventing the wheel over and over again. Hence the decision
to try to get it done with existing ns_config code.

Basically,  most important information is: from the two (are there  
more?)
issues (locking and security) which of those two is more
important/sensitive/whatever to everybody. So I know where I will
have to make tradeoffs.

Basically, all this comes down to:

a.
  should I make everything compile-time option so only people who need
  it would compile-in that code, everybody else being not affected

or

b.
  should I make it runtime selectable, because people would like to
  have both, depending on the site/application

or

c.
  should I make it read/write entirely, basically ignoring both
  locking and security issues.

The simplest is of course c. The a. is next,  with b. being more/most
complex. I would of course not want to invest much work with that and
both security/locking are no problem for me, so the logical choice
for me would be c. But I do not know if everybody is ok with that.
Hence all this email ping-pong.

Cheers
Zoran 

Re: [naviserver-devel] read/write ns_config

[Rick C. <rc...@Kn...>]

My apologies. I was actually responding more to my perception of Bernd's
idea than to the feature of allowing writability of the ns_config system
in general.

My concern with the idea of trying to identify typos or other problems
in configuration is that there are a number of ns_param sets which
aren't a set of fixed names.  In the AOLserver world, the ns_section for
"cgi", for example, allows you to set an ns_param with any name and any
value, none of which are verified by the C module, all of which are just
copied to the environment array for the cgi.

Now, looking @ Bernd's idea, I see, perhaps, a bit where he's going: if,
for example, the ns_param dictionary had a bit saying "I've been read by
a module" (no matter which language the module was in), then at server
*shutdown*, you could print a report of unused ns_params and
ns_sections.  But really, you can't tell before then (as you've pointed
out, Zoran).

I'll note that in the configuration system for LiveServer, we used our
publish/subscribe facilities (the stuff we sell) to handle the "write
back" issues you're solving.  We allow coders to register a (C-only, at
present) callback for their variables, so if they change, they're
informed and can immediately change behavior.  We also actually write
the physical configuration file; if no callback was provided, we inform
the administrators (presumably one of whom is making the change) that
the server needs to be restarted before their change will take effect.

To do this, we use a completely separate layer of TCL variables, procs,
and sourced TCL files to construct ns_section/ns_param calls.  It's not
elegant, and doesn't fit the real NaviServer/AOLServer world very well,
but we've found it quite useful.

-- ReC

-----Original Message-----
From: nav...@li...
[mailto:nav...@li...] On Behalf Of
Zoran Vasiljevic
Sent: Tuesday, August 22, 2006 8:03 AM
To: nav...@li...
Subject: Re: [naviserver-devel] read/write ns_config


On 22.08.2006, at 16:46, Rick Cobb wrote:

> Uh, and I may be missing something about how you're thinking of doing
> this, but how would you handle open configuration sets like the
> environment variables for the CGI module, or the map of mime-type/file
> extensions?

?? ( =3D=3D do not understand)

What do you mean by "open configuration sets" ??
Using ns_config, nothing is "open". It is just
a query into a in-memory data structure.

The fact that this query now uses no locks, and is thus
"cheap", modifying it to use locks is one issue
(and you need locks at that place because somebody
could read and other could modify the same structure).
Another issue is: some module already "consumed"
the parameter at startup and will never ask for it
again: what is the point in changing that particular
parameter?
Also, should this all be allowed anyway (security issue)?

This is what I see as things to take in account.

For the locking issue: I believe this is not that
important as it can be easily "fixed" by storing the
value got from config in a private variable and then
reusing this for the time being.

For the security issue: well, there might be a
config option to allow this, either as runtime
or compile-time option (the latter will solve
any locking issues implicitly)

For the sense of changing a once-read config value?
Well, there isn't (sense). But that has nothing to
do with the implementation.

Cheers
Zoran




------------------------------------------------------------------------
-
Using Tomcat but need to do more? Need to support web services,
security?
Get stuff done quickly with pre-integrated technology to make your job
easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache
Geronimo
http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=3D120709&bid=3D263057&dat=3D=
121642
_______________________________________________
naviserver-devel mailing list
nav...@li...
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] read/write ns_config

[Bernd E. <eid...@we...>]

> There is no such thing as "server" configuration and the "rest" of the
> configuraton. There is one configuration parameters store. Everybody
> can (and does) use it during the startup. Your Tcl modules, your C-
> modules
> core-server modules etc. pp.

Yes, this is the point. I will have to agree that it only "works" if I assume 
the first call (with the "first default") to be "authoritative" - and this is 
probably crap. 

But it also means that the resulting list in the logs (the excerpt in my mail) 
is just a snapshot of the last Ns_Config* call?

Example (4 calls in 4 different parts during startup in this order):

    nsconf.shutdowntimeout =
        Ns_ConfigIntRange(path, "shutdowntimeout", 20, 0, 40);
    nsconf.shutdowntimeout =
        Ns_ConfigIntRange(path, "shutdowntimeout", 21, 1, 41;
    nsconf.shutdowntimeout =
        Ns_ConfigIntRange(path, "shutdowntimeout", 22, 2, 42);
    nsconf.shutdowntimeout =
        Ns_ConfigIntRange(path, "shutdowntimeout", 23, 3, INT_MAX);

Then the logs show the 23,3,INT_MAX call as result???

> I believe we are still not in consensus :-(

We are. Except for one thing, but I have to leave the office now ;-)

Bernd.

Re: [naviserver-devel] read/write ns_config

[Zoran V. <zv...@ar...>]

On 22.08.2006, at 17:19, Bernd Eidenschink wrote:

>
> Ok Zoran, the important part first:
>
>> I'm thinking about allowing *changes* to/of this repository
>> during the runtime. This way one could override a ns_param
>> value assigned at startup or create entirely new ns_section
>> and stuff it with bunch of ns_param so others may call
>> [ns_config] to get it.
>
> I'm all for it if it is possible. It saves us server restarts for  
> dumb issues
> like increasing maximum post sizes etc.

Right. This is very good and trivial example.

>
> Problems: I can change stacksize to "0". I can tell fastpath to use  
> 1 byte of
> memory instead 5 megs. Is memory freed? I guess it comes down to  
> "You are
> able to change everything - but be sure you now what you are doing".

As said: if the module has already consumed the value,
changing it has no real sense. If the fastpath module
already got that 5 MB and allocated the cache, it will
most definitely not ask for the size again, unless it
is programmed to do so. But, since no modules so far
assumed that config values can be changed after startup,
there will be no problems. So, you *could* change it
to 0 bytes but it will mean nothing.

>
>
>
>
> The other part: It would be nice to have the dictionary of config  
> sections and
> options along with their defaults that is created when you set  
> logmaxlevel
> to "6" (dev) or "5" (debug) in section ns/parameters...
>
> -----------------------------------------------------------------
>
> [...]
> Debug: config section: ns/server/default
> Debug: config: ns/server/default:realm value="(null)"  
> default="default"
> (string)
> Debug: config: ns/server/default:checkmodifiedsince value=(null)  
> default=true
> (bool)
> Debug: config: ns/server/default:flushcontent value=false  
> default=false (bool)
> Debug: config: ns/server/default:noticedetail value=(null)  
> default=true (bool)
> Debug: config: ns/server/default:errorminsize value=(null)  
> min=-2147483648
> max=2147483647 default=514 (int)
> Debug: config: ns/server/default:headercase value="(null)"  
> default="preserve"
> (string)
> Debug: config: ns/server/default:outputCharset value=(null) (string)
> Debug: config: ns/server/default:HackContentType value=(null) (bool)
> Debug: config: ns/server/default:urlCharset value=(null) (string)
> [...]
>
> -----------------------------------------------------------------
>
> ...available as TCL command like ns_configsections.
> For the sake of the argument, e.g. "ns_startupconfigsections".
>
> It must be the list created at server startup that DOES NOT CONTAIN  
> entries
> like
>     ns_section milly/vanilly
>     ns_param fate 1
>
> As far as I can tell this is possible because simply because a  
> value was not
> requested by Ns_Config* (Int/Range whatever) commands means it is  
> irrelevant
> for the server. At least in a sense of "get things right for startup".
> Default values are set by these commands.

HE! This is the stumbling point! The fact that something is requested by
Ns_Config is not of any importance! It could have been also requested by
the "ns_config" out of (your own) Tcl startup file!

There is no such thing as "server" configuration and the "rest" of the
configuraton. There is one configuration parameters store. Everybody
can (and does) use it during the startup. Your Tcl modules, your C- 
modules
core-server modules etc. pp.

>
> I think the listing above does also only list those and would not list
> milly/vanilly section and options. Maybe it's already there.
>
> Only then I am able to utilize TCL to diff and compare and return  
> results
> like "milly/vanilly" section may be bogus or a typo. Or that
> parameter "stacksize" does not belong to section "fastpath".

You CANNOT know if a section/parameter is a typo!
If the section (or parameter) is not found and a paramter value is  
*assigned*,
then the section (and the parameter) is/are created! Point.
This is the way how all this stuff works.
Otherwise, some central authority must declare *upfront* all possible  
sections
and parameter values. We do not have this "authority". Every module  
declares
its own.

I believe we are still not in consensus :-(
Give me an example what you mean. This is perhaps easier.

Cheers
Zoran

Re: [naviserver-devel] read/write ns_config

[Zoran V. <zv...@ar...>]

On 22.08.2006, at 17:11, Vlad Seryakov wrote:

> What is the requirement for changing config array, did i miss it?
>
> Can you dump all config tree at startup into ns_cache and then work  
> with
> it instead of ns_config later?

I can't.

The very important aspect is: C *and* Tcl API existence.
Numerous C modules we have, peek into the config. Tcl
modules as well. I would like to allow them to poke
there as well.


Cheers
Zoran

Re: [naviserver-devel] read/write ns_config

[Bernd E. <b.e...@ki...>]

Ok Zoran, the important part first:

> I'm thinking about allowing *changes* to/of this repository
> during the runtime. This way one could override a ns_param
> value assigned at startup or create entirely new ns_section
> and stuff it with bunch of ns_param so others may call
> [ns_config] to get it.

I'm all for it if it is possible. It saves us server restarts for dumb issues 
like increasing maximum post sizes etc.

Problems: I can change stacksize to "0". I can tell fastpath to use 1 byte of 
memory instead 5 megs. Is memory freed? I guess it comes down to "You are 
able to change everything - but be sure you now what you are doing".




The other part: It would be nice to have the dictionary of config sections and 
options along with their defaults that is created when you set logmaxlevel 
to "6" (dev) or "5" (debug) in section ns/parameters...

-----------------------------------------------------------------

[...]
Debug: config section: ns/server/default
Debug: config: ns/server/default:realm value="(null)" default="default" 
(string)
Debug: config: ns/server/default:checkmodifiedsince value=(null) default=true 
(bool)
Debug: config: ns/server/default:flushcontent value=false default=false (bool)
Debug: config: ns/server/default:noticedetail value=(null) default=true (bool)
Debug: config: ns/server/default:errorminsize value=(null) min=-2147483648 
max=2147483647 default=514 (int)
Debug: config: ns/server/default:headercase value="(null)" default="preserve" 
(string)
Debug: config: ns/server/default:outputCharset value=(null) (string)
Debug: config: ns/server/default:HackContentType value=(null) (bool)
Debug: config: ns/server/default:urlCharset value=(null) (string)
[...]

-----------------------------------------------------------------

...available as TCL command like ns_configsections.
For the sake of the argument, e.g. "ns_startupconfigsections".

It must be the list created at server startup that DOES NOT CONTAIN entries 
like
    ns_section milly/vanilly
    ns_param fate 1

As far as I can tell this is possible because simply because a value was not 
requested by Ns_Config* (Int/Range whatever) commands means it is irrelevant 
for the server. At least in a sense of "get things right for startup". 
Default values are set by these commands.

I think the listing above does also only list those and would not list 
milly/vanilly section and options. Maybe it's already there.

Only then I am able to utilize TCL to diff and compare and return results 
like "milly/vanilly" section may be bogus or a typo. Or that 
parameter "stacksize" does not belong to section "fastpath". 

And, as a bonus, I can create a table of options, their defaults, types and 
minimum or maximum value.

Bernd.

Re: [naviserver-devel] read/write ns_config

[Vlad S. <vl...@cr...>]

What is the requirement for changing config array, did i miss it?

Can you dump all config tree at startup into ns_cache and then work with 
it instead of ns_config later?

Zoran Vasiljevic wrote:
> On 22.08.2006, at 16:46, Rick Cobb wrote:
> 
>> Uh, and I may be missing something about how you're thinking of doing
>> this, but how would you handle open configuration sets like the
>> environment variables for the CGI module, or the map of mime-type/file
>> extensions?
> 
> ?? ( == do not understand)
> 
> What do you mean by "open configuration sets" ??
> Using ns_config, nothing is "open". It is just
> a query into a in-memory data structure.
> 
> The fact that this query now uses no locks, and is thus
> "cheap", modifying it to use locks is one issue
> (and you need locks at that place because somebody
> could read and other could modify the same structure).
> Another issue is: some module already "consumed"
> the parameter at startup and will never ask for it
> again: what is the point in changing that particular
> parameter?
> Also, should this all be allowed anyway (security issue)?
> 
> This is what I see as things to take in account.
> 
> For the locking issue: I believe this is not that
> important as it can be easily "fixed" by storing the
> value got from config in a private variable and then
> reusing this for the time being.
> 
> For the security issue: well, there might be a
> config option to allow this, either as runtime
> or compile-time option (the latter will solve
> any locking issues implicitly)
> 
> For the sense of changing a once-read config value?
> Well, there isn't (sense). But that has nothing to
> do with the implementation.
> 
> Cheers
> Zoran
> 
> 
> 
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] read/write ns_config

[Zoran V. <zv...@ar...>]

On 22.08.2006, at 16:46, Rick Cobb wrote:

> Uh, and I may be missing something about how you're thinking of doing
> this, but how would you handle open configuration sets like the
> environment variables for the CGI module, or the map of mime-type/file
> extensions?

?? ( == do not understand)

What do you mean by "open configuration sets" ??
Using ns_config, nothing is "open". It is just
a query into a in-memory data structure.

The fact that this query now uses no locks, and is thus
"cheap", modifying it to use locks is one issue
(and you need locks at that place because somebody
could read and other could modify the same structure).
Another issue is: some module already "consumed"
the parameter at startup and will never ask for it
again: what is the point in changing that particular
parameter?
Also, should this all be allowed anyway (security issue)?

This is what I see as things to take in account.

For the locking issue: I believe this is not that
important as it can be easily "fixed" by storing the
value got from config in a private variable and then
reusing this for the time being.

For the security issue: well, there might be a
config option to allow this, either as runtime
or compile-time option (the latter will solve
any locking issues implicitly)

For the sense of changing a once-read config value?
Well, there isn't (sense). But that has nothing to
do with the implementation.

Cheers
Zoran

Re: [naviserver-devel] read/write ns_config

[Rick C. <rc...@Kn...>]

Uh, and I may be missing something about how you're thinking of doing
this, but how would you handle open configuration sets like the
environment variables for the CGI module, or the map of mime-type/file
extensions?

Thanks --
-- ReC

-----Original Message-----
From: nav...@li...
[mailto:nav...@li...] On Behalf Of
Zoran Vasiljevic
Sent: Tuesday, August 22, 2006 6:41 AM
To: nav...@li...
Subject: Re: [naviserver-devel] read/write ns_config


On 22.08.2006, at 15:26, Bernd Eidenschink wrote:

>
> Because only during startup we have a chance to gather a knowledge =20
> of "core"
> configuration options via the Ns_Config* C-command calls.

I might be stupid. (A valid explanation why) I still do not
understand what you need?

The config machinery is rather dumb as it (only) has
a notion of the ns_section, one or more ns_param's
in it and some basic type checking on ns_params.
Not more.
There are no "hardwired" or "default" or C-level or
Tcl-level sections or parameters. All of them are
created equal.

The config repository is built during the startup and
later-on never changed (current behaviour)

I'm thinking about allowing *changes* to/of this repository
during the runtime. This way one could override a ns_param
value assigned at startup or create entirely new ns_section
and stuff it with bunch of ns_param so others may call
[ns_config] to get it.  (Obviously, changing some of the
parameters may have no sense as they will be read only during
the startup but this is not important at this time).

Now, what would you need EXACTLY, in addition to that?

Can you describe this shortly and possibly give one example
so that I can understand it as well (I'm getting old, you know...)

Cheers
Zoran





------------------------------------------------------------------------
-
Using Tomcat but need to do more? Need to support web services,
security?
Get stuff done quickly with pre-integrated technology to make your job
easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache
Geronimo
http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=3D120709&bid=3D263057&dat=3D=
121642
_______________________________________________
naviserver-devel mailing list
nav...@li...
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] read/write ns_config

[Zoran V. <zv...@ar...>]

On 22.08.2006, at 15:26, Bernd Eidenschink wrote:

>
> Because only during startup we have a chance to gather a knowledge  
> of "core"
> configuration options via the Ns_Config* C-command calls.

I might be stupid. (A valid explanation why) I still do not
understand what you need?

The config machinery is rather dumb as it (only) has
a notion of the ns_section, one or more ns_param's
in it and some basic type checking on ns_params.
Not more.
There are no "hardwired" or "default" or C-level or
Tcl-level sections or parameters. All of them are
created equal.

The config repository is built during the startup and
later-on never changed (current behaviour)

I'm thinking about allowing *changes* to/of this repository
during the runtime. This way one could override a ns_param
value assigned at startup or create entirely new ns_section
and stuff it with bunch of ns_param so others may call
[ns_config] to get it.  (Obviously, changing some of the
parameters may have no sense as they will be read only during
the startup but this is not important at this time).

Now, what would you need EXACTLY, in addition to that?

Can you describe this shortly and possibly give one example
so that I can understand it as well (I'm getting old, you know...)

Cheers
Zoran

Re: [naviserver-devel] read/write ns_config

[Bernd E. <b.e...@ki...>]

> If you say you can collect all sections and all params with
> Tcl and do your cross-checking, why bother?

Because only during startup we have a chance to gather a knowledge of "core" 
configuration options via the Ns_Config* C-command calls.

Only at a stage somewhere at the end of the startup process I can identify 
options belonging to sections I did not Ns_Config-request and therefor assume 
that a particular option is
a) "unknown" aka. not belonging to a section or
b) "unknown" in section x but was Ns_Config-requested elsewhere, where it was 
not found and the default value was used

The C-level-Ns_Config-calls-during-startup here are my "authoritative" guides 
to build a list of configuration options. Later on in TCL I have to accept 
milly/vanilly or any other part in the soup unless I create a mandatory list 
from the source code by hand. This is the way now for everyone of us to 
identify 
a) parameters that are not used any more!
b) default values!
c) even options you never heard of!

Bernd.

Re: [naviserver-devel] read/write ns_config

[Zoran V. <zv...@ar...>]

On 22.08.2006, at 14:44, Bernd Eidenschink wrote:

> As it is possible to list all options of all sections (from C and  
> TCL), at the
> end I can detect config options that I did not request and set during
> startup.
> Therefor I can lookup if the options belong to other sections or if  
> the
> options are unknown in a section or totally unknown.

If something can be done in couple of Tcl lines and
if this is not something you'd do 1000.000 times a day
in a tight loop I'd leave it on the Tcl level.

Mingling with C-code is about 10-times the work you'd
invest for Tcl and it has a side-effect of gross
destabilization as every single/small error is fatal.
Therefore, I could/would do this in C only if it is really
issue of speed (you need light-fast code) or data-fiddling
(can be done with Tcl but is dead-inneficient and unelegant)
or lack of Tcl interface.

If you say you can collect all sections and all params with
Tcl and do your cross-checking, why bother?

Cheers
Zoran

Re: [naviserver-devel] read/write ns_config

[Bernd E. <eid...@we...>]

> I would say: this is pretty impossible! Nobody imposes any known set of
> config options to the underlying implemntation so the implementation
> cannot
> do any "sanity" checks beside basic type-checking.
> You can pretty much say:
>
>     ns_section milly/vanilly
>     ns_param fate 1

But the point is: At a certain stage the server has finished reading all 
configuration options. You would be able to detect that

ns_section milly/vanilly

is a section you have no knowledge of or no use for:
"unknown config section"

This is of course only true when it is really not used, e.g. at a later point 
in time by  an "external" TCL module.

If, during startup, a configuration value is requested for the first time, the 
value OR the default value is set. This knowledge is buried in the C call:

    nsconf.shutdowntimeout =
        Ns_ConfigIntRange(path, "shutdowntimeout", 20, 0, INT_MAX);

A list of all options, btw., is created then the server logs debug notices.

As it is possible to list all options of all sections (from C and TCL), at the 
end I can detect config options that I did not request and set during 
startup.
Therefor I can lookup if the options belong to other sections or if the 
options are unknown in a section or totally unknown.

This should be possible! The question is, if it makes sense, as only options 
are detected that are requested and set during startup and TCL module ones 
will be missing.

Bernd.

Re: [naviserver-devel] read/write ns_config

[Zoran V. <zv...@ar...>]

On 22.08.2006, at 14:03, Bernd Eidenschink wrote:

>>> If, as a side-effect of your work, you will find it easy to
>>> implement a way to
>>> let ns_config return all
>>> a) non-core/non-default/non-c-module-specific
>>
>> you mean only those set from the "outside" i.e. config file?
>
> Basically a way to detect a typo of the ns_param key. Something  
> that _could_
> lead to a basic configuration check.
> Now a typo is just a missing parameter, the server sets a default  
> value and
> you'll likely not notice it.
> Currently the knowledge about configuration options and their  
> defaults is
> burried in the source, the setting of defaults is done during  
> startup so a
> check without starting a server may be impossible;
>
> beside that, collecting "default" keys and their values during  
> startup should
> be possible - but the identification of unknown keys would only be  
> possible
> when no more config options are read/set (see next).


I would say: this is pretty impossible! Nobody imposes any known set of
config options to the underlying implemntation so the implementation  
cannot
do any "sanity" checks beside basic type-checking.
You can pretty much say:

    ns_section milly/vanilly
    ns_param fate 1

and the ns_config will swallow this peacefull, althought you really  
wanted
to say:

    ns_param fake 1

If we could define all possbile options of "milly/vanilly" section
somehow, then it would be possible to do checks like that.
Hm... perhaps this is not that bad idea at all. What about:

   ns_section name ?args?

where args would give a list of all supported parameters for that
section? Then the implementation could easily complain if you
misspelled the parameter. But this leaves you with abother trouble...
What if you say:

    ns_section mily/vanilly

(note one "el" missing in section name) then you are also out of luck.
To cover that, you would need to be able to give all sections in advance
which is, of course pretty weird. Given the amount of work to do that
*properly* I'd rather opt with the basic type-checking we have now.


>
>>> b) not-belonging-to-this-section
>>
>> give me one example about what you mean.
>
> Similar to the one above, but it would be the result of an extra  
> lookup: does
> the ns_param key (of which we think it is a typo) in fact belong to  
> another
> section?
> It could currently only be done at a stage where we know all config  
> options
> are set up.
>
> ns_section ns/parameters
> # a fastpath entry in fact belonging to ns/server/$server/fastpath:
> ns_param cachemaxentry 8192
>
> Hm. The lookup may result in n-results, as some key-names are used  
> in multiple
> sections.
>
> "Unkown config option x in section y"
> "Config option x not known in section y, check section[s] z, z' or  
> z''"
>
> Frankly, it is probably not worth to keep an eye on that...

Exactly.

Cheers
Zoran

Re: [naviserver-devel] read/write ns_config

[Bernd E. <eid...@we...>]

> > If, as a side-effect of your work, you will find it easy to
> > implement a way to
> > let ns_config return all
> > a) non-core/non-default/non-c-module-specific
>
> you mean only those set from the "outside" i.e. config file?

Basically a way to detect a typo of the ns_param key. Something that _could_ 
lead to a basic configuration check. 
Now a typo is just a missing parameter, the server sets a default value and 
you'll likely not notice it.
Currently the knowledge about configuration options and their defaults is 
burried in the source, the setting of defaults is done during startup so a 
check without starting a server may be impossible; 

beside that, collecting "default" keys and their values during startup should 
be possible - but the identification of unknown keys would only be possible 
when no more config options are read/set (see next).

> > b) not-belonging-to-this-section
>
> give me one example about what you mean.

Similar to the one above, but it would be the result of an extra lookup: does 
the ns_param key (of which we think it is a typo) in fact belong to another 
section? 
It could currently only be done at a stage where we know all config options 
are set up.

ns_section ns/parameters
# a fastpath entry in fact belonging to ns/server/$server/fastpath:
ns_param cachemaxentry 8192  

Hm. The lookup may result in n-results, as some key-names are used in multiple 
sections.

"Unkown config option x in section y"
"Config option x not known in section y, check section[s] z, z' or z''"

Frankly, it is probably not worth to keep an eye on that...

Bernd.

Re: [naviserver-devel] read/write ns_config

[Zoran V. <zv...@ar...>]

On 22.08.2006, at 12:56, Bernd Eidenschink wrote:

>
> My only suggestion would be to be somehow able to let the current  
> behaviour
> (read-only-access) be configurable or default; read-only-access in  
> some
> situations is a means of security - in others of course it is more  
> or less
> useless, as one can drop or at least delete the database anyway.

Yes. Actually, security on one and performance on the other
side are major reasons against such a change. But in some
cases, as in ours, neither of those really apply. Yet we could
benefit a lot from already existing code which is both Tcl and
C-level accesible.

I do not know if it will be possible to do select this as
runtime option w/o performance (locking) penalty. I may need
to make it a compile-time option. But I haven't started on that
yet, so I can't really say which way it will go, if at all.

>
> A change of an option should also be worth a "ns_log notice" in  
> that respect.
>
> If, as a side-effect of your work, you will find it easy to  
> implement a way to
> let ns_config return all
> a) non-core/non-default/non-c-module-specific

you mean only those set from the "outside" i.e. config file?

> b) not-belonging-to-this-section

give me one example about what you mean.

Cheers
Zoran

Re: [naviserver-devel] read/write ns_config

[Bernd E. <eid...@we...>]

Hi Zoran,

> Idea is to allow ns_config to actually write the
> config, instead of only reading it.
[...]
>Unfortunately
>this works only during the startup i.e. things can be
>added to the config only at startup and later on you can
>only read it. What I would need is the ability to write
>new and/or change the existing values.

Sounds good! 

My only suggestion would be to be somehow able to let the current behaviour 
(read-only-access) be configurable or default; read-only-access in some 
situations is a means of security - in others of course it is more or less 
useless, as one can drop or at least delete the database anyway.

A change of an option should also be worth a "ns_log notice" in that respect.

If, as a side-effect of your work, you will find it easy to implement a way to 
let ns_config return all 
a) non-core/non-default/non-c-module-specific
b) not-belonging-to-this-section 
configuration options, I'll ask you to implement that, too! :-)

Cheerio!
Bernd. 

[naviserver-devel] read/write ns_config

[Zoran V. <zv...@ar...>]

Hi!

This one stands very long on my list but I'm not
sure if everybody would be pleased...

Idea is to allow ns_config to actually write the
config, instead of only reading it.

Without going into details how this would look like
on the Tcl level, I just wanted to poke what you
think about the underlying implemnentaion, as the
current one would have to be changed i.e. locks added.

The thing I like about ns_config is that it has pretty
nice tree-like structured repository of config options
which you can peek from C and Tcl level. Unfortunately
this works only during the startup i.e. things can be
added to the config only at startup and later on you can
only read it. What I would need is the ability to write
new and/or change the existing values.

Cheers
zoran 

Re: [naviserver-devel] whole lotta errors

[Bernd E. <eid...@we...>]

Hi Stephen,

> In the documents you serve, do you specify the encoding *within* the
> document, at th etop of the HTML file for example? Or are you serving
> XML in which case the default for that is utf-8 anyway (I think, off
> the top of my head...).

Usually we specify it both ways, in the
     meta http-equiv
part of the HTML header and the
     Content-Type
header of the HTTP response.

> Another possibility is that you happen to be using browsers which are
> smart enough to reparse a document if it doesn't happen to be in the
> encoding it first expected.  I think the big guys do this -- not sure
> your mobile phone will be so forgiving.

I'd say we are perfectly happy with just setting up the config file via the 
ns/encodings + ns/mimetypes sections and let the server handle the rest. The 
less knobs the better. We know (or can control) the encoding of files on 
disk, we set up the encoding of the database - and then we simply want to 
return the specified encoding. We have different sites running with 
iso-8859-1, -15 and utf-8.
Usually we have no need to do runtime changes, but if so, I would like to see 
ns_conn to do the expected thing.

Only relying on (aka. being forced to use) UTF-8 would not be optimal as a 
potential naviserver user might want to use another specified encoding or 
avoid a UTF/unicode database setup for whatever reason, e.g. performance, 
storage or to avoid collation issues (sorting orders). 

For us using only web and http moving with every installation to UTF-8 is 
nevertheless the way to go.

> (This applies to case 3: supporting multiple encodings)
>
>
> I agree with Zoran. ns_conn encoding should be the way to change the
> encoding (input or output) at runtime.

yes.

> Another place this trips up: In the config for the tests Michael added:
>
>   ns_section "ns/mimetypes"
>   ns_param   .utf2utf_adp  "text/plain; charset=utf-8"
>   ns_param   .iso2iso_adp  "text/plain; charset=iso-8859-1"
>
>   ns_section "ns/encodings"
>   ns_param   .utf2utf_adp  "utf-8"
>   ns_param   .iso2iso_adp  "iso-8859-1"
>
> The ns/encodings are the encoding to use to read an ADP file from
> disk, accoring to extension. It solves the problem: the web designers
> editor doesn't support utf-8. 

If you focus here only on web designers and adp files. It could be every other 
kind of usage as well (file exports etc.).

> But, the code is actually expecting Tcl encoding names here, not a
> charset, so this config is busted. It doesn't show up in the tests
> because the only alternative encoding we're using is iso-8859-1, which
> also happens to be the default.

this is correct, an annoying thing to be aware of.

> The strategy of driving the encoding from the mime-type has some other
> problems.  You have to create a whole bunch of fake mime-types /
> extension mappings just to support multiple encodings (the
> ns/mimetypes above).
>
> What if there is no extension? Or you want to keep the .adp (or
> whatever) extension, but serve content in different encodings from
> different parts of the URL tree? Currently you have to put code in
> each ADP to set the mime-type (which is always the same) explicitly,
> to set the charset as a side effect.

this is true. it does not affect our apps, as we commit to one encoding and 
then cache the HTML output to files on disk, but it is not nice if you have 
the need to change it.

> * utf-8 by default
> * mime-types are just mime-types
> * always hack the mime-type for text data to add the charset
> * text is anything sent via Ns_ConnReturnCharData()
> * binary is a Tcl bytearray object
> * static files are served as-is, text or binary
> * multiple encodings are handled via calling ns_conn encoding
> * folks need to do this manually. no more file extension magic

> I think a nice way for folks to handle multiple encodings is to
> register a filter, which you can of course use to simulate the file
> extension scheme in place now, the AOLserver 4.5 ns_register_encoding
> stuff, and more, because it's a filter. You can also do things like
> check query data or cookies for the charset to use.

As our app has one main filter that handles the file dispatching we simply 
would place it there.
But we should find a solution that is both flexible and compatible in respect 
of the "file extension magic", if possible!

> Questions that need answered:
>
> * can we junk charset aliases in nsd/encodings.c and use a dir of symlinks?

i would vote for non filesystem based lookup function.

> * can we junk ns/encodings in 2006?

i would not recommend it as the server loses purposes.

Bernd.

Re: [naviserver-devel] whole lotta errors

[Bernd E. <eid...@we...>]

Am Sonntag, 13. August 2006 21:44 schrieb Gustaf Neumann:
> Michael Lex schrieb:
> > It's only going to hit you if you use some other encoding than utf-8
> > and then only if you use a HTTP-client, that does not understand
> > chunked encoding. But every web-browser and virtually every other
> > client-software should understand this (It's a MUST part of the rfc).
>
> ... to be more precise, of HTTP/1.1, not HTTP/1.0
>
> do you have any statistics, what your clients use? we have still a lot
> of HTTP/1.0
> requests (around 5%).

On one customer server with about 13 Million requests this month I have also 
5,17% of http/1.0 requests...
This one will be replaced at the end of the year with naviserver, it still 
runs AS.

Bernd.

Re: [naviserver-devel] whole lotta errors

[Bernd E. <eid...@we...>]

> I would ask the commiter (we will found who that is soon) to always
> update ChangeLog file so we know what is going on. Otherwise it
> is just a waste of time and nerves...

Sorry for not updating the ChangeLog! :-(

Guess I should consider the medical usage part of 
http://en.wikipedia.org/wiki/Ginkgo

Bernd.