naviserver-devel — Developer discussion

Re: [naviserver-devel] read/write ns_config

[Zoran V. <zv...@ar...>]

On 24.08.2006, at 21:36, Stephen Deasey wrote:

> So I think you can't get away from specialisation, unfortunately.

IOW, move to ns_logctl kind of commands?
Identify a module, then provide a control/setup
konbs via Tcl command?
Is this the new "mantra" we should follow?

Cheers
Zoran

Re: [naviserver-devel] read/write ns_config

[Stephen D. <sd...@gm...>]

On 8/24/06, Zoran Vasiljevic <zv...@ar...> wrote:
>
> This MUST also internally lock, or? You can't have shared access
> w/o locking, or?
> Is this limits stuff yet another "specialized" thing like this
> config, which costed me so much hair pulling? If yes, then we
> should avoid using it at any cost!
>

It does lock internally, and it is specialised. But that's the point
-- you can't have a simplistic locking scheme apply to all subsystems.

The logctl stuff, for example has *no* locking (for what it currently
does), because it doesn't need it. I gave an example of allowing the
config to be set/changed only at start up, which is another way of
handling locking -- completely forbid it after startup.  You could
easily imagine more complicated schemes, with reference counting etc.

So I think you can't get away from specialisation, unfortunately.

Re: [naviserver-devel] read/write ns_config

[Zoran V. <zv...@ar...>]

On 24.08.2006, at 20:26, Stephen Deasey wrote:

>
> That's because it's a bad idea... :-)

Ah, it is. The "original" idea is not though.
I hate to reboot the server just to turn on
or off a bit. We have a module doing SCSI
communication. It can be turned on to make
extended logging. This is controlled by a
ns_config parameter. So, on customer site
I might need to turn this in in the middle
of the job. The processed some TB of data
and got stucked, I need to find out why.
The job ran for some 10's of hours (perhaps
2 or 3 days) and consumed about 10-15 tapes.
Now I'd need to STOP the server, killing the job?
No way. There has to be a better solution.
Hence the need for changing in-memory config.

>
> Maybe the limits stuff is a possible model. Unlike the aols4.5
> version, the one is our cvs handles config by default in the standard
> fashion by looking up values in the config file. However, it does this
> from Tcl in tcl/limits.tcl.
>
> The limits code creates a new tool. The tricky stuff, the locking, is
> handled in C. The tool is presented as Tcl commands, and Tcl is used
> to configure it at start up. It looks just like any other server
> subsystem:
>
>   ns_section "ns/limits"
>   ns_param  default         "The default limits"
>   ns_param  restrict         "A restricted set of limits"
>
>   ns_section "ns/limits/default"
>   ns_param  maxrun        100
>   ns_param  maxwait       100
>   ns_param  maxupload    [expr 10*1024*1000]
>   ns_param  timeout          60
>
>   ns_section "ns/limits/restrict"
>   ...
>
> In the majority of cases, that's probably all that's needed. Config
> goes in the config file, not buried in some Tcl code.
>
> However! You can also, at runtime, call:
>
>   ns_limits_set -maxrun 10 -timeout 30 -- restrict
>
>
> This is what you're trying to achieve with the read/write config. The
> defaults get set at start up, and then at runtime, you call
> ns_config_set and everything magicaly updates.

This MUST also internally lock, or? You can't have shared access
w/o locking, or?
Is this limits stuff yet another "specialized" thing like this
config, which costed me so much hair pulling? If yes, then we
should avoid using it at any cost!

>
> The tool based approach is more flexible though. You could, if you
> wanted, have a very minimal config file to bootstrap the server and
> config everything yourself, getting the values from an HTTP server (as
> someone mentioned), from a local SQLite db, whatever you can dream up.
>
> A tool based approach also fits in better as a stand alone Tcl module.
> There's usually not 'config' file when you 'package require' a module.

The tool-based approach is exactly what?
Is that something like ns_ictl or ns_logctl where you
have an extra configurator command?
If yes, then yes, I understand this. This is perhaps better
way to go. It is more logical and versatile than the config
stuff. To treat the config as a big "default" seems logical
in this light.

>
>
> Anyway, the idea is that maybe forcing everything to go through the
> config system is too restricting, and that the config, as it stands,
> is a great *default*.  And recognising that the locking requirements
> can be very different for different subsystems. Some config options
> are dependant on others. A change in some should initiate an action,
> such as purge a cache down to it's new maximum size.

Yes.

>
> Maybe what we need is to start moving more things to this model. For
> example, remove the calls to Ns_Config*() from nsd/log.c and instead
> use ns_logctl from within tcl/init.tcl etc. (move the limits stuff
> there).

Yes.


>
> It may be the case that not all the logging options can be dynamically
> adjusted. That's OK, maybe they can be set via ns_logctl only at start
> up, which is a special single threaded environment.
>
> Perhaps there's some helper functions we can add that would make
> writing config functions easier, in the way that Ns_ParseObjv() makes
> parsing options easier. Seems like there's a whole bunch of manual
> twiddling going on in nsproxy/nsproxylib.c:ConfigureObjCmd() for
> example.

Ah... this is right. I will rewrite that. Old habits are
difficult to handle!

>
>
> Now, climb down off the balcony and have a nice cup of tea and a  
> biscuit!

Only after I get a clear picture :-) But the hint with tool approach
is a good way to look at. I will have to give it more thoughts.

Thanks for the encouraging ideas!
Cheers
Zoran

Re: [naviserver-devel] read/write ns_config

[Stephen D. <sd...@gm...>]

On 8/24/06, Zoran Vasiljevic <zv...@ar...> wrote:
>
> On 22.08.2006, at 22:13, Stephen Deasey wrote:
>
> >
> > I don't see how you will handle the locking...  :-(
>
> And I don't see how I'm going to handle config values
> returned to caller *per reference* (as for the string
> values) as this is what happens to be the case now :-(
>
> If we *ever* want to make this beast shared read/write
> then we need copy-on-write and copy-on-read access
> (I guess ref-counting would be simply overkill).
> Which means radical change to how the config values
> are handled throughout the server code *and* extensions.
>
> Ah, I really hate it!
>

That's because it's a bad idea... :-)

Ns_Sets allow you to store multiple identically named keys.  They also
preserve the order of the entries.

Finding the 50th element of a 100 element set isn't a representative
test. The primary use case for sets is HTTP headers and forms. The
sets are as likely to have 6 elements, need to be created quickly,
preserve order, and for headers, be case insensitive.

Sets are just completely different data structures to hash tables!


This is hard because you're writing a database: you have a data model,
concurrency, and persistence.  Don't do it!

Persistence is broken.  Config files are Tcl scripts. The 'section'
and 'param' form a DOM. But the other parts of the script are also
important, from comments to Tcl variables. If you write out the DOM
you completely loose all the Tcl.


Even so, you're still only tackling a subset of the problem, which
seems to be code which runs enfrequently enough that it can be as slow
as you like.

What about code which needs to be fast? Seems to be out of scope...
The Tcl config stuff is *already* too slow, adding locking can only
slow it down.



> Which leaves me 3 possible choices:
>
>    1. Go climb the Eiffel tower and throw myself down
>    2. Go climb the Empire State Building and throw myself down
>    3. Write our own module for handling configuration


Maybe the limits stuff is a possible model. Unlike the aols4.5
version, the one is our cvs handles config by default in the standard
fashion by looking up values in the config file. However, it does this
from Tcl in tcl/limits.tcl.

The limits code creates a new tool. The tricky stuff, the locking, is
handled in C. The tool is presented as Tcl commands, and Tcl is used
to configure it at start up. It looks just like any other server
subsystem:

  ns_section "ns/limits"
  ns_param  default         "The default limits"
  ns_param  restrict         "A restricted set of limits"

  ns_section "ns/limits/default"
  ns_param  maxrun        100
  ns_param  maxwait       100
  ns_param  maxupload    [expr 10*1024*1000]
  ns_param  timeout          60

  ns_section "ns/limits/restrict"
  ...

In the majority of cases, that's probably all that's needed. Config
goes in the config file, not buried in some Tcl code.

However! You can also, at runtime, call:

  ns_limits_set -maxrun 10 -timeout 30 -- restrict


This is what you're trying to achieve with the read/write config. The
defaults get set at start up, and then at runtime, you call
ns_config_set and everything magicaly updates.

The tool based approach is more flexible though. You could, if you
wanted, have a very minimal config file to bootstrap the server and
config everything yourself, getting the values from an HTTP server (as
someone mentioned), from a local SQLite db, whatever you can dream up.

A tool based approach also fits in better as a stand alone Tcl module.
There's usually not 'config' file when you 'package require' a module.


Anyway, the idea is that maybe forcing everything to go through the
config system is too restricting, and that the config, as it stands,
is a great *default*.  And recognising that the locking requirements
can be very different for different subsystems. Some config options
are dependant on others. A change in some should initiate an action,
such as purge a cache down to it's new maximum size.

Maybe what we need is to start moving more things to this model. For
example, remove the calls to Ns_Config*() from nsd/log.c and instead
use ns_logctl from within tcl/init.tcl etc. (move the limits stuff
there).

It may be the case that not all the logging options can be dynamically
adjusted. That's OK, maybe they can be set via ns_logctl only at start
up, which is a special single threaded environment.

Perhaps there's some helper functions we can add that would make
writing config functions easier, in the way that Ns_ParseObjv() makes
parsing options easier. Seems like there's a whole bunch of manual
twiddling going on in nsproxy/nsproxylib.c:ConfigureObjCmd() for
example.


Now, climb down off the balcony and have a nice cup of tea and a biscuit!

Re: [naviserver-devel] read/write ns_config

[Zoran V. <zv...@ar...>]

On 22.08.2006, at 22:13, Stephen Deasey wrote:

>
> I don't see how you will handle the locking...  :-(

And I don't see how I'm going to handle config values
returned to caller *per reference* (as for the string
values) as this is what happens to be the case now :-(

If we *ever* want to make this beast shared read/write
then we need copy-on-write and copy-on-read access
(I guess ref-counting would be simply overkill).
Which means radical change to how the config values
are handled throughout the server code *and* extensions.

Ah, I really hate it!

Which leaves me 3 possible choices:

   1. Go climb the Eiffel tower and throw myself down
   2. Go climb the Empire State Building and throw myself down
   3. Write our own module for handling configuration

I desperately wanted to avoid 3. (for simple reasons).
I have very little chices left.

(Confused and disappointed)
Zoran

Re: [naviserver-devel] read/write ns_config

[Zoran V. <zv...@ar...>]

On 24.08.2006, at 11:14, Zoran Vasiljevic wrote:

> The b. turns out to be problematic because of the case
> sensitivity.

Just for your information, the section names are always
lowercased. Paramters are not. They are stored as-is
and searched either case-sensitive or case-insensitive.

Even more, the sections are allowed to be in form
    ns\section\whatever
as the code will also swap backslashes...

I would say: this is a complete and ultimate CRAP.
This smells strong at some ill-concept borrowed from
you know what company...

I do really and deeply hate this junk. Really.
If I would be asked, I'd go to extent of junking all
this mess and making sections AND parameters case
sensitive and properly slash-separated.  This way we
could trash ns_sets and all the troubles we have with
them and we could use nsv instead.
But I can't make this decision by myself. If we don't
get all agree then it's better not to do it.

Hugh.
Zoran

Re: [naviserver-devel] read/write ns_config

[Zoran V. <zv...@ar...>]

On 23.08.2006, at 20:44, Zoran Vasiljevic wrote:

> But lets see if there are another ideas or remarks.

Ah... damn... sometimes it's not easy to fit in new shoe
on the old foot!

I have been investigating all those options:

   a. using DOM tree as config repository
   b. using nsv's as config repository
   c. continue to use (but modify for speed) the ns_set

and this is what I've found...

The a. is most complex yet, will not effectively bring much
as it will be almost the fast as the c. (if not slower).
It needs lots of work and would definitely bring in some
other nice side-effects, but at this point, and for this
purpose *only*, this may be an overkill.

The b. turns out to be problematic because of the case
sensitivity. We need to retain user-given input, yet be
able to search in case-insensitive way. The nsv's will
not be the right choice for that, unfortunately. We'd have
to make too many tricks to make this work and this does not
pay off.


The c. is also problematic as if we would like to improve
ns_set speed, we'd need to modify Ns_Set strucure which is
public and would require all extension that use it to
be recompiled. There are some other minor issues to that
as well which complicate the change, adding to the trouble.

Conclusion:
It seems to me that we will have to live with ns_set as the
configuration repository for some time to come.
Again, I'd like to know who was that smart guy who wrote
this piece of code in the first place... grmpf...


Now, the question about adding shared access to the config
repository is simple to answer:  I will not make this by default.
Instead, I will make it as a compile time option and set it as
disabled per default.
This way, nobody will experience any (whatever) side-effects.
If however the support is being compiled in, there will be one
additional command available:

   ns_configctl

and it will contain subcommands:

   ns_configctl save channel
   ns_configctl load channel

This is self-explanatory, I believe.

The ns_config command will be extended in the form:

   ns_config ?-force? section key ?value?

I did not want to include yet another command to explictily
set the new configuration value. Therefore I tried to squeeze
it in the existing ns_config (the -force flag).
This flag, if set, will require the syntax of command to be:

   ns_config -force section key value

the value isn't optional any more; it must be given and it
will be used to set/override existing config value.
This also means that [ns_config -force section key] will
actually trigger syntax error.
(I'm not very happy about the syntax of the ns_config but
I could not find a better one. If there are good alternatives
please get them in.)

I will also inspect callers of config from the C code and
try to cache the values privately, thus avoiding frequent
config lookups. This will not hurt anybody as in the
non-writable config this does not matter.


Basically, for all of you, the changes I'm willing to do
will be invisible, unless you turn this on at compile time.
I find this fair as it will give us option to get long
awaited feature on board w/o impairing other users.

Cheers
Zoran

Re: [naviserver-devel] read/write ns_config

[Zoran V. <zv...@ar...>]

On 23.08.2006, at 19:41, Mike wrote:

> "Two" has to remain with-case because it is a value, but its intent is
> to point the reader of the config at ns/two, which is now lowercased.
>

Urks! I would never come to that usage pattern,
but you are absolutely right.
Another reason we must preserve the user-given case.

I have an idea how this can still be achieved
with nsv but it would cost some extra memory...
But lets see if there are another ideas or remarks.

Cheers
Zoran

Re: [naviserver-devel] read/write ns_config

[Mike <nee...@gm...>]

On 8/23/06, Zoran Vasiljevic <zv...@ar...> wrote:
>
> On 23.08.2006, at 18:35, Mike wrote:
>
> > I do have one concern with this change - what happens when the value
> > (where case must be preserved) for some key is actually the name of
> > another section or key?  I do not have enough experience to tell if
> > this is ever a problem, but I am not sure how this situation would be
> > handled.  Perhaps by the code that uses that config value?
>
> Hm... this does not compute.
>
> If we look at the how nsv's would replace the existing ns_set
> as backend for storage: each ns_section would receive a separate
> nsv_array, so to speak. In details, this would be handled on the
> C instead of Tcl level, but it can be described this way for
> simplicity.

ns_section ns/one
ns_param look_in_section "Two"
ns_section ns/two
ns_param look_for_me "value"

"Two" has to remain with-case because it is a value, but its intent is
to point the reader of the config at ns/two, which is now lowercased.

Re: [naviserver-devel] read/write ns_config

[Zoran V. <zv...@ar...>]

On 23.08.2006, at 19:20, Rick Cobb wrote:

> Potential objection to normalizing the case: ns_params for CGI
> environment variables.  Environment variable names are case sensitive,
> and it's perfectly possible that there are applications that use
> lower-case names for these.  Would require changing nscgi to something
> more like
> Ns_param set {{variable_name} {value}}
>
> Don't know if there are more like this. We have used a similar
> configuration for setting java properties, though, and IIRC, at least
> some of those required mixed-case names (ClassStuff.propertyName).
>

Yes. This is right. This would be a problem. We must pass them
thru as-is w/o modification. This of course limits the choice
of backends...

I wonder who was that bright guy who wrote that in the beggining
and allowed people to play with case?! Damn.

Thanks for pointing this out!
Cheers,
Zoran

Re: [naviserver-devel] read/write ns_config

[Rick C. <rc...@Kn...>]

Potential objection to normalizing the case: ns_params for CGI
environment variables.  Environment variable names are case sensitive,
and it's perfectly possible that there are applications that use
lower-case names for these.  Would require changing nscgi to something
more like
Ns_param set {{variable_name} {value}}

Don't know if there are more like this. We have used a similar
configuration for setting java properties, though, and IIRC, at least
some of those required mixed-case names (ClassStuff.propertyName).

-- ReC

-----Original Message-----
From: nav...@li...
[mailto:nav...@li...] On Behalf Of
Mike
Sent: Wednesday, August 23, 2006 9:35 AM
To: nav...@li...
Subject: Re: [naviserver-devel] read/write ns_config

On 8/23/06, Zoran Vasiljevic <zv...@ar...> wrote:
>
> On 23.08.2006, at 17:20, Andrew Piskorski wrote:
>
> > That's only necessary to preserve the current ns_config
> > case-insensitivity, right?  Wouldn't it be easier to just
canonicalize
> > the key values on both write and read?  E.g., create a wrapper
around
> > nsv which forces all keys to all lowercase.  Then the key "Foo_Bar"
> > will be automatically translated to "foo_bar" in all situations.
>
> This is also a posibility I was thinking of. This would work, but then
> if you would want to generate a dump of the config later on, you would
> loose the case.
>
> If somebody made this:
>
>   ns_config ns/section/something
>   ns_param MyParam 1
>
> and if we would to re-create the setup
> in order to put it in the file again,
> the case would be lost, so the param
> will not be called "MyParam" but "MYPARAM"
> or "myparam" whatever case you selected.
>
> I do not know if this is/would-be an issue
> for anybody, but this is important to know.
>
> If you ask me: I would go with that and with
> the nsv as I believe this is most optimal in
> case of needed work and final result. But I
> must see that we do not shoot ourselves in
> the foot by some wrong design decision. Hence
> so much fuss.

Reading today's developments on this thread, I am glad to know that by
the time I got to the bottom others had the same thoughts as I.
Reusing nsv for the config makes most sense to me - removes code to
add functionality and flexibility.  This always seems like the correct
path.  Making the config file canonical isntead of the silly case
insensitivity also makes a lot of sense.

I do think a sanity check is in order when reading the config file -
while dealing with config file and collapsing the case, it makes sense
to check that no other case-version of the same key has been seen and
at least issue a warning.  This might be a bit too expensive to
include, but maybe a good idea.

I do have one concern with this change - what happens when the value
(where case must be preserved) for some key is actually the name of
another section or key?  I do not have enough experience to tell if
this is ever a problem, but I am not sure how this situation would be
handled.  Perhaps by the code that uses that config value?

------------------------------------------------------------------------
-
Using Tomcat but need to do more? Need to support web services,
security?
Get stuff done quickly with pre-integrated technology to make your job
easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache
Geronimo
http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=3D120709&bid=3D263057&dat=3D=
121642
_______________________________________________
naviserver-devel mailing list
nav...@li...
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] read/write ns_config

[Zoran V. <zv...@ar...>]

On 23.08.2006, at 18:23, Zoran Vasiljevic wrote:

>
> All measurements are done on my Mac PB 1.5GHZ powerpc.

WRONG! That was on an Linux box (SuSE9.1)

devlinux# cat /proc/cpuinfo
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 6
model           : 8
model name      : AMD Athlon(tm) XP 2200+
stepping        : 1
cpu MHz         : 1795.351
cache size      : 256 KB

Re: [naviserver-devel] read/write ns_config

[Zoran V. <zv...@ar...>]

On 23.08.2006, at 18:35, Mike wrote:

> I do have one concern with this change - what happens when the value
> (where case must be preserved) for some key is actually the name of
> another section or key?  I do not have enough experience to tell if
> this is ever a problem, but I am not sure how this situation would be
> handled.  Perhaps by the code that uses that config value?

Hm... this does not compute.

If we look at the how nsv's would replace the existing ns_set
as backend for storage: each ns_section would receive a separate
nsv_array, so to speak. In details, this would be handled on the
C instead of Tcl level, but it can be described this way for
simplicity.

Now, each nsv_array (i.e. ns_section) will hold any number
of ns_param's.

Where do you see the collision???
Values and names of sections and parameters are all entirely
separated in their own spaces. There is no collision I see there.

The only concern I have with changing (abandoning) case-insensitivity
is that many people's config out there would break. Hence we can't do
that. Idea to store the keys in canonical lowercase way would not hurt
anybody but MAY pose some unwanted side effects if we decide at any
point to serialize the in-memory structures back to files.  I do not
see this as a problem but we must be aware of that.

Cheers,
Zoran

Re: [naviserver-devel] read/write ns_config

[Mike <nee...@gm...>]

On 8/23/06, Zoran Vasiljevic <zv...@ar...> wrote:
>
> On 23.08.2006, at 17:20, Andrew Piskorski wrote:
>
> > That's only necessary to preserve the current ns_config
> > case-insensitivity, right?  Wouldn't it be easier to just canonicalize
> > the key values on both write and read?  E.g., create a wrapper around
> > nsv which forces all keys to all lowercase.  Then the key "Foo_Bar"
> > will be automatically translated to "foo_bar" in all situations.
>
> This is also a posibility I was thinking of. This would work, but then
> if you would want to generate a dump of the config later on, you would
> loose the case.
>
> If somebody made this:
>
>   ns_config ns/section/something
>   ns_param MyParam 1
>
> and if we would to re-create the setup
> in order to put it in the file again,
> the case would be lost, so the param
> will not be called "MyParam" but "MYPARAM"
> or "myparam" whatever case you selected.
>
> I do not know if this is/would-be an issue
> for anybody, but this is important to know.
>
> If you ask me: I would go with that and with
> the nsv as I believe this is most optimal in
> case of needed work and final result. But I
> must see that we do not shoot ourselves in
> the foot by some wrong design decision. Hence
> so much fuss.

Reading today's developments on this thread, I am glad to know that by
the time I got to the bottom others had the same thoughts as I.
Reusing nsv for the config makes most sense to me - removes code to
add functionality and flexibility.  This always seems like the correct
path.  Making the config file canonical isntead of the silly case
insensitivity also makes a lot of sense.

I do think a sanity check is in order when reading the config file -
while dealing with config file and collapsing the case, it makes sense
to check that no other case-version of the same key has been seen and
at least issue a warning.  This might be a bit too expensive to
include, but maybe a good idea.

I do have one concern with this change - what happens when the value
(where case must be preserved) for some key is actually the name of
another section or key?  I do not have enough experience to tell if
this is ever a problem, but I am not sure how this situation would be
handled.  Perhaps by the code that uses that config value?

Re: [naviserver-devel] read/write ns_config

[Zoran V. <zv...@ar...>]

On 23.08.2006, at 17:11, Vlad Seryakov wrote:

> In the case when ns_set has only few items, sequential search can be
> much faster than hash overhead, so i am not sure about this one

Most interesting:

lexxsrv:nscp 3> ns_set create
d0

lexxsrv:nscp 7> for {set i 0} {$i < 100} {incr i} {ns_set put d0 test 
$i test$i}

lexxsrv:nscp 10> time {ns_set find d0 test50} 1000
3.269 microseconds per iteration

lexxsrv:nscp 16>  time {ns_set find d0 test99} 1000
4.462 microseconds per iteration

lexxsrv:nscp 18> time {ns_set find d0 test1} 1000
2.093 microseconds per iteration

lexxsrv:nscp 11> for {set i 0} {$i < 100} {incr i} {set xx(test$i)  
test$i}

lexxsrv:nscp 21> time {set xx(test1)} 1000
1.527 microseconds per iteration

lexxsrv:nscp 25>  time {set xx(test50)} 1000
1.628 microseconds per iteration

lexxsrv:nscp 23>  time {set xx(test99)} 1000
1.628 microseconds per iteration

As you see, the ns_set is *way* inferior to
hash tables. Allright, the point is perhaps
that the "set" command is part of the Tcl byte
code so it is inherently faster that ns_set.
But, even then, a moderate set of 100 elements
needs about 3 microseconds to find the middle key
whereas the hash-table needs about 2 msec flat
to locate ANY key.

This all means that even for moderate-sized
sets it pays of to speed them up for searching.
(we'd have to accept some speed penalty for
adding/updating/deleting sets allright, but
nothing is for free).

All measurements are done on my Mac PB 1.5GHZ powerpc.

Cheers,
Zoran

Re: [naviserver-devel] read/write ns_config

[Zoran V. <zv...@ar...>]

On 23.08.2006, at 17:20, Andrew Piskorski wrote:

> That's only necessary to preserve the current ns_config
> case-insensitivity, right?  Wouldn't it be easier to just canonicalize
> the key values on both write and read?  E.g., create a wrapper around
> nsv which forces all keys to all lowercase.  Then the key "Foo_Bar"
> will be automatically translated to "foo_bar" in all situations.

This is also a posibility I was thinking of. This would work, but then
if you would want to generate a dump of the config later on, you would
loose the case.

If somebody made this:

  ns_config ns/section/something
  ns_param MyParam 1

and if we would to re-create the setup
in order to put it in the file again,
the case would be lost, so the param
will not be called "MyParam" but "MYPARAM"
or "myparam" whatever case you selected.

I do not know if this is/would-be an issue
for anybody, but this is important to know.

If you ask me: I would go with that and with
the nsv as I believe this is most optimal in
case of needed work and final result. But I
must see that we do not shoot ourselves in
the foot by some wrong design decision. Hence
so much fuss.

Cheers
Zoran

Re: [naviserver-devel] read/write ns_config

[Andrew P. <at...@pi...>]

On Wed, Aug 23, 2006 at 05:05:55PM +0200, Zoran Vasiljevic wrote:

> I'm interesed to find out what do you think about
> speeding up the set operation by a help hash-table?
> This would minimize the time the lock is held and
> would therefore lower the impact of locking altogether.

That's only necessary to preserve the current ns_config
case-insensitivity, right?  Wouldn't it be easier to just canonicalize
the key values on both write and read?  E.g., create a wrapper around
nsv which forces all keys to all lowercase.  Then the key "Foo_Bar"
will be automatically translated to "foo_bar" in all situations.

-- 
Andrew Piskorski <at...@pi...>
http://www.piskorski.com/

Re: [naviserver-devel] read/write ns_config

[Zoran V. <zv...@ar...>]

On 23.08.2006, at 17:11, Vlad Seryakov wrote:

>
> In the case when ns_set has only few items, sequential search can be
> much faster than hash overhead, so i am not sure about this one

Yes. This is true. But how would you know?
Say you have 100 element set and the thing
you are looking for is on the 1 place?

Obviously, the sequential search is instant.

But if the thing is at the end, then hash
*could* be faster.

The real problem is: set with a handful
of items (2-5). There the sequential search would
be almost always faster.

At some point (depending on the size of the
set), a hash lookup could pay off. the problem
is how to locate that point!

OTOH, how MUCH is the hash lookup slower for
a say, 10 or 20 elements set if, for example,
the thing is sitting in the middle? I believe
this is the question to answer... Perhaps
I should write some test code...

Cheers
Zoran

Re: [naviserver-devel] read/write ns_config

[Vlad S. <vl...@cr...>]

Agree on up-to-the clinet policy of config usage


> I'm interesed to find out what do you think about
> speeding up the set operation by a help hash-table?
> This would minimize the time the lock is held and
> would therefore lower the impact of locking altogether.

In the case when ns_set has only few items, sequential search can be 
much faster than hash overhead, so i am not sure about this one

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] read/write ns_config

[Zoran V. <zv...@ar...>]

On 23.08.2006, at 16:55, Vlad Seryakov wrote:

> Then we can just go through the code(fastpath, log) and change it  
> so it
> copies config at startup into local variables (as in server.c) if that
> particular code does not care about realtime changes. This way it will
> be a compromise between locking and performance.

I believe this is possible.

The config user (in that case the fastpath or log code)
should actually decide what is more important: reaction
to config changes or absolute avoidance of locking
wherever possible.

In some cases, reaction to config-chage is pretty
difficult, if possble at all! In such cases, getting
the config value once and never asking for it again
is quite natural.

In some cases (like Stephens example with log) it is
actually desireable to react to changes instantly.
Actually, here the locking would be unavoidable.
But, even then, in this particular case, we have
ns_logctl which would for example be used to query
the config and update private log data once and then
have other log calls examine that data w/o locking.

I'm interesed to find out what do you think about
speeding up the set operation by a help hash-table?
This would minimize the time the lock is held and
would therefore lower the impact of locking altogether.


Cheers
Zoran

Re: [naviserver-devel] read/write ns_config

[Vlad S. <vl...@cr...>]

Then we can just go through the code(fastpath, log) and change it so it 
copies config at startup into local variables (as in server.c) if that 
particular code does not care about realtime changes. This way it will 
be a compromise between locking and performance.

Zoran Vasiljevic wrote:
> On 23.08.2006, at 16:17, Vlad Seryakov wrote:
> 
>> Looking thru the code i see that mostly only fastpath and log uses
>> NsConfig reads in real time, all others are copy values into local
>> structures. funny, but these who uses config are performance critical
>> and will introduce lock contention because thet are figthing for the
>> same config values.
>>
>> Another raw/uncooked idea: is it possible to keep changed values
>> separately, no lock default config as is now and whoever changed value
>> just keep it in the lock-guarded hash. If i care about locking and
>> changed value i will check new hash for new value, others who does not
>> may still use read-only config list.
> 
> I do not know how this could work. If thread A changes
>    ns/some/thing
> and thread B is constantly looking up the
>    ns/some/thing
> then the B must always lock to get the value, as A may
> be fiddling with it at the same time.
> 
> Thread B can of course copy the value in its
> private (tsd) copy on the first fetch but
> then again, HOW is he going to find out that the
> thing was changed by the A in the meantime?
> 
> I do not believe you can avoid locking if you want
> to have consistent behaviour.
> 
> Also, the changes done by thread A would be pretty
> meaningless if only A is able to see them...
> I do not believe this is worth examining.
> 
> As said, locking cannot be avoided. It is just the question
> of: can we tolerate it or not? I can. Therefore for me it
> is irelevant. But I do not if you can. Therefore I'm looking
> to see if we can do this as runtime or compile-time option.
> Also, because the locking is needed, we have to see how we
> can minimize it, also minimizing time spent in the locked
> sections of the code (by speeding up certain operations for
> example).
> 
> Cheers
> Zoran
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] read/write ns_config

[Zoran V. <zv...@ar...>]

On 23.08.2006, at 16:17, Vlad Seryakov wrote:

> Looking thru the code i see that mostly only fastpath and log uses
> NsConfig reads in real time, all others are copy values into local
> structures. funny, but these who uses config are performance critical
> and will introduce lock contention because thet are figthing for the
> same config values.
>
> Another raw/uncooked idea: is it possible to keep changed values
> separately, no lock default config as is now and whoever changed value
> just keep it in the lock-guarded hash. If i care about locking and
> changed value i will check new hash for new value, others who does not
> may still use read-only config list.

I do not know how this could work. If thread A changes
   ns/some/thing
and thread B is constantly looking up the
   ns/some/thing
then the B must always lock to get the value, as A may
be fiddling with it at the same time.

Thread B can of course copy the value in its
private (tsd) copy on the first fetch but
then again, HOW is he going to find out that the
thing was changed by the A in the meantime?

I do not believe you can avoid locking if you want
to have consistent behaviour.

Also, the changes done by thread A would be pretty
meaningless if only A is able to see them...
I do not believe this is worth examining.

As said, locking cannot be avoided. It is just the question
of: can we tolerate it or not? I can. Therefore for me it
is irelevant. But I do not if you can. Therefore I'm looking
to see if we can do this as runtime or compile-time option.
Also, because the locking is needed, we have to see how we
can minimize it, also minimizing time spent in the locked
sections of the code (by speeding up certain operations for
example).

Cheers
Zoran

Re: [naviserver-devel] read/write ns_config

[Zoran V. <zv...@ar...>]

On 23.08.2006, at 14:46, Zoran Vasiljevic wrote:

> Why wouldn't we making nsv interface
> public and use it for configuration store?

Eh... there is always yet-another gotcha!

Because... the nature of the config interface
is to allow case-insensitive or case sensitive
lookups. Hence, sets are used in the current
implementation. Peh!

But, sets are SLOW. Because every time you need
to get one "thing" out of the set, a linear
lookup with a custom compare function is performed!

But what if we expand the set to contain a small
helping hash table? The hash table would cache the
results of lookup so the very next time the same
"thing" is looked up, the result would be delivered
out of the cache? This may all be lazily computed so
it would not bloat the memory... Hmhmhmhmhmhm...

Perhaps this is more optimal solution? It would
require very minor changes to the set implementation,
would have a side effect of speeding up set operation
(at least the lookup) generally and would allow for
relatively fast lookup for a given "thing" (in this
case config parameter).  This will allow us to hold
the global lock for duration of the entire operation
on the set and implement read/write configuration with
less work.

The locking: we can employ the same locking mechanism
as in nsv: compute a hash of the section name and map
it to fixed number of buckets, each holding a lock.

So effectively, we have numerous options:

   a. stay with sets but improve set lookup speed
   b. use nsv-code (would need to ditch case-insensitivity)
   c. use DOM based tree

For a. and c. the locking contention can be minimized
by splitting the entire "thing" into fixed number of
buckets as in the nsv (the b.) option...

Also, the complexity of implementation is a->b->c
with a. being most simple.

Now, what do you think about that?
I hope I'm not overloading you with too much information!

Cheers
Zoran

Re: [naviserver-devel] read/write ns_config

[Vlad S. <vl...@cr...>]

Looking thru the code i see that mostly only fastpath and log uses 
NsConfig reads in real time, all others are copy values into local 
structures. funny, but these who uses config are performance critical 
and will introduce lock contention because thet are figthing for the 
same config values.

Another raw/uncooked idea: is it possible to keep changed values 
separately, no lock default config as is now and whoever changed value 
just keep it in the lock-guarded hash. If i care about locking and 
changed value i will check new hash for new value, others who does not 
may still use read-only config list.

Zoran Vasiljevic wrote:
> On 23.08.2006, at 13:01, Zoran Vasiljevic wrote:
> 
>> Alternatively, the DOM tree can be replaced by a combination
>> of hash-tables and C-structures. Each ns_section will
>> effectively be a hash table of parameters with a parameter
>> being a simple C-struct. Also, there will be one global hashtable
>> needed to resolve ns_section names to hash-tables holding parameters.
>> I doubt however that this would be any faster then a good DOM
>> implementation.
> 
> 
> I see... this is more-less the nsv as we have it today.
> 
> Why wouldn't we making nsv interface
> public and use it for configuration store?
> After all, the code is there and is well
> tested. We might need to tweak some pieces
> but, in principle, this is it.
> 
> I would need to refresh my view of nsv locking
> but generally it uses fixed number of buckets
> to avoid everybody locking the same global mutex.
> Increasing number of buckets lowers the contention.
> 
> The only part we need to think about is actual
> parameter storage as this will not be a plain
> value, but also some housekeeping info like type,
> ranges etc, in addition.
> 
> Given relatively fast lookup and relatively cheap
> locking, we would not need to have private and shared
> copies. I believe just one shared copy would be
> perfectly OK.
> 
> Hm? What do you mean?
> 
> Cheors
> Zoran
> 
> 
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] read/write ns_config

[Zoran V. <zv...@ar...>]

On 23.08.2006, at 13:01, Zoran Vasiljevic wrote:

> Alternatively, the DOM tree can be replaced by a combination
> of hash-tables and C-structures. Each ns_section will
> effectively be a hash table of parameters with a parameter
> being a simple C-struct. Also, there will be one global hashtable
> needed to resolve ns_section names to hash-tables holding parameters.
> I doubt however that this would be any faster then a good DOM
> implementation.


I see... this is more-less the nsv as we have it today.

Why wouldn't we making nsv interface
public and use it for configuration store?
After all, the code is there and is well
tested. We might need to tweak some pieces
but, in principle, this is it.

I would need to refresh my view of nsv locking
but generally it uses fixed number of buckets
to avoid everybody locking the same global mutex.
Increasing number of buckets lowers the contention.

The only part we need to think about is actual
parameter storage as this will not be a plain
value, but also some housekeeping info like type,
ranges etc, in addition.

Given relatively fast lookup and relatively cheap
locking, we would not need to have private and shared
copies. I believe just one shared copy would be
perfectly OK.

Hm? What do you mean?

Cheors
Zoran