Menu
▾
▴
naviserver-devel
|
From: Zoran V. <zv...@ar...> - 2006-08-22 09:58:53
|
Hi! This one stands very long on my list but I'm not sure if everybody would be pleased... Idea is to allow ns_config to actually write the config, instead of only reading it. Without going into details how this would look like on the Tcl level, I just wanted to poke what you think about the underlying implemnentaion, as the current one would have to be changed i.e. locks added. The thing I like about ns_config is that it has pretty nice tree-like structured repository of config options which you can peek from C and Tcl level. Unfortunately this works only during the startup i.e. things can be added to the config only at startup and later on you can only read it. What I would need is the ability to write new and/or change the existing values. Cheers zoran |
|
From: Bernd E. <eid...@we...> - 2006-08-22 10:53:07
|
Hi Zoran, > Idea is to allow ns_config to actually write the > config, instead of only reading it. [...] >Unfortunately >this works only during the startup i.e. things can be >added to the config only at startup and later on you can >only read it. What I would need is the ability to write >new and/or change the existing values. Sounds good! My only suggestion would be to be somehow able to let the current behaviour (read-only-access) be configurable or default; read-only-access in some situations is a means of security - in others of course it is more or less useless, as one can drop or at least delete the database anyway. A change of an option should also be worth a "ns_log notice" in that respect. If, as a side-effect of your work, you will find it easy to implement a way to let ns_config return all a) non-core/non-default/non-c-module-specific b) not-belonging-to-this-section configuration options, I'll ask you to implement that, too! :-) Cheerio! Bernd. |
|
From: Zoran V. <zv...@ar...> - 2006-08-22 11:00:47
|
On 22.08.2006, at 12:56, Bernd Eidenschink wrote: > > My only suggestion would be to be somehow able to let the current > behaviour > (read-only-access) be configurable or default; read-only-access in > some > situations is a means of security - in others of course it is more > or less > useless, as one can drop or at least delete the database anyway. Yes. Actually, security on one and performance on the other side are major reasons against such a change. But in some cases, as in ours, neither of those really apply. Yet we could benefit a lot from already existing code which is both Tcl and C-level accesible. I do not know if it will be possible to do select this as runtime option w/o performance (locking) penalty. I may need to make it a compile-time option. But I haven't started on that yet, so I can't really say which way it will go, if at all. > > A change of an option should also be worth a "ns_log notice" in > that respect. > > If, as a side-effect of your work, you will find it easy to > implement a way to > let ns_config return all > a) non-core/non-default/non-c-module-specific you mean only those set from the "outside" i.e. config file? > b) not-belonging-to-this-section give me one example about what you mean. Cheers Zoran |
|
From: Bernd E. <eid...@we...> - 2006-08-22 12:00:26
|
> > If, as a side-effect of your work, you will find it easy to > > implement a way to > > let ns_config return all > > a) non-core/non-default/non-c-module-specific > > you mean only those set from the "outside" i.e. config file? Basically a way to detect a typo of the ns_param key. Something that _could_ lead to a basic configuration check. Now a typo is just a missing parameter, the server sets a default value and you'll likely not notice it. Currently the knowledge about configuration options and their defaults is burried in the source, the setting of defaults is done during startup so a check without starting a server may be impossible; beside that, collecting "default" keys and their values during startup should be possible - but the identification of unknown keys would only be possible when no more config options are read/set (see next). > > b) not-belonging-to-this-section > > give me one example about what you mean. Similar to the one above, but it would be the result of an extra lookup: does the ns_param key (of which we think it is a typo) in fact belong to another section? It could currently only be done at a stage where we know all config options are set up. ns_section ns/parameters # a fastpath entry in fact belonging to ns/server/$server/fastpath: ns_param cachemaxentry 8192 Hm. The lookup may result in n-results, as some key-names are used in multiple sections. "Unkown config option x in section y" "Config option x not known in section y, check section[s] z, z' or z''" Frankly, it is probably not worth to keep an eye on that... Bernd. |
|
From: Zoran V. <zv...@ar...> - 2006-08-22 12:13:12
|
On 22.08.2006, at 14:03, Bernd Eidenschink wrote:
>>> If, as a side-effect of your work, you will find it easy to
>>> implement a way to
>>> let ns_config return all
>>> a) non-core/non-default/non-c-module-specific
>>
>> you mean only those set from the "outside" i.e. config file?
>
> Basically a way to detect a typo of the ns_param key. Something
> that _could_
> lead to a basic configuration check.
> Now a typo is just a missing parameter, the server sets a default
> value and
> you'll likely not notice it.
> Currently the knowledge about configuration options and their
> defaults is
> burried in the source, the setting of defaults is done during
> startup so a
> check without starting a server may be impossible;
>
> beside that, collecting "default" keys and their values during
> startup should
> be possible - but the identification of unknown keys would only be
> possible
> when no more config options are read/set (see next).
I would say: this is pretty impossible! Nobody imposes any known set of
config options to the underlying implemntation so the implementation
cannot
do any "sanity" checks beside basic type-checking.
You can pretty much say:
ns_section milly/vanilly
ns_param fate 1
and the ns_config will swallow this peacefull, althought you really
wanted
to say:
ns_param fake 1
If we could define all possbile options of "milly/vanilly" section
somehow, then it would be possible to do checks like that.
Hm... perhaps this is not that bad idea at all. What about:
ns_section name ?args?
where args would give a list of all supported parameters for that
section? Then the implementation could easily complain if you
misspelled the parameter. But this leaves you with abother trouble...
What if you say:
ns_section mily/vanilly
(note one "el" missing in section name) then you are also out of luck.
To cover that, you would need to be able to give all sections in advance
which is, of course pretty weird. Given the amount of work to do that
*properly* I'd rather opt with the basic type-checking we have now.
>
>>> b) not-belonging-to-this-section
>>
>> give me one example about what you mean.
>
> Similar to the one above, but it would be the result of an extra
> lookup: does
> the ns_param key (of which we think it is a typo) in fact belong to
> another
> section?
> It could currently only be done at a stage where we know all config
> options
> are set up.
>
> ns_section ns/parameters
> # a fastpath entry in fact belonging to ns/server/$server/fastpath:
> ns_param cachemaxentry 8192
>
> Hm. The lookup may result in n-results, as some key-names are used
> in multiple
> sections.
>
> "Unkown config option x in section y"
> "Config option x not known in section y, check section[s] z, z' or
> z''"
>
> Frankly, it is probably not worth to keep an eye on that...
Exactly.
Cheers
Zoran
|
|
From: Bernd E. <eid...@we...> - 2006-08-22 12:40:38
|
> I would say: this is pretty impossible! Nobody imposes any known set of
> config options to the underlying implemntation so the implementation
> cannot
> do any "sanity" checks beside basic type-checking.
> You can pretty much say:
>
> ns_section milly/vanilly
> ns_param fate 1
But the point is: At a certain stage the server has finished reading all
configuration options. You would be able to detect that
ns_section milly/vanilly
is a section you have no knowledge of or no use for:
"unknown config section"
This is of course only true when it is really not used, e.g. at a later point
in time by an "external" TCL module.
If, during startup, a configuration value is requested for the first time, the
value OR the default value is set. This knowledge is buried in the C call:
nsconf.shutdowntimeout =
Ns_ConfigIntRange(path, "shutdowntimeout", 20, 0, INT_MAX);
A list of all options, btw., is created then the server logs debug notices.
As it is possible to list all options of all sections (from C and TCL), at the
end I can detect config options that I did not request and set during
startup.
Therefor I can lookup if the options belong to other sections or if the
options are unknown in a section or totally unknown.
This should be possible! The question is, if it makes sense, as only options
are detected that are requested and set during startup and TCL module ones
will be missing.
Bernd.
|
|
From: Zoran V. <zv...@ar...> - 2006-08-22 12:53:30
|
On 22.08.2006, at 14:44, Bernd Eidenschink wrote: > As it is possible to list all options of all sections (from C and > TCL), at the > end I can detect config options that I did not request and set during > startup. > Therefor I can lookup if the options belong to other sections or if > the > options are unknown in a section or totally unknown. If something can be done in couple of Tcl lines and if this is not something you'd do 1000.000 times a day in a tight loop I'd leave it on the Tcl level. Mingling with C-code is about 10-times the work you'd invest for Tcl and it has a side-effect of gross destabilization as every single/small error is fatal. Therefore, I could/would do this in C only if it is really issue of speed (you need light-fast code) or data-fiddling (can be done with Tcl but is dead-inneficient and unelegant) or lack of Tcl interface. If you say you can collect all sections and all params with Tcl and do your cross-checking, why bother? Cheers Zoran |
|
From: Bernd E. <b.e...@ki...> - 2006-08-22 13:23:12
|
> If you say you can collect all sections and all params with > Tcl and do your cross-checking, why bother? Because only during startup we have a chance to gather a knowledge of "core" configuration options via the Ns_Config* C-command calls. Only at a stage somewhere at the end of the startup process I can identify options belonging to sections I did not Ns_Config-request and therefor assume that a particular option is a) "unknown" aka. not belonging to a section or b) "unknown" in section x but was Ns_Config-requested elsewhere, where it was not found and the default value was used The C-level-Ns_Config-calls-during-startup here are my "authoritative" guides to build a list of configuration options. Later on in TCL I have to accept milly/vanilly or any other part in the soup unless I create a mandatory list from the source code by hand. This is the way now for everyone of us to identify a) parameters that are not used any more! b) default values! c) even options you never heard of! Bernd. |
|
From: Zoran V. <zv...@ar...> - 2006-08-22 13:41:38
|
On 22.08.2006, at 15:26, Bernd Eidenschink wrote: > > Because only during startup we have a chance to gather a knowledge > of "core" > configuration options via the Ns_Config* C-command calls. I might be stupid. (A valid explanation why) I still do not understand what you need? The config machinery is rather dumb as it (only) has a notion of the ns_section, one or more ns_param's in it and some basic type checking on ns_params. Not more. There are no "hardwired" or "default" or C-level or Tcl-level sections or parameters. All of them are created equal. The config repository is built during the startup and later-on never changed (current behaviour) I'm thinking about allowing *changes* to/of this repository during the runtime. This way one could override a ns_param value assigned at startup or create entirely new ns_section and stuff it with bunch of ns_param so others may call [ns_config] to get it. (Obviously, changing some of the parameters may have no sense as they will be read only during the startup but this is not important at this time). Now, what would you need EXACTLY, in addition to that? Can you describe this shortly and possibly give one example so that I can understand it as well (I'm getting old, you know...) Cheers Zoran |
|
From: Rick C. <rc...@Kn...> - 2006-08-22 14:46:22
|
Uh, and I may be missing something about how you're thinking of doing this, but how would you handle open configuration sets like the environment variables for the CGI module, or the map of mime-type/file extensions? Thanks -- -- ReC -----Original Message----- From: nav...@li... [mailto:nav...@li...] On Behalf Of Zoran Vasiljevic Sent: Tuesday, August 22, 2006 6:41 AM To: nav...@li... Subject: Re: [naviserver-devel] read/write ns_config On 22.08.2006, at 15:26, Bernd Eidenschink wrote: > > Because only during startup we have a chance to gather a knowledge =20 > of "core" > configuration options via the Ns_Config* C-command calls. I might be stupid. (A valid explanation why) I still do not understand what you need? The config machinery is rather dumb as it (only) has a notion of the ns_section, one or more ns_param's in it and some basic type checking on ns_params. Not more. There are no "hardwired" or "default" or C-level or Tcl-level sections or parameters. All of them are created equal. The config repository is built during the startup and later-on never changed (current behaviour) I'm thinking about allowing *changes* to/of this repository during the runtime. This way one could override a ns_param value assigned at startup or create entirely new ns_section and stuff it with bunch of ns_param so others may call [ns_config] to get it. (Obviously, changing some of the parameters may have no sense as they will be read only during the startup but this is not important at this time). Now, what would you need EXACTLY, in addition to that? Can you describe this shortly and possibly give one example so that I can understand it as well (I'm getting old, you know...) Cheers Zoran ------------------------------------------------------------------------ - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=3D120709&bid=3D263057&dat=3D= 121642 _______________________________________________ naviserver-devel mailing list nav...@li... https://lists.sourceforge.net/lists/listinfo/naviserver-devel |
|
From: Zoran V. <zv...@ar...> - 2006-08-22 15:02:38
|
On 22.08.2006, at 16:46, Rick Cobb wrote: > Uh, and I may be missing something about how you're thinking of doing > this, but how would you handle open configuration sets like the > environment variables for the CGI module, or the map of mime-type/file > extensions? ?? ( == do not understand) What do you mean by "open configuration sets" ?? Using ns_config, nothing is "open". It is just a query into a in-memory data structure. The fact that this query now uses no locks, and is thus "cheap", modifying it to use locks is one issue (and you need locks at that place because somebody could read and other could modify the same structure). Another issue is: some module already "consumed" the parameter at startup and will never ask for it again: what is the point in changing that particular parameter? Also, should this all be allowed anyway (security issue)? This is what I see as things to take in account. For the locking issue: I believe this is not that important as it can be easily "fixed" by storing the value got from config in a private variable and then reusing this for the time being. For the security issue: well, there might be a config option to allow this, either as runtime or compile-time option (the latter will solve any locking issues implicitly) For the sense of changing a once-read config value? Well, there isn't (sense). But that has nothing to do with the implementation. Cheers Zoran |
|
From: Rick C. <rc...@Kn...> - 2006-08-22 16:31:14
|
My apologies. I was actually responding more to my perception of Bernd's idea than to the feature of allowing writability of the ns_config system in general. My concern with the idea of trying to identify typos or other problems in configuration is that there are a number of ns_param sets which aren't a set of fixed names. In the AOLserver world, the ns_section for "cgi", for example, allows you to set an ns_param with any name and any value, none of which are verified by the C module, all of which are just copied to the environment array for the cgi. Now, looking @ Bernd's idea, I see, perhaps, a bit where he's going: if, for example, the ns_param dictionary had a bit saying "I've been read by a module" (no matter which language the module was in), then at server *shutdown*, you could print a report of unused ns_params and ns_sections. But really, you can't tell before then (as you've pointed out, Zoran). I'll note that in the configuration system for LiveServer, we used our publish/subscribe facilities (the stuff we sell) to handle the "write back" issues you're solving. We allow coders to register a (C-only, at present) callback for their variables, so if they change, they're informed and can immediately change behavior. We also actually write the physical configuration file; if no callback was provided, we inform the administrators (presumably one of whom is making the change) that the server needs to be restarted before their change will take effect. To do this, we use a completely separate layer of TCL variables, procs, and sourced TCL files to construct ns_section/ns_param calls. It's not elegant, and doesn't fit the real NaviServer/AOLServer world very well, but we've found it quite useful. -- ReC -----Original Message----- From: nav...@li... [mailto:nav...@li...] On Behalf Of Zoran Vasiljevic Sent: Tuesday, August 22, 2006 8:03 AM To: nav...@li... Subject: Re: [naviserver-devel] read/write ns_config On 22.08.2006, at 16:46, Rick Cobb wrote: > Uh, and I may be missing something about how you're thinking of doing > this, but how would you handle open configuration sets like the > environment variables for the CGI module, or the map of mime-type/file > extensions? ?? ( =3D=3D do not understand) What do you mean by "open configuration sets" ?? Using ns_config, nothing is "open". It is just a query into a in-memory data structure. The fact that this query now uses no locks, and is thus "cheap", modifying it to use locks is one issue (and you need locks at that place because somebody could read and other could modify the same structure). Another issue is: some module already "consumed" the parameter at startup and will never ask for it again: what is the point in changing that particular parameter? Also, should this all be allowed anyway (security issue)? This is what I see as things to take in account. For the locking issue: I believe this is not that important as it can be easily "fixed" by storing the value got from config in a private variable and then reusing this for the time being. For the security issue: well, there might be a config option to allow this, either as runtime or compile-time option (the latter will solve any locking issues implicitly) For the sense of changing a once-read config value? Well, there isn't (sense). But that has nothing to do with the implementation. Cheers Zoran ------------------------------------------------------------------------ - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=3D120709&bid=3D263057&dat=3D= 121642 _______________________________________________ naviserver-devel mailing list nav...@li... https://lists.sourceforge.net/lists/listinfo/naviserver-devel |
|
From: Zoran V. <zv...@ar...> - 2006-08-22 17:05:19
|
On 22.08.2006, at 18:31, Rick Cobb wrote: > > Now, looking @ Bernd's idea, I see, perhaps, a bit where he's > going: if, > for example, the ns_param dictionary had a bit saying "I've been > read by > a module" (no matter which language the module was in), then at server > *shutdown*, you could print a report of unused ns_params and > ns_sections. But really, you can't tell before then (as you've > pointed > out, Zoran). This is true. If we had a counter on an parameter, we could dump the whole config tree where counter == 0. This would probably mean that those have been set by somebody, but never used. Although I can understand this, somebody has to show me any practical use of it. You can catch typos with that, allright but this is at the wrong side of the street! I would expect the server to refuse booting in such case. Or, emit a big fat error log during startup. Not at shutdown. > > I'll note that in the configuration system for LiveServer, we used our > publish/subscribe facilities (the stuff we sell) to handle the "write > back" issues you're solving. We allow coders to register a (C- > only, at > present) callback for their variables, so if they change, they're > informed and can immediately change behavior. We also actually write > the physical configuration file; if no callback was provided, we > inform > the administrators (presumably one of whom is making the change) that > the server needs to be restarted before their change will take effect. > > To do this, we use a completely separate layer of TCL variables, > procs, > and sourced TCL files to construct ns_section/ns_param calls. It's > not > elegant, and doesn't fit the real NaviServer/AOLServer world very > well, > but we've found it quite useful. You know, I have this "requirement" is sitting for about 4 years in my todo list. I was not very keen on getting it out as I know that it might bear some security (more important) and locking (less important) issues. But we've come to a point where we either have to abandon the whole ns_config stuff and write everything ourselves OR to get this modification in the core server. Needles to say that I hate reinventing the wheel over and over again. Hence the decision to try to get it done with existing ns_config code. Basically, most important information is: from the two (are there more?) issues (locking and security) which of those two is more important/sensitive/whatever to everybody. So I know where I will have to make tradeoffs. Basically, all this comes down to: a. should I make everything compile-time option so only people who need it would compile-in that code, everybody else being not affected or b. should I make it runtime selectable, because people would like to have both, depending on the site/application or c. should I make it read/write entirely, basically ignoring both locking and security issues. The simplest is of course c. The a. is next, with b. being more/most complex. I would of course not want to invest much work with that and both security/locking are no problem for me, so the logical choice for me would be c. But I do not know if everybody is ok with that. Hence all this email ping-pong. Cheers Zoran |
|
From: Stephen D. <sd...@gm...> - 2006-08-22 20:13:07
|
On 8/22/06, Zoran Vasiljevic <zv...@ar...> wrote: > > You know, I have this "requirement" is sitting for about 4 years in my > todo list. I was not very keen on getting it out as I know that it > might bear some security (more important) and locking (less important) > issues. But we've come to a point where we either have to abandon > the whole ns_config stuff and write everything ourselves OR to > get this modification in the core server. Needles to say that I > hate reinventing the wheel over and over again. Hence the decision > to try to get it done with existing ns_config code. > > Basically, most important information is: from the two (are there > more?) > issues (locking and security) which of those two is more > important/sensitive/whatever to everybody. So I know where I will > have to make tradeoffs. > > Basically, all this comes down to: > > a. > should I make everything compile-time option so only people who need > it would compile-in that code, everybody else being not affected > > or > > b. > should I make it runtime selectable, because people would like to > have both, depending on the site/application > > or > > c. > should I make it read/write entirely, basically ignoring both > locking and security issues. > > The simplest is of course c. The a. is next, with b. being more/most > complex. I would of course not want to invest much work with that and > both security/locking are no problem for me, so the logical choice > for me would be c. But I do not know if everybody is ok with that. > Hence all this email ping-pong. > I think the C code does the right thing: create a struct to hold the config values and use the Ns_Config* routines to populate it at startup. Use the custom struct directly in code. The Tcl level ns_config is kinda broken. *Every* time you call it the code does a linear, case insensitive search for the right set; then it does a linear case insensitive search for the correct key in that set; then it parses the string value and converts it into an int or bool or whatever. ArsDigita figured out this was hurting them back in '99 and created a wrapper which on first access put the config value in an nsv array. And *that* is kinda broken as now you suffer locking overhead on hundreds of config values which are essentially static. So instead you call ns_config at startup and put the result in a name spaced Tcl variable. And *that* is kinda broken as the only reason it works is an accident of implementation -- no one could figure out how to clean out name spaced variables between requests as efficiently as global ones. Anyway, I don't see how you're going to do the locking. Do you have one global lock around the whole config? A lock per section? Do you free updated config values/sets, or do you leak them? The idea is good -- don't reboot the server when config values change. Looks hard to me as there are different locking requirements for different pieces of code. For example, the ns_log code can now be updated at runtime! You turn on/off debug logging, maybe to isolate a specific problem. It's easy, as updating the boolean value is atomic, and even if it's not it doesn't matter as it's just a boolean value. But even there you call ns_logctl -- the underlying config isn't changed. If every time Ns_Log() was called it in turn called Ns_Conf* to figure out whether debug logging was on or off, performance would be unacceptably slow. Yes, the Tcl code at the moment *does* scan the sets every time (unless it doesn't because you hacked around it), but it probably shouldn't. It should also be fast. How do you handle notifying the code which has parsed and stashed away a copy of the config value that the config has been re-written? I don't see how you will handle the locking... :-( |
|
From: Zoran V. <zv...@ar...> - 2006-08-22 20:52:28
|
On 22.08.2006, at 22:13, Stephen Deasey wrote: > > I don't see how you will handle the locking... :-( Me neiter. I haven't looked at the beast in detail yet. What I cannot imagine, though, is that there is no way to have a propoerly locked config depository. Let me have a look there and see how this could be done the best. Cheers Zoran |
|
From: Zoran V. <zv...@ar...> - 2006-08-23 11:01:39
|
On 22.08.2006, at 22:13, Stephen Deasey wrote:
>
> I don't see how you will handle the locking... :-(
I imagine the entire configuration tree being implemented
as a global master (in-memory) tree, close to the DOM.
Actually, it can be really a DOM structure (see below for
the alternative).
I'm assuming that reads (queries into the) config tree will
be a majority of operations whereas writes will be occasional.
I expect this to be far more than 80/20 ratio in favour
of readers vs. writers.
Every thread will have a copy (in TSD) of the master tree
structure and will use it to read config parameters from.
The master tree structure will have a single global
mutex and associated epoch counter.
A thread wanting to read some config param will have
to lock the mutex, examine the epoch and compare it with
the epoch of the copy of the tree it's holding in the TSD.
If there is a match, the mutex is released.
If there is no match, the entire private copy is trashed,
new copy is made and the mutex is released.
In any case, the result is returned from the *local* copy.
A thread wanting to change a parameter will lock the mutex,
bump the epoch, change the parameter in the master copy,
then copy the entire master DOM to its private storage and
release the mutex.
Having the configuration storage in tree means that we can
easily serialize it to XML and store it as a plain file
on the filesystem. This can be done either:
never
at server shutdown (or some signal perhaps)
periodically over some scheduled procedure
Alternatively, one can attach some kind of permanent
persistence (a (g)dbm file for example) to the tree
so every modification is immediately saved.
Due to the assumption that readers will be far more
frequent than writers, so the tree copying between
the master and private copy will be very rare, all we
have to "accept" would be a short lock on a global mutex
in order to examine the epoch counter and decide wether
our local copy is up-to-date or not.
Alternatively, the DOM tree can be replaced by a combination
of hash-tables and C-structures. Each ns_section will
effectively be a hash table of parameters with a parameter
being a simple C-struct. Also, there will be one global hashtable
needed to resolve ns_section names to hash-tables holding parameters.
I doubt however that this would be any faster then a good DOM
implementation.
This is not a "piece of cake" project hence I would like
to hear some comments before proceeding.
Cheers
Zoran
|
|
From: Zoran V. <zv...@ar...> - 2006-08-23 11:07:01
|
On 23.08.2006, at 13:01, Zoran Vasiljevic wrote: > > The master tree structure will have a single global > mutex and associated epoch counter. To reduce contention on that mutex, one can compute a simple checksum of the section name and use this to address fixed number of buckets, each holding one mutex and the epoch counter. The number of buckets can be tuned to fight the contention, if needed. This technique is already used in nsv implementation. Cheers Zoran |
|
From: Zoran V. <zv...@ar...> - 2006-08-23 12:46:55
|
On 23.08.2006, at 13:01, Zoran Vasiljevic wrote: > Alternatively, the DOM tree can be replaced by a combination > of hash-tables and C-structures. Each ns_section will > effectively be a hash table of parameters with a parameter > being a simple C-struct. Also, there will be one global hashtable > needed to resolve ns_section names to hash-tables holding parameters. > I doubt however that this would be any faster then a good DOM > implementation. I see... this is more-less the nsv as we have it today. Why wouldn't we making nsv interface public and use it for configuration store? After all, the code is there and is well tested. We might need to tweak some pieces but, in principle, this is it. I would need to refresh my view of nsv locking but generally it uses fixed number of buckets to avoid everybody locking the same global mutex. Increasing number of buckets lowers the contention. The only part we need to think about is actual parameter storage as this will not be a plain value, but also some housekeeping info like type, ranges etc, in addition. Given relatively fast lookup and relatively cheap locking, we would not need to have private and shared copies. I believe just one shared copy would be perfectly OK. Hm? What do you mean? Cheors Zoran |
|
From: Vlad S. <vl...@cr...> - 2006-08-23 14:16:18
|
Looking thru the code i see that mostly only fastpath and log uses NsConfig reads in real time, all others are copy values into local structures. funny, but these who uses config are performance critical and will introduce lock contention because thet are figthing for the same config values. Another raw/uncooked idea: is it possible to keep changed values separately, no lock default config as is now and whoever changed value just keep it in the lock-guarded hash. If i care about locking and changed value i will check new hash for new value, others who does not may still use read-only config list. Zoran Vasiljevic wrote: > On 23.08.2006, at 13:01, Zoran Vasiljevic wrote: > >> Alternatively, the DOM tree can be replaced by a combination >> of hash-tables and C-structures. Each ns_section will >> effectively be a hash table of parameters with a parameter >> being a simple C-struct. Also, there will be one global hashtable >> needed to resolve ns_section names to hash-tables holding parameters. >> I doubt however that this would be any faster then a good DOM >> implementation. > > > I see... this is more-less the nsv as we have it today. > > Why wouldn't we making nsv interface > public and use it for configuration store? > After all, the code is there and is well > tested. We might need to tweak some pieces > but, in principle, this is it. > > I would need to refresh my view of nsv locking > but generally it uses fixed number of buckets > to avoid everybody locking the same global mutex. > Increasing number of buckets lowers the contention. > > The only part we need to think about is actual > parameter storage as this will not be a plain > value, but also some housekeeping info like type, > ranges etc, in addition. > > Given relatively fast lookup and relatively cheap > locking, we would not need to have private and shared > copies. I believe just one shared copy would be > perfectly OK. > > Hm? What do you mean? > > Cheors > Zoran > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > naviserver-devel mailing list > nav...@li... > https://lists.sourceforge.net/lists/listinfo/naviserver-devel > -- Vlad Seryakov 571 262-8608 office vl...@cr... http://www.crystalballinc.com/vlad/ |
|
From: Zoran V. <zv...@ar...> - 2006-08-23 14:37:20
|
On 23.08.2006, at 16:17, Vlad Seryakov wrote: > Looking thru the code i see that mostly only fastpath and log uses > NsConfig reads in real time, all others are copy values into local > structures. funny, but these who uses config are performance critical > and will introduce lock contention because thet are figthing for the > same config values. > > Another raw/uncooked idea: is it possible to keep changed values > separately, no lock default config as is now and whoever changed value > just keep it in the lock-guarded hash. If i care about locking and > changed value i will check new hash for new value, others who does not > may still use read-only config list. I do not know how this could work. If thread A changes ns/some/thing and thread B is constantly looking up the ns/some/thing then the B must always lock to get the value, as A may be fiddling with it at the same time. Thread B can of course copy the value in its private (tsd) copy on the first fetch but then again, HOW is he going to find out that the thing was changed by the A in the meantime? I do not believe you can avoid locking if you want to have consistent behaviour. Also, the changes done by thread A would be pretty meaningless if only A is able to see them... I do not believe this is worth examining. As said, locking cannot be avoided. It is just the question of: can we tolerate it or not? I can. Therefore for me it is irelevant. But I do not if you can. Therefore I'm looking to see if we can do this as runtime or compile-time option. Also, because the locking is needed, we have to see how we can minimize it, also minimizing time spent in the locked sections of the code (by speeding up certain operations for example). Cheers Zoran |
|
From: Vlad S. <vl...@cr...> - 2006-08-23 14:53:40
|
Then we can just go through the code(fastpath, log) and change it so it copies config at startup into local variables (as in server.c) if that particular code does not care about realtime changes. This way it will be a compromise between locking and performance. Zoran Vasiljevic wrote: > On 23.08.2006, at 16:17, Vlad Seryakov wrote: > >> Looking thru the code i see that mostly only fastpath and log uses >> NsConfig reads in real time, all others are copy values into local >> structures. funny, but these who uses config are performance critical >> and will introduce lock contention because thet are figthing for the >> same config values. >> >> Another raw/uncooked idea: is it possible to keep changed values >> separately, no lock default config as is now and whoever changed value >> just keep it in the lock-guarded hash. If i care about locking and >> changed value i will check new hash for new value, others who does not >> may still use read-only config list. > > I do not know how this could work. If thread A changes > ns/some/thing > and thread B is constantly looking up the > ns/some/thing > then the B must always lock to get the value, as A may > be fiddling with it at the same time. > > Thread B can of course copy the value in its > private (tsd) copy on the first fetch but > then again, HOW is he going to find out that the > thing was changed by the A in the meantime? > > I do not believe you can avoid locking if you want > to have consistent behaviour. > > Also, the changes done by thread A would be pretty > meaningless if only A is able to see them... > I do not believe this is worth examining. > > As said, locking cannot be avoided. It is just the question > of: can we tolerate it or not? I can. Therefore for me it > is irelevant. But I do not if you can. Therefore I'm looking > to see if we can do this as runtime or compile-time option. > Also, because the locking is needed, we have to see how we > can minimize it, also minimizing time spent in the locked > sections of the code (by speeding up certain operations for > example). > > Cheers > Zoran > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > naviserver-devel mailing list > nav...@li... > https://lists.sourceforge.net/lists/listinfo/naviserver-devel > -- Vlad Seryakov 571 262-8608 office vl...@cr... http://www.crystalballinc.com/vlad/ |
|
From: Vlad S. <vl...@cr...> - 2006-08-23 15:09:57
|
Agree on up-to-the clinet policy of config usage > I'm interesed to find out what do you think about > speeding up the set operation by a help hash-table? > This would minimize the time the lock is held and > would therefore lower the impact of locking altogether. In the case when ns_set has only few items, sequential search can be much faster than hash overhead, so i am not sure about this one -- Vlad Seryakov 571 262-8608 office vl...@cr... http://www.crystalballinc.com/vlad/ |
|
From: Zoran V. <zv...@ar...> - 2006-08-23 15:19:06
|
On 23.08.2006, at 17:11, Vlad Seryakov wrote: > > In the case when ns_set has only few items, sequential search can be > much faster than hash overhead, so i am not sure about this one Yes. This is true. But how would you know? Say you have 100 element set and the thing you are looking for is on the 1 place? Obviously, the sequential search is instant. But if the thing is at the end, then hash *could* be faster. The real problem is: set with a handful of items (2-5). There the sequential search would be almost always faster. At some point (depending on the size of the set), a hash lookup could pay off. the problem is how to locate that point! OTOH, how MUCH is the hash lookup slower for a say, 10 or 20 elements set if, for example, the thing is sitting in the middle? I believe this is the question to answer... Perhaps I should write some test code... Cheers Zoran |
|
From: Zoran V. <zv...@ar...> - 2006-08-23 16:23:50
|
On 23.08.2006, at 17:11, Vlad Seryakov wrote:
> In the case when ns_set has only few items, sequential search can be
> much faster than hash overhead, so i am not sure about this one
Most interesting:
lexxsrv:nscp 3> ns_set create
d0
lexxsrv:nscp 7> for {set i 0} {$i < 100} {incr i} {ns_set put d0 test
$i test$i}
lexxsrv:nscp 10> time {ns_set find d0 test50} 1000
3.269 microseconds per iteration
lexxsrv:nscp 16> time {ns_set find d0 test99} 1000
4.462 microseconds per iteration
lexxsrv:nscp 18> time {ns_set find d0 test1} 1000
2.093 microseconds per iteration
lexxsrv:nscp 11> for {set i 0} {$i < 100} {incr i} {set xx(test$i)
test$i}
lexxsrv:nscp 21> time {set xx(test1)} 1000
1.527 microseconds per iteration
lexxsrv:nscp 25> time {set xx(test50)} 1000
1.628 microseconds per iteration
lexxsrv:nscp 23> time {set xx(test99)} 1000
1.628 microseconds per iteration
As you see, the ns_set is *way* inferior to
hash tables. Allright, the point is perhaps
that the "set" command is part of the Tcl byte
code so it is inherently faster that ns_set.
But, even then, a moderate set of 100 elements
needs about 3 microseconds to find the middle key
whereas the hash-table needs about 2 msec flat
to locate ANY key.
This all means that even for moderate-sized
sets it pays of to speed them up for searching.
(we'd have to accept some speed penalty for
adding/updating/deleting sets allright, but
nothing is for free).
All measurements are done on my Mac PB 1.5GHZ powerpc.
Cheers,
Zoran
|
|
From: Zoran V. <zv...@ar...> - 2006-08-23 16:55:40
|
On 23.08.2006, at 18:23, Zoran Vasiljevic wrote: > > All measurements are done on my Mac PB 1.5GHZ powerpc. WRONG! That was on an Linux box (SuSE9.1) devlinux# cat /proc/cpuinfo processor : 0 vendor_id : AuthenticAMD cpu family : 6 model : 8 model name : AMD Athlon(tm) XP 2200+ stepping : 1 cpu MHz : 1795.351 cache size : 256 KB |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X