Menu
▾
▴
naviserver-devel
|
From: Vasiljevic Z. <zv...@ar...> - 2007-10-12 19:36:40
|
So: The configuration is global and read-only. Parameters may be retrieved at run-time using [cmd ns_config], although usually configuration parameters are used by Tcl library files at start-up. I submitted a patch/module to RFE section since some time already, that implements the config storage that is read/write. The emphasis was to minimize locking for readers (making readers fast) on the cost of making writers slow(er). Has anybody checked that? Because I think that we might really use that module as a replacement for the read-only version that is currently in place. We may even make the (new module) read-only per switch (default) but allow read-write for those who might need it (we, for example). Any ideas, objections? |
|
From: Vlad S. <vl...@cr...> - 2007-10-12 22:05:58
|
In it will not hurt performance when constantly asking for config parameter with locking i am fine with it. Can be also compile time option for configure Vasiljevic Zoran wrote: > So: > > > The configuration is global and read-only. Parameters may be > retrieved at > run-time using [cmd ns_config], although usually configuration > parameters are > used by Tcl library files at start-up. > > I submitted a patch/module to RFE section since some time > already, that implements the config storage > that is read/write. The emphasis was to minimize > locking for readers (making readers fast) on the cost > of making writers slow(er). > > Has anybody checked that? Because I think that we might > really use that module as a replacement for the read-only > version that is currently in place. We may even make the > (new module) read-only per switch (default) but allow > read-write for those who might need it (we, for example). > > Any ideas, objections? > > > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. > Still grepping through log files to find problems? Stop. > Now Search log events and configuration files using AJAX and a browser. > Download your FREE copy of Splunk now >> http://get.splunk.com/ > _______________________________________________ > naviserver-devel mailing list > nav...@li... > https://lists.sourceforge.net/lists/listinfo/naviserver-devel > |
|
From: Vasiljevic Z. <zv...@ar...> - 2007-10-13 08:56:20
|
On 13.10.2007, at 00:06, Vlad Seryakov wrote: > In it will not hurt performance when constantly asking for config > parameter with locking i am fine with it. Locking is of course necessary. But it is limited to locking one mutex and, more importantly, the locking contention is only between one reading and one or more writing threads. Locking a mutex is per-se not problematic, speedwise. It is the contention on the mutex that can slow you down. Assuming that writing (changing config) is far less to expect tnat reading, I'm sure there will be no speed penalty. I can also make the thing read-only per-config parameter so to avoid locking altogether if any concerns arise in future. It is good to peek in the code so you can see why I think this approach will not hurt. Cheers Zoran |
|
From: Stephen D. <sd...@gm...> - 2007-10-14 21:34:55
|
On 10/12/07, Vasiljevic Zoran <zv...@ar...> wrote:
>
> So:
>
> The configuration is global and read-only. Parameters may be
> retrieved at run-time using [cmd ns_config], although usually
> configuration parameters are used by Tcl library files at start-up.
This is from here:
http://naviserver.sourceforge.net/n/naviserver/files/ns_config.html
(which describes how it is, not necessarily how it should be)
> I submitted a patch/module to RFE section since some time
> already, that implements the config storage
> that is read/write. The emphasis was to minimize
> locking for readers (making readers fast) on the cost
> of making writers slow(er).
http://sourceforge.net/tracker/index.php?func=detail&aid=1549952&group_id=130646&atid=719006
(Also, there's now an nsconf module in CVS, but that's something different...)
> Has anybody checked that? Because I think that we might
> really use that module as a replacement for the read-only
> version that is currently in place. We may even make the
> (new module) read-only per switch (default) but allow
> read-write for those who might need it (we, for example).
>
> Any ideas, objections?
Here's the thread from last time (lots of good ideas):
http://sourceforge.net/mailarchive/message.php?msg_id=9AD66047-D15A-4C93-B705-AFF1F2E5F330%40archiware.com
Did you eventually use this code? Did it work for you?
|
|
From: Vasiljevic Z. <zv...@ar...> - 2007-10-15 16:40:20
|
On 14.10.2007, at 23:34, Stephen Deasey wrote: > Did you eventually use this code? Did it work for you? It is still "lab" quality. I have it running here for some time but not in the production (i.e in the product). I guess it is safe but you obviously never know. I also have no speed metrics although the speed should not be the problem, given the low mutex contention. Except if you have very large number of simultaneous writers which is practically never the case... Of course, before I do any step I wanted to hear some comments on the design to see if everybody really understands what is going on there and judge if this is going to open some new benefits, or (god forbid) troubles. The latter must/should be really zero. |
|
From: Vasiljevic Z. <zv...@ar...> - 2007-10-15 16:58:43
|
On 14.10.2007, at 23:34, Stephen Deasey wrote: > Here's the thread from last time (lots of good ideas): Yep. Too bad nobody peeked at the suggested (bare-bones) implementation. I guess I should just go and get it in, tagging the tree before the change so we can easily revert back if this founds to be problematic? Out of the numerous ideas, the really important one would be an option (compile or runtime) wether to allow config to be writable or not.... This I may need to implement beforehand. |
|
From: Stephen D. <sd...@gm...> - 2007-10-15 18:37:33
|
On 10/15/07, Vasiljevic Zoran <zv...@ar...> wrote:
>
> On 14.10.2007, at 23:34, Stephen Deasey wrote:
>
> > Here's the thread from last time (lots of good ideas):
>
> Yep. Too bad nobody peeked at the suggested
> (bare-bones) implementation. I guess I should
> just go and get it in, tagging the tree before
> the change so we can easily revert back if this
> founds to be problematic?
:-/
> Out of the numerous ideas, the really important
> one would be an option (compile or runtime) wether
> to allow config to be writable or not....
I don't think this is the most important problem.
One problem is that there seems to be a lot of memory allocation and
copying going on, even on the fast path. For example, NsConf_GetBool()
does three mallocs, a couple of copies, conversion back and forth
between string and int rep, two hash lookups and a lock/unlock. It's
not exactly (opt & FOO_OPT) :-(
Another problem is that the model is extremely basic. It seems to be
identical to nsv (with different locking) so the only way to have
dynamic configuration is to do this:
AtRuntime()
{
if (ConfigGet("enabled")) {
...
}
}
- This will be a problem for some code because it's too slow.
- Some configuration parameters depend on other ones or on some
computation like converting units. ATM this is done once at startup.
But with this scheme it would have to be inline with the code that
used the param (or perhaps wrappers..).
- Some changes in config values should trigger other code to run -- it
is an event, it is not polled. For example, some threads might need
killed off or created within a pool, or a proc should be scheduled.
- One of the problems mentioned was inventorying config params. For
example, some config params are guarded behind a check for 'enabled'
-- if it's not enabled the other params won't be checked, and they
remain hidden. I guess this module doesn't help with this, maybe it
shouldn't..?
One way of looking at it might be that this will not handle all config
situations. I think the problem with that is:
1: It is confusing. We are saying: config is read write. But when you
write some config params, nothing changes. It is doubly confusing
because some of those params *will* be changeable, but only by calling
ns_fooctl.
2: This scheme would nicely handle non performance critical key/value
config called from Tcl code. But if you can already do that with
nsv...?
I still agree that a more runtime configurable server is desirable.
|
|
From: Jeff R. <dv...@di...> - 2007-10-15 19:53:59
|
Vasiljevic Zoran wrote: > Do you have some other idea how (if) we should build the chnageable > config? It seems pretty silly to me that we need to restart the server > to change some marginal parameter... In the 21. century... However this gets resolved, I think there should also be a way to easily write out a config file based on the current dynamic configuration, so that if the server does get restarted the configuration can stay the same. -J |
|
From: Rick C. <rc...@Kn...> - 2007-10-15 20:15:29
|
In the version we did for AOLServer 3.4.2, which relies on our proprietary event-notification subsystem, we: (1) Put a place in the startup .tcl script where over-rides from a local file would be used (using "source"). That way, the file output from live settings could be the equivalent of: ns_section "x" ns_param p v ... without changing the file we ship for customers. This made it somewhat easier to edit with Tcl (since it only uses these two commands and is always generated, not hand-edited). (I say "the equivalent" above because we do some fairly strange things to map between sections/parameters and our configuration "topicspace" that I don't want to go into here.) (2) Our configuration variables are marked with whether they are "no restart" or not. The backend that implements "change this variable" returns a flag indicating whether restart is required, and our GUI puts an indicator up if that's happened. Actually, we also set another global state variable to "configuration changes make restart required" so any instance of any console can show an appropriate flag. This mostly works OK, though we do occasionally have to train administrators with the question "When you changed that, did the console tell you you needed to restart? What happened when you restarted?" As far as calculated or enabled configuration settings go, we model all of those as "restart required" settings. -- ReC -----Original Message----- From: nav...@li... [mailto:nav...@li...] On Behalf Of Jeff Rogers Sent: Monday, October 15, 2007 12:54 PM To: nav...@li... Subject: Re: [naviserver-devel] ns_config read/write Vasiljevic Zoran wrote: > Do you have some other idea how (if) we should build the chnageable > config? It seems pretty silly to me that we need to restart the server > to change some marginal parameter... In the 21. century... However this gets resolved, I think there should also be a way to easily write out a config file based on the current dynamic configuration, so=20 that if the server does get restarted the configuration can stay the same. -J ------------------------------------------------------------------------ - This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ _______________________________________________ naviserver-devel mailing list nav...@li... https://lists.sourceforge.net/lists/listinfo/naviserver-devel |
|
From: Andrew P. <at...@pi...> - 2007-10-15 21:30:03
|
On Mon, Oct 15, 2007 at 12:53:57PM -0700, Jeff Rogers wrote: > However this gets resolved, I think there should also be a way to easily > write out a config file based on the current dynamic configuration, so > that if the server does get restarted the configuration can stay the same. Excellent idea. It's also awfully nice to be able to just diff two files to see what you changed. -- Andrew Piskorski <at...@pi...> http://www.piskorski.com/ |
|
From: Vasiljevic Z. <zv...@ar...> - 2007-10-17 16:40:44
|
On 15.10.2007, at 21:53, Jeff Rogers wrote: > However this gets resolved, I think there should also be a way to > easily > write out a config file based on the current dynamic configuration, so > that if the server does get restarted the configuration can stay > the same. ... which is called persistence and I thought about that of course. I haven't figure out how this would be achieved but one can think of anything between generating the ns_section/ns_param lines in a Tcl file on one extreme, to putting all in somekind of database (gdbm or similar). Cheers Zoran |
|
From: Vasiljevic Z. <zv...@ar...> - 2007-10-15 19:41:14
|
On 15.10.2007, at 20:37, Stephen Deasey wrote:
>
> One problem is that there seems to be a lot of memory allocation and
> copying going on, even on the fast path. For example, NsConf_GetBool()
> does three mallocs, a couple of copies, conversion back and forth
> between string and int rep, two hash lookups and a lock/unlock. It's
> not exactly (opt & FOO_OPT) :-(
This all is true in the current implementation just
because I was lazy... The important part where you
need to look at is the:
ConfGetParam(section, key, &defpar, &outpar);
This one we must/should take apart. Even here we have
some headroom.
>
> Another problem is that the model is extremely basic. It seems to be
> identical to nsv (with different locking) so the only way to have
> dynamic configuration is to do this:
>
> AtRuntime()
> {
> if (ConfigGet("enabled")) {
> ...
> }
> }
>
>
> - This will be a problem for some code because it's too slow.
It very different from nsv. Locking is completely different.
The only similarity is that it stores key/value pairs.
>
> - Some configuration parameters depend on other ones or on some
> computation like converting units. ATM this is done once at startup.
> But with this scheme it would have to be inline with the code that
> used the param (or perhaps wrappers..).
True. But, again, this would be done only once, as I cache
the values in the per-thread private hash table.
>
> - Some changes in config values should trigger other code to run -- it
> is an event, it is not polled. For example, some threads might need
> killed off or created within a pool, or a proc should be scheduled.
By all means. Bu this is not in the scope of the config module,
rather the "user" code. If the user-code is written so that it
reacts to changes of the config, then there is no problem.
At the moment no code does that because the config is read-only.
>
> - One of the problems mentioned was inventorying config params. For
> example, some config params are guarded behind a check for 'enabled'
> -- if it's not enabled the other params won't be checked, and they
> remain hidden. I guess this module doesn't help with this, maybe it
> shouldn't..?
This is the functionality that I did not handle at all. The emphasis
was to create read/write config more/less with the same functionality
as the current one.
>
>
> One way of looking at it might be that this will not handle all config
> situations. I think the problem with that is:
>
> 1: It is confusing. We are saying: config is read write. But when you
> write some config params, nothing changes. It is doubly confusing
> because some of those params *will* be changeable, but only by calling
> ns_fooctl.
This is true. If the rest of the code is not programmed with the
notion of changeable config, there will be "confusing" situations.
>
> 2: This scheme would nicely handle non performance critical key/value
> config called from Tcl code. But if you can already do that with
> nsv...?
Well, not really. This needs to be also accessible from C and from
various plugin-modules.
>
>
> I still agree that a more runtime configurable server is desirable.
One thing to remember: if we set the funtional-bar too high we will
not achieve much on the short/midterm.
What I see is that in order to make the config writable, all kind of
parts in the server needs to be rewritten in order to gain benefit
from it (less important) and behave "logically" (more important).
Given number of such places that would be potentially changed are
so high that I will not attempt to do so. Simply because of the lack
of time.
Do you have some other idea how (if) we should build the chnageable
config? It seems pretty silly to me that we need to restart the server
to change some marginal parameter... In the 21. century...
>
> ----------------------------------------------------------------------
> ---
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems? Stop.
> Now Search log events and configuration files using AJAX and a
> browser.
> Download your FREE copy of Splunk now >> http://get.splunk.com/
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
|
|
From: Andrew P. <at...@pi...> - 2007-10-15 21:27:52
|
On Mon, Oct 15, 2007, Zoran Vasiljevic wrote: > On 15.10.2007, at 20:37, Stephen Deasey wrote: >> 2: This scheme would nicely handle non performance critical key/value >> config called from Tcl code. But if you can already do that with >> nsv...? > > Well, not really. This needs to be also accessible from C and from > various plugin-modules. Making nsv usable from C code is very useful, and not all that hard. I've done it in the past (for AOLserver 3.x and 4.0.x). So, just how are the use cases for dynamic ns_config different from those for nsv? Since the existig ns_config is totally static, it's safe to take the pointer returned by Ns_ConfigGetValue() and just assume it never changes and is always valid process-wide. But of course that can't work when calling Nsv from C code. What does your dynamic ns_config code do for that case? -- Andrew Piskorski <at...@pi...> http://www.piskorski.com/ |
|
From: Vasiljevic Z. <zv...@ar...> - 2007-10-17 16:38:57
|
On 15.10.2007, at 23:27, Andrew Piskorski wrote: > > Making nsv usable from C code is very useful, and not all that hard. > I've done it in the past (for AOLserver 3.x and 4.0.x). So, just how > are the use cases for dynamic ns_config different from those for nsv? Good question. Tt happens that we already use the ns_section/ns_param all over the code (c and tcl). furthermore we need to be able to tune server params during runtime as well. All this is not what nsv's are. > > Since the existig ns_config is totally static, it's safe to take the > pointer returned by Ns_ConfigGetValue() and just assume it never > changes and is always valid process-wide. But of course that can't > work when calling Nsv from C code. What does your dynamic ns_config > code do for that case? It gets the value from it's per-thread cache. But, the source-code is pretty simple and documented so you should be able to get the idea by skipping thru the code (takes about 10-15 minutes). It tries to avoid lock contention by greedily caching all conf params in all threads that access them. OTOH, writers have a "tough" job of updating all those private thread caches everytime some config parameter is changed. Cheers Zoran |
|
From: Stephen D. <sd...@gm...> - 2007-10-16 23:09:19
|
I cleaned up the module a bit and imported it to CVS. http://naviserver.cvs.sourceforge.net/naviserver/nsconfigrw http://naviserver.sourceforge.net/n/nsconfigrw/files/ns_getconfig.html (btw. if you create a man page for your module and put it in the right place, doc/src/mann or doc/src/man3, it will appear on the website...) Most of the tests are failing. Looks like the code can't distinguish missing values for bool and int because a default is always required. Is this OK..? Step two would be to fix the excessive copying. |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X