naviserver-devel — Developer discussion

Re: [naviserver-devel] positioning naviserver

[Jeff R. <dv...@di...>]

Gustaf Neumann wrote:

> Forward-looking does certainly not mean that we do not want
> to keep compatibility (we have all large code bases using
> naviserver) but i would certainly hate to see e.g. the
> useless $conn argument to be re-introduced in naviserver
> just because some 10+ year old code expects it.
>
> The general rule of change should be
>      major-versions-are-allowed-to-break-backward-compatibility
> and
>      minor-versions-should-keep-backward-compatbility
> where we are talking about intra-naviserver compatibility.

This is pretty standard and sensible for a project supported by volunteers.

However, what I'd like to see - when reasonable - is an option for 
backwards compatibility, either by setting a config flag or loading a 
compatibility module, as nsshare appears to be shaping up into.  Adding 
in a version of ns_register_filter that passes aolserver-style args 
would not be a stretch.

This I think could give the best of both worlds - improvements in 
functionality without alienation of user base.

-J

Re: [naviserver-devel] ns_set ... -persistent

[Zoran V. <zv...@ar...>]

On 10.10.2012, at 15:58, Zoran Vasiljevic wrote:

> The nsshare module we can deliver 
> as-is and declare unsupported and
> deprecated.

I think that people who want that 
level of compatibility should take
care of the whole thing. 
This is really not going to bring 
us any further in terms of functionality 
(n)or stability. I would not have 
anything against to "host" a module
in the distribution but I can definitely 
not support it.

Stephen was very kind to provide a nice 
starting point. So we can leave it at
that level IMO.

Cheers
Zoran 



 

Re: [naviserver-devel] ns_set ... -persistent

[Gustaf N. <ne...@wu...>]

Am 10.10.12 18:33, schrieb Zoran Vasiljevic:
> On 10.10.2012, at 18:15, Jeff Rogers wrote:
>
>> could that renaming of the core
>> tcl command be done as part of the module
> sure it can
there is no need for an explicit rename,
one can register the new command as "ns_set".

once the command is overloaded, all handlings
of potentially shared ns_sets have to go through
the new command which provides the locking....

if there is a place in naviserver, where a ns-set handle
is passed to a c-function other than ns_set, it could
break. but since i guess, this does not happen,
this should be fine.

-gustaf

Re: [naviserver-devel] ns_set ... -persistent

[Zoran V. <zv...@ar...>]

On 10.10.2012, at 18:15, Jeff Rogers wrote:

> could that renaming of the core 
> tcl command be done as part of the module

sure it can

Re: [naviserver-devel] ns_set ... -persistent

[Jeff R. <dv...@di...>]

Zoran Vasiljevic wrote:
>
> On 10.10.2012, at 12:11, Stephen Deasey wrote:
>
>> I was thinking that the 2 or 3 people in the world that need backward
>> compatibility would load the nsshare module and then:
>>
>>   rename _ns_set ns_set
>>
>> You wouldn't use them both, you just want your old code to work the
>> way it did in 1999.
>>
>
> I find this to be a good compromise.

I agree, this is a good compromise.  But could that renaming of the core 
tcl command be done as part of the module?  So the instructions would be 
just "load this module in your config" rather than "load this module in 
your config and add this line somewhere in your library code".  The goal 
being to make backwards compatibility settable just in the server config.

-J

> The nsshare module we can deliver
> as-is and declare unsupported and
> deprecated. People that really depend
> on it can then take the responsibility
> of keeping it up-to-date. And since it
> is not in the main server code, we can
> (mostly) forget about it.
>
> Cheers,
> Zoran

Re: [naviserver-devel] ns_set ... -persistent

[Zoran V. <zv...@ar...>]

On 10.10.2012, at 12:11, Stephen Deasey wrote:

> I was thinking that the 2 or 3 people in the world that need backward
> compatibility would load the nsshare module and then:
> 
>  rename _ns_set ns_set
> 
> You wouldn't use them both, you just want your old code to work the
> way it did in 1999.
> 

I find this to be a good compromise.
The nsshare module we can deliver 
as-is and declare unsupported and
deprecated. People that really depend
on it can then take the responsibility
of keeping it up-to-date. And since it
is not in the main server code, we can
(mostly) forget about it.

Cheers,
Zoran

Re: [naviserver-devel] ns_set ... -persistent

[Stephen D. <sd...@gm...>]

On Wed, Oct 10, 2012 at 10:09 AM, Gustaf Neumann <ne...@wu...> wrote:
>
> Using a "_ns_set" capable of handling "-shared" which just calls
> existing "ns_set" with the unlocked semantics would not work,
> using a lock with for all "_ns_set" commands is not good either
> without modifying the existing "ns_set" semantics (one should
> not be able to access variables created by "_ns_set" from
> "ns_set"). If we check for the shared flag in "ns_set", we are
> essentially at the old "ns_set" implementation.

I was thinking that the 2 or 3 people in the world that need backward
compatibility would load the nsshare module and then:

  rename _ns_set ns_set

You wouldn't use them both, you just want your old code to work the
way it did in 1999.

Re: [naviserver-devel] ns_set ... -persistent

[Gustaf N. <ne...@wu...>]

On 09.10.12 22:43, Stephen Deasey wrote:
>
>> Separately, I was experimenting with a performance enhancement to
>> ns_sets, creating a hashtable mapping the keys to indexes.  There is a
>> slight cost in memory and on the first lookup, but it should get to
>> breakeven after only around 3 lookups on average, with decreasing
>> amortized cost after that.  The difference is measurable and can be
>> significant, as much as 4x faster on a large set.  However, that's
>> measured on a very large set, 1000 keys;  and the real savings is pretty
>> small, around 15 microseconds per lookup on average hardware.  Is that
>> kind of micro-optimization worth the additional complexity?
> Sets are used in a few core places where this sounds like a disadvantage:
>
> - http headers (in/out)
> - html forms
> - database rows
>
> All low numbers of rows with few lookups. Even config values rarely
> have huge numbers of elements, or duplicates, and often are looked up
> only once at startup and then stashed away in a struct.
also, in our experience ns_sets are typically quite small.

Since the ns_sets are currently used just per thread, the 
question is for
what kind of purposes needing large amounts of keys are 
these used.
Tcl has already arrays and dicts. Handling multiple values 
per key
can be realized with "dict lappend".

-gustaf neumann

Re: [naviserver-devel] ns_set ... -persistent

[Gustaf N. <ne...@wu...>]

On 09.10.12 14:18, Stephen Deasey wrote:
>
> If I remember rightly, the code has long been depreciated in
> AOLserver, and as no one used it and it complicates the arg parsing
> and locking we removed it, along with the ns_share and ns_var command.
> There's Tcl wrapper for ns_var because it was trivial.
>
> I've created a new module nsshare which reimplements the ns_share
> command using the old C code:
>
>    https://bitbucket.org/naviserver/nsshare/overview
>
> Haven't put much effort into testing it, but the skeleton of the
> module is ready to go, it compiles and the sanity tests work.
Great, many thanks, i hope, the people asking for it will 
appreciate it.
> I think the way to add back the -shared switch to ns_set would be to
> add a new file to this module: setcmd.c, add the C code that was
> removed, export an _ns_set ?-shared? ?args ...? command which, if the
> -shared flag is present does it's thing, otherwise calls the
> underlying ns_set command. If you want to do this, that would be
> great.
well, the parsing is here not a real issue, i was more 
concerned about
thread-safety, scalability, etc. (similar to Jeff). In my 
opinion, if we support
it, it should go into the "ns_set" command, if we don't 
support it (due to
instability, etc.), drop it from documentation and suggest 
alternatives.

When i read through the change-set, the term "-persist" means
actually "-shared", all ns-sets created via the 
tcl-interface are
"dynamic", and there was a single lock per server for all 
(shared)
ns_sets. The mutex is used for EnterSet (adding a set to an 
interp),
"ns_set list" and all lookup operations. So, aside of checking
thread-saftey of  the long list of subcommands of ns_set,
and cleanup-semantics/memory leaks, the code looks pretty
straight forward. The single lock for all ns_set is not very 
scalable,
reintroducting "ns_set ... -shared" would mean to slow down 
existing
ns_set a little (checking the flag whether a lock is needed).

Using a "_ns_set" capable of handling "-shared" which just calls
existing "ns_set" with the unlocked semantics would not work,
using a lock with for all "_ns_set" commands is not good either
without modifying the existing "ns_set" semantics (one should
not be able to access variables created by "_ns_set" from
"ns_set"). If we check for the shared flag in "ns_set", we are
essentially at the old "ns_set" implementation.

I guess the driver for removing the "-persistent" flag was
that we have already so many ways of handling shared vars
either in naviserver or in libthread, that this additional way
with the misleading name (that should be changed) was dropped.

>An alternative might be to provide equivalent functionality
>(order-preserving, multiple entries for keys, access by index or key or
>case-insensitive key) as a new nsv subcommand, perhaps 'nsv_multiset'.

Note that there are as well in libthread the keyed list 
commands.
Serializeíng the ns_set in a nsv, and create/save it in a thread
when needed, as jeff suggested in the aolserver thread seems
feasible. But these approaches require some code 
modification in the
applications...

-gustaf neumann

[naviserver-devel] positioning naviserver

[Gustaf N. <ne...@wu...>]

Dear all,

As our audience gets larger, i think it is important to 
reiterate,
what is a common understanding in the naviserver community
(at least how i see it). While the basic attitude in the
aolserver community is backward looking (be as conservative
as possible, never change anything user-visible, even if this is
complete useless) the attitude of the naviserver community
is rather forward looking (clean up interface, improve the 
way how
to handle problems, be competitive with other server 
environments,
be among the best in terms of scalability, etc).
The need of doing things differently in some real-world
applications was the driver of the split of naviserver from
aolserver.

Forward-looking does certainly not mean that we do not want
to keep compatibility (we have all large code bases using
naviserver) but i would certainly hate to see e.g. the
useless $conn argument to be re-introduced in naviserver
just because some 10+ year old code expects it.

The general rule of change should be
    major-versions-are-allowed-to-break-backward-compatibility
and
    minor-versions-should-keep-backward-compatbility
where we are talking about intra-naviserver compatibility.

Not every change in the server has the same urgency
for every participant. For example Zoran has a very different
usage pattern for the server than we have, so several changes
important for us are effectively useless for him. But still, 
there
is a nice long-going cooperation between the main stakeholders,
and the tradition of (although informal) code-reviews
works very well.

-gustaf neumann

Re: [naviserver-devel] Greetings

[Gustaf N. <ne...@wu...>]

On 09.10.12 23:02, Stephen Deasey wrote:
> On Tue, Oct 9, 2012 at 7:49 PM, Zoran Vasiljevic <zv...@ar...> wrote:
>
>>> Propose
>>> changes on the development list and then make changes to the main
>> This is how we have worked so far. This mostly covers bug fixes.
>> Whole-sale changes are little bit different... but they happen
>> seldom. In that case, a branch consisting of all the changes is
>> prefered.
> Conceptually yes, but don't actually use a mercurial branch for
> temporary development. You can't delete branches.
>
> On your local computer just clone an existing checkout into a new
> directory, it'll use hard links so it's fast and efficient. If you
> want to publish it for feedback then click the 'fork' button on the
> naviserver project at bitbucket and create a new repo under your own
> account: push your changes directly to it. If it's a small change,
> just post it here.
For larger changes (not just cleanup or fixes) feature branches
are the best way. That works nicely and perfectly in our
(git) developents, and keeps a change set focused to a
topic (it's telling a story, as stephen said) while still
being able to improve the change set.

We have not used feature branches with hg/naviserver so
far, but it seems certainly to be a good idea to start with this
Bitbucket advertises the work via via pull requests prominently
in its new web-design

https://bitbucket.org/naviserver/naviserver/pull-requests

Pull requests work with branches and forks
https://confluence.atlassian.com/display/BITBUCKET/Working+with+pull+requests

but it seems the approach with forks is better, since 
conceptually
branches (mercurial calls it named branches) are in 
mercurial thought
to be long-living:

http://stackoverflow.com/questions/6357012/in-mercurial-how-do-i-merge-remove-a-feature-branch-so-i-can-commit

-gustaf neumann

Re: [naviserver-devel] Greetings

[Stephen D. <sd...@gm...>]

On Tue, Oct 9, 2012 at 7:49 PM, Zoran Vasiljevic <zv...@ar...> wrote:
>
> On 09.10.2012, at 20:10, Jeff Rogers wrote:
>
> Hi Jeff!
>
>> Propose
>> changes on the development list and then make changes to the main
>
> This is how we have worked so far. This mostly covers bug fixes.
> Whole-sale changes are little bit different... but they happen
> seldom. In that case, a branch consisting of all the changes is
> prefered.

Conceptually yes, but don't actually use a mercurial branch for
temporary development. You can't delete branches.

On your local computer just clone an existing checkout into a new
directory, it'll use hard links so it's fast and efficient. If you
want to publish it for feedback then click the 'fork' button on the
naviserver project at bitbucket and create a new repo under your own
account: push your changes directly to it. If it's a small change,
just post it here.


Anyway, if you think of vc as backup and approval then it sounds like
a drag, but it's really more like your editor - it helps you code.

You often don't know what you're going to end up with when you start
so you use your editor and hg to manipulate the code.
- You start to add a feature but half way through you realise if you
refactored some existing functions it would make the addition easier.
- So, pop-off what you've done so far, refactor the code, push your
pending changes back.
- Then you notice a bug in some existing code. Pop off both changes,
fix the bug, push them back on again.
Now when you share the code there's 3 separate changes and they tell a
story: here's a simple bug fix, here's a refactoring which should not
change existing behaviour, and here's a new feature.

The easier it is to read the more people are likely to read it and the
more feedback you'll get, which is invaluable. In addition, everyone
who touches the code end up with a mental model of how it works, so
when the code changes the model needs to be synced up again. It's not
scalable to have each person slog through a big old dump of code that
took the original changer 6 hours to grok. You need to present it like
a story so it makes sense.

Check out the lkml to see how the ninjas do it.

Re: [naviserver-devel] ns_set ... -persistent

[Stephen D. <sd...@gm...>]

On Tue, Oct 9, 2012 at 7:49 PM, Jeff Rogers <dv...@di...> wrote:
> I don't know that people love it so much as that there is no exact
> replacement for the functionality.
>
> I'm a big proponent of compatibility but shared sets as they previously
> existed are problematic.  I wouldn't be at all surprised if they are
> thread-unsafe to the point of being able to crash the server.
>
> An alternative might be to provide equivalent functionality
> (order-preserving, multiple entries for keys, access by index or key or
> case-insensitive key) as a new nsv subcommand, perhaps 'nsv_multiset'.

Hmm, sounds a bit like reintroducing the old code under a new name. I
think people are just unwilling to change old code, not specifically
looking for case-folding multi-sets.

> Separately, I was experimenting with a performance enhancement to
> ns_sets, creating a hashtable mapping the keys to indexes.  There is a
> slight cost in memory and on the first lookup, but it should get to
> breakeven after only around 3 lookups on average, with decreasing
> amortized cost after that.  The difference is measurable and can be
> significant, as much as 4x faster on a large set.  However, that's
> measured on a very large set, 1000 keys;  and the real savings is pretty
> small, around 15 microseconds per lookup on average hardware.  Is that
> kind of micro-optimization worth the additional complexity?

Sets are used in a few core places where this sounds like a disadvantage:

- http headers (in/out)
- html forms
- database rows

All low numbers of rows with few lookups. Even config values rarely
have huge numbers of elements, or duplicates, and often are looked up
only once at startup and then stashed away in a struct.

How are you planing to use these key/value structures?

If you're going to use them from a Tcl API within one interp then you
can use the Tcl object shimmering technique. If you make the lookup
key fully describe the value instead of using a two-part array-key
then after the first lookup you can store a pointer to the value in
the spare Tcl_Obj ptr. You can see this technique used in the arg
parsing machinery.

If you're sharing between interps you can't use that technique, but
there may be other things you can speed up. For example, here's a
project I never finished to create an API identical to Tcl hash tables
except that you can pre-allocate space for the value in the key
struct:

  https://bitbucket.org/groks/naviserver-nshash

The idea was to reduce memory fragmentation and be more cache
friendly, which gets more important as modern computers with numa
memory layouts and a large difference between cpu-memory speeds.

There's also this r/w config replacement:

  https://bitbucket.org/naviserver/nsconfigrw

which has links to some discussion about implementation, including
support for multiple values, case folding etc.

Or maybe you're trying to recreate PHP's do-everything hash-list
container... :-)

Re: [naviserver-devel] ns_set ... -persistent

[Zoran V. <zv...@ar...>]

On 09.10.2012, at 20:49, Jeff Rogers wrote:

>  Is that 
> kind of micro-optimization worth the additional complexity?

For me personally: not really. I would not have anything
against, though, if somebody (you) give it a try ;-)

Cheers,
Zoran

Re: [naviserver-devel] ns_set ... -persistent

[Jeff R. <dv...@di...>]

Zoran Vasiljevic wrote:
>
> On 09.10.2012, at 20:49, Jeff Rogers wrote:
>
>> creating a hashtable mapping the keys to indexes
>
> I assume you used Tcl hash-tables with TCL_STRING_KEYS?
> You know that ns_set keys can be case-insensitive?

Yes - I've only half-implemented it so far, but the full implementation 
would need to create two hashtables, one with case preserved and one 
with all keys tolower-ed.  I expect the costs and benefits would be similar.

-J

Re: [naviserver-devel] ns_set ... -persistent

[Zoran V. <zv...@ar...>]

On 09.10.2012, at 20:49, Jeff Rogers wrote:

> creating a hashtable mapping the keys to indexes

I assume you used Tcl hash-tables with TCL_STRING_KEYS?
You know that ns_set keys can be case-insensitive?

Cheers,
Zoran

Re: [naviserver-devel] Greetings

[Zoran V. <zv...@ar...>]

On 09.10.2012, at 20:10, Jeff Rogers wrote:

Hi Jeff!

> Propose 
> changes on the development list and then make changes to the main 

This is how we have worked so far. This mostly covers bug fixes.
Whole-sale changes are little bit different... but they happen
seldom. In that case, a branch consisting of all the changes is
prefered. In any case, we must see the changes *clearly* and everybody
should be able to inspect and judge if this is potentially breaking
something or heading in the "wrong" direction. What is "wrong" is
difficult to tell... But people usually get the picture soon.
Coding style is clear from the code and it follows the AOLserver
and Tcl conventions mostly.

We are very concerned about the stability. 

Some people here (including myself) use this code in commercial 
products which are mission-critial for many companies arround 
the world. Lots of effort and time has been invested in debugging, 
fixing races and memory leaks in the current Naviserver code. 
This does not mean we don't want any changes. On the contrary!
But just that one has to be a little bit extra careful.
I assume you know all this but it is good to repeat this every
once in a while :-)

Cheers,
Zoran

Re: [naviserver-devel] ns_set ... -persistent

[Jeff R. <dv...@di...>]

I don't know that people love it so much as that there is no exact 
replacement for the functionality.

I'm a big proponent of compatibility but shared sets as they previously 
existed are problematic.  I wouldn't be at all surprised if they are 
thread-unsafe to the point of being able to crash the server.

An alternative might be to provide equivalent functionality 
(order-preserving, multiple entries for keys, access by index or key or 
case-insensitive key) as a new nsv subcommand, perhaps 'nsv_multiset'.

Separately, I was experimenting with a performance enhancement to 
ns_sets, creating a hashtable mapping the keys to indexes.  There is a 
slight cost in memory and on the first lookup, but it should get to 
breakeven after only around 3 lookups on average, with decreasing 
amortized cost after that.  The difference is measurable and can be 
significant, as much as 4x faster on a large set.  However, that's 
measured on a very large set, 1000 keys;  and the real savings is pretty 
small, around 15 microseconds per lookup on average hardware.  Is that 
kind of micro-optimization worth the additional complexity?

-J

Gustaf Neumann wrote:
> Stephen,
>
> do you remember why you took out the -shared flag from ns_set?
>
> https://bitbucket.org/naviserver/naviserver/changeset/1cbaf1acc09436f2a1c56102269a8b7fab0be168
>
>
> it seems that some people love it. We have either to take it
> out of
> the documentation (and give sensible explanation) or
> reintroduce it in the code...
>
> -gustaf

[naviserver-devel] Greetings

[Jeff R. <dv...@di...>]

Hi all,

I've exchanged correspondence with a number of people here but I'd like 
to extend a formal introduction - well, as formal as mailing lists get.

I suspect most of the people on this list also read the aolserver list 
and have seen the recent interest in re-merging aolserver and naviserver 
development and community.  I'm happy to be a part of this effort.  I've 
only kept up with developments in naviserver intermittently, so I'm just 
now starting to learn and explore the differences.  From what I can tell 
a lot of what I'd like to see in aolserver has already been done in 
naviserver.

I have worked with aolserver for a while, mostly as a user and 
occasional patch-submitter / bugfixer, but only recently as an active 
developer.  Other than what I've been doing, AOLserver has been seeing 
very little development.  This, along with other unrelated 
considerations has led me to adopt some sloppy practices, notably a 
tendency to commit working but incomplete (i.e., poor style, 
documentation) changes earlier and clean up on later commits.

While this has worked ok for me in the past, naviserver is a different, 
more active community and development style.  I've already made a few 
missteps and been advised of commit guidelines, but I'm still learning 
what the preferred development style is (and also getting used to the 
quirks of mercurial).   So then - what is the preferred style?  Propose 
changes on the development list and then make changes to the main 
branch,  make changes on a dev branch and merge after discussion, or 
create a private forked repository and merge changes in as patches/pull 
requests, or something else entirely.  I'm happy to follow whatever the 
convention is, tho I certainly prefer to avoid additional administrative 
overhead and making changes in multiple places.

I'm looking forward to working with everyone.

-J

Re: [naviserver-devel] ns_set ... -persistent

[Stephen D. <sd...@gm...>]

On Tue, Oct 9, 2012 at 11:11 AM, Gustaf Neumann <ne...@wu...> wrote:
> Stephen,
>
> do you remember why you took out the -shared flag from ns_set?
>
> https://bitbucket.org/naviserver/naviserver/changeset/1cbaf1acc09436f2a1c56102269a8b7fab0be168
>
>
> it seems that some people love it. We have either to take it
> out of
> the documentation (and give sensible explanation) or
> reintroduce it in the code...


If I remember rightly, the code has long been depreciated in
AOLserver, and as no one used it and it complicates the arg parsing
and locking we removed it, along with the ns_share and ns_var command.
There's Tcl wrapper for ns_var because it was trivial.

I've created a new module nsshare which reimplements the ns_share
command using the old C code:

  https://bitbucket.org/naviserver/nsshare/overview

Haven't put much effort into testing it, but the skeleton of the
module is ready to go, it compiles and the sanity tests work.

I think the way to add back the -shared switch to ns_set would be to
add a new file to this module: setcmd.c, add the C code that was
removed, export an _ns_set ?-shared? ?args ...? command which, if the
-shared flag is present does it's thing, otherwise calls the
underlying ns_set command. If you want to do this, that would be
great.

As there is now somewhere to put it, the ns_var code could be moved here too.

There's some compatibility info here:

  http://wiki.tcl.tk/22567

[naviserver-devel] ns_set ... -persistent

[Gustaf N. <ne...@wu...>]

Stephen,

do you remember why you took out the -shared flag from ns_set?

https://bitbucket.org/naviserver/naviserver/changeset/1cbaf1acc09436f2a1c56102269a8b7fab0be168 


it seems that some people love it. We have either to take it 
out of
the documentation (and give sensible explanation) or 
reintroduce it in the code...

-gustaf

[naviserver-devel] Recent changes of team/admin memers on Bitbucket

[Zoran V. <zv...@ar...>]

Hi friends,

Some days ago we had some non-communicated changes of 
admins/team-members lists of the project on the source
repository at bitbucket.org.

As not all of the existing admins of the project are of 
the opinion that increasing number of people with admin 
rights will do anything good for the project state, 
I decided to revert those changes (with one notable exception)
and put people who want to contribute new or improve and
fix existing code as developers with commit rights, leaving
the admin list small.  
Two of initial project admins have decided to pull back
from the active status because of their own reasons. 
I really regret this but will keep them on the admin list
as it makes me feel better. They have *significantly* 
contributed to the whole project and I'd like their names
to be associated with the project in the future as well.

If somebody knows people that what to participate as 
developers, I encourage you to tell them to first 
join the Navidevel mail list:

  <nav...@li...>

on SourceForge. This is where the most member communication
is taking place. Here is where we discuss, approve or
reject changes to the code and discuss other related things.
There, they can simply (and informally) request developer 
status, if needed.

At the moment we do not have tighter organisational infrastructure 
that would channel the work in a more formal manner (like for 
example the TIP infrastructure of the Tcl project) because of the
size of the user-base. This may change over the time, depending
on how things develop. But I think that we are all very well
off with the informal way as we have now. And that is... changes
are done only after having communicated the intention on the list
and passing the "human filter" of other involved people. Usually
the changes are silently accepted. In some cases, a longer 
discussion is needed. I rare cases (never happened so far) the
change is rejected. This is how we did the work in past few years
and this proved good enough for everybody involved.

For the current state of the code... 

We have a pretty stable code-base. Stable in the sense that it 
performs rock-solid in various large installations, which is good
and which is what I personally want to keep.
But, OTOH, stable in the sense that it does not develop further
mostly because there are little or no requests from the user base.

I see some movements in the latter, and I hope this will result
in a better and more versatile software than we have now. 

Cheers,
Zoran

Re: [naviserver-devel] tracking locks on nsv arrays

[Gustaf N. <ne...@wu...>]

Am 15.06.12 20:07, schrieb Zoran Vasiljevic:
> Interesting would be to see lock contention,
> i.e. lock requests that could not be satisfied
> (by counting trylock's).
The nsvs don't have own locks, but their buckets do. Therefore
we have a lock contention on a bucket rather than on a nsv.

The lock contention is measured on the bucket as usual, and
nsstats reports it like ever. The contention rate is often not very
meaningful either, if e.g. the locks occur seldom (e.g. when
starting sched procs) or the lock-time is small. Therefore,
since about half  a year the stats reported from naviserver
contain the waiting times as well.

The reason, i started looking into this was that we saw
larger waiting times on the mutex for nsv buckets, but so
far, naviserver had no means to tell a user, what nsvs
are kept in the bucket.

it is possible on the scripting level of nsstats to add the
mutex statistics for the arrays, but many array will
report the same lock statistics.....

well, this was simple enough, so i just added this
(see the graphics below). one has now "Locks" and
"Bucket Locks", which are for the top cases below
quite similar, meaning that the busy arrays are
in different buckets... Maybe, one could as well
add the ratio of these two values as well.

The results look strange, when one does a sorting by
some of the mutex statistics, since one sees then
then essentially the list of all arrays in that bucket.

-gustaf neumann

Re: [naviserver-devel] tracking locks on nsv arrays

[Zoran V. <zv...@ar...>]

On 15.06.2012, at 13:31, Gustaf Neumann wrote:

> The changes are on bitbucket.

Interesting would be to see lock contention,
i.e. lock requests that could not be satisfied
(by counting trylock's).

[naviserver-devel] tracking locks on nsv arrays

[Gustaf N. <ne...@wu...>]

Dear all,

There is now support for naviserver and nsstats.tcl
to track the source for high locks/contention
rate/lock waiting time on nsvs. This is implemented
through a new command ns_bucket, which returns
the contents of all or the specified bucket(s) with lock
rates caused by the nsv.

After the startup and a few clicks on my local machine,
i see the following table in nsstats (sorted by locks).

The changes are on bitbucket.
-gn