Re: [naviserver-devel] using rwlocks for nsv variables

[Gustaf N. <ne...@wu...>]

Dear all,

on June 30, i wrote:

> However, there might be NaviServer applications with nsvs out there, 
> for which the change of using rwlocks for nsv variables might lead to 
> a reduced performance. So we have in general the following options for 
> the forthcoming release:
>
> a) hardwire nsvs to rwlocks
> b) make it a compile-time decision to choose between rwlocks and mutex 
> locks for nsvs
> c) provide a configuration variable in the config file to choose 
> between rwlocks and mutex locks for nsvs at startup
> d) provide a runtime API for creating nsv arrays with rwlock or mutex
>
Since there were some concerns that rwlocks might not be the best choice 
for all nsvs, i did some more extensive tests and metering on several 
servers. These tests convinced me that rwlocks are best default value 
for web server applications, since there is an overwhelming amount of 
read operations compared to write operations (see below, e.g. less the 
5% writer operations, these are real-world figures from our production 
code at the university):

     Name	                Locks	Busy	Read	Write	Write %
     nsv:56:live	                192.91M	14	192.87M	47.79K	0.02%
     nsv:138:live	        92.15M	1	92.13M	16.21K	0.02%
     nsv:169:live	        44M	0	43.99M	14.46K	0.03%
     nsv:164:live	        31.81M	21	31.52M	294.13K	0.92%
     nsv:157:live	        26.96M	0	26.96M	1.5K	0.01%
     nsv:76:live	                23.53M	0	23.53M	1.26K	0.01%
     nsv:146:live	        20.59M	0	20.55M	38.33K	0.19%
     nsv:27:live	                13.26M	0	13.26M	801	0.01%
     nsv:50:live	                8.98M	0	8.98M	1.15K	0.01%
     nsv:185:live	        8.57M	0	8.28M	288.21K	3.36%
     nsv:184:live	        8.02M	0	7.72M	290.49K	3.62%
     nsv:107:live	        7.05M	0	7.05M	924	0.01%

One more interesting fact is we see very little number of busy 
operations, which was in the case of mutex locks on the same server as 
least by a factor of 1000 higher. So, we can achieve a much higher 
degree of parallelism using rwlocks. These numbers can be obtained from 
the updated versions of the nstats module.

One more interesting comparison is potting different kind of operations 
into relation. The numbers below are in the sense of "Numbers everyone 
should know" in [1].

As we can see one very simple DB query (0.068ms) costs as much as 500 
Tcl variable lookups (130 ns), but is about 70 times faster then an 
"exec ls /". The operations in the table are either NaviServer 
primitives, NSF commands, or simple OpenACS commands, from the point of 
view of an application developer (the exec is over nsproxy).

The results show that the nsv read operation (based on rwlock) is faster 
than a proc invocation or an "info command", where in the case of the 
mutex, it is slower.

           86 ns   time {dict get {a 1 b 2 c 3} b} 100000
          130 ns   set x 1; time {info exists x} 100000
          131 ns   set x 1; time {set x} 100000
          140 ns   time {set x 1} 100000
          198 ns   time {ns_quotehtml "hello world"} 100000
          214 ns   set x 1; time {expr {$x + $x}} 100000
          216 ns   nsv_set foo x 1; time {nsv_get foo x} 100000
          248 ns   proc foo {x} {return $x}; time {foo 1} 100000
          273 ns   time {info commands ::db_string} 100000
          313 ns   time {ns_cache_eval ns:memoize 1 {set x 1}} 100000
          319 ns   time {nsv_set foo x 1} 100000
          322 ns   time {array set x {a 1 b 2 c 3}} 100000
          348 ns   time {ns_md5 foo} 100000
          362 ns   time {ns_sha1 foo} 100000
          373 ns   time {lang::util::localize "hello world"} 100000
          485 ns   nx::Class create Foo {:public method bar {} {return 0};:create ::foo}; time {::foo bar} 100000
          776 ns   time {ad_conn subsite_id} 100000
         1820 ns   time {nx::Object create ::o} 100000
         4104 ns   time {nx::Object new} 100000
         6945 ns   time {parameter::get -package_id [ad_conn subsite_id] -parameter DefaultMaster -default "x"} 100000
        25611 ns   time {md5::md5 foo} 100000
        27423 ns   time {sha1::sha1 foo} 100000
        68492 ns   set id 252; time {xo::dc get_value -prepare int qn {select title from acs_objects where object_id=:id}} 100000
        90712 ns   time {xo::dc get_value dbqd..qn {select title from acs_objects where object_id=252}} 100000
       103241 ns   time {db_string dbqd..qn {select title from acs_objects where object_id=252}} 100000
       156529 ns   time {set F [open /tmp/nix w]; puts $F x; close $F} 10000
      4760448 ns   time {exec ls /} 1000

times with mutex locks

          293 ns   nsv_set foo x 1; time {nsv_get foo x} 100000
          354 ns   time {nsv_set foo x 1} 100000

This is a test with very little contention, where the previous tests 
i've posted were with very high contention. But certainly, applications 
might be different.

Since we want to have on the longer range binary distributions of 
NaviServer, i implemented the option (c) from above, such that the 
decision of using rwlocks or mutex operation can be done (1) at startup 
time and (2) per server. Also, the documentation of NaviServer is updated.

all the best

-g

[1] 
https://stackoverflow.com/questions/4087280/approximate-cost-to-access-various-caches-and-main-memory