naviserver-devel — Developer discussion

Re: [naviserver-devel] dbi_dml

[Stephen <yo...@gr...>]

Well, that doesn't make it available to the code that needs it...

So add numRowsHint to the public portion of Dbi_Handle. Set it to
DBI_NUM_ROWS_UNKNOWN before passing it to the driver in Exec. Driver
may fill it in. Upper layers use it if available, but don't rely on
it.

On Fri, Feb 7, 2014 at 1:25 AM, Stephen <yo...@gr...> wrote:
> This used to work. I *think* what happened is it got overlooked when I
> implemented the sqlite driver and had to change the driver interface
> to accommodate it (sqlite doesn't return nrows - you have to step the
> state machine yourself).
>
> Maybe numRows should be added back as a *numRowsHint arg in the driver
> Exec() callback, and one valid value should be -1 or
> DBI_NUM_ROWS_UNKNOWN.
>
> The sqlite driver would set -1 for a normal query, but would set it
> correctly for DML. The other drivers could always set this correctly,
> but the higher level code would treat it as a hint only, to for
> example pre-allocate a Tcl list of the correct size for results. The
> counting code needs to be maintained for sqlite.
>
> I don't think it existed at the time (hence the FIXME in tclcmds.c)
> but sqlite3_changes() now gives you rows-affected for DML.
>
>
>
> On Thu, Feb 6, 2014 at 9:56 PM, Gustaf Neumann <ne...@wu...> wrote:
>> For the time being, i've used (abused?) the public
>> Dbi_Handle->rowIdx to return in the DML case the
>> number of affected rows. This has the advantage,
>> that all data-structures are unchanged, and the
>> modifications are quite local.
>>
>> -g
>>
>> Am 06.02.14 14:12, schrieb Gustaf Neumann:
>>> Hi Stephen,
>>>
>>> in contrary to the documentation, dbi_dml does not return the number
>>> of rows affected by the query. Typically the dbi_* tcl commands
>>> are implemented via Exec() followed by a retrieving cmd, like in
>>>
>>> dbi_1row does an Exec() + NextRow()
>>> dbi_rows does an Exec() + many NextRow(),
>>> etc.
>>>
>>> but dbi_dml does the Exec() and nothing after that to
>>> retrieve the result (which is in pg: PQcmdTuples(res)).
>>>
>>> What is your suggestion to fix this?
>>> - extend the record of Dbi_DriverProc? (new function)
>>> - mangle NextRow() to return the result in the dml case
>>> - extend the Dbi_Handle structure such that Exec()
>>>   can return this value directly?
>>> others?
>>>
>>> -gustaf
>>>
>>
>>
>> --
>> Univ.Prof. Dr. Gustaf Neumann
>> WU Vienna
>> Institute of Information Systems and New Media
>> Welthandelsplatz 1, A-1020 Vienna, Austria
>>
>>
>> ------------------------------------------------------------------------------
>> Managing the Performance of Cloud-Based Applications
>> Take advantage of what the Cloud has to offer - Avoid Common Pitfalls.
>> Read the Whitepaper.
>> http://pubads.g.doubleclick.net/gampad/clk?id=121051231&iu=/4140/ostg.clktrk
>> _______________________________________________
>> naviserver-devel mailing list
>> nav...@li...
>> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] dbi_dml

[Stephen <yo...@gr...>]

This used to work. I *think* what happened is it got overlooked when I
implemented the sqlite driver and had to change the driver interface
to accommodate it (sqlite doesn't return nrows - you have to step the
state machine yourself).

Maybe numRows should be added back as a *numRowsHint arg in the driver
Exec() callback, and one valid value should be -1 or
DBI_NUM_ROWS_UNKNOWN.

The sqlite driver would set -1 for a normal query, but would set it
correctly for DML. The other drivers could always set this correctly,
but the higher level code would treat it as a hint only, to for
example pre-allocate a Tcl list of the correct size for results. The
counting code needs to be maintained for sqlite.

I don't think it existed at the time (hence the FIXME in tclcmds.c)
but sqlite3_changes() now gives you rows-affected for DML.



On Thu, Feb 6, 2014 at 9:56 PM, Gustaf Neumann <ne...@wu...> wrote:
> For the time being, i've used (abused?) the public
> Dbi_Handle->rowIdx to return in the DML case the
> number of affected rows. This has the advantage,
> that all data-structures are unchanged, and the
> modifications are quite local.
>
> -g
>
> Am 06.02.14 14:12, schrieb Gustaf Neumann:
>> Hi Stephen,
>>
>> in contrary to the documentation, dbi_dml does not return the number
>> of rows affected by the query. Typically the dbi_* tcl commands
>> are implemented via Exec() followed by a retrieving cmd, like in
>>
>> dbi_1row does an Exec() + NextRow()
>> dbi_rows does an Exec() + many NextRow(),
>> etc.
>>
>> but dbi_dml does the Exec() and nothing after that to
>> retrieve the result (which is in pg: PQcmdTuples(res)).
>>
>> What is your suggestion to fix this?
>> - extend the record of Dbi_DriverProc? (new function)
>> - mangle NextRow() to return the result in the dml case
>> - extend the Dbi_Handle structure such that Exec()
>>   can return this value directly?
>> others?
>>
>> -gustaf
>>
>
>
> --
> Univ.Prof. Dr. Gustaf Neumann
> WU Vienna
> Institute of Information Systems and New Media
> Welthandelsplatz 1, A-1020 Vienna, Austria
>
>
> ------------------------------------------------------------------------------
> Managing the Performance of Cloud-Based Applications
> Take advantage of what the Cloud has to offer - Avoid Common Pitfalls.
> Read the Whitepaper.
> http://pubads.g.doubleclick.net/gampad/clk?id=121051231&iu=/4140/ostg.clktrk
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] ns_dbi

[Stephen <yo...@gr...>]

On Thu, Feb 6, 2014 at 5:17 PM, Wolfgang Winkler
<wol...@di...> wrote:
> Hello!
>
> I've added a datatype "dict" to the list of possible DbiResults. "dicts"
> returns a list of dicts, whereas "dict" returns a dict. The keys for the first
> level is a simple counter. So instead of
>
> {user_id 1 user_name Max} {user_id 2 user_name Lisa}
>
> 1 {user_id 1 user_name Max} 2 {user_id 2 user_name Lisa}
>
> There is a slight performance hit compared to "dicts", but the result is more
> useful, as you can do the following:
>
> dict for {k v} $result {
>         set user_id [dict get $v user_id]
>         dict set  result $k user_url "user?user_id=$user_id"
> }

Hmm, is the above better than the following:

    set rows [list [list c1 v1 c2 v2] [list c1 vv1 c2 vv2]]

    foreach row $rows {
        lappend rowdicts [dict set row c3 "c3=[dict get $row c2]"]
    }

    set rowdicts
    {c1 v1 c2 v2 c3 c3=v2} {c1 vv1 c2 vv2 c3 c3=vv2}

I guess the idea is that 'dict set result k1 k2 v1' is efficient, but
I would guess that the implementation above is about the same, due to
non-obvious dict duplication.

dicts are copy-on-write. If the dict is unshared then no one will see
the mutation so no copy is needed, but in the case above the row-dict
is shared.

dbi_rows returns a list and all the elements in the list have their
ref count incremented to  by it, and they are therefore unshared.

The v in 'dict for {k v} ...' refers to the list which holds the keys
and values of the row, and as v is a new variable passed to the script
block it has it's ref count incremented, which is now 2.

'dict get $v ...' shimmers the list to a dict, ref-count still 2.

'dict set result ...' navigates the outer dict, which was shimmered by
'dict for ...', with refcount hopefully 1, to the inner dict, with
refcount 2, copies it because it won't mutate a shared object, adds
the new key and value to the copy, puts the copy back in the outer
dict.

So, the row is created as a list. The the row is created as a dict. It
is created again as a dict, and the old dict is discarded.

You can see the effect here:

    set rows [list 1 [list c1 v1 c2 v2] 2 [list c1 vv1 c2 vv2]]

    dict for {rownum listydict} $rows {
        puts "(before) rownum: $rownum listydict: $listydict"
        puts "         rows: $rows"

        set c2 [dict get $listydict c2]
        dict set rows $rownum c3 "c3=$c2"

        puts "(after) rownum: $rownum listydict: $listydict"
        puts "         rows: $rows\n"
    }

    (before) rownum: 1 listydict: c1 v1 c2 v2
             rows: 1 {c1 v1 c2 v2} 2 {c1 vv1 c2 vv2}
    (after) rownum: 1 listydict: c1 v1 c2 v2
             rows: 1 {c1 v1 c2 v2 c3 c3=v2} 2 {c1 vv1 c2 vv2}

    (before) rownum: 2 listydict: c1 vv1 c2 vv2
             rows: 1 {c1 v1 c2 v2 c3 c3=v2} 2 {c1 vv1 c2 vv2}
    (after) rownum: 2 listydict: c1 vv1 c2 vv2
             rows: 1 {c1 v1 c2 v2 c3 c3=v2} 2 {c1 vv1 c2 vv2 c3 c3=vv2}

In 'after' the first row of rows has the new column but listydict does
not, even though it was a reference to it at the start of the loop.


How about something more like this:

    set rows [dbi_rows -extend {{c3 c4} {
        set c3 "c3=$c1"
        set c4 "c4=$c2"
    }} {
        select c1, c2 from table limit 2
    }]

    v1 v2 c3=v1 c4=v2 vv1 vv2 c3=vv1 c4=vv2

This is the flat-list version, which has advantages over the array-get
format mentioned in the other thread, but for the purposes of -extend
it doesn't matter.

It could works like this:

After the first row of values has been appended to the rows list, if
-extend was given then that number of empty objects are appended
before starting the next row.

Once all rows have been appended the db handle is released.

Now skip back to the empty objects and run the extend-script to fill
them in (should look closely at how the modern Tcl does this -- looks
like they have some new stuff to prevent variable clobbering).

I think this is an acceptable trick for something like -extend but not
dbi_foreach because there is no expectation you are working on one row
at a time: dbi_rows returns the whole result set. Nested queries here
are the equivalent of serial queries.

Now, you could do the above also with listy-dicts, but how about this:

    dbi_dicts -enumerate 1 $cols $rows

...which would return a list of 2-element lists, the first being
either a 0 or 1 based counter, the second an actual dict composed of
the zipped cols 'n row. All the keys 'n values in the dicts are just
ref-counted from the cols 'n rows, so the cols values will be well
shared. The dicts are created exactly once.

Re: [naviserver-devel] dbi_dml

[Gustaf N. <ne...@wu...>]

For the time being, i've used (abused?) the public
Dbi_Handle->rowIdx to return in the DML case the
number of affected rows. This has the advantage,
that all data-structures are unchanged, and the
modifications are quite local.

-g

Am 06.02.14 14:12, schrieb Gustaf Neumann:
> Hi Stephen,
>
> in contrary to the documentation, dbi_dml does not return the number
> of rows affected by the query. Typically the dbi_* tcl commands
> are implemented via Exec() followed by a retrieving cmd, like in
>
> dbi_1row does an Exec() + NextRow()
> dbi_rows does an Exec() + many NextRow(),
> etc.
>
> but dbi_dml does the Exec() and nothing after that to
> retrieve the result (which is in pg: PQcmdTuples(res)).
>
> What is your suggestion to fix this?
> - extend the record of Dbi_DriverProc? (new function)
> - mangle NextRow() to return the result in the dml case
> - extend the Dbi_Handle structure such that Exec()
>   can return this value directly?
> others?
>
> -gustaf
>


-- 
Univ.Prof. Dr. Gustaf Neumann
WU Vienna
Institute of Information Systems and New Media
Welthandelsplatz 1, A-1020 Vienna, Austria

Re: [naviserver-devel] ns_dbi

[Gustaf N. <ne...@wu...>]

Am 06.02.14 18:17, schrieb Wolfgang Winkler:
> I've added a datatype "dict" to the list of possible DbiResults.
...
> dict for {k v} $result {
> 	set user_id [dict get $v user_id]
> 	dict set  result $k user_url "user?user_id=$user_id"
> }
ah, nice... updating the dict as result set is conveniant
I hear Stephen in the background saying, why not

    set flat [dbi_rows -cols cols { select .... }]
    set dict [dbi_convert -format dict $cols $flat]
    dict for {k v} $dict {
       ....
    }

will do some more tests about the "built-in" result types vs. the flat 
list converter
over the weekend....

all the best
-gn

Re: [naviserver-devel] ns_dbi

[Wolfgang W. <wol...@di...>]

Hello!

I've added a datatype "dict" to the list of possible DbiResults. "dicts" 
returns a list of dicts, whereas "dict" returns a dict. The keys for the first 
level is a simple counter. So instead of

{user_id 1 user_name Max} {user_id 2 user_name Lisa}

1 {user_id 1 user_name Max} 2 {user_id 2 user_name Lisa}

There is a slight performance hit compared to "dicts", but the result is more 
useful, as you can do the following:

dict for {k v} $result {
	set user_id [dict get $v user_id]
	dict set  result $k user_url "user?user_id=$user_id"
}

We are using dict result sets in our templating system frequently. With the 
patch we could switch to ns_dbi very easily and get the performance bonus with 
only a few hours of work.

Regards,

Wolfgang

[naviserver-devel] dbi_dml

[Gustaf N. <ne...@wu...>]

Hi Stephen,

in contrary to the documentation, dbi_dml does not return the number
of rows affected by the query. Typically the dbi_* tcl commands
are implemented via Exec() followed by a retrieving cmd, like in

dbi_1row does an Exec() + NextRow()
dbi_rows does an Exec() + many NextRow(),
etc.

but dbi_dml does the Exec() and nothing after that to
retrieve the result (which is in pg: PQcmdTuples(res)).

What is your suggestion to fix this?
- extend the record of Dbi_DriverProc? (new function)
- mangle NextRow() to return the result in the dml case
- extend the Dbi_Handle structure such that Exec()
   can return this value directly?
others?

-gustaf

Re: [naviserver-devel] commit/nsdbi: 3 new changesets

[Gustaf N. <ne...@wu...>]

Am 04.02.14 01:35, schrieb Stephen:
>
> On Mon, Feb 3, 2014 at 2:40 PM, Gustaf Neumann <ne...@wu... 
> <mailto:ne...@wu...>> wrote:
>
>
>     converting ns-sets naively into local variables will clobber other
>     locals,
>     using some strange array names etc. would still require to modify
>     all functions accessing the ns-sets, or passing the ns-sets around
>     etc.
>
>
>
> db_foreach already unpacks it's row sets into local variables.
somehow we are arguing in circles. yes, db_foreach does it.
When dbi is used for OpenACS, there are situations, where
one wants set all column values into local variables (as it is
the case when implementing db_foreach on top of dbi), there are other
cases, where one does not want this, therefore dicts, sets, etc
are needed.
> Without a C implementation only Tcl command dispatch overhead is being 
> tested.
sorry, did not have the time so far to implement the structure conversion
routines in C. By implementing this in C, one make more optimizations
when processing the flat list, since the is no need to set the columns
into local variables (what the "foreach $columns ..." does), etc.
Maybe i find some time-slots towards end of the week or on
the weekend.
>
>     in our current production environment (with ns_db) requests to the
>     application server require on average about 9 SQL queries
>     (but up to *700* SQL queries uncached for the personalized start
>     page).
>
>
>
> This is what happens when you make it easy to nest db_foreach :-)
you are quick in drawing conclusions. About 10% of these SQL
queries are from setup (obtaining names of user, communities,
applications, ...), 40 % are from collecting and summarizing
information from about 50 communities  (courses,
organizational units, clubs, ...) and the largest number is
from permission checking. With a warm cache, less than 10%
of the queries are needed in the current implementation (which
can/should/will be improved)

It is "real world" code. By replacing db_foreach by a dbi_rows,
the number of queries won't change, the nesting can be reduced
slightly (the 700 SQL queries use two handles, with the warm
cache, one handle).

-gn

Re: [naviserver-devel] commit/nsdbi: 3 new changesets

[Stephen <yo...@gr...>]

On Mon, Feb 3, 2014 at 2:40 PM, Gustaf Neumann <ne...@wu...> wrote:

>
> converting ns-sets naively into local variables will clobber other locals,
> using some strange array names etc. would still require to modify
> all functions accessing the ns-sets, or passing the ns-sets around etc.
>


db_foreach already unpacks it's row sets into local variables.


 No, this:
>
>     set rows [dbi_rows -columns cols -- $sql]
>     foreach d [dbi_dicts $cols $rows] {
>         puts [dict get $d k]
>     }
>
> ...is faster and uses less memory than this:
>
>     foreach row [dbi_rows -result dicts -- $sql] {
>         puts [dict get $row k]
>     }
>
>  No, its not. The data duplication and double iteration are still needed
> in your suggested approach. Due to the lazy dict conversion there
> might be many cases, where not all dicts are created.
> But even, when all lists are converted to dicts, you suggestion is slower.
> Look at the following tests:
>
> p1: dbi_rows returns a flat list, convert the flat list to a list of
> lists, convert  lazy to dicts
> p2: dbi_rows returns a flat list, convert the flat list to a list of
> dicts, use the dicts
> p3: dbi_returns a list of lists, convert  lazy to dicts
>
> The times for retrieving 1000 values with 14 columns via SQL and to
> process as indicated are
> (micro seconds average of 100 runs).
>
>    p1    35692
>    p2    45767
>    p3    29407
>
> The experiment indicates, that p2 = p3 * 1.56.
> The built-in solution (p3) is much faster, appending to the list and
> convert lazy to the dict (p1) is faster than straight dict creation (p2)
>


All you're measuring here is 14,000 extra calls to the dict command in p2
because it's written in Tcl not C.



>  If we look at ns_sets, we see that ns-sets better than some people
> might suggest:
>
> p4: dbi_rows returns a flat list, convert the flat list to a list of
> ns-sets, access all columns in the loop
> p5: dbi_returns a list of lists, convert  rows to ns-sets, access all
> columns in the loop
> p6: dbi_returns a list of ns-sets, access all columns in the loop
>
>    p4    41607
>    p5    25508
>    p6    16095
>
> Interestingly, p6 is the fastest of all tests, and p4 is the slowest of
> the ns-set tests
> (p4 = p6 * 2.59). also unexpected to me, p5 is faster than p3, since the
> list-of-lists is converted to an ns-set, which is faster than the dict
> creation.
>


Similar problem. p4 is slower than p5 because because there's an
intermediate step coded in Tcl.

p5 is faster than p3 because an ns_set is faster to construct: no hash
table, no Tcl objects.

However, although the comments say "access all columns in the loop" the
test code shows only one column is accessed (object_id, which might well be
the first column). The ns_set uses a linear scan to find keys so I would
expect that the more lookups, the more columns, the slower sets will be
compared to dicts.



>
> These results can be made more similar by coding the flat-to-struct
> statements
> in C, but i see no indication that the eager dict generation will be
> better than
> the list operations used now.
>


Without a C implementation only Tcl command dispatch overhead is being
tested.



> in our current production environment (with ns_db) requests to the
> application server require on average about 9 SQL queries
> (but up to *700* SQL queries uncached for the personalized start
> page).
>


This is what happens when you make it easy to nest db_foreach :-)

If you could get rid of that, you could reduce the number of database
backends, reduce contention, and use some of the saved memory for more
buffer cache, or extra indexes.

Re: [naviserver-devel] commit/nsdbi: 3 new changesets

[Gustaf N. <ne...@wu...>]

If someone want to make experiments, here is the code used for testing
-gn



set ::sql {select * from acs_objects limit 10000}

proc dbi_dicts1 {cols rows} {
     set dicts {}
     foreach $cols $rows {
	set dict ""
	foreach __c $cols {
	    lappend dict $__c [set $__c]
	}
	lappend dicts $dict
     }
     return $dicts
}

proc dbi_dicts2 {cols rows} {
     set dicts {}
     foreach $cols $rows {
	foreach __c $cols {
	    dict set dict $__c [set $__c]
	}
	lappend dicts $dict
     }
     return $dicts
}

proc dbi_sets {cols rows} {
     set sets {}
     foreach $cols $rows {
	set dict ""
	foreach __c $cols {
	    lappend dict $__c [set $__c]
	}
	lappend sets [ns_set create r {*}$dict]
     }
     return $sets
}

proc p1 {} {
     set sum 0
     set count 0
     set rows [dbi_rows -columns cols -max 1000000 -- $::sql]
     foreach d [dbi_dicts1 $cols $rows] {
	incr sum [dict get $d object_id]
	incr count
     }
     return [list $count $sum]
}

proc p2 {} {
     set sum 0
     set count 0
     set rows [dbi_rows -columns cols -max 1000000 -- $::sql]
     foreach d [dbi_dicts2 $cols $rows] {
	incr sum [dict get $d object_id]
	incr count
     }
     return [list $count $sum]
}

proc p3 {} {
     set sum 0
     set count 0
     foreach row [dbi_rows -result dicts -max 1000000 -- $::sql] {
         incr sum [dict get $row object_id]
	incr count
     }
     return [list $count $sum]
}

proc p4 {} {
     set sum 0
     set count 0
     set rows [dbi_rows -columns cols -max 1000000 -- $::sql]
     foreach row [dbi_sets $cols $rows] {
	incr sum [ns_set get $row object_id]
	incr count
     }
     return [list $count $sum]
}

proc p5 {} {
     set sum 0
     set count 0
     foreach dict [dbi_rows -result dicts -max 1000000 -- $::sql] {
	set row [ns_set create x {*}$dict]
	incr sum [ns_set get $row object_id]
	incr count
     }
     return [list $count $sum]
}

proc p6 {} {
     set sum 0
     set count 0
     foreach row [dbi_rows -result sets -max 1000000 -- $::sql] {
         incr sum [ns_set get $row object_id]
	incr count
     }
     return [list $count $sum]
}

Re: [naviserver-devel] commit/nsdbi: 3 new changesets

[Gustaf N. <ne...@wu...>]

Am 03.02.14 02:39, schrieb Stephen:
> How about introducing a _db_foreach helper to replace the
> stereotypical while loop outlined above -- something with an interface
> more amenable to talking with both dbi and nsdb. Looking briefly at
> db_foreach and db_multirow, it looks like some kind of helper could
> clean up turning the sets into local variables without touching the
> bulk of the code around it. Then sets could be avoided entirely when
> using dbi.
converting ns-sets naively into local variables will clobber other locals,
using some strange array names etc. would still require to modify
all functions accessing the ns-sets, or passing the ns-sets around etc.
> By legacy I mean before dicts, sets were a useful alternative to the
> weird by-name arrays, but not any more. And it looks like you're
> focused on some kind of xo orm, so the bits under the hood which
> connect db_* with dbi are to keep legacy code working as more stuff is
> moved to xo. With that in mind, it seems a shame to re-surface sets in
> the dbi api, to be used in code no one's going to see, especially when
> there are alternatives.
i am still exploring, what's feasible for OpenACS. there is both, a orm 
mapping
and a db-abstraction layer in the xotcl-core packages, which work together.
Since code-wise, most parts of OpenACS are not object oriented, this db 
abstraction
layer is a good candidate for reuse in wider parts of OpenACS. This code is
handle-free, and simpler (it not deal with the huge number seldomly
used features of ns_db*)
>
>>> 2) New generic xo functions
>>>
>>> I'm not following how the implementation as it is now, which returns a
>>> list of lists with duplicated column names, is faster than just
>>> returning a list of dicts. You have to feed the lists to dict create
>>> before you use it, which is extra work and always going to be slower.
>> i am not following you here. The print string of a tcl dict is a tcl
>> list. there is no need to use "dict create". try:
>>
>>     dict get {a 1 b 2} b
>>
>> what i said in the comment was that adding elements to a list is faster
>> than adding entries to the dict in the loop iterating over the columns.
>> Furthermore, the list without the internal representation of the dict
>> (essentially the hash table) requires less memory.
> 'dict get' is implemented by DictGetCommand:
sure, i know what "dict get" does. I (mis?)read from your argument, that
an explicit "dict create" is required. this lazy implementation is
quite useful (see below).
>> you say that
>>
>>       set results [dbi_rows -colums cols -- *SQL*]
>>       set dicts [dbi_dicts $results $cols]
>>
>> is faster than
>>
>>       set dicts [dbi_rows -result dicts -- *SQL*]
> No, this:
>
>      set rows [dbi_rows -columns cols -- $sql]
>      foreach d [dbi_dicts $cols $rows] {
>          puts [dict get $d k]
>      }
>
> ...is faster and uses less memory than this:
>
>      foreach row [dbi_rows -result dicts -- $sql] {
>          puts [dict get $row k]
>      }
No, its not. The data duplication and double iteration are still needed
in your suggested approach. Due to the lazy dict conversion there
might be many cases, where not all dicts are created.
But even, when all lists are converted to dicts, you suggestion is slower.
Look at the following tests:

p1: dbi_rows returns a flat list, convert the flat list to a list of 
lists, convert  lazy to dicts
p2: dbi_rows returns a flat list, convert the flat list to a list of 
dicts, use the dicts
p3: dbi_returns a list of lists, convert  lazy to dicts

The times for retrieving 1000 values with 14 columns via SQL and to 
process as indicated are
(micro seconds average of 100 runs).

    p1    35692
    p2    45767
    p3    29407

The experiment indicates, that p2 = p3 * 1.56.
The built-in solution (p3) is much faster, appending to the list and
convert lazy to the dict (p1) is faster than straight dict creation (p2)

If we look at ns_sets, we see that ns-sets better than some people
might suggest:

p4: dbi_rows returns a flat list, convert the flat list to a list of 
ns-sets, access all columns in the loop
p5: dbi_returns a list of lists, convert  rows to ns-sets, access all 
columns in the loop
p6: dbi_returns a list of ns-sets, access all columns in the loop

    p4    41607
    p5    25508
    p6    16095

Interestingly, p6 is the fastest of all tests, and p4 is the slowest of 
the ns-set tests
(p4 = p6 * 2.59). also unexpected to me, p5 is faster than p3, since the
list-of-lists is converted to an ns-set, which is faster than the dict 
creation.
p6 has the advantage that for 1000 tuples and 14 cols the result list
contain just 1000 (small) entries, where the flat list contains 14000 
entries.

Below are some more tests with result-set sizes 100, 1,000 and 10,000.
All results are normalized to the fastest results (always p6).





These results can be made more similar by coding the flat-to-struct 
statements
in C, but i see no indication that the eager dict generation will be 
better than
the list operations used now. My internal drive to drop the fastest solution
(with ns-sets) is not too big. ns-sets have the disadvantage of not being
cachable, but i am not a big fan of caching raw sql row-sets at all.


>> what is the problem with a dbi_foreach in the dbi infrastructure of the
>> form:
>>
>>      dbi_foreach ?-db name? ?-autonull? ?-timeout t? ?-bind bindSource?
>> ?-max nrows? ?--? query ?body?
>>
> Because client-side nested queries are a bad idea in a web server
> environment and it was an explicit design goal of nsdbi to prevent
> them.
>
> ....
Non of these arguments look as show-stoppers to me. I am not a fan
of db_foreach either, and yes, globbering etc. is bad. In OpenACS core
+ packages i count 1490 db_foreach and 1697 db_multirow
(where the latter concerns me less). The 1490 db_foreach
have these properties, and it is unclear, how many of
these require break etc.

It is to easy to say, "get your sql queries right" than you need
no calls to the tcl api to restrict the answer set. This means
that for every call in the tcl-api in the body of the db_foreach loop
partitioning the results, one needs as well a SQL implementation
of that same functionality. OpenACS on our production
environment has more than 1000 tables and views,
many tables have 10mio+ tuples, and our environment is
not special about this. Relying there on a maxrow limit
and returning the full sets does not seem practical.

...
> Another downside of foreach is that it clobbers local variables, which
> was the original motivation for adding dicts. (dbi_rows with a
> template doesn't have this problem).
dbi_rows + templating is pretty limited, since it supports
just hard-coded output, not some computed output
(widgets with a tcl api, tdom etc.). it is not feasible to
code this in sql.
>> PS: using the OpenACS based xowiki with dbi (and the functions in
>> mercurial head) lead to a 20% performance improvement with very little
>> changes.
> Neat.
>
> I wonder if you'll notice improvements under load, in real life
> conditions.
that would require to get more of the OpenACS ns_db interface
kind of working with dbi_*

> The original micro benchmarks were impressive, but there's
> also the effects of handle management.
sure, these tests benefit also over-proportional from prepare
(running every query 100 times in sequence), but running
every query just once would be even less realistic.

The overall speedup is hard to compare, especially when
comparing OpenACS sites with non-OpenACS sites.
If one has a smaller tcl layer than OpenACS, the speed
improvement will be much higher than the measured 20%.
but this is already very notable.
> You have a choice between fewer
> back-ends, due to shorter lifetimes, or more with the 1-1 conn-thread
> to db backend mode, if you discover that on average every conn thread
> uses a db backend.
in our current production environment (with ns_db) requests to the
application server require on average about 9 SQL queries
(but up to 700 SQL queries uncached for the personalized start
page). We are still running there behind an nginx for static files.
These SQL queries are often served from multiple handles
(OpenACS has three pools for primary, secondary and
tertiary requests, which are used for nesting sql queries,
e.g. in db_foreach). Therefore every active connection
thread on the backend uses typically 1 or more db-backends.
The db-pools have up to 100 SQL handles.

all the best
-gn

Re: [naviserver-devel] commit/nsdbi: 3 new changesets

[Stephen <yo...@gr...>]

On Sun, Feb 2, 2014 at 9:16 PM, Gustaf Neumann <ne...@wu...> wrote:
>
> Am 02.02.14 19:36, schrieb Stephen:
> >
> > The original concern was that a generic utility function which wraps
> > dbi_rows and uses the typical foreach pattern might have it's private
> > variables clobbered by column names it can't predict. Returning a list
> > of dicts from dbi_rows is one way around that.
> >
> > Looks like there's a couple of different situations where wrappers are
> > wanted:
> >
> > 1) Legacy ACS db_* functions
> >
> > Looks like a lot of that code unpacks row ns_sets into local vars
> > anyway. db_multirow even has an -unclobber switch which restores the
> > original values of any vars which clash with column names.
> >
> > It seems like a bit of a backwards step to add legacy sets to the dbi
> > api only to have the legacy db_* stuff unpack it again.
>
> The best approach for OpenACS would be to rewrite the db-interface based
> on dbi_*. but this has currently at least two show-stoppers: dbi has no
> oracle support, and time (too many ns_db calls scattered all over the
> code, and ns_db has a very wide interface).
>
> i guess you refer by "legacy sets" to the added option "-result sets".
> i've done this to cope with legacy code of the following form:
>
>          while { [::db_getrow $db $answers] } {
>              ...
>              ... large body doing something with the ns_set $answers
>              ...
>          }
>
> To keep the code working with the old ns_db interface and to support as
> well additionally dbi, having "dbi_rows" to return a list of ns_sets
> comes very handy. One can replace the ns_set collecting code by dbi, but
> one does not have to rewrite the code extracting values from the ns_sets.
>
> One could certainly as well use the dicts option for dbi_* and covert
> the list of dicts to a list of ns_sets in tcl, which is certainly much
> slower (same with the flat result list).

How about introducing a _db_foreach helper to replace the
stereotypical while loop outlined above -- something with an interface
more amenable to talking with both dbi and nsdb. Looking briefly at
db_foreach and db_multirow, it looks like some kind of helper could
clean up turning the sets into local variables without touching the
bulk of the code around it. Then sets could be avoided entirely when
using dbi.

By legacy I mean before dicts, sets were a useful alternative to the
weird by-name arrays, but not any more. And it looks like you're
focused on some kind of xo orm, so the bits under the hood which
connect db_* with dbi are to keep legacy code working as more stuff is
moved to xo. With that in mind, it seems a shame to re-surface sets in
the dbi api, to be used in code no one's going to see, especially when
there are alternatives.

> > 2) New generic xo functions
> >
> > I'm not following how the implementation as it is now, which returns a
> > list of lists with duplicated column names, is faster than just
> > returning a list of dicts. You have to feed the lists to dict create
> > before you use it, which is extra work and always going to be slower.
>
> i am not following you here. The print string of a tcl dict is a tcl
> list. there is no need to use "dict create". try:
>
>    dict get {a 1 b 2} b
>
> what i said in the comment was that adding elements to a list is faster
> than adding entries to the dict in the loop iterating over the columns.
> Furthermore, the list without the internal representation of the dict
> (essentially the hash table) requires less memory.

'dict get' is implemented by DictGetCommand:

    http://core.tcl.tk/tcl/artifact/cd2f2a05bbb75127

The first substantial thing it does is call TclTraceDictPath to
resolve the keys, and the first thing that does is call
SetDictFromAny, which uses the elements of the list to create the dict
internal representation.

So although it is faster to create nested lists than dicts, you create
the dicts lazily on first access anyway.

> > For any other purpose nested lists are no different than the original
> > flat list with column list, and don't address the problem of variable
> > clobbering.
> >
> > How about this:
> >
> >     dbi_dicts cols rows
> >
> > Returns a list of dicts by zipping up the cols and rows lists returned
> > by dbi_rows. It's faster than the current implementation as dbi_rows
> > only has to construct a flat list rather than the nested list with
> > duplicated keys, and that means less data to cache in ns_cache, and
> > parse again when deserialising.
>
> you say that
>
>      set results [dbi_rows -colums cols -- *SQL*]
>      set dicts [dbi_dicts $results $cols]
>
> is faster than
>
>      set dicts [dbi_rows -result dicts -- *SQL*]

No, this:

    set rows [dbi_rows -columns cols -- $sql]
    foreach d [dbi_dicts $cols $rows] {
        puts [dict get $d k]
    }

...is faster and uses less memory than this:

    foreach row [dbi_rows -result dicts -- $sql] {
        puts [dict get $row k]
    }

..the way you have it currently implemented with nested lists, because
$row shimmers to a dict on first access.

> hard to believe for me. the first form has to create an two more
> variables and has to iterate over the solutions twice.
> btw, the code in the tip behaves as ever when the option "-result" is
> not used.
> >
> > Generalising, it could be something like:
> >
> >   dbi_foreach ?-dict name | -array name | -set name? cols rows ?body?
> >
> > I avoided adding the obvious implementation of this because it
> > encourages stuff like nested queries. With dbi_rows you either get a
> > list result or a template filled with subst, but no eval. There's only
> > ever one query active and no visible handle management.
>
> what is the problem with a dbi_foreach in the dbi infrastructure of the
> form:
>
>     dbi_foreach ?-db name? ?-autonull? ?-timeout t? ?-bind bindSource?
> ?-max nrows? ?--? query ?body?
>
> i've prototyped such a thing, and it seems to work fine with nested
> queries.

Because client-side nested queries are a bad idea in a web server
environment and it was an explicit design goal of nsdbi to prevent
them.

One problem is that sql newbies tend to fall back on procedural code
in the client rather than figure out how to write the correct sql.

Another problem is the poor performance of n sub-queries, one for each
row in the driver loop.

Another is that two db backends and two handles are tied up for as
long as it takes the last (slow) sub-query to complete.

> i would not be surprised if there would problems with
> transactions, but i see no reason these can be adressed as well.
>
> dbi_foreach allows break-able loops, where one can step over a query
> with potentially huge amount of results and stop depending on the result
> of an tcl-call.

This is almost always a bad idea because:

- it encourages people to write the wrong sql and fix it up in tcl

- with reasonably sized result sets the query runs to completion in
the database anyway (unless you explicitly use cursors) and transfers
to the client in batches, not row by row, so an early break saves you
nothing

- you need to run code to break, and by allowing code to run you also
allow sub-queries, or even just slow code which keeps the handle busy

nsdbi even has a -max option and a default config of 1000 rows to
guard against faulty queries blowing up the web server (which is the
usual situation with large result sets) and force people to think
about what sort of result they're expecting. This is the n-rows to the
1row and 0or1row.

The remaining use case is bulk loading and extraction, and that's
probably not best handled by a front-end web server. Or at least,
there's probably a solution which doesn't undo the nice properties of
the api which are helpful day to day.

> There is no need to get all results first.
>
>     dbi_foreach *SQL* {
>        ....
>         if {[some-tcl-call $a $b $c]} { break }
>         ...
>     }
>
> How can this be handled with the exising dbi driver? i've no problem
> dropping the code in favor of some better implementation.

Another downside of foreach is that it clobbers local variables, which
was the original motivation for adding dicts. (dbi_rows with a
template doesn't have this problem).

Due to the above points, 'foreach' is probably not the right word, but
maybe something like dbi_zip (like the functional zip), working as
outlined originally with $cols and $rows, would be a tool you could
use to implement an xo orm.

> PS: using the OpenACS based xowiki with dbi (and the functions in
> mercurial head) lead to a 20% performance improvement with very little
> changes.

Neat.

I wonder if you'll notice improvements under load, in real life
conditions. The original micro benchmarks were impressive, but there's
also the effects of handle management. You have a choice between fewer
back-ends, due to shorter lifetimes, or more with the 1-1 conn-thread
to db backend mode, if you discover that on average every conn thread
uses a db backend.

Re: [naviserver-devel] commit/nsdbi: 3 new changesets

[Gustaf N. <ne...@wu...>]

Am 02.02.14 19:36, schrieb Stephen:
>
> The original concern was that a generic utility function which wraps 
> dbi_rows and uses the typical foreach pattern might have it's private 
> variables clobbered by column names it can't predict. Returning a list 
> of dicts from dbi_rows is one way around that.
>
> Looks like there's a couple of different situations where wrappers are 
> wanted:
>
> 1) Legacy ACS db_* functions
>
> Looks like a lot of that code unpacks row ns_sets into local vars 
> anyway. db_multirow even has an -unclobber switch which restores the 
> original values of any vars which clash with column names.
>
> It seems like a bit of a backwards step to add legacy sets to the dbi 
> api only to have the legacy db_* stuff unpack it again.
The best approach for OpenACS would be to rewrite the db-interface based 
on dbi_*. but this has currently at least two show-stoppers: dbi has no 
oracle support, and time (too many ns_db calls scattered all over the 
code, and ns_db has a very wide interface).

i guess you refer by "legacy sets" to the added option "-result sets". 
i've done this to cope with legacy code of the following form:

         while { [::db_getrow $db $answers] } {
             ...
             ... large body doing something with the ns_set $answers
             ...
         }

To keep the code working with the old ns_db interface and to support as 
well additionally dbi, having "dbi_rows" to return a list of ns_sets 
comes very handy. One can replace the ns_set collecting code by dbi, but 
one does not have to rewrite the code extracting values from the ns_sets.

One could certainly as well use the dicts option for dbi_* and covert 
the list of dicts to a list of ns_sets in tcl, which is certainly much 
slower (same with the flat result list).
>
> 2) New generic xo functions
>
> I'm not following how the implementation as it is now, which returns a 
> list of lists with duplicated column names, is faster than just 
> returning a list of dicts. You have to feed the lists to dict create 
> before you use it, which is extra work and always going to be slower.
i am not following you here. The print string of a tcl dict is a tcl 
list. there is no need to use "dict create". try:

   dict get {a 1 b 2} b

what i said in the comment was that adding elements to a list is faster 
than adding entries to the dict in the loop iterating over the columns. 
Furthermore, the list without the internal representation of the dict 
(essentially the hash table) requires less memory.
>
> For any other purpose nested lists are no different than the original 
> flat list with column list, and don't address the problem of variable 
> clobbering.
>
> How about this:
>
>     dbi_dicts cols rows
>
> Returns a list of dicts by zipping up the cols and rows lists returned 
> by dbi_rows. It's faster than the current implementation as dbi_rows 
> only has to construct a flat list rather than the nested list with 
> duplicated keys, and that means less data to cache in ns_cache, and 
> parse again when deserialising.
you say that

     set results [dbi_rows -colums cols -- *SQL*]
     set dicts [dbi_dicts $results $cols]

is faster than

     set dicts [dbi_rows -result dicts -- *SQL*]

hard to believe for me. the first form has to create an two more 
variables and has to iterate over the solutions twice.
btw, the code in the tip behaves as ever when the option "-result" is 
not used.
>
> Generalising, it could be something like:
>
>   dbi_foreach ?-dict name | -array name | -set name? cols rows ?body?
>
> I avoided adding the obvious implementation of this because it 
> encourages stuff like nested queries. With dbi_rows you either get a 
> list result or a template filled with subst, but no eval. There's only 
> ever one query active and no visible handle management.

what is the problem with a dbi_foreach in the dbi infrastructure of the 
form:

    dbi_foreach ?-db name? ?-autonull? ?-timeout t? ?-bind bindSource? 
?-max nrows? ?--? query ?body?

i've prototyped such a thing, and it seems to work fine with nested 
queries. i would not be surprised if there would problems with 
transactions, but i see no reason these can be adressed as well.

dbi_foreach allows break-able loops, where one can step over a query 
with potentially huge amount of results and stop depending on the result 
of an tcl-call. There is no need to get all results first.

    dbi_foreach *SQL* {
       ....
        if {[some-tcl-call $a $b $c]} { break }
        ...
    }

How can this be handled with the exising dbi driver? i've no problem 
dropping the code in favor of some better implementation.
-gn

PS: using the OpenACS based xowiki with dbi (and the functions in 
mercurial head) lead to a 20% performance improvement with very little 
changes.

Re: [naviserver-devel] commit/nsdbi: 3 new changesets

[Stephen <yo...@gr...>]

On Sat, Feb 1, 2014 at 10:42 AM, Bitbucket <com...@bi...>wrote:

> 3 new commits in nsdbi:
>
> https://bitbucket.org/naviserver/nsdbi/commits/c81a764b2dad/
> Changeset:   c81a764b2dad
> User:        gustafn
> Date:        2014-02-01 11:41:41
> Summary:     - added "-result flat|sets|dicts?" to dbi_rows to specify
> output  format
> - added "dict" to allowed option values in "-bind" (allowed are
> array|set|dict?)
> - added the option "-autonull" to the options applicable to all
>    queries. This option automatically converts missing bind variables
>    as null values in the SQL statement
> - extended regression test
> - updated documentation
> Affected #:  4 files


Regarding the back 'n forth on these changes...


https://bitbucket.org/naviserver/nsdbi/commits/c81a764b2dad/#Ltclcmds.cT657

The original concern was that a generic utility function which wraps
dbi_rows and uses the typical foreach pattern might have it's private
variables clobbered by column names it can't predict. Returning a list of
dicts from dbi_rows is one way around that.

Looks like there's a couple of different situations where wrappers are
wanted:

1) Legacy ACS db_* functions

Looks like a lot of that code unpacks row ns_sets into local vars anyway.
db_multirow even has an -unclobber switch which restores the original
values of any vars which clash with column names.

It seems like a bit of a backwards step to add legacy sets to the dbi api
only to have the legacy db_* stuff unpack it again.

2) New generic xo functions

I'm not following how the implementation as it is now, which returns a list
of lists with duplicated column names, is faster than just returning a list
of dicts. You have to feed the lists to dict create before you use it,
which is extra work and always going to be slower.

For any other purpose nested lists are no different than the original flat
list with column list, and don't address the problem of variable clobbering.

How about this:

    dbi_dicts cols rows

Returns a list of dicts by zipping up the cols and rows lists returned by
dbi_rows. It's faster than the current implementation as dbi_rows only has
to construct a flat list rather than the nested list with duplicated keys,
and that means less data to cache in ns_cache, and parse again when
deserialising.

It could have more utility:

  dbi_dicts cols rows ?dict-name body?

...would evaluate body with a dict containing the current row, so that
rownum dicts do not have to exist in memory at once.

Generalising, it could be something like:

  dbi_foreach ?-dict name | -array name | -set name? cols rows ?body?

I avoided adding the obvious implementation of this because it encourages
stuff like nested queries. With dbi_rows you either get a list result or a
template filled with subst, but no eval. There's only ever one query active
and no visible handle management.

[naviserver-devel] forward secrecy for nsssl

[Gustaf N. <ne...@wu...>]

Dear friends,

Google has implemented in 2011 "forward secrecy" via ephemeral keys and 
Diffie-Hellman key exchange in OpenSSL [1].Since this feature of OpenSSL 
this is easy to use, i added support for forward secrecy to nsssl. One 
can new use these improved security features by adding DH parameters [2] 
to the server.pem file (see example in README [3]) and by using the 
"right" ciphers (*E*DH*, see e.g. [4]).

By using these features, a web site can improve its security ratings as 
measured e.g. by Qualys' SSL Labs.

all the best
-gustaf neumann

[1] 
http://googleonlinesecurity.blogspot.co.at/2011/11/protecting-data-for-long-term-with.html
[2] https://bitbucket.org/naviserver/nsssl/src
[3] http://en.wikibooks.org/wiki/OpenSSL/Diffie-Hellman_parameters
[4] https://wiki.mozilla.org/Security/Server_Side_TLS

[naviserver-devel] quick comparison for nsdbi/nsdbipg

[Gustaf N. <ne...@wu...>]

Dear friends,

Here are some nice figures about Stephens great nsdbi/nsdbipg drivers:
http://openacs.org/forums/message-view?message_id=4112487

-gn

[naviserver-devel] naviserver on the raspberry pi

[Gustaf N. <ne...@wu...>]

Dear Friends,

I did some tests with NaviServer on the raspberry pi (see picture 
below), which are quite promising. (The raspberry pi is a " 
credit-card-sized" single-board computer with a ARMv6-compatible 
processor, costing less than 40 euro at amazon).

NaviServer (and Tcl 8.5.15) compiles out of the box (libnsd is 1.9 MB) 
on 2014-01-07-wheezy-raspbian and runs quite well on the pi as the 
following test show. All tests were executed with "ab -n 1000 -c 10 
http://......" where "ab" was executed on my notebook. In test "WLAN", 
the notebook connected via WLAN to the raspberry pi, in test LAN, both 
were on the same LAN. For all tests, I've used the unmodified default 
configuration nsd-config.tcl.

                WLAN      LAN
timer2.adp:   214.56    216.22
mini.html:    286.13    294.53
5k.html:      245.29    236.09


The reported values are "reqs/sec". One can certainly question the 
usefulness of the number that "ab" returns (which are not  very stable) 
but overall this tiny machine feels quite fast, when clicking around....

all the best
-gustaf neumann

Re: [naviserver-devel] dtplite

[David O. <da...@qc...>]

Thanks Gustaf
I'd never come across this before since I always use a debian packaged
tcllib.
It's easy enough to pass DTPLITE=dtplite during the build process so it's a
simple workaround whether the change stays or goes.
Thanks for filling me in.

Regards,
-- 
David

On 9 January 2014 11:06, Gustaf Neumann <ne...@wu...> wrote:

>
> The problem are installations with multiple tcl and multiple tcllib
> installations, where e.g. the public installation of tcl does not contain
> tclllib, or an ancient version of tcllib, or installations not having tcl
> or tcllib installed in the public path. The change has a self-contained
> install dir for naviserver in mind, where e.g. /usr/local/ns contains a
> tclsh and a tcllib (using "--prefix=/usr/local/ns", as it is the case with
> the install scripts for naviserver + openacs from openacs.org). ... but
> you are right, one cannot assume this in general ... but "build-doc" is
> just for developers (the tar file releases of naviserver contain the
> generated manual pages)
>
> i am somewhat reluctant to extend "autoconf" for developers to require
> tcllib (in a matching version).
> For now, one can use "make DTPLITE=dtplite build-doc"; maybe it is better
> to keep the old version around for us few developers, and pass the a
> matching "DTPLITE=..." in for the full install script, that knows where tcl
> and tcllib is residing, ... unless someone has a better suggestion.
>
> -g
>

Re: [naviserver-devel] dtplite

[Gustaf N. <ne...@wu...>]

The problem are installations with multiple tcl and multiple tcllib 
installations, where e.g. the public installation of tcl does not 
contain tclllib, or an ancient version of tcllib, or installations not 
having tcl or tcllib installed in the public path. The change has a 
self-contained install dir for naviserver in mind, where e.g. 
/usr/local/ns contains a tclsh and a tcllib (using 
"--prefix=/usr/local/ns", as it is the case with the install scripts for 
naviserver + openacs from openacs.org). ... but you are right, one 
cannot assume this in general ... but "build-doc" is just for developers 
(the tar file releases of naviserver contain the generated manual pages)

i am somewhat reluctant to extend "autoconf" for developers to require 
tcllib (in a matching version).
For now, one can use "make DTPLITE=dtplite build-doc"; maybe it is 
better to keep the old version around for us few developers, and pass 
the a matching "DTPLITE=..." in for the full install script, that knows 
where tcl and tcllib is residing, ... unless someone has a better 
suggestion.

-g

Am 09.01.14 10:43, schrieb David Osborne:
> Hello,
>
> With regards to the following change... I've since been having 
> problems with make build-doc not being able to find tclsh when trying 
> to run dtplite.
>
> https://bitbucket.org/naviserver/naviserver/commits/43ac5ab18616/
> Changeset:   43ac5ab18616
> User:        gustafn
> Date:        2014-01-04 17:33:39
> Summary:     - use dtplite from $(NAVISERVER)/bin to avoid confusion 
> with other tcl installations on the same machine
> Affected #:  1 file
>
> The build now gives errors like the following:
> /bin/sh: 9: /usr/local/ns/bin/tclsh: not found
> /bin/sh: 13: /usr/local/ns/bin/tclsh: not found
>
> Previously it would pick it up from the installed tcllib, which it 
> sounds like your specifically trying to avoid.
> What is the intention for how tclsh and dtplite is installed to 
> $(NAVISERVER)/bin? Wondering if I am missing a build step at some point..
>
> Regards,
> -- 
> David
>
>
> ------------------------------------------------------------------------------
> CenturyLink Cloud: The Leader in Enterprise Cloud Services.
> Learn Why More Businesses Are Choosing CenturyLink Cloud For
> Critical Workloads, Development Environments & Everything In Between.
> Get a Quote or Start a Free Trial Today.
> http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel


-- 
Univ.Prof. Dr. Gustaf Neumann
WU Vienna
Institute of Information Systems and New Media
Welthandelsplatz 1, A-1020 Vienna, Austria

[naviserver-devel] dtplite

[David O. <da...@qc...>]

Hello,

With regards to the following change... I've since been having problems
with make build-doc not being able to find tclsh when trying to run dtplite.

https://bitbucket.org/naviserver/naviserver/commits/43ac5ab18616/
Changeset:   43ac5ab18616
User:        gustafn
Date:        2014-01-04 17:33:39
Summary:     - use dtplite from $(NAVISERVER)/bin to avoid confusion with
other tcl installations on the same machine
Affected #:  1 file

The build now gives errors like the following:
/bin/sh: 9: /usr/local/ns/bin/tclsh: not found
/bin/sh: 13: /usr/local/ns/bin/tclsh: not found

Previously it would pick it up from the installed tcllib, which it sounds
like your specifically trying to avoid.
What is the intention for how tclsh and dtplite is installed to
$(NAVISERVER)/bin? Wondering if I am missing a build step at some point..

Regards,
-- 
David

Re: [naviserver-devel] tcmalloc and jemalloc results with naviserver

[John B. <jo...@ma...>]

Hi Gustaf, thanks for the suggestion.

I tried adding a zero, and then another zero, to 
export TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES= 167772160

and still had terrible naviserver performance at 100 minthreads.  

When I dropped a digit, ie:
export TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES= 1677721

naviserver failed to work at all, which serves as a good proof that the setting was taking effect.

That being said, I went with your 4xCPU cores suggestion, and set minthreads at 32 to test again. 

At that setting, performance was the same as at minthreads=10, so I can live with that!

-john


On Jan 6, 2014, at 3:05 PM, Gustaf Neumann <ne...@WU...> wrote:

> The "best" number of threads depends on many things, including the hardware (e.g. number of cores). For most cases, i recommend not more than than 4 threads/core. With high number of threads a low numbers of cores, context switching       might degrade performance. 
> 
> concerning 100 threads vs. tcmalloc: you might try to increase TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES (defaults to 16MB). For more details, see:
> http://gperftools.googlecode.com/svn/trunk/doc/tcmalloc.html
> 
> concerning waiting time in threads: a good number of threads can be determined via observing the queuing time and the number of queue requests recorded by naviserver (See in nsstats (from bitbucket) the "process" page). The best number of theads is the lowest number where the number of queued requests is low (e.g. less than 0.1%) or the queuing time is acceptable (e.g. less than 0.1 ms). The numbers certainly depend on the application, but when one has essentially no queued requests with 30 connection threads, then there is no benefit in increasing the number of connection threads.

Re: [naviserver-devel] tcmalloc and jemalloc results with naviserver

[Gustaf N. <ne...@wu...>]

The "best" number of threads depends on many things, including the 
hardware (e.g. number of cores). For most cases, i recommend not more 
than than 4 threads/core. With high number of threads a low numbers of 
cores, context switching might degrade performance.

concerning 100 threads vs. tcmalloc: you might try to increase 
|TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES| (defaults to 16MB). For more 
details, see:
http://gperftools.googlecode.com/svn/trunk/doc/tcmalloc.html

concerning waiting time in threads: a good number of threads can be 
determined via observing the queuing time and the number of queue 
requests recorded by naviserver (See in nsstats (from bitbucket) the 
"process" page). The best number of theads is the lowest number where 
the number of queued requests is low (e.g. less than 0.1%) or the 
queuing time is acceptable (e.g. less than 0.1 ms). The numbers 
certainly depend on the application, but when one has essentially no 
queued requests with 30 connection threads, then there is no benefit in 
increasing the number of connection threads.

-gustaf


Am 06.01.14 14:36, schrieb John Buckman:
>
> On Jan 5, 2014, at 2:27 PM, Wolfgang Winkler 
> <wol...@di... 
> <mailto:wol...@di...>> wrote:
>
>> I was wondering, why you'll want to have 100 threads, because the 
>> number seems a little high to me. So I've just conducted some test on 
>> two of our development system.
>
> You're right, 100 is high, but I do use SQL quite a lot, and so long 
> running threads, where the cpu is blocking, waiting for a SQL server 
> response, can generate a lot of pending threads.
>
> I wanted enough threads to be around to handle short running page 
> requests too.
>
> But besides that, it's very odd that tcmalloc had this huge decrease 
> in performance at 100 threads.  It's perhaps not enough of a reason to 
> avoid tcmalloc, but it's a cause to worry, nonetheless.
>
> Gustaf's tests didn't show that same slowdown result, and the big 
> slowdown I have at 100 threads only happens with tcmalloc, not with 
> jemalloc or zippy, with a simple [clock seconds] ADP page.
>
> -john
>
>

Re: [naviserver-devel] tcmalloc and jemalloc results with naviserver

[John B. <jo...@ma...>]

On Jan 5, 2014, at 2:27 PM, Wolfgang Winkler <wol...@di...> wrote:

> I was wondering, why you'll want to have 100 threads, because the number seems a little high to me. So I've just conducted some test on two of our development system.

You're right, 100 is high, but I do use SQL quite a lot, and so long running threads, where the cpu is blocking, waiting for a SQL server response, can generate a lot of pending threads.

I wanted enough threads to be around to handle short running page requests too.

But besides that, it's very odd that tcmalloc had this huge decrease in performance at 100 threads.  It's perhaps not enough of a reason to avoid tcmalloc, but it's a cause to worry, nonetheless.

Gustaf's tests didn't show that same slowdown result, and the big slowdown I have at 100 threads only happens with tcmalloc, not with jemalloc or zippy, with a simple [clock seconds] ADP page.

-john

Re: [naviserver-devel] tcmalloc and jemalloc results with naviserver

[Wolfgang W. <wol...@di...>]

I was wondering, why you'll want to have 100 threads, because the number 
seems a little high to me. So I've just conducted some test on two of our 
development system.

A test. similar to your simple adp page with maxthreads of 5 yields 
between 1500 and 2500 requests per seconds, including session handling 
in the preauth filter. With 24 maxthreads I get between 6800 and 7500 
reqs/sec.

For a real world not cached CMS page on a 6 core hyperthreaded CPU we 
still get more than 500 pages/sec with 5 maxthreads and 770-780  with 12 
threads, 820 - 830 up to 950 with 24 maxthreads. From there it doesn't 
change much, until performance degrades slightly again at around 100 
maxthreads (780 reqs/sec).

I used ab with -n 20000 -c 50|200|500|1000

My conclusion: Having a higher number of maxthreads than the number of 
cores is helping performance, more than twice the amount isn't helping 
anymore. 

It might be different, when you've got some long running threads, where 
you have to pull information from other servers and the threads are idling.

This was all done with tcmalloc, If I'll find the time, I'll rerun the tests with 
jemalloc as well.

wolfgang


Am Samstag, 4. Januar 2014, 20:35:05 schrieb John Buckman:
> >> After rebuilding tcl and naviserver (current) from source, with
> >> -ltcmalloc, my naviserver is stable.> 
> > great. the tip version of naviserver have already some modifications
> > to help against concurrency bugs in tcl (e.g. reducing the frequency 
of
> > interp create operations, serializing these etc.).
> 
> With Gustaf's patch in place, I was able to test tcmalloc vs jemalloc vs
> zippy in naviserver.
> 
> Both tclmalloc and jemalloc gave me a 20% speed up on a simple [clock
> seconds] adp page.  Both gave me about a 40% speedup on a much 
more
> complicated, unoptimized Tcl index.adp page.
> 
> Once I received an openssl() assertion failure on startup with jemalloc 
and
> "minthreads 100" but was not able to recreate it.
> 
> tcmalloc under 20 threads worked well, but performance was *terrible* 
at
> "minthreads 100", (about a 90% slowdown, even with apachebench -c 
10). 
> Neither zippy nor jemalloc had this problem.  There's something about
> having many naivserver threads launched that makes tcmalloc not work 
well.
> 
> Conclusion:
> 
> For now, I'm using jemalloc, though the openssl assertion failure I had
> makes me nervous.
> 
> tcmalloc's terrible performance with 100 threads makes me not want to 
use
> it.
> 
> At the same time, I don't want to stick with zippy, as I've experienced the
> "bloat" problem that Gustaf talks about, solving it at BookMooch.com 
with
> 64GB of RAM (!) but that's not a very good solution, so it's worth my 
while
> to try other mallocs.
> 
> -john
> 
> 
> 
> ----------------------------------------------------------------------------
> -- Rapidly troubleshoot problems before they affect your business. Most 
IT
> organizations don't have a clear picture of how application performance
> affects their revenue. With AppDynamics, you get 100% visibility into your
> Java,.NET, & PHP application. Start your 15-day FREE TRIAL of 
AppDynamics
> Pro!
> 
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.
clktrk
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

-- 
digital concepts OG
Software & Design
Landstrasse 68 / 5. Stock
A - 4020 Linz

[naviserver-devel] tcmalloc and jemalloc results with naviserver

[John B. <jo...@ma...>]

>> After rebuilding tcl and naviserver (current) from source, with -ltcmalloc, my naviserver is stable.
> great. the tip version of naviserver have already some modifications
> to help against concurrency bugs in tcl (e.g. reducing the frequency of interp
> create operations, serializing these etc.). 

With Gustaf's patch in place, I was able to test tcmalloc vs jemalloc vs zippy in naviserver.

Both tclmalloc and jemalloc gave me a 20% speed up on a simple [clock seconds] adp page.  Both gave me about a 40% speedup on a much more complicated, unoptimized Tcl index.adp page.

Once I received an openssl() assertion failure on startup with jemalloc and "minthreads 100" but was not able to recreate it.

tcmalloc under 20 threads worked well, but performance was *terrible* at "minthreads 100", (about a 90% slowdown, even with apachebench -c 10).  Neither zippy nor jemalloc had this problem.  There's something about having many naivserver threads launched that makes tcmalloc not work well.

Conclusion:

For now, I'm using jemalloc, though the openssl assertion failure I had makes me nervous.  

tcmalloc's terrible performance with 100 threads makes me not want to use it.  

At the same time, I don't want to stick with zippy, as I've experienced the "bloat" problem that Gustaf talks about, solving it at BookMooch.com with 64GB of RAM (!) but that's not a very good solution, so it's worth my while to try other mallocs.

-john