naviserver-devel — Developer discussion

Re: [naviserver-devel] Changing pageroot

[Gustaf N. <ne...@wu...>]

hmm, i found two occurrences of  "nsparam pageroot", and 
changed these.

% egrep -R "ns_param *pageroot" *
openacs-config.tcl:    ns_param   pageroot        $pageroot
tests/http-test-config.tcl:ns_param   pageroot        $pageroot

i have added as well "ns_info pagedir" as an alias for 
"ns_info pageroot".
One should mark the latter as deprecated.

-gustaf

On 01.11.12 18:00, Jeff Rogers wrote:
> It's "pagedir" rather than "pageroot", and it's relative to the server
> root for virtual hosts (it can still be absolute).  The documentation
> appears to be not up to date on this item.
>
> -J
>
>
> David Osborne wrote:
>> Hi again,
>>
>> In Aolserver we used to change pageroot in the config as so:
>> ns_param pageroot /new/page/root
>>
>> Should the same work in Naviserver?
>> ns_info pageroot is returning /usr/local/ns/pages no matter what I put
>> in the config for pageroot (built from the latest version in bitbucket).
>> Trying to work out if this is a config error or a code difference... any
>> pointers?
>>
>> Thanks,
>> - David
>>
>>
>> ------------------------------------------------------------------------------
>> Everyone hates slow websites. So do we.
>> Make your web apps faster with AppDynamics
>> Download AppDynamics Lite for free today:
>> http://p.sf.net/sfu/appdyn_sfd2d_oct
>>
>>
>>
>> _______________________________________________
>> naviserver-devel mailing list
>> nav...@li...
>> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>>
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_sfd2d_oct
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

[naviserver-devel] naviserver with connection thread queue

[Gustaf N. <ne...@wu...>]

Dear all,

There is now a version on bitbucket, which works quite nice 
and stable, as far i can tell. I have split up the rather 
coarse lock of all pools and introduced finer locks for 
waiting queue (wqueue) and thread queue (tqueue) per pool.  
The changes lead to significant finer lock granularity and 
improve scalability.

I have tested this new version with a synthetic load of 120 
requests per seconds, some slower requests and some faster 
ones, and it appears to be pretty stable. This load keeps 
about 20 connection threads quite busy on my home machine. 
The contention of the new locks is very little: on this test 
we saw 12 busy locks on 217.000 locks on the waiting queue, 
and 9 busy locks out of 83.000 locks on the thread queue. 
These measures are much better than in current naviserver, 
which has on the same test on the queue 248.000 locks with 
190 busy ones. The total waiting time for locks is reduced 
by a factor of 10. One has to add, that it was not so bad 
before either. The benefit will be larger when multiple 
pools are used.

Finally i think, the code is clearer than before, where the 
lock duration was quite tricky to determine.

opinions?
-gustaf neumann

PS: For the changes, see: 
https://bitbucket.org/gustafn/naviserver-connthreadqueue/changesets 

PS2: have not addressed the server exit signaling yet.

On 29.10.12 13:41, Gustaf Neumann wrote:
> A version of this is in the following fork:
>
> https://bitbucket.org/gustafn/naviserver-connthreadqueue/changesets
>
> So far, the competition on the pool mutex is quite high, but
> i think, it can be improved. Currently primarily the pool
> mutex is used for conn thread life-cycle management, and it
> is needed from the main/drivers/spoolers as well from the
> connection threads to update the idle/running/.. counters
> needed for controlling thread creation etc. Differentiating
> these mutexes should help.
>
> i have not addressed the termination signaling, but that's
> rather simple.
>
> -gustaf neumann
>
> On 28.10.12 03:08, Gustaf Neumann wrote:
>> i've just implemented lightweight version of the above (just
>> a few lines of code) by extending the connThread Arg
>> structure; ....
>
> ------------------------------------------------------------------------------
> The Windows 8 Center - In partnership with Sourceforge
> Your idea - your app - 30 days.
> Get started!
> http://windows8center.sourceforge.net/
> what-html-developers-need-to-know-about-coding-windows-8-metro-style-apps/
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] Changing pageroot

[Jeff R. <dv...@di...>]

It's "pagedir" rather than "pageroot", and it's relative to the server 
root for virtual hosts (it can still be absolute).  The documentation 
appears to be not up to date on this item.

-J


David Osborne wrote:
> Hi again,
>
> In Aolserver we used to change pageroot in the config as so:
> ns_param pageroot /new/page/root
>
> Should the same work in Naviserver?
> ns_info pageroot is returning /usr/local/ns/pages no matter what I put
> in the config for pageroot (built from the latest version in bitbucket).
> Trying to work out if this is a config error or a code difference... any
> pointers?
>
> Thanks,
> - David
>
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_sfd2d_oct
>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>

[naviserver-devel] Changing pageroot

[David O. <da...@qc...>]

Hi again,

In Aolserver we used to change pageroot in the config as so:
ns_param pageroot /new/page/root

Should the same work in Naviserver?
ns_info pageroot is returning /usr/local/ns/pages no matter what I put in
the config for pageroot (built from the latest version in bitbucket).
Trying to work out if this is a config error or a code difference... any
pointers?

Thanks,
- David

Re: [naviserver-devel] ns_return* documentation and ns_register_* "default" method

[Gustaf N. <ne...@wu...>]

On 31.10.12 13:54, Stefan Sobernig wrote:
> I have noticed that some of the documentation files are being updated.
> By accident, I noticed that the ns_returnerror documentation is out of
> date, e.g., listing "ns_returnmoved" which is not available as a Tcl
> command:
i've added ns_returnmoved
>
> I haven't checked the others ... (leaving aside that documenting a
> command "ns_returnok" under "ns_returnerror" is not necessarily obvious).
the table of contents lists the sections, not commands. For 
most commands, there is one section per command, so it looks 
like a command reference. One might have noticed as well, 
that the sections are sorted by their titles, which is as 
well somewhat wierd.

The documentation is built with dtplite, which is in turn 
built on top of tcl's doctools. The latter seem to support 
command references, but not via dtplite. One can use cross 
reverences ("see_also") and index entries ("keywords"). One 
could add all commands as index entries, but this has the 
bad effect, that if one clicks on the command in the 
synopsis, the browser does not jump to the command 
definition in the documentation page, but to the index. The 
current markup is not consistent, what to put into keywords 
or see_also, it requires more work.

i think, that adding a command reference would be a nice 
addition, and is most likely possible with reasonable amount 
of work. The other alternative would be to split up the 
documentation pages with multiple commands into single pages...

any input/help is welcome

-gustaf neumann

[naviserver-devel] ns_return* documentation and ns_register_* "default" method

[Stefan S. <ste...@wu...>]

I have noticed that some of the documentation files are being updated. 
By accident, I noticed that the ns_returnerror documentation is out of 
date, e.g., listing "ns_returnmoved" which is not available as a Tcl 
command:

https://bitbucket.org/naviserver/naviserver/src/fbfcaea8cccc808cc79534e81f61cdd50a1d9714/doc/src/naviserver/ns_returnerror.man?at=default#cl-46


I haven't checked the others ... (leaving aside that documenting a 
command "ns_returnok" under "ns_returnerror" is not necessarily obvious).

I'd also appreciate a heads-up on a previous thread:

http://www.mail-archive.com/nav...@li.../msg01811.html

Was the idea of a "default" method ("*" or the like) for the 
ns_register_* cmd family ever implemented?

(I briefly skimmed over the code base, I could not find a hint)

thx
//stefan

Re: [naviserver-devel] ns_urldecode -charset

[David O. <da...@qc...>]

On 30 October 2012 20:39, Stephen Deasey <sd...@gm...> wrote:

> On Tue, Oct 30, 2012 at 7:59 PM, Stephen Deasey <sd...@gm...> wrote:
> >
> > But the code points of iso88591 are a subset of utf8...
>
> Actually, this doesn't make sense. The byte encoding of code points
> above 128 uses two bytes for utf8, but only one byte for iso88591.
>

Yes this is exactly it Stephen. We're communicating with an external system
which uses iso8859-1.
So for the extended character set the results are different. The external
system gets confused if it tries to decode something we encoded using utf-8
eg:

nscp 52> ns_urlencode -charset iso8859-1 ú
%fa
nscp 53> ns_urlencode -charset utf-8 ú
%c3%ba
nscp 54> ns_urldecode -charset iso8859-1 %c3%ba
ú

Thanks very much Gustaf for making this change!

Re: [naviserver-devel] ns_urldecode -charset

[Gustaf N. <ne...@wu...>]

The changes are committed to bitbucket.
see 
https://bitbucket.org/naviserver/naviserver/changeset/7b89b89802beebeb3db4a37c77f3d2d63c944494

all the best -gustaf

On 30.10.12 16:03, David Osborne wrote:
> Hi Gustaf,
> Yes, that looks great. Along with the ns_urlencode 
> equivalent I think it would solve our problem.
> Thanks very much for the reply.
>
> - David

Re: [naviserver-devel] ns_urldecode -charset

[Stephen D. <sd...@gm...>]

On Tue, Oct 30, 2012 at 7:59 PM, Stephen Deasey <sd...@gm...> wrote:
>
> But the code points of iso88591 are a subset of utf8...

Actually, this doesn't make sense. The byte encoding of code points
above 128 uses two bytes for utf8, but only one byte for iso88591.

Looks like you need -charset.

Re: [naviserver-devel] ns_urldecode -charset

[Stephen D. <sd...@gm...>]

On Tue, Oct 30, 2012 at 10:38 AM, David Osborne <da...@qc...> wrote:
> Hi,
>
> We're currently in the process of porting a fairly large code base from
> Aolserver to Naviserver for testing (using Naviserver v4.99.4 on Debian
> Squeeze).
>
> One thing that has come up so far is that ns_urldecode seems to have dropped
> the "-charset" switch.
>
> I'm assuming it used to be present since some of your documentation mentions
> it:
> (eg. http://naviserver.sourceforge.net/n/naviserver/files/ns_urldecode.html
> )
> I can't find any mention of why it was dropped?


It was more than 7 years ago so I can't remember the details, but I
think there were some other bugs to do with character sets that
basically meant forcing everything to be utf-8. Strictly speaking the
-charset switch to ns_urldecode might still be needed, and I think it
got removed by mistake, but it's usually not needed:

> So in Naviserver, is there an alternative to achieve the following:
>
> nscp 1> ns_urldecode -charset iso8859-1 "%FA"
> ú

Here for example, if you don't pass -charset then naviserver assumes
utf8. But the code points of iso88591 are a subset of utf8 (and ascii
is a subset of both), so the result is identical. So you should never
have to specify iso88591, because I think you can no longer set the
notion of a global default character set -- it's always utf8, and then
in some places you can specifically choose if really needed.

Where are you getting %FA from? Is it something you're encoding
yourself or are you interacting with another system?


> PS. This is not really a devel question - is there a more appropriate place
> to ask config/user questions?


This is it.

Re: [naviserver-devel] ns_urldecode -charset

[David O. <da...@qc...>]

Hi Gustaf,
Yes, that looks great. Along with the ns_urlencode equivalent I think it
would solve our problem.
Thanks very much for the reply.

- David

On 30 October 2012 14:44, Gustaf Neumann wrote:

>  Dear David,
>
> would the following change help you?
>
> Before i finalize this change (do this on encode as well, add to
> documentation, etc.),
> was this omitted on purpose in naviserver?
>
> -gustaf neumann
>
> --- a/nsd/urlencode.c	Mon Oct 29 13:46:08 2012 +0100
> +++ b/nsd/urlencode.c	Tue Oct 30 15:41:06 2012 +0100
> @@ -504,8 +504,9 @@
>  NsTclUrlDecodeObjCmd(ClientData arg, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[])
>  {
>      Ns_DString   ds;
> -    char        *string;
> +    char        *string, *charset = NULL;
>      int          part = 'q';
> +    Tcl_Encoding encoding = NULL;
>
>      Ns_ObjvTable parts[] = {
>          {"query",    'q'},
> @@ -514,7 +515,8 @@
>      };
>      Ns_ObjvSpec opts[] = {
>          {"-part",    Ns_ObjvIndex,   &part,   &parts},
> -        {"--",       Ns_ObjvBreak,    NULL,    NULL},
> +        {"-charset", Ns_ObjvString,  &charset, NULL},
> +        {"--",       Ns_ObjvBreak,   NULL,    NULL},
>          {NULL, NULL, NULL, NULL}
>      };
>      Ns_ObjvSpec args[] = {
> @@ -526,7 +528,11 @@
>      }
>
>      Ns_DStringInit(&ds);
> -    UrlDecode(&ds, string, NULL, part);
> +    if (charset) {
> +        encoding = Ns_GetUrlEncoding(charset);
> +    }
> +
> +    UrlDecode(&ds, string, encoding, part);
>      Tcl_DStringResult(interp, &ds);
>
>      return TCL_OK;
>
>
>

Re: [naviserver-devel] ns_urldecode -charset

[Gustaf N. <ne...@wu...>]

Dear David,

would the following change help you?

Before i finalize this change (do this on encode as well, 
add to documentation, etc.),
was this omitted on purpose in naviserver?

-gustaf neumann

--- a/nsd/urlencode.c	Mon Oct 29 13:46:08 2012 +0100
+++ b/nsd/urlencode.c	Tue Oct 30 15:41:06 2012 +0100
@@ -504,8 +504,9 @@
  NsTclUrlDecodeObjCmd(ClientData arg, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[])
  {
      Ns_DString   ds;
-    char        *string;
+    char        *string, *charset = NULL;
      int          part = 'q';
+    Tcl_Encoding encoding = NULL;
  
      Ns_ObjvTable parts[] = {
          {"query",    'q'},
@@ -514,7 +515,8 @@
      };
      Ns_ObjvSpec opts[] = {
          {"-part",    Ns_ObjvIndex,   &part,   &parts},
-        {"--",       Ns_ObjvBreak,    NULL,    NULL},
+        {"-charset", Ns_ObjvString,  &charset, NULL},
+        {"--",       Ns_ObjvBreak,   NULL,    NULL},
          {NULL, NULL, NULL, NULL}
      };
      Ns_ObjvSpec args[] = {
@@ -526,7 +528,11 @@
      }
  
      Ns_DStringInit(&ds);
-    UrlDecode(&ds, string, NULL, part);
+    if (charset) {
+        encoding = Ns_GetUrlEncoding(charset);
+    }
+
+    UrlDecode(&ds, string, encoding, part);
      Tcl_DStringResult(interp, &ds);
  
      return TCL_OK;



On 30.10.12 11:38, David Osborne wrote:
> Hi,
>
> We're currently in the process of porting a fairly large 
> code base from Aolserver to Naviserver for testing (using 
> Naviserver v4.99.4 on Debian Squeeze).
>
> One thing that has come up so far is that ns_urldecode 
> seems to have dropped the "-charset" switch.
>
> I'm assuming it used to be present since some of your 
> documentation mentions it:
> (eg. 
> http://naviserver.sourceforge.net/n/naviserver/files/ns_urldecode.html 
> )
> I can't find any mention of why it was dropped?
>
> So in Naviserver, is there an alternative to achieve the 
> following:
>
> nscp 1> ns_urldecode -charset iso8859-1 "%FA"
> ú
>
> PS. This is not really a devel question - is there a more 
> appropriate place to ask config/user questions?
>
> Thanks in advance.
>
> - David
>

[naviserver-devel] ns_urldecode -charset

[David O. <da...@qc...>]

Hi,

We're currently in the process of porting a fairly large code base from
Aolserver to Naviserver for testing (using Naviserver v4.99.4 on Debian
Squeeze).

One thing that has come up so far is that ns_urldecode seems to have
dropped the "-charset" switch.

I'm assuming it used to be present since some of your documentation
mentions it:
(eg. http://naviserver.sourceforge.net/n/naviserver/files/ns_urldecode.html)
I can't find any mention of why it was dropped?

So in Naviserver, is there an alternative to achieve the following:

nscp 1> ns_urldecode -charset iso8859-1 "%FA"
ú

PS. This is not really a devel question - is there a more appropriate place
to ask config/user questions?

Thanks in advance.

- David

Re: [naviserver-devel] lurking bugs: conn threads

[Gustaf N. <ne...@wu...>]

A version of this is in the following fork:

https://bitbucket.org/gustafn/naviserver-connthreadqueue/changesets

So far, the competition on the pool mutex is quite high, but 
i think, it can be improved. Currently primarily the pool 
mutex is used for conn thread life-cycle management, and it 
is needed from the main/drivers/spoolers as well from the 
connection threads to update the idle/running/.. counters 
needed for controlling thread creation etc. Differentiating 
these mutexes should help.

i have not addressed the termination signaling, but that's 
rather simple.

-gustaf neumann

On 28.10.12 03:08, Gustaf Neumann wrote:
> i've just implemented lightweight version of the above (just
> a few lines of code) by extending the connThread Arg
> structure; ....

Re: [naviserver-devel] lurking bugs: conn threads

[Gustaf N. <ne...@wu...>]

On 27.10.12 15:56, Gustaf Neumann wrote:
> Changing the notification structure (adding a
> connection-thread-queue and extra condition) is a relatively
> small change, compared to general redesign.
i've just implemented lightweight version of the above (just 
a few lines of code) by extending the connThread Arg 
structure; i have not handled the spooling and stopping 
server cases (e.g. NsWaitServer), but this looks promising 
and can be still optimized further. it runs already the 
regression test. When maxthreads connThread Arg structures 
are allocated per pool at start-time, one can iterate over 
these for the stopping cases to compensate for the needed 
Ns_CondBroadcasts. No additional thread is needed.

-gustaf neumann

Re: [naviserver-devel] compression

[Gustaf N. <ne...@wu...>]

the version on the tip handles now identity and q-values.
The logic sketched below does not handle cases, where e.g. 
identiy or * is
higher than gzip qvalue, or explicit forbidding of gzip. The 
values are doubles,
so one has to be careful with comparisons. I am not sure 
about the logic for
http/1.1, if we really should assume gzip by default.

-gustaf

On 27.10.12 15:03, Stephen Deasey wrote:
> On Sat, Oct 27, 2012 at 1:13 PM, Gustaf Neumann <ne...@wu...> wrote:
>> On 26.10.12 15:47, Stephen Deasey wrote:
>>> I think the spec says that for HTTP/1.1, if the client doesn't
>>> explicitly say to NOT send the body gzipped, say by using a q value of
>>> 0 (which we don't actually check for, woops...), then the server can
>>> choose the encoding, although it should prefer the identity, unless
>>> that's unavailable. So we're being very pushy...
>> now we do check for the qvalues used in connection with
>> gzip. So, a client can now specify explicitly, that gzip is
>> NOT wanted via "gzip; q=0". What the code still does not do
>> is comparing the identity-qvalue with the gzip-qvalue or
>> combination with wildcards "....*;q=..."
> Jeff had a go at this as well:
>
>    https://bitbucket.org/jeffr/naviserver-queues/changesets
>
> Here's the feedback I tacked on to the end of a mercurial question via email:
>
>
>
> It bothers me a bit that a fresh Ns_Set has to be allocated, and also
> the parsing code is pretty gnarly and hard to verify. It looks about
> right, but I'd have to resort to pencil and paper to be more sure, and
> the fact I haven't done that reminds me that people tend to not look
> too closely at these things and that's where bugs can fester. I wonder
> if there's another way of doing this...
>
> You'll have to double check the details, but IIRC q values go from
> 0-999. If 'gzip' is present without a q value then it's as if it had
> q=1 -- that's the default. If it's not present, it's as if it were but
> with q=0. So, how about a function like Ns_HeaderQ(header, attr) which
> returns the integer q value for the specified atttribute. It simply
> uses strstr to find gzip. If it doesn't, return 0. If it does, then
> check for a q=. If it's the character '0', return 0. If there's no q,
> return 1. If it's something else, parse the int and return that. Then
> you can use it something like:
>
>
> if (!(connPtr->flags & NS_CONN_SENTHDRS)
>         && !(connPtr->flags & NS_CONN_SKIPBODY)) {
>
>      contentEncoding = Ns_SetIGet(Ns_ConnHeaders(conn), "Accept-Encoding");
>
>      if (Ns_HeaderQ(contentEncoding, "gzip") > 0 ||
> Ns_HeaderQ(contentEncoding, "*") > 0) {
>          gzip = 1;
>
>
> I'm not sure how eager the server should be sending gzipped content.
> An alternative to the above would be to keep the existing eager use of
> gzip, but respect the client's negative assertion:
>
>
> if (!(connPtr->flags & NS_CONN_SENTHDRS)
>         && !(connPtr->flags & NS_CONN_SKIPBODY)) {
>
>      contentEncoding = Ns_SetIGet(Ns_ConnHeaders(conn), "Accept-Encoding");
>
>      if (connPtr->request->version >= 1.1) {
>          if ((!Ns_HeaderQ(contentEncoding, "gzip", &q) || q > 0)
>              && (!Ns_HeaderQ(contentEncoding, "*", &q) || q > 0)) {
>              gzip = 1;
>          }
>      } else if ((Ns_HeaderQ(contentEncoding, "gzip", &q) && q > 0)
>                 || (Ns_HeaderQ(contentEncoding, "*", &q) && q > 0)) {
>          gzip = 1;
>      }
>
>
> Here Ns_HeaderQ is modified to return NS_TRUE or NS_FALSE depending on
> whether the attribute is present, and the q value is returned into the
> given variable. HTTP 1.0 clients have to explicitly ask for gzip, 1.1
> clients get it unless the say they don't want it.
>
> ------------------------------------------------------------------------------
> WINDOWS 8 is here.
> Millions of people.  Your app in 30 days.
> Visit The Windows 8 Center at Sourceforge for all your go to resources.
> http://windows8center.sourceforge.net/
> join-generation-app-and-make-money-coding-fast/
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] Hello

[Agustin L. <Agu...@uv...>]

Thanks, Gustaf.

We have too compiled for naviserver one module (ported from aolserver)
for unzip / zip that we are using on our cluster.

Regards,
Agustin


El 27/10/12 19:52, Gustaf Neumann escribió:
> Agustin,
>
> here is a port from nsldap_v0_r8 to the naviserver interface.
>
> https://bitbucket.org/naviserver/nsldap/src
>
> This should make it easier fro users to use this module...
> -gustaf neumann
>
>
> On 27.10.12 13:42, Agustin Lopez wrote:
>> Thanks, Stephen.
>>
>> Finally I find I have to replace  Ns_ConfigSection with Ns_ConfigGetSection.
>> Moreover, I have compiled the nsldap module and I am testing
>> Navisever in one cluster node. All Ok.
>>
>>
>> Regards,
>> Agustin
>>
>>
>>
>> El 26/10/12 15:49, Stephen Deasey escribió:
>>> On Wed, Oct 24, 2012 at 5:01 PM, Agustin Lopez<Agu...@uv...>  wrote:
>>>> When I run the server I get error with Ns_ConfigSection from my compiled
>>>> nsldap.
>>>> I suppose that it is a known problem. Any pointer to work it?
>>> What is the exact error message?
>>>
>>> ------------------------------------------------------------------------------
>>> Everyone hates slow websites. So do we.
>>> Make your web apps faster with AppDynamics
>>> Download AppDynamics Lite for free today:
>>> http://p.sf.net/sfu/appdyn_sfd2d_oct
>>> _______________________________________________
>>> naviserver-devel mailing list
>>> nav...@li...
>>> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>>>
>> ------------------------------------------------------------------------------
>> WINDOWS 8 is here.
>> Millions of people.  Your app in 30 days.
>> Visit The Windows 8 Center at Sourceforge for all your go to resources.
>> http://windows8center.sourceforge.net/
>> join-generation-app-and-make-money-coding-fast/
>> _______________________________________________
>> naviserver-devel mailing list
>> nav...@li...
>> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>
>
>
> ------------------------------------------------------------------------------
> WINDOWS 8 is here.
> Millions of people.  Your app in 30 days.
> Visit The Windows 8 Center at Sourceforge for all your go to resources.
> http://windows8center.sourceforge.net/
> join-generation-app-and-make-money-coding-fast/
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] Hello

[Gustaf N. <ne...@wu...>]

Agustin,

here is a port from nsldap_v0_r8 to the naviserver interface.

https://bitbucket.org/naviserver/nsldap/src

This should make it easier fro users to use this module...
-gustaf neumann


On 27.10.12 13:42, Agustin Lopez wrote:
> Thanks, Stephen.
>
> Finally I find I have to replace  Ns_ConfigSection with Ns_ConfigGetSection.
> Moreover, I have compiled the nsldap module and I am testing
> Navisever in one cluster node. All Ok.
>
>
> Regards,
> Agustin
>
>
>
> El 26/10/12 15:49, Stephen Deasey escribió:
>> On Wed, Oct 24, 2012 at 5:01 PM, Agustin Lopez <Agu...@uv...> wrote:
>>> When I run the server I get error with Ns_ConfigSection from my compiled
>>> nsldap.
>>> I suppose that it is a known problem. Any pointer to work it?
>> What is the exact error message?
>>
>> ------------------------------------------------------------------------------
>> Everyone hates slow websites. So do we.
>> Make your web apps faster with AppDynamics
>> Download AppDynamics Lite for free today:
>> http://p.sf.net/sfu/appdyn_sfd2d_oct
>> _______________________________________________
>> naviserver-devel mailing list
>> nav...@li...
>> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>>
>
> ------------------------------------------------------------------------------
> WINDOWS 8 is here.
> Millions of people.  Your app in 30 days.
> Visit The Windows 8 Center at Sourceforge for all your go to resources.
> http://windows8center.sourceforge.net/
> join-generation-app-and-make-money-coding-fast/
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] lurking bugs: conn threads

[Gustaf N. <ne...@wu...>]

On 26.10.12 21:30, Stephen Deasey wrote:
> I was thinking it could work something like this:
>
>
> - driver acquires lock, takes first conn thread off queue, releases lock
>
> - driver thread puts new socket in conn structure and the signals on
> cond to that one thread (no locking, I don't think)
>
> - that conn thread wakes up, takes no locks, runs connection
>
> - conn thread acquires driver lock, puts conn back on front of queue,
> releases lock
>
>
> Four lock/unlock, lock/unlock sequences, two threads.
you are talking here about "lock" as if we would have a 
single mutex. i guess, you mean connection-thread-queue-lock 
in the first item. What happens with the request queue (the 
queue of the parsed connections, realized currently via the 
array of connections)? Is the request queue handling missing 
in the paragraph above, or are you considering to get rid of 
it at all (which would raise further questions).

This picture is drawn from the assumption that the request 
queue contains one request and there is at least one 
connection thread waiting in the connection-thread-queue. 
But this is not always the case. Connection threads are not 
starting in 0-time; in the meanwhile many requests may be 
queued already. So, if a connection thread is ready to work, 
it should acquire a lock on the connection-thread-queue and 
add itself). Also, what happens if the connection thread 
terminates. Should it put itself in front of the 
connection-thread-queue as well to signal the driver to 
create fresh connection threads? This could address the 
problem we discussed recently, when the request queue is 
still quite full, but no new requests are coming in, but all 
connection threads are gone. If it adds itself as well, we 
could control in the driver thread and controls the 
liveliness of the connection threads (at least min threads 
running, etc.).

So, i think the picture above is oversimplifying too much, 
but i guess the the overall message is certainly right. we 
can reduce the needless context switches this way. The 
question remains: how much is this really a problem? How 
much performance can be gained?

Changing the notification structure (adding a 
connection-thread-queue and extra condition) is a relatively 
small change, compared to general redesign.

-gustaf neumann

Re: [naviserver-devel] compression

[Stephen D. <sd...@gm...>]

On Sat, Oct 27, 2012 at 1:13 PM, Gustaf Neumann <ne...@wu...> wrote:
> On 26.10.12 15:47, Stephen Deasey wrote:
>> I think the spec says that for HTTP/1.1, if the client doesn't
>> explicitly say to NOT send the body gzipped, say by using a q value of
>> 0 (which we don't actually check for, woops...), then the server can
>> choose the encoding, although it should prefer the identity, unless
>> that's unavailable. So we're being very pushy...
>
> now we do check for the qvalues used in connection with
> gzip. So, a client can now specify explicitly, that gzip is
> NOT wanted via "gzip; q=0". What the code still does not do
> is comparing the identity-qvalue with the gzip-qvalue or
> combination with wildcards "....*;q=..."

Jeff had a go at this as well:

  https://bitbucket.org/jeffr/naviserver-queues/changesets

Here's the feedback I tacked on to the end of a mercurial question via email:



It bothers me a bit that a fresh Ns_Set has to be allocated, and also
the parsing code is pretty gnarly and hard to verify. It looks about
right, but I'd have to resort to pencil and paper to be more sure, and
the fact I haven't done that reminds me that people tend to not look
too closely at these things and that's where bugs can fester. I wonder
if there's another way of doing this...

You'll have to double check the details, but IIRC q values go from
0-999. If 'gzip' is present without a q value then it's as if it had
q=1 -- that's the default. If it's not present, it's as if it were but
with q=0. So, how about a function like Ns_HeaderQ(header, attr) which
returns the integer q value for the specified atttribute. It simply
uses strstr to find gzip. If it doesn't, return 0. If it does, then
check for a q=. If it's the character '0', return 0. If there's no q,
return 1. If it's something else, parse the int and return that. Then
you can use it something like:


if (!(connPtr->flags & NS_CONN_SENTHDRS)
       && !(connPtr->flags & NS_CONN_SKIPBODY)) {

    contentEncoding = Ns_SetIGet(Ns_ConnHeaders(conn), "Accept-Encoding");

    if (Ns_HeaderQ(contentEncoding, "gzip") > 0 ||
Ns_HeaderQ(contentEncoding, "*") > 0) {
        gzip = 1;


I'm not sure how eager the server should be sending gzipped content.
An alternative to the above would be to keep the existing eager use of
gzip, but respect the client's negative assertion:


if (!(connPtr->flags & NS_CONN_SENTHDRS)
       && !(connPtr->flags & NS_CONN_SKIPBODY)) {

    contentEncoding = Ns_SetIGet(Ns_ConnHeaders(conn), "Accept-Encoding");

    if (connPtr->request->version >= 1.1) {
        if ((!Ns_HeaderQ(contentEncoding, "gzip", &q) || q > 0)
            && (!Ns_HeaderQ(contentEncoding, "*", &q) || q > 0)) {
            gzip = 1;
        }
    } else if ((Ns_HeaderQ(contentEncoding, "gzip", &q) && q > 0)
               || (Ns_HeaderQ(contentEncoding, "*", &q) && q > 0)) {
        gzip = 1;
    }


Here Ns_HeaderQ is modified to return NS_TRUE or NS_FALSE depending on
whether the attribute is present, and the q value is returned into the
given variable. HTTP 1.0 clients have to explicitly ask for gzip, 1.1
clients get it unless the say they don't want it.

Re: [naviserver-devel] lurking bugs: conn threads

[Stephen D. <sd...@gm...>]

On Fri, Oct 26, 2012 at 11:41 PM, Jeff Rogers <dv...@di...> wrote:
> Stephen Deasey wrote:
>> .., but I wonder if
>> we're even attempting to do the right thing?
>
> Do we even know what the right thing is?  It could be any of
> - ...
> - minimize resource usage
> - adapt to dynamically changing workload
> - ...

Right, but you can't actually use the memory that temporarily running
fewer threads frees up because the threads may be restarted at any
moment, so the goal of adapting and minimizing is not being met.
Memory is being wasted, not recycled. (maybe, that's what I'm
suggesting...)

> Not only that, but memory isn't always released back to the system when
> free()d.  (vtamlloc is supposed to be able to, but I haven't had too
> much success with it so far.)  So freeing memory by shutting down
> threads won't necessarily make that available to your database.

I think glibc these days uses POSIX_MADV_DONTNEED on ranges within
mmap'd areas it uses for malloc. It doesn't reduce the address usage,
so this only shows up as lower RSS under top.


> Server resources (memory) is either 'small' for requests that do not
> need a tcl interp (although tcl filters could tend to make this a
> nonexistent set), or 'big' for those that do.  Time is either slow or
> fast, by some arbitrary measure.
>
> So a small/fast pool could be set up to serve static resources, a
> big/fast pool for non-database scripts, and a big/slow pool for database
> stuff.

Naviserver has some stuff which AOLserver doesn't to help with this
partitioning. Gustaf gave an example the other day where a server gets
a small burst of traffic, a couple of page requests, but the browser
simultaneously requests all the css and javascript etc., which causes
a bunch of new threads to be created and a stall as they all have
their interps allocated.

You can create a non-tcl pool as well as the default pool and then use
some tricks to force certain types of requests not to use Tcl.

- ACS registers a default handler for /*, but naviserver exposes
ns_register_fastpath with which you can re-register the pure C
fastpath handler for /*.css etc.

- ACS has an elaborate search path for files so the above is not
enough. But naviserver provides the url2file callback interface. A
pure C version of the algorithm which maps a url path to a file path
code be coded and then the fastpath code would correctly find package
specific static assets.

- There's auth filters and so on registered for /* but again they
aren't needed for /*.css. You can use ns_shortcut_filter to push a
null filter to the front of the filter queue which simply returns OK
and prevents the rest from running.


> The small/fast pool would only need a small number of threads with a
> high maxconnsperthread, while the large/slow pool might have many
> threads as most of those will be blocking on database access at any
> given time.

This is sort of a recreation of the memory situation I was suggesting
may bot be working. Balancing memory between naviserver and postgress
is like balancing the threads (memory) in two pools. The max number of
threads across all pools can't be so high that it overwhelms the
server. Within that constraint you have to balance the threads between
the pools. If you have two pools each with 10 threads, and one pool is
busy but the other is idle, then the server is not running to
capacity, but you may have to reject requests. If you increase the
threads in the busy pool, the other pool may also become busy and now
the server is overwhelmed.

Tcl-using conn threads are often so memory intensive it seems like it
would always be a win to have two conn thread pools for Tcl and
non-Tcl threads. To partition further there's ns_limits but that's not
hooked up and needs more work.

Re: [naviserver-devel] compression

[Gustaf N. <ne...@wu...>]

On 26.10.12 15:47, Stephen Deasey wrote:
> I think the spec says that for HTTP/1.1, if the client doesn't
> explicitly say to NOT send the body gzipped, say by using a q value of
> 0 (which we don't actually check for, woops...), then the server can
> choose the encoding, although it should prefer the identity, unless
> that's unavailable. So we're being very pushy...
now we do check for the qvalues used in connection with 
gzip. So, a client can now specify explicitly, that gzip is 
NOT wanted via "gzip; q=0". What the code still does not do 
is comparing the identity-qvalue with the gzip-qvalue or 
combination with wildcards "....*;q=..."

>> Also on compression, a separate compression stream is pre-allocated and
>> initialized for each pre-allocated Conn, whether or not compression is
>> even enabled for the server.  The causes a fairly large initial memory
>> footprint.  I think this pre-initialization could be made optional, and
>> bypassed entirely when compression isn't enabled.  Any thoughts?
> Seems reasonable.
the initialization is on the server level during startup for 
every connection structure, the actual need comes up in a 
connection thread. One could defer the Ns_CompressInit until 
a connection thread needs it (by checking, whether the 
compress stream is available/initialized.

-gustaf neumann

Re: [naviserver-devel] Hello

[Agustin L. <Agu...@uv...>]

Thanks, Stephen.

Finally I find I have to replace  Ns_ConfigSection with Ns_ConfigGetSection.
Moreover, I have compiled the nsldap module and I am testing
Navisever in one cluster node. All Ok.


Regards,
Agustin



El 26/10/12 15:49, Stephen Deasey escribió:
> On Wed, Oct 24, 2012 at 5:01 PM, Agustin Lopez <Agu...@uv...> wrote:
>> When I run the server I get error with Ns_ConfigSection from my compiled
>> nsldap.
>> I suppose that it is a known problem. Any pointer to work it?
> What is the exact error message?
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_sfd2d_oct
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>

Re: [naviserver-devel] lurking bugs: conn threads

[Stephen D. <sd...@gm...>]

On Fri, Oct 26, 2012 at 11:44 PM, Jeff Rogers <dv...@di...> wrote:
> Andrew Piskorski wrote:
>> On Fri, Oct 26, 2012 at 08:30:26PM +0100, Stephen Deasey wrote:
>>
>>> I was thinking it could work something like this:
>>>
>>> - driver acquires lock, takes first conn thread off queue, releases lock
>>
>> What if there are no conn threads waiting in the queue?
>>
>
> Same as currently I'd think: the driver holds on to them as waiting
> sockets.  I think the handling of this is a bit less efficient than
> putting them on the conn queue tho, as it creates more work for the
> driver to do on every spin and it needs to get woken up once threads are
> available.

- driver takes the lock, sees that there are no threads in the thread
queue, puts conn on the back of the conn queue, does not signal
anything, releases the lock

- conn thread completes a request, takes the driver lock.
-- If the conn queue is empty it puts itself on the front of the
thread queue and releases the lock.
-- Otherwise it takes then next conn and releases the lock.

Re: [naviserver-devel] lurking bugs: conn threads

[Jeff R. <dv...@di...>]

Andrew Piskorski wrote:
> On Fri, Oct 26, 2012 at 08:30:26PM +0100, Stephen Deasey wrote:
>
>> I was thinking it could work something like this:
>>
>> - driver acquires lock, takes first conn thread off queue, releases lock
>
> What if there are no conn threads waiting in the queue?
>

Same as currently I'd think: the driver holds on to them as waiting 
sockets.  I think the handling of this is a bit less efficient than 
putting them on the conn queue tho, as it creates more work for the 
driver to do on every spin and it needs to get woken up once threads are 
available.

-J