naviserver-devel — Developer discussion

Re: [naviserver-devel] nspostgres database driver

[Bernd E. <eid...@we...>]

> Default PG driver for naviserver is nsdbpg now, this one works fine but
> it does not include ACS code

Thanks Vlad!

Re: [naviserver-devel] nspostgres database driver

[Vlad S. <vl...@cr...>]

Default PG driver for naviserver is nsdbpg now, this one works fine but 
it does not include ACS code

Bernd Eidenschink wrote:
> Hi friends!
> 
> Anybody out there who successfully compiled nspostgres-4.0 driver with 
> NaviServer (including -DFOR_ACS_USE CFLAG) ?
> 
> Bernd.
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by xPML, a groundbreaking scripting language
> that extends applications into web and mobile media. Attend the live webcast
> and join the prime developer group breaking into this new coding territory!
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

[naviserver-devel] nspostgres database driver

[Bernd E. <eid...@we...>]

Hi friends!

Anybody out there who successfully compiled nspostgres-4.0 driver with 
NaviServer (including -DFOR_ACS_USE CFLAG) ?

Bernd.

[naviserver-devel] test cases

[Zoran V. <zv...@ar...>]

==== ns_thread-3.1 create and destroy a mutex FAILED
==== Contents of test case:

     set mutex [ns_mutex create]
     ns_mutex destroy $mutex
     set mutex

---- Result was:
tff3089f8 a33f7a0 ns:mutex
---- Result should have been (regexp matching):
t0x[a-z0-9]+ a0x[a-z0-9]+ ns:mutex
==== ns_thread-3.1 FAILED


The solaris sprintf(buf, "%p", ptr) does not
retrun 0xabcdef, it returns abcdef instead.

Re: [naviserver-devel] spooler thread

[Vlad S. <vl...@cr...>]

Great work Zoran, thanks

Zoran Vasiljevic wrote:
> Hi !
> 
> I have dome some work on the problem of test-suite blocking
> while uploading 1MB data to server (http test 4.4, 4.5, 4.6).
> 
> I have fixed some Tcl code and corrected one endless
> loop in the spooler C-code. Also I reformatted the code
> to be more readable (I can't easily follow lines > 120 chars
> and multiple struct->struct->struct->struct pointers,
> I guess I'm just old...).
> 
> But... I have found that by using the very same nstest_http
> code to exercise the "normal" server, there are no problems.
> (by "normal" I mean NOT the "make test" environment).
> The same if using ns_httpopen from tcl/http.tcl.
> 
> The problems are however present when running the tests from
> within the test server. So I guess there has to be some special
> case with the test env which makes those test fail (i.e. block).
> 
> I haven't found the reason, but I'm half-satisfied that the code
> actually works very fine when stress-tested outside the test env.
> Just out of the curiosity, I will try to nail this down tomorrow
> but this is low priority.
> 
> Cheers
> Zoran
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by xPML, a groundbreaking scripting language
> that extends applications into web and mobile media. Attend the live 
> webcast
> and join the prime developer group breaking into this new coding territory!
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 


-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

[naviserver-devel] spooler thread

[Zoran V. <zv...@ar...>]

Hi !

I have dome some work on the problem of test-suite blocking
while uploading 1MB data to server (http test 4.4, 4.5, 4.6).

I have fixed some Tcl code and corrected one endless
loop in the spooler C-code. Also I reformatted the code
to be more readable (I can't easily follow lines > 120 chars
and multiple struct->struct->struct->struct pointers,
I guess I'm just old...).

But... I have found that by using the very same nstest_http
code to exercise the "normal" server, there are no problems.
(by "normal" I mean NOT the "make test" environment).
The same if using ns_httpopen from tcl/http.tcl.

The problems are however present when running the tests from
within the test server. So I guess there has to be some special
case with the test env which makes those test fail (i.e. block).

I haven't found the reason, but I'm half-satisfied that the code
actually works very fine when stress-tested outside the test env.
Just out of the curiosity, I will try to nail this down tomorrow
but this is low priority.

Cheers
Zoran

[naviserver-devel] spooler upload problem

[Zoran V. <zv...@ar...>]

Hi!

I'm still not done, but I have a funny feeling that
we are looking at the wrong end.

Both, driver AND spooler threads block i.e. read VERY slowly
bytes from the socket. So, it is not the question of spooler
being broken.
Moreover, I think that the client side does have some problem
in sending the data to the server (the nstest_http proc).
It appears that it is perfectly fast in sending about 16 buffers
with 4096 chars and then it suddenly stops and gets about 48bytes
per call. That means that we have some problem on the sending side
which has difficulties in sending files larger than 64K.
This is all very suspicious.

I will invest more time here, but now I have to go
otherwise my girlfriend will cook me for supper.

Cheers
Zoran

Re: [naviserver-devel] Test failures: while executing "close $wfd" invoked from within "nstest_http ..."

[Zoran V. <zv...@ar...>]

Am 10.03.2006 um 00:13 schrieb Stephen Deasey:

> Zoran!  Why do you hate my test suite?    :-)
>
>
> RCS file: /cvsroot/naviserver/naviserver/tcl/http.tcl,v
> date: 2006/03/09 11:25:55;  author: vasiljevic;  state: Exp;   
> lines: +356 -312
>
> ...The http.tcl and sendmail.tcl are rewritten to take advantage of
> automatic end-of-line handling of Tcl channels...
>

Since WHEN we are supporting "_ns_" as part of the API?

This is in testtinghttp.tcl:

         if {[string equal $http ""]} {
             set request "$method $url\r"
         } else {
             set request "$method $url HTTP/$http\r"
         }
         _ns_http_puts $timeout $wfd $request ; ######################
         for {set i 0} {$i < [ns_set size $hdrs]} {incr i} {


Yes, I have modified http.tcl/sendmail.tcl code given the fact
that Tcl will perfectly well handle the network-line-break
with on-board tools. But I did not expect anybody using internal
helper procedures from outside.

I guess the simple and easy fix to that is to:
   a. provide complete custom support for http conns in testtinghttp.tcl
   b. trash it entirely and use standard ns_httpget

I will opt for the a. for the moment and fix this in about 15 minutes.
Please: do not use non API calls in the future.

Cheers
Zoran

Re: [naviserver-devel] Test failures: while executing "close $wfd" invoked from within "nstest_http ..."

[Zoran V. <zv...@ar...>]

Am 10.03.2006 um 00:13 schrieb Stephen Deasey:

> Zoran!  Why do you hate my test suite?    :-)
>

I dont! I just keep forget to run the damn thing
BEFORE I checkin! I run it for some weird reason
always AFTER. Yes, I noticed that yesterday very
late and did not bother th fix it until today.

Sorry. I must work on my bad habit!

Cheers
Zoran

Re: [naviserver-devel] Test failures: while executing "close $wfd" invoked from within "nstest_http ..."

[Stephen D. <sd...@gm...>]

Zoran!  Why do you hate my test suite?    :-)


RCS file: /cvsroot/naviserver/naviserver/tcl/http.tcl,v
date: 2006/03/09 11:25:55;  author: vasiljevic;  state: Exp;  lines: +356 -=
312

...The http.tcl and sendmail.tcl are rewritten to take advantage of
automatic end-of-line handling of Tcl channels...


 proc _ns_http_gets {timeout sock} {
-    set line ""
-    set done 0
-    while {!$done} {
-       set nline [_ns_http_readable $timeout $sock]
-       if {!$nline} {set done 1}
-       while {!$done && $nline > 0} {
-           set char [read $sock 1]
-           if {$char eq "\n"} {set done 1}
-           append line $char
-           incr nline -1
-       }
+
+    while {[gets $sock line] =3D=3D -1} {
+        if {[eof $sock]} {
+            return -code error "_hs_http_gets: premature end of data"
+        }
+        _ns_http_readable $timeout $sock
     }
-    string trimright $line
+
+    return $line
 }


On 3/9/06, Vlad Seryakov <vl...@cr...> wrote:
> Setting this removed all errors:
>
> set line [string trim [_ns_http_gets $timeout $rfd]]
>
> Before that _ns_http_gets returned \r in the string
>
> Vlad Seryakov wrote:
> > It should close only when post limit exceeds, other requests should not
> > be affected. It looks like broke something
> >
> > Stephen Deasey wrote:
> >> On 3/9/06, Vlad Seryakov <vl...@cr...> wrote:
> >>> Stephen Deasey wrote:
> >>>> I'm getting a lot (123) of what looks like bogus test failures on a
> >>>> fresh checkout.  Are the recent changes to the nstest_http proc OK?
> >>> Yes, i noticed this as well, this is because server now can close
> >>> connection during client sending the body and i tried to put catch
> >>> around close in all places but still it complains on close.
> >>
> >>
> >> This is new behaviour?  Why would the server close the connection
> >> while the client is sending data?
> >>
> >> I'm getting in failures in e.g. test/url2file.test:url2file-2.1 which
> >> looks like:
> >>
> >>
> >> test url2file-2.1 {ns_url2file} -setup {
> >>     ns_register_proc GET /url2file {ns_return 200 text/plain
> >> [ns_url2file /foo] ;#}
> >> } -body {
> >>     nstest_http -getbody 1 GET /url2file
> >> } -cleanup {
> >>     ns_unregister_proc GET /url2file
> >> } -result [list 200 [ns_pagepath foo]]
> >>
> >>
> >> This is just a simple GET request, no body.  Test error is:
> >>
> >>
> >> ---- Test generated error; Return code was: 1
> >> ---- Return code should have been one of: 0 2
> >> ---- errorInfo: can not find channel named "sock24"
> >>     while executing
> >> "close $wfd"
> >>     invoked from within
> >> "nstest_http -getbody 1 GET /url2file"
> >>     ("uplevel" body line 2)
> >>     invoked from within
> >> "uplevel 1 $script"
> >> ---- errorCode: NONE
> >> =3D=3D=3D=3D url2file-2.1 FAILED

Re: [naviserver-devel] Test failures: while executing "close $wfd" invoked from within "nstest_http ..."

[Vlad S. <vl...@cr...>]

Setting this removed all errors:

set line [string trim [_ns_http_gets $timeout $rfd]]

Before that _ns_http_gets returned \r in the string

Vlad Seryakov wrote:
> It should close only when post limit exceeds, other requests should not 
> be affected. It looks like broke something
> 
> Stephen Deasey wrote:
>> On 3/9/06, Vlad Seryakov <vl...@cr...> wrote:
>>> Stephen Deasey wrote:
>>>> I'm getting a lot (123) of what looks like bogus test failures on a
>>>> fresh checkout.  Are the recent changes to the nstest_http proc OK?
>>> Yes, i noticed this as well, this is because server now can close
>>> connection during client sending the body and i tried to put catch
>>> around close in all places but still it complains on close.
>>
>>
>> This is new behaviour?  Why would the server close the connection
>> while the client is sending data?
>>
>> I'm getting in failures in e.g. test/url2file.test:url2file-2.1 which
>> looks like:
>>
>>
>> test url2file-2.1 {ns_url2file} -setup {
>>     ns_register_proc GET /url2file {ns_return 200 text/plain
>> [ns_url2file /foo] ;#}
>> } -body {
>>     nstest_http -getbody 1 GET /url2file
>> } -cleanup {
>>     ns_unregister_proc GET /url2file
>> } -result [list 200 [ns_pagepath foo]]
>>
>>
>> This is just a simple GET request, no body.  Test error is:
>>
>>
>> ---- Test generated error; Return code was: 1
>> ---- Return code should have been one of: 0 2
>> ---- errorInfo: can not find channel named "sock24"
>>     while executing
>> "close $wfd"
>>     invoked from within
>> "nstest_http -getbody 1 GET /url2file"
>>     ("uplevel" body line 2)
>>     invoked from within
>> "uplevel 1 $script"
>> ---- errorCode: NONE
>> ==== url2file-2.1 FAILED
>>
>>
>> -------------------------------------------------------
>> This SF.Net email is sponsored by xPML, a groundbreaking scripting 
>> language
>> that extends applications into web and mobile media. Attend the live 
>> webcast
>> and join the prime developer group breaking into this new coding 
>> territory!
>> http://sel.as-us.falkag.net/sel?cmd=k&kid0944&bid$1720&dat1642
>> _______________________________________________
>> naviserver-devel mailing list
>> nav...@li...
>> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>>
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by xPML, a groundbreaking scripting language
> that extends applications into web and mobile media. Attend the live 
> webcast
> and join the prime developer group breaking into this new coding territory!
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

Re: [naviserver-devel] Test failures: while executing "close $wfd" invoked from within "nstest_http ..."

[Vlad S. <vl...@cr...>]

It should close only when post limit exceeds, other requests should not 
be affected. It looks like broke something

Stephen Deasey wrote:
> On 3/9/06, Vlad Seryakov <vl...@cr...> wrote:
>> Stephen Deasey wrote:
>>> I'm getting a lot (123) of what looks like bogus test failures on a
>>> fresh checkout.  Are the recent changes to the nstest_http proc OK?
>> Yes, i noticed this as well, this is because server now can close
>> connection during client sending the body and i tried to put catch
>> around close in all places but still it complains on close.
> 
> 
> This is new behaviour?  Why would the server close the connection
> while the client is sending data?
> 
> I'm getting in failures in e.g. test/url2file.test:url2file-2.1 which
> looks like:
> 
> 
> test url2file-2.1 {ns_url2file} -setup {
>     ns_register_proc GET /url2file {ns_return 200 text/plain
> [ns_url2file /foo] ;#}
> } -body {
>     nstest_http -getbody 1 GET /url2file
> } -cleanup {
>     ns_unregister_proc GET /url2file
> } -result [list 200 [ns_pagepath foo]]
> 
> 
> This is just a simple GET request, no body.  Test error is:
> 
> 
> ---- Test generated error; Return code was: 1
> ---- Return code should have been one of: 0 2
> ---- errorInfo: can not find channel named "sock24"
>     while executing
> "close $wfd"
>     invoked from within
> "nstest_http -getbody 1 GET /url2file"
>     ("uplevel" body line 2)
>     invoked from within
> "uplevel 1 $script"
> ---- errorCode: NONE
> ==== url2file-2.1 FAILED
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by xPML, a groundbreaking scripting language
> that extends applications into web and mobile media. Attend the live webcast
> and join the prime developer group breaking into this new coding territory!
> http://sel.as-us.falkag.net/sel?cmd=k&kid0944&bid$1720&dat1642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

Re: [naviserver-devel] Test failures: while executing "close $wfd" invoked from within "nstest_http ..."

[Stephen D. <sd...@gm...>]

On 3/9/06, Vlad Seryakov <vl...@cr...> wrote:
>
> Stephen Deasey wrote:
> > I'm getting a lot (123) of what looks like bogus test failures on a
> > fresh checkout.  Are the recent changes to the nstest_http proc OK?
>
> Yes, i noticed this as well, this is because server now can close
> connection during client sending the body and i tried to put catch
> around close in all places but still it complains on close.


This is new behaviour?  Why would the server close the connection
while the client is sending data?

I'm getting in failures in e.g. test/url2file.test:url2file-2.1 which
looks like:


test url2file-2.1 {ns_url2file} -setup {
    ns_register_proc GET /url2file {ns_return 200 text/plain
[ns_url2file /foo] ;#}
} -body {
    nstest_http -getbody 1 GET /url2file
} -cleanup {
    ns_unregister_proc GET /url2file
} -result [list 200 [ns_pagepath foo]]


This is just a simple GET request, no body.  Test error is:


---- Test generated error; Return code was: 1
---- Return code should have been one of: 0 2
---- errorInfo: can not find channel named "sock24"
    while executing
"close $wfd"
    invoked from within
"nstest_http -getbody 1 GET /url2file"
    ("uplevel" body line 2)
    invoked from within
"uplevel 1 $script"
---- errorCode: NONE
=3D=3D=3D=3D url2file-2.1 FAILED

Re: [naviserver-devel] Re: [naviserver-commits] naviserver/tests http.test,1.6,1.7

[Vlad S. <vl...@cr...>]

Yes, this is temporary, testing is not going very fast because i am busy 
at work but i keep trying to resolve this issue. Of course all your help 
is appreciated, eventually test will do overflow and weird size.

Stephen Deasey wrote:
> On 3/7/06, Vlad Seryakov <ser...@us...> wrote:
>> Update of /cvsroot/naviserver/naviserver/tests
>> In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv17927/tests
>>
>> Modified Files:
>>         http.test
>> Log Message:
>> More experiments in attempt to narrow down spooler bug
>>
>>
>> Index: http.test
>> ===================================================================
>> RCS file: /cvsroot/naviserver/naviserver/tests/http.test,v
>> retrieving revision 1.6
>> retrieving revision 1.7
>> diff -C2 -d -r1.6 -r1.7
>> *** http.test   3 Mar 2006 02:04:50 -0000       1.6
>> --- http.test   7 Mar 2006 19:37:00 -0000       1.7
>> ***************
>> *** 163,170 ****
>>       }
>>   } -body {
>> !     nstest_http -http 1.1 -getbody 1 PUT /put [string repeat x 1000000]
>>   } -cleanup {
>>       ns_unregister_proc PUT /put
>> ! } -result [list 200 [string repeat x 1000000]]
>>
>>   ns_log warning ---> about to run http-4.6
>> --- 163,170 ----
>>       }
>>   } -body {
>> !     nstest_http -http 1.1 -getbody 1 PUT /put [string repeat x 10000]
>>   } -cleanup {
>>       ns_unregister_proc PUT /put
>> ! } -result [list 200 [string repeat x 10000]]
>>
>>   ns_log warning ---> about to run http-4.6
> 
> 
> Test http-4.5 sends the most data possible, without actually going
> over the maxinput which is set in the config file as 1000001.  The
> following test, http-4.6 sends 1000002 bytes.
> 
> I'm guessing here you dropped it to 10000 bytes to make testing with
> Ns_Log statements easier.  But don't forget to also drop the maxinput
> limit and test http-4.6.
> 
> Not sure what we can reasonably lower this to.  It needs to be large
> enough not to interfere with other tests, and also to completely fill
> any internal buffers and run through any loops at least once,
> readahead limits etc.
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by xPML, a groundbreaking scripting language
> that extends applications into web and mobile media. Attend the live webcast
> and join the prime developer group breaking into this new coding territory!
> http://sel.as-us.falkag.net/sel?cmd=k&kid0944&bid$1720&dat1642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

[naviserver-devel] Re: [naviserver-commits] naviserver/tests http.test,1.6,1.7

[Stephen D. <sd...@gm...>]

On 3/7/06, Vlad Seryakov <ser...@us...> wrote:
> Update of /cvsroot/naviserver/naviserver/tests
> In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv17927/tests
>
> Modified Files:
>         http.test
> Log Message:
> More experiments in attempt to narrow down spooler bug
>
>
> Index: http.test
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> RCS file: /cvsroot/naviserver/naviserver/tests/http.test,v
> retrieving revision 1.6
> retrieving revision 1.7
> diff -C2 -d -r1.6 -r1.7
> *** http.test   3 Mar 2006 02:04:50 -0000       1.6
> --- http.test   7 Mar 2006 19:37:00 -0000       1.7
> ***************
> *** 163,170 ****
>       }
>   } -body {
> !     nstest_http -http 1.1 -getbody 1 PUT /put [string repeat x 1000000]
>   } -cleanup {
>       ns_unregister_proc PUT /put
> ! } -result [list 200 [string repeat x 1000000]]
>
>   ns_log warning ---> about to run http-4.6
> --- 163,170 ----
>       }
>   } -body {
> !     nstest_http -http 1.1 -getbody 1 PUT /put [string repeat x 10000]
>   } -cleanup {
>       ns_unregister_proc PUT /put
> ! } -result [list 200 [string repeat x 10000]]
>
>   ns_log warning ---> about to run http-4.6


Test http-4.5 sends the most data possible, without actually going
over the maxinput which is set in the config file as 1000001.  The
following test, http-4.6 sends 1000002 bytes.

I'm guessing here you dropped it to 10000 bytes to make testing with
Ns_Log statements easier.  But don't forget to also drop the maxinput
limit and test http-4.6.

Not sure what we can reasonably lower this to.  It needs to be large
enough not to interfere with other tests, and also to completely fill
any internal buffers and run through any loops at least once,
readahead limits etc.

Re: [naviserver-devel] Test failures: while executing "close $wfd" invoked from within "nstest_http ..."

[Vlad S. <vl...@cr...>]

Yes, i noticed this as well, this is because server now can close 
connection during client sending the body and i tried to put catch 
around close in all places but still it complains on close.

Stephen Deasey wrote:
> I'm getting a lot (123) of what looks like bogus test failures on a
> fresh checkout.  Are the recent changes to the nstest_http proc OK?
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by xPML, a groundbreaking scripting language
> that extends applications into web and mobile media. Attend the live webcast
> and join the prime developer group breaking into this new coding territory!
> http://sel.as-us.falkag.net/sel?cmd=k&kid0944&bid$1720&dat1642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

[naviserver-devel] Test failures: while executing "close $wfd" invoked from within "nstest_http ..."

[Stephen D. <sd...@gm...>]

I'm getting a lot (123) of what looks like bogus test failures on a
fresh checkout.  Are the recent changes to the nstest_http proc OK?

[naviserver-devel] New nstrace

[Zoran V. <zv...@ar...>]

Hi!

I have included new version of nstrace module.
I also moved (and rewritten) the ugly introspective
script in nsd/init.tcl to the nstrace.tcl.
This way, the nstrace is now sole source of scripts
for new Tcl interpreter initialization.

I hope nothing is broken with the change. If yes,
please tell me so I can fix it. Normally there are
no visible changes from the outside.
For the courious, there are now plenty of comments
in the nstrace.tcl. If there is still something unclear
about how the interp init scripts are generated, please
ask and I will be happy to answer.

I took the time to review, re-comment and partially
re-write most of the Tcl code living in the tcl/
directory. Please try to maintain coding convention
we all agreed upon when writing Tcl code. Some of the
files were very difficult to read/understand so I had
to reformat and in some cases rewrite them to understand
what is going on. I took the time to extensively document
such places for the sake of future viewers.

I noticed quite a few junk ns_* procedures polluting the
Tcl API which are mostly found in util.tcl. Plese do not
use this file as a genral-purpose-sink and think twice
before naming a Tcl procedure ns_* as this will put it
(per definition) into the public Tcl API which we will
have to a. document and b. maintain for longer time.

Cheers,
Zoran

Re: [naviserver-devel] Re: Test failures: maxinput, maxheaders not honoured

[Vlad S. <vl...@cr...>]

I was struggling with this today but still do not see the reason. It 
slows down after second big upload when spooler is enabled. By setting 
pollto = 1; in the SpoolerThread it started working but this is 
completely strange. Under strace while hanging it show only send 
requests, nobody is reading which tells that SpoolerThread is inside 
poll system call.

Stephen Deasey wrote:
> On 3/1/06, Vlad Seryakov <vl...@cr...> wrote:
>> I played with this and still not sure what is the problem. With small
>> TCP buffers and starting with 2 request it slws down and reads be small
>> chunks. netstat shows Send-Q so it might be TCP stack playing on Linux,
> 
> 
> The error is present with 2048 byte buffers, but goes away if you set
> spoolerthreads to 0 in the config file.  I don't think this is a Linux
> TCP stack problem...
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by xPML, a groundbreaking scripting language
> that extends applications into web and mobile media. Attend the live webcast
> and join the prime developer group breaking into this new coding territory!
> http://sel.as-us.falkag.net/sel?cmd=k&kid0944&bid$1720&dat1642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 


-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] Re: Test failures: maxinput, maxheaders not honoured

[Zoran V. <zv...@ar...>]

Am 02.03.2006 um 00:21 schrieb Vlad Seryakov:

> I played with this and still not sure what is the problem. With  
> small TCP buffers and starting with 2 request it slws down and  
> reads be small chunks. netstat shows Send-Q so it might be TCP  
> stack playing on Linux,
> Zoran, can you check this on Solaris.
>
If you tell me what exactly to do, I can do that.
In the meantime I just made fresh checkout and test
on Darwin. Here is the output:

[02/Mar/2006:16:16:19][13224.2684415336][-main-] Notice: nsmain: Tcl  
version: 8.4.12
[02/Mar/2006:16:16:19][13224.2684415336][-main-] Notice: nsmain:  
NaviServer/4.99.2 starting
[02/Mar/2006:16:16:19][13224.2684415336][-main-] Notice: nsmain:  
security info: uid=1005, euid=1005, gid=101, egid=101
[02/Mar/2006:16:16:19][13224.2684415336][-main-] Notice: nsmain: max  
files: FD_SETSIZE = 1024, rl_cur = 10240, rl_max = 10240
[02/Mar/2006:16:16:19][13224.2684415336][-main-] Warning: nsmain:  
rl_max > FD_SETSIZE
[02/Mar/2006:16:16:19][13224.2684415336][-main-] Notice: vhost 
[testvhost]: www.example.com:80 -> /Users/zoran/sf/naviserver/tests/ 
testserver/vhosts/e/x/a/example.com/pages
[02/Mar/2006:16:16:19][13224.2684415336][-main-] Notice: nsd/ 
init.tcl: Booting virtual server: testvhost...
[02/Mar/2006:16:16:19][13224.2684415336][-main-] Notice: conf: [ns/ 
server/testvhost]enabletclpages = 1
[02/Mar/2006:16:16:19][13224.2684415336][-main-] Notice: tcl:  
enabling .tcl pages
[02/Mar/2006:16:16:19][13224.2684415336][-main-] Notice: nsd/ 
init.tcl: Booting virtual server: testvhost2...
[02/Mar/2006:16:16:19][13224.2684415336][-main-] Notice: conf: [ns/ 
server/testvhost2]enabletclpages = 1
[02/Mar/2006:16:16:19][13224.2684415336][-main-] Notice: tcl:  
enabling .tcl pages
[02/Mar/2006:16:16:19][13224.2684415336][-main-] Notice: adp[test]:  
mapped *.adp
[02/Mar/2006:16:16:19][13224.2684415336][-main-] Notice: nsd/ 
init.tcl: Booting virtual server: test...
[02/Mar/2006:16:16:19][13224.2684415336][-main-] Notice: modload:  
loading /Users/zoran/sf/naviserver/tests/../nslog/nslog.so
[02/Mar/2006:16:16:19][13224.2684415336][-main-] Notice: nslog:  
opened '/Users/zoran/sf/naviserver/tests/testserver/access.log'
[02/Mar/2006:16:16:19][13224.2684415336][-main-] Notice: modload:  
loading /Users/zoran/sf/naviserver/tests/../nsdb/nsdb.so
[02/Mar/2006:16:16:19][13224.2684415336][-main-] Notice: modload:  
loading /Users/zoran/sf/naviserver/tests/../nsdbtest/nsdbtest.so
[02/Mar/2006:16:16:19][13224.2684415336][-main-] Notice: conf: [ns/ 
server/test]enabletclpages = 1
[02/Mar/2006:16:16:19][13224.2684415336][-main-] Notice: tcl:  
enabling .tcl pages
[02/Mar/2006:16:16:19][13224.2684415336][-main-] Notice: modload:  
loading /Users/zoran/sf/naviserver/tests/../nssock/nssock.so
[02/Mar/2006:16:16:19][13224.2684415336][-main-] Notice: nssock:  
enable 3 spooler thread(s) for uploads >= 1025 bytes
[02/Mar/2006:16:16:19][13224.2684415336][-main-] Notice: nssock:  
enable 3 writer thread(s) for downloads >= 1025 bytes, bufsize=512 bytes
[02/Mar/2006:16:16:19][13224.2684415336][-main-] Notice: nsmain:  
NaviServer/4.99.2 running
[02/Mar/2006:16:16:19][13224.2684415336][-main-] Notice: nsmain:  
security info: uid=1005, euid=1005, gid=101, egid=101
[02/Mar/2006:16:16:19][13224.41950208][-sched-] Notice: sched: starting
[02/Mar/2006:16:16:19][13224.2684415336][-main-] Notice: nssock:  
listening on 127.0.0.1:8001
[02/Mar/2006:16:16:19][13224.41951232][-spooler2-] Notice: spooler2:  
accepting connections
[02/Mar/2006:16:16:19][13224.41967104][-spooler1-] Notice: spooler1:  
accepting connections
[02/Mar/2006:16:16:19][13224.41968128][-spooler0-] Notice: spooler0:  
accepting connections
[02/Mar/2006:16:16:19][13224.41969152][-writer2-] Notice: writer2:  
accepting connections
[02/Mar/2006:16:16:19][13224.41970176][-writer1-] Notice: writer1:  
accepting connections
[02/Mar/2006:16:16:19][13224.41971200][-writer0-] Notice: writer0:  
accepting connections
[02/Mar/2006:16:16:19][13224.41972224][-driver-] Notice: starting
[02/Mar/2006:16:16:19][13224.41972224][-driver-] Notice: driver:  
accepting connections
Tests running in interp:  /Users/zoran/sf/naviserver/nsd/nsd
Tests located in:  /Users/zoran/sf/naviserver/tests
Tests running in:  /Users/zoran/sf/naviserver/tests
Temporary files stored in /Users/zoran/sf/naviserver/tests
Test files sourced into current interpreter
Running tests that match:  *
Skipping test files that match:  l.*.test
Only running test files that match:  *.test
Tests began at Thu Mar 02 16:16:20 CET 2006
encoding.test
[02/Mar/2006:16:16:20][13224.41973248][-conn:test:0] Notice:  
encoding: loaded: iso8859-2
http.test
[02/Mar/2006:16:16:20][13224.41972224][-driver-] Error: Releasing  
Socket; Bad Request (-11/0), FD = 27, Peer = 127.0.0.1:63732
[02/Mar/2006:16:16:20][13224.41969152][-writer2-] Notice: Writer:  
closed fd=30, error=0/0, sent=2048
[02/Mar/2006:16:16:20][13224.41973248][-conn:test:0] Notice: Writer:  
2: started fd=30: 2048 bytes: /2048bytes
[02/Mar/2006:16:16:20][13224.41972224][-driver-] Error: Releasing  
Socket; Request-URI Too Long (-10/0), FD = 27, Peer = 127.0.0.1:63738
[02/Mar/2006:16:16:20][13224.41972224][-driver-] Error: Releasing  
Socket; Bad Request (-11/0), FD = 27, Peer = 127.0.0.1:63739
[02/Mar/2006:16:16:20][13224.41972224][-driver-] Error: Releasing  
Socket; Bad Request (-11/0), FD = 27, Peer = 127.0.0.1:63740
[02/Mar/2006:16:16:21][13224.2684415336][-main-] Warning: ---> about  
to run http-4.5
[02/Mar/2006:16:16:21][13224.41972224][-driver-] Error: Releasing  
Socket; Request Entity Too Large (-12/0), FD = 28, Peer =  
127.0.0.1:63745
make: *** [test] Broken pipe

Re: [naviserver-devel] Re: Test failures: maxinput, maxheaders not honoured

[Stephen D. <sd...@gm...>]

On 3/1/06, Vlad Seryakov <vl...@cr...> wrote:
> I played with this and still not sure what is the problem. With small
> TCP buffers and starting with 2 request it slws down and reads be small
> chunks. netstat shows Send-Q so it might be TCP stack playing on Linux,


The error is present with 2048 byte buffers, but goes away if you set
spoolerthreads to 0 in the config file.  I don't think this is a Linux
TCP stack problem...

Re: [naviserver-devel] Re: Test failures: maxinput, maxheaders not honoured

[Vlad S. <vl...@cr...>]

I played with this and still not sure what is the problem. With small 
TCP buffers and starting with 2 request it slws down and reads be small 
chunks. netstat shows Send-Q so it might be TCP stack playing on Linux,
Zoran, can you check this on Solaris.

I put debug in SockRead and see lines like this
[01/Mar/2006:18:20:14][32388.3065797552][-spooler2-] Error: n=1228/4096 
avail=924176 max=1000001
[01/Mar/2006:18:20:14][32388.3065797552][-spooler2-] Error: n=1228/4096 
avail=925404 max=1000001
[01/Mar/2006:18:20:14][32388.3065797552][-spooler2-] Error: n=103/4096 
avail=925507 max=1000001
[01/Mar/2006:18:20:14][32388.3065797552][-spooler2-] Error: n=1536/4096 
avail=927043 max=1000001
[01/Mar/2006:18:20:14][32388.3065797552][-spooler2-] Error: n=1536/4096 
avail=928579 max=1000001

It could be the client as well, i tried to submit this from separate 
process with the same result.

Stephen Deasey wrote:
>> On 2/28/06, Stephen Deasey <sd...@gm...> wrote:
>>> Looks like the maxinput setting which is supposed to limit the max
>>> size of a file upload is not being honoured.  That's potentially
>>> nasty...
>>>
>>> With the spooler threads disabled, test http-4.5 just fails.  It
>>> returns 200 OK and the content where as it should return 413 Request
>>> Entity Too Big.
>>>
>>> With the spooler threads enabled, the test hangs.  Sometimes...  It
>>> was doing this occasionally, now it does it consistently for me.
>>> Once, it did hang just prior to this test beginning to run.
>>>
>>> Something racy going on there...
>>
>>
>> Modified Files:
>>        ChangeLog
>> Log Message:
>> Fixed error reporting during
>> request parsing, now server returns 414/400 return codes and honors
>> maxinput/maxheaders parameters. Test http-4.5 does not hang actually
>> but with sndbuf/rcvbug set to so small values it takes very logn time to
>> read the request, by putting Ns_Log in SockRead i was seeing reading by
>> 64/192 bytes but the server was operational.
> 
> 
> I don't think that's the case.  Test http-4.4 is almost exactly the
> same as http-4.5 except it sends 1000001 bytes instead of 1000003 and
> it takes only a second to run.
> 
> Experimenting further, if I drop the snd/rcv buffer sizes from 9 to 8,
> the test no longer stalls.
> 
> Increasing the buffer sizes to 16, 32, 64, 128, 256, 512, 1024 and
> 2048 the test still stalls.  So this is clearly not an issue of small
> buffers taking a long time to fill.  4096 seems to work.
> 
> Running top while the server is stalled shows that nsd is not buzzing
> the processor.  The server will run another request while the current
> one is stalled.
> 
> I can also get the stalling to go away by disabling the spooler
> threads, even with the odd buffer sizes.
> 
> Remember, occasionally this works for me if I run the test suite
> enough times.  This suggests that there is a locking or socket timeout
> race or, there's more than one bug interacting...
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by xPML, a groundbreaking scripting language
> that extends applications into web and mobile media. Attend the live webcast
> and join the prime developer group breaking into this new coding territory!
> http://sel.as-us.falkag.net/sel?cmd=k&kid0944&bid$1720&dat1642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

Re: [naviserver-devel] Re: Test Failures: maxinput, maxheaders not honoured (naviserver/nsd driver.c,1.43,1.44)

[Vlad S. <vl...@cr...>]

Right, good hint

Stephen Deasey wrote:
> On 3/1/06, Vlad Seryakov <vl...@cr...> wrote:
>>> I think this is checking that the amount of data *already* received is
>>> larger than maxinput, and if so, then rejecting.  The idea is to not
>>> read that much data in the first place.
>>
>> Yes, by the last buffer size only, it may not read the whole buffer, so
>> cheking befor for requested size is not accurate yet, i may ask for 4096
>> bytes but read only 100 for whatever reasons.
> 
> 
> What happens if you set maxinput to 20MB, and a user tries to upload a
> 21MB image?  Do they get an error message after uploading 20MB + ~8k? 
> That's how I'm reading it.
> 
> All modern browsers send a content-length header.  Maybe we should
> check the reqPtr->length field as soon as it's parsed.  Round about
> line 1760 in driver.c:
> 
>             s = Ns_SetIGet(reqPtr->headers, "content-length");
>             if (s != NULL) {
>                 int length;
> 
>                 /*
>                  * Honour meaningfull remote
>                  * content-length hints only.
>                  */
> 
>                 length = atoi(s);
>                 if (length >= 0) {
>                     reqPtr->length = length;
>                 }
>                 /* Check for maxinput here? */
>             }
> 
> 
>>> I didn't look carefully so it may only be overshooting by the receive
>>> buffer size or some such, but the same principle applies to the
>>> following check for max headers.  Here the headers have been
>>> completely parsed into an Ns_Set structure, and *then* the size is
>>> checked.  Too late...  :-(
>> It does it after each line only, whatever amount was read is in the
>> buffer already, so we parse it line by line and i check after each line.
> 
> 
> It's kind of odd when reading the code that it overshoots by one, but OK...
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by xPML, a groundbreaking scripting language
> that extends applications into web and mobile media. Attend the live webcast
> and join the prime developer group breaking into this new coding territory!
> http://sel.as-us.falkag.net/sel?cmd=k&kid0944&bid$1720&dat1642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

Re: [naviserver-devel] Re: Test Failures: maxinput, maxheaders not honoured (naviserver/nsd driver.c,1.43,1.44)

[Stephen D. <sd...@gm...>]

On 3/1/06, Vlad Seryakov <vl...@cr...> wrote:
> > I think this is checking that the amount of data *already* received is
> > larger than maxinput, and if so, then rejecting.  The idea is to not
> > read that much data in the first place.
>
>
> Yes, by the last buffer size only, it may not read the whole buffer, so
> cheking befor for requested size is not accurate yet, i may ask for 4096
> bytes but read only 100 for whatever reasons.


What happens if you set maxinput to 20MB, and a user tries to upload a
21MB image?  Do they get an error message after uploading 20MB + ~8k?=20
That's how I'm reading it.

All modern browsers send a content-length header.  Maybe we should
check the reqPtr->length field as soon as it's parsed.  Round about
line 1760 in driver.c:

            s =3D Ns_SetIGet(reqPtr->headers, "content-length");
            if (s !=3D NULL) {
                int length;

                /*
                 * Honour meaningfull remote
                 * content-length hints only.
                 */

                length =3D atoi(s);
                if (length >=3D 0) {
                    reqPtr->length =3D length;
                }
                /* Check for maxinput here? */
            }


> > I didn't look carefully so it may only be overshooting by the receive
> > buffer size or some such, but the same principle applies to the
> > following check for max headers.  Here the headers have been
> > completely parsed into an Ns_Set structure, and *then* the size is
> > checked.  Too late...  :-(
>
> It does it after each line only, whatever amount was read is in the
> buffer already, so we parse it line by line and i check after each line.


It's kind of odd when reading the code that it overshoots by one, but OK...

Re: [naviserver-devel] Re: Test Failures: maxinput, maxheaders not honoured (naviserver/nsd driver.c,1.43,1.44)

[Vlad S. <vl...@cr...>]

> I think this is checking that the amount of data *already* received is
> larger than maxinput, and if so, then rejecting.  The idea is to not
> read that much data in the first place.


Yes, by the last buffer size only, it may not read the whole buffer, so 
cheking befor for requested size is not accurate yet, i may ask for 4096 
bytes but read only 100 for whatever reasons.

> I didn't look carefully so it may only be overshooting by the receive
> buffer size or some such, but the same principle applies to the
> following check for max headers.  Here the headers have been
> completely parsed into an Ns_Set structure, and *then* the size is
> checked.  Too late...  :-(

It does it after each line only, whatever amount was read is in the 
buffer already, so we parse it line by line and i check after each line.