naviserver-devel — Developer discussion

Re: [naviserver-devel] TclOO in NaviServer, How to use?

[Anthony B. <to...@br...>]

I worked with TclOO for a little bit and made some libraries to get it 
working.  Here's some of my old code.  Ultimately I found TclOO wasn't a 
good fit for Naviserver.  Someone please chime in if things have changed 
and the garbage collection is no longer needed.  Also note that class 
creation needs to be in the ns_ictl trace create because it doesn't copy 
over to other interpreters.

ns_ictl trace create {
	
	# ModelClass - Base class. Adds garbage collection for [ClassName new]
	# Use with:
	# superclass BaseClass
	::oo::class create BaseClass {
		
		construct {} {
			# Add to garbage collection.
			lappend ::garbage::objects [self namespace]

			set className [info object class [self]]

			# Don't allow [ModelClass new]
			if {![lindex [self call] 1]} {
				return -code error "Class '$className' is abstract. Use with 'superclass $className' inside '::oo::class create DerievedClass'."
			}
		}
	}
	
}

#
# garbage
#
# TclOO does not clear temporary objects when the interpreter is deallocated.
# This namespace keeps track of temporary objects and destroys them when
# the connection closes.
#
# If you are taking up a lot of space with temporary objects then you may need to manually
# when they're no longer needed.  The garbage collector only destroys objects on connection
# end and doesn't cleanup objects you lost references to. For instance
# the following loop replaces the objet reference 1000 times. Those objects will persist
# through the duration of the connection unless manually destroyed.
#
# set value ""
# for {set i 0} {$i < 1000} {incr i} {
#  set obj [MyClass new]
#  append value [$obj myMethod]
#  $obj destroy
# }
#
# To see the memory leak without garbage collection on. Comment out
# The deallocate garbage::collect and uncomment the ns_log in allocate.
#
# Note: Your constructor should have the line "lappend ::garbage::objects [self namespace]"
# If the object is to be garbage collected. You should use "set obj [Classname new]" when
# instantiating the object instead of [Classname create obj]. Then use it with
# "$obj method" etc.
#
namespace eval garbage {
	variable objects [list]
	
	proc collect {} {
		
		set total 0
		set withError 0
		
		foreach {obj} $::garbage::objects {

			if {[namespace exists $obj] && [catch {$obj destroy} err]} {
				# NOTE: on error the object is still destroyed.
				ns_log Error "destroy $obj.\n$err."
				incr withError
			}
			
			incr total
		}
		
		if {$withError > 0} {
			ns_log Notice  "Cleared '$total' objects. With error = '$withError'."
			ns_log Notice ":oo::Obj count is [llength [namespace children ::oo Obj*]]"
		}
		
		set ::garbage::objects [list]
	}
}

ns_ictl trace deallocate {
	# Collect garbage when we leave.
	::garbage::collect
}

ns_ictl trace allocate {

	# Test for object destruction.
	#ns_log Notice ":oo::Obj count is [llength [namespace children ::oo Obj*]]"
}


- Tony

On 12/15/17 3:25 AM, Pavel Jurečka wrote:
> Hi!
>
> How can i use TclOO in Naviserver?
>
> I have tcllib file: init.tcl, and this code:
>
> namespace eval ::test {
>     oo::class create greeter {
>         method say {} {
>             ns_return 200 text/html "Hello"
>         }
>     }
> }
>
> ns_register_proc -noinherit GET /ootest {
>     set g [::test::greeter new]
>     $g say
> }
>
> Then, when i visit: 127.0.0.1/ootest <http://127.0.0.1/ootest>, i get 
> error:
>
> Error: GET /ootest, PeerAddress: 127.0.0.1
> invalid command name "::test::greeter"
>     while executing
> "::test::greeter new"
>     invoked from within
> "set g [::test::greeter new]"
>     while executing callback
> ns:tclrequest {
>     set g [::test::greeter new]
>     $g say
> }
>
> So, where should i place classes?
>
>
> Thanks for help!
> Pj.
>
>
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

[naviserver-devel] TclOO in NaviServer, How to use?

[Pavel J. <jur...@gm...>]

Hi!

How can i use TclOO in Naviserver?

I have tcllib file: init.tcl, and this code:

namespace eval ::test {
    oo::class create greeter {
        method say {} {
            ns_return 200 text/html "Hello"
        }
    }
}

ns_register_proc -noinherit GET /ootest {
    set g [::test::greeter new]
    $g say
}

Then, when i visit: 127.0.0.1/ootest, i get error:

Error: GET /ootest, PeerAddress: 127.0.0.1
invalid command name "::test::greeter"
    while executing
"::test::greeter new"
    invoked from within
"set g [::test::greeter new]"
    while executing callback
ns:tclrequest {
    set g [::test::greeter new]
    $g say
}

So, where should i place classes?


Thanks for help!
Pj.

Re: [naviserver-devel] ns_http and SNI

[Gustaf N. <ne...@wu...>]

Dear Michael,

the only sane solution is to add client SNI support to nsssl. I will try 
to look into
what this means in detail over the weekend.

all the best
-g

Am 12/6/17 um 8:35 PM schrieb Steigman, Michael:
> Hi,
>
> I am updating an application that uses TLS (with -servername option) and Tcllib's rest package to post data to an HTTPS URI. I’d like to use ns_http but based on my tinkering, it does not appear that ns_http handles SNI. I tried plain old ns_http and also ns_http with TLS, e.g.
>
> http::register https 443 ::tls::socket
> tls::init -tls1 true -ssl2 false -ssl3 false -servername my-host.com
> set h [ns_http queue -method POST -headers $queryHeaders -body "test" https://my-host.com/]
> ns_http wait -status S -result R $h
>
> I get a 503 in either case. Am I missing something?
>
> TIA,
>
> Michael
>
>
>
> The information in this e-mail is intended only for the person to whom it is
> addressed. If you believe this e-mail was sent to you in error and the e-mail
> contains patient information, please contact the Partners Compliance HelpLine at
> http://www.partners.org/complianceline . If the e-mail was sent to you in error
> but does not contain patient information, please contact the sender and properly
> dispose of the e-mail.
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel


-- 
Univ.Prof. Dr. Gustaf Neumann
WU Vienna
Institute of Information Systems and New Media
Welthandelsplatz 1, A-1020 Vienna, Austria

[naviserver-devel] ns_http and SNI

[Steigman, M. <MST...@mg...>]

Hi,

I am updating an application that uses TLS (with -servername option) and Tcllib's rest package to post data to an HTTPS URI. I’d like to use ns_http but based on my tinkering, it does not appear that ns_http handles SNI. I tried plain old ns_http and also ns_http with TLS, e.g.

http::register https 443 ::tls::socket
tls::init -tls1 true -ssl2 false -ssl3 false -servername my-host.com
set h [ns_http queue -method POST -headers $queryHeaders -body "test" https://my-host.com/]
ns_http wait -status S -result R $h

I get a 503 in either case. Am I missing something? 

TIA,

Michael



The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.

Re: [naviserver-devel] Source of non executable CGI scripts are shown

[Roderick <hr...@gm...>]

> This is different to other servers, which do not allow this. ... and 
> apparently, this is for you unexpected behavior - which can lead to
> revealing unwanted information, when not carefully set up.

Well, I though it was a bug, but it is indeed a feature. One needs only
to be carefull. I do not have a problem with it.

Rodrigo.

Re: [naviserver-devel] Source of non executable CGI scripts are shown

[Gustaf N. <ne...@wu...>]

Am 17.10.17 um 1:15 PM schrieb Roderick:
> If a cgi script is readable, but not executable, the server
> sends its source as text.
>
> Is this not a security problem? 
NaviServer allows to serve cgi-programs AND included content (images, 
css, ...) from a cgi-bin
directory. In order to identify in a cgi-directory whether a a file 
should be treated as a file or as
a cgi-script, it uses the executable flag. The source code says

    Evidently people are storing images and such in
    their cgi bin directory and they expect us to
    return these files directly.

This is different to other servers, which do not allow this. ... and 
apparently, this is
for you unexpected behavior - which can lead to revealing unwanted 
information,
when not carefully set up.

One can certainly change this, but that would break existing 
applications relying on that feature.
We can consider adding an config option to make this behavior 
configurable, where by default
serving static content this way is disallowed. I would still prefer to 
require the executable flag
to be set.

More opinions about this ?

all the best
-gn

[naviserver-devel] Source of non executable CGI scripts are shown

[Roderick <hr...@gm...>]

Dear Sirs,

If a cgi script is readable, but not executable, the server
sends its source as text.

Is this not a security problem?

Rodrigo.

Re: [naviserver-devel] "ns_server active" request IP - proxy aware?

[Gustaf N. <ne...@wu...>]

Am 06.10.17 um 1:47 PM schrieb David Osborne:
> That doesn't seem to quite do it... still getting a segfault in the 
> same place using naviserver built from tip.
>
Just to let you know: the problem seems to be resolved with the latest 
commits.

The problems were race-conditions with half-initialized connection 
structures,
since ns_server lets inspect the conn structures of all concurrently 
running
threads while these are potentially modifying/freeing these. There is 
now more
locking needed, but performance looks still ok. I get on my notebook in a VM
with debian sid still more than 7k requests/sec with ab in a stress test,
where a preauth filter calls "ns_server active" for every request.

-gn

Re: [naviserver-devel] "ns_server active" request IP - proxy aware?

[Gustaf N. <ne...@wu...>]

I've to check on the weekend, what happens here exactly. The submitted 
fix helped obviously just for "my" testcase, so it needs more 
exploration. Nevertheless, it seems to need much more locking, or more 
copying and general overhead in advance to avoid concurrency on the 
connPtr at all.

-gn

Re: [naviserver-devel] "ns_server active" request IP - proxy aware?

[David O. <da...@qc...>]

That doesn't seem to quite do it... still getting a segfault in the same
place using naviserver built from tip.

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffed783700 (LWP 25127)]
Ns_SetFindCmp (set=0x0, key=0x7ffff7bae306 "X-Forwarded-For",
cmp=0x7ffff63945a0 <__strcasecmp_avx>) at set.c:281
281 for (i = 0u; i < set->size; i++) {
(gdb) bt
#0  Ns_SetFindCmp (set=0x0, key=0x7ffff7bae306 "X-Forwarded-For",
cmp=0x7ffff63945a0 <__strcasecmp_avx>) at set.c:281
#1  0x00007ffff7b77009 in Ns_SetGetCmp (set=0x0, key=key@entry=0x7ffff7bae306
"X-Forwarded-For", cmp=<optimized out>) at set.c:330
#2  0x00007ffff7b7708c in Ns_SetIGet (set=<optimized out>,
key=key@entry=0x7ffff7bae306
"X-Forwarded-For") at set.c:497
#3  0x00007ffff7b716c0 in AppendConn (dsPtr=dsPtr@entry=0x7fffed782590,
connPtr=connPtr@entry=0x6a4ca8, state=state@entry=0x7ffff7bb1b99 "running",
checkforproxy=checkforproxy@entry=1)
    at queue.c:2416
#4  0x00007ffff7b6f114 in AppendConnList (checkforproxy=<optimized out>,
state=<optimized out>, firstPtr=0x6a4ca8, dsPtr=<optimized out>) at
queue.c:2490
#5  ServerListActive (dsPtr=0x7fffed782590, interp=<optimized out>,
objc=<optimized out>, objv=<optimized out>, poolPtr=0x6910b0,
nargs=<optimized out>) at queue.c:1041
#6  0x00007ffff7b6fc5f in NsTclServerObjCmd (clientData=0x0,
interp=0x7fffcc004f60, objc=7, objv=0x7ffff7ff914b <gettimeofday+635>) at
queue.c:1343
#7  0x00007ffff71c2dfd in TclEvalObjvInternal () from
/usr/lib/x86_64-linux-gnu/libtcl8.5.so
#8  0x00007ffff72095ce in TclExecuteByteCode () from
/usr/lib/x86_64-linux-gnu/libtcl8.5.so
#9  0x00007ffff724c6d0 in TclObjInterpProcCore () from
/usr/lib/x86_64-linux-gnu/libtcl8.5.so
#10 0x00007ffff71c2dfd in TclEvalObjvInternal () from
/usr/lib/x86_64-linux-gnu/libtcl8.5.so
#11 0x00007ffff71c3b4a in TclEvalEx () from /usr/lib/x86_64-linux-gnu/libt
cl8.5.so
#12 0x00007ffff71c3373 in Tcl_EvalEx () from /usr/lib/x86_64-linux-gnu/libt
cl8.5.so
#13 0x00007ffff7b8de10 in NsTclFilterProc (arg=0x709e70, conn=0x6a44f8,
why=NS_FILTER_PRE_AUTH) at tclrequest.c:537
#14 0x00007ffff7b63ddf in NsRunFilters (conn=conn@entry=0x6a44f8,
why=why@entry=NS_FILTER_PRE_AUTH) at filter.c:160
#15 0x00007ffff7b7131d in ConnRun (connPtr=0x6a44f8) at queue.c:2157
#16 NsConnThread (arg=0x6bbab0) at queue.c:1867
#17 0x00007ffff74bf76d in NsThreadMain (arg=<optimized out>) at thread.c:232
#18 0x00007ffff74c0799 in ThreadMain (arg=<optimized out>) at pthread.c:829
#19 0x00007ffff5e57064 in start_thread (arg=0x7fffed783700) at
pthread_create.c:309
#20 0x00007ffff635862d in clone () at ../sysdeps/unix/sysv/linux/
x86_64/clone.S:111
(gdb) f 3
#3  0x00007ffff7b716c0 in AppendConn (dsPtr=dsPtr@entry=0x7fffed782590,
connPtr=connPtr@entry=0x6a4ca8, state=state@entry=0x7ffff7bb1b99 "running",
checkforproxy=checkforproxy@entry=1)
    at queue.c:2416
2416                         p = Ns_SetIGet(connPtr->headers,
"X-Forwarded-For");
(gdb) list
2411             if ((connPtr->flags & NS_CONN_CONFIGURED) != 0u) {
2412                 const char *p;
2413
2414                 if ( checkforproxy ) {
2415                     if ((connPtr->flags & NS_CONN_CONFIGURED) != 0u) {
2416                         p = Ns_SetIGet(connPtr->headers,
"X-Forwarded-For");
2417                     } else {
2418                         Ns_Log(Notice, "AppendConn falls back to
physical peer address, since connection has no header fields");
2419                         p = NULL;
2420                     }

On 6 October 2017 at 12:18, Gustaf Neumann <ne...@wu...> wrote:

> Hi David,
>
> This was a classical race condition, where one thread was accessing the
> connPtr of another thread while it
> was de-allocating. The committed version should not lead to this situation
> anymore.
>
> In general, it is not the best approach to check in every connection the
> state of all connections, which
> is a quadratic operation on the number of running connections. I would
> rather recommend to use
> either nsvs to track connections/peers, or to use a separate monitoring
> thread keeping this information
> up to date, receiving information about starting and ending threads.
>
> please check, if the changes work as well with your setup and tests.
> -g
>
> Am 06.10.17 um 11:46 AM schrieb David Osborne:
>
> However, the ns_server call is the first thing it does. In fact I can
> strip the filter down to only do the following and still get the same
> signal 11 (it also segfaults if returning  filter_ok but takes longer)
>
> proc filter_dos_check { event args } {
>     set requests_active [ns_server -server [ns_info server] -pool {}
> active -checkforproxy]
>     return filter_return
> }
>
> Thanks for the patch.
>
>
>

Re: [naviserver-devel] "ns_server active" request IP - proxy aware?

[Gustaf N. <ne...@wu...>]

Hi David,

This was a classical race condition, where one thread was accessing the 
connPtr of another thread while it
was de-allocating. The committed version should not lead to this 
situation anymore.

In general, it is not the best approach to check in every connection the 
state of all connections, which
is a quadratic operation on the number of running connections. I would 
rather recommend to use
either nsvs to track connections/peers, or to use a separate monitoring 
thread keeping this information
up to date, receiving information about starting and ending threads.

please check, if the changes work as well with your setup and tests.
-g

Am 06.10.17 um 11:46 AM schrieb David Osborne:
> However, the ns_server call is the first thing it does. In fact I can 
> strip the filter down to only do the following and still get the same 
> signal 11 (it also segfaults if returning filter_ok but takes longer)
>
> proc filter_dos_check { event args } {
>     set requests_active [ns_server -server [ns_info server] -pool {} 
> active -checkforproxy]
>     return filter_return
> }
>
> Thanks for the patch.
>

Re: [naviserver-devel] "ns_server active" request IP - proxy aware?

[David O. <da...@qc...>]

Thanks Gustaf,

I can only really reproduce it by hammering my development instance using
apache bench (and only with a concurrency level of >100 after a few hundred
requests) - not very scientific.

The purpose of the filter_dos_check filter is to examine the server load
(via ns_server active) and potentially return a 503 if we're too busy.

However, the ns_server call is the first thing it does. In fact I can strip
the filter down to only do the following and still get the same signal 11
(it also segfaults if returning  filter_ok but takes longer)

proc filter_dos_check { event args } {
    set requests_active [ns_server -server [ns_info server] -pool {} active
-checkforproxy]
    return filter_return
}

Thanks for the patch.

If there's anything I could capture during the segfault which would help
let me know.



On 5 October 2017 at 20:22, Gustaf Neumann <ne...@wu...> wrote:

>
> Hi David,
>
> so far, i could not recreate the problem. By reading the backtrace, the
> problem
> seems to be that the connection has no ns_set for the header fields. This
> is strange,
> since every ConnRun() sets these fields, an these are cleared after the
> ConnRun(),
> while the problem happens within ConnRun().
> So i wonder, how connPtr->headers can be lost. I see from the backtrace,
> that the issue
> happens in a preauth filter named  "filter_dos_check". Can it be, that
> this filter somehow
> causes a cleanup of the connection? I can't see in the code, where this
> might happen.
>
> For the time being, i've protected the crashing function call with test
> for the
> headers and added a debug line for the error.log. So, the crash at this
> very situation should disappear.  But probably, the function should not
> be called in such situations at all, there might be as well other places,
> that depend on the assumption, that the ns_set for the header exists
> during the run.
>
> Do you see anything in your filter code related with this?
> -g
>
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>
>

Re: [naviserver-devel] "ns_server active" request IP - proxy aware?

[Gustaf N. <ne...@wu...>]

Hi David,

so far, i could not recreate the problem. By reading the backtrace, the 
problem
seems to be that the connection has no ns_set for the header fields. 
This is strange,
since every ConnRun() sets these fields, an these are cleared after the 
ConnRun(),
while the problem happens within ConnRun().
So i wonder, how connPtr->headers can be lost. I see from the backtrace, 
that the issue
happens in a preauth filter named  "filter_dos_check". Can it be, that 
this filter somehow
causes a cleanup of the connection? I can't see in the code, where this 
might happen.

For the time being, i've protected the crashing function call with test 
for the
headers and added a debug line for the error.log. So, the crash at this
very situation should disappear.  But probably, the function should not
be called in such situations at all, there might be as well other places,
that depend on the assumption, that the ns_set for the header exists 
during the run.

Do you see anything in your filter code related with this?
-g

Re: [naviserver-devel] "ns_server active" request IP - proxy aware?

[David O. <da...@qc...>]

Hi,

After an initial rollout we made a change to our code which massive
increased the number of times ns_server is called with the -checkforproxy
switch - we than started to get intermittent seg faults.

I can reproduce this is test with gdb.
It does seem to be related to the new section of code which checks the
headers for the X-Forwarded-For value.

The backtrace is here:
https://gist.github.com/davidqc/16cdd932509654f3d124e067a9a781bd
Naviserver is built from tip as of 3rd Oct.

I should mention I'm also getting some warnings during build on debian
jessie - not sure if serious or not:
https://gist.github.com/davidqc/a4569fa50d786349b3797c9105754be1

Can you see anything obvious going on here?

Will paste the contents of the links here also:

Backtrace:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffa67fc700 (LWP 15992)]
Ns_SetFindCmp (set=set@entry=0x0, key=0x7ffff7ba851f "X-Forwarded-For",
cmp=0x7ffff63885a0 <__strcasecmp_avx>) at set.c:281
281     set.c: No such file or directory.
(gdb) bt
#0  Ns_SetFindCmp (set=set@entry=0x0, key=0x7ffff7ba851f "X-Forwarded-For",
cmp=0x7ffff63885a0 <__strcasecmp_avx>) at set.c:281
#1  0x00007ffff7b6d659 in Ns_SetGetCmp (set=0x0, key=<optimized out>,
cmp=<optimized out>) at set.c:330
#2  0x00007ffff7b67ce0 in AppendConn (dsPtr=dsPtr@entry=0x7fffa67fb570,
connPtr=connPtr@entry=0x6ae908, state=state@entry=0x7ffff7bafcb6 "running",
checkforproxy=checkforproxy@entry=1) at queue.c:2393
#3  0x00007ffff7b65254 in AppendConnList (checkforproxy=<optimized out>,
state=<optimized out>, firstPtr=0x6ae908, dsPtr=<optimized out>) at
queue.c:2456
#4  ServerListActive (dsPtr=0x7fffa67fb570, interp=<optimized out>,
objc=<optimized out>, objv=<optimized out>, poolPtr=0x651820,
nargs=<optimized out>) at queue.c:1041
#5  0x00007ffff7b662a3 in NsTclServerObjCmd (clientData=0x0,
interp=0x7fffc80054f0, objc=-164067936, objv=0x7fffc9a7bca0) at queue.c:1343
#6  0x00007ffff71b5e59 in TclEvalObjvInternal (interp=0x7fffc80054f0,
objc=-138771169, objv=0x7fffc9a7bca0, command=0xffffffffffffffff <error:
Cannot access memory at address 0xffffffffffffffff>,
    length=-1, flags=0) at
/tmp/buildd/tcl8.5-8.5.17/unix/../generic/tclBasic.c:3727
#7  0x00007ffff71fc95e in TclExecuteByteCode (interp=0x0,
interp@entry=0x7fffc80054f0,
codePtr=0x7ffff7ba851f, codePtr@entry=0x7fffb4367690)
    at /tmp/buildd/tcl8.5-8.5.17/unix/../generic/tclExecute.c:2416
#8  0x00007ffff723fce9 in TclObjInterpProcCore (interp=0x7fffc80054f0,
procNameObj=0x7fffd74b8f70, skip=1, errorProc=0x7ffff723fee0
<MakeProcError>)
    at /tmp/buildd/tcl8.5-8.5.17/unix/../generic/tclProc.c:1763
#9  0x00007ffff71b5e59 in TclEvalObjvInternal (interp=0x7fffc80054f0,
objc=-138771169, objv=0x7fffc9a7b9c0, command=0x7fffa67fbc20
"filter_dos_check preauth", length=24, flags=0)
    at /tmp/buildd/tcl8.5-8.5.17/unix/../generic/tclBasic.c:3727
#10 0x00007ffff71b6b29 in TclEvalEx (interp=0x2, script=0x7ffff7ba851f
"X-Forwarded-For", numBytes=-164067936, flags=-939502352, line=-911754816,
clNextOuter=clNextOuter@entry=0x0,
    outerScript=0x7fffa67fbc20 "filter_dos_check preauth") at
/tmp/buildd/tcl8.5-8.5.17/unix/../generic/tclBasic.c:4426
#11 0x00007ffff71b6473 in Tcl_EvalEx (interp=interp@entry=0x7fffc80054f0,
script=<optimized out>, numBytes=<optimized out>, flags=flags@entry=0)
    at /tmp/buildd/tcl8.5-8.5.17/unix/../generic/tclBasic.c:4083
#12 0x00007ffff7b84f58 in NsTclFilterProc (arg=0x29da610, conn=0x6ace20,
why=NS_FILTER_PRE_AUTH) at tclrequest.c:537
#13 0x00007ffff7b59c3f in NsRunFilters (conn=conn@entry=0x6ace20,
why=why@entry=NS_FILTER_PRE_AUTH) at filter.c:160
#14 0x00007ffff7b6775d in ConnRun (connPtr=0x6ace20) at queue.c:2143
#15 NsConnThread (arg=0x6af660) at queue.c:1861
#16 0x00007ffff74b397d in NsThreadMain (arg=<optimized out>) at thread.c:232
#17 0x00007ffff74b49a9 in ThreadMain (arg=<optimized out>) at pthread.c:829
#18 0x00007ffff5e4b064 in start_thread (arg=0x7fffa67fc700) at
pthread_create.c:309
#19 0x00007ffff634c62d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:111


Warnings:


gcc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -g
-Wall -fPIC -g -O2 -fstack-protector-strong -Wformat
-Werror=format-security -fno-unit-at-a-time -pipe -D_FORTIFY_SOURCE=2
-I../include -I"/usr/include/tcl8.5"  -DHAVE_CONFIG_H  -g -Wall -fPIC -g
-O2 -fstack-protector-strong -Wformat -Werror=format-security
-fno-unit-at-a-time -pipe -D_FORTIFY_SOURCE=2 -I../include
-I"/usr/include/tcl8.5"  -DHAVE_CONFIG_H   -D_FORTIFY_SOURCE=2  -c -o
cache.o cache.c
In file included from ../include/nsthread.h:503:0,
                 from ../include/ns.h:41,
                 from nsd.h:33,
                 from cache.c:37:
cache.c: In function 'Ns_CacheSetValueExpires':
cache.c:602:65: warning: passing argument 2 of
'cachePtr->uncommittedTable.createProc' from incompatible pointer type
         (void) Tcl_CreateHashEntry(&cachePtr->uncommittedTable, ePtr,
&isNew);
                                                                 ^
/usr/include/tcl8.5/tcl.h:2388:40: note: in definition of macro
'Tcl_CreateHashEntry'
  (*((tablePtr)->createProc))(tablePtr, key, newPtr)
                                        ^
cache.c:602:65: note: expected 'const char *' but argument is of type
'struct Entry *'
         (void) Tcl_CreateHashEntry(&cachePtr->uncommittedTable, ePtr,
&isNew);
                                                                 ^
/usr/include/tcl8.5/tcl.h:2388:40: note: in definition of macro
'Tcl_CreateHashEntry'
  (*((tablePtr)->createProc))(tablePtr, key, newPtr)
                                        ^
cache.c: In function 'Ns_CacheDeleteEntry':
cache.c:751:65: warning: passing argument 2 of
'ePtr->cachePtr->uncommittedTable.findProc' from incompatible pointer type
     hPtr = Tcl_FindHashEntry(&ePtr->cachePtr->uncommittedTable, ePtr);
                                                                 ^
/usr/include/tcl8.5/tcl.h:2385:38: note: in definition of macro
'Tcl_FindHashEntry'
  (*((tablePtr)->findProc))(tablePtr, key)
                                      ^
cache.c:751:65: note: expected 'const char *' but argument is of type
'struct Entry *'
     hPtr = Tcl_FindHashEntry(&ePtr->cachePtr->uncommittedTable, ePtr);
                                                                 ^
/usr/include/tcl8.5/tcl.h:2385:38: note: in definition of macro
'Tcl_FindHashEntry'
  (*((tablePtr)->findProc))(tablePtr, key)
                                      ^
cache.c: In function 'CacheTransaction':
/usr/include/tcl8.5/tcl.h:2373:2: warning: initialization from incompatible
pointer type
  ((char *) (((tablePtr)->keyType == TCL_ONE_WORD_KEYS || \
  ^
cache.c:939:28: note: in expansion of macro 'Tcl_GetHashKey'
         Ns_Entry  *entry = Tcl_GetHashKey(&cachePtr->uncommittedTable,
hPtr);

Re: [naviserver-devel] "ns_server active" request IP - proxy aware?

[David O. <da...@qc...>]

Thanks very much for reviewing the pull request. We've got it rolled out
now and it's working well.
Regards,
David


On 30 September 2017 at 18:34, Gustaf Neumann <ne...@wu...> wrote:

> Dear David,
>
> this is in essence a good addition - i though about this as well in the
> past.
>
> One alterative to the option would be a global configuration parameter
> (maybe generalizing "checkforproy" for nslog), such that a webmaster
> can say "i am running behind a proxy, i don't care about the physical
> upstream peer". However, it might as well be that in some cases
> the true peer address is needed, and sometimes the proxied one ...
> which is possible with your modifications.
>
> I have made one essential change by moving "-checkforproxy" to the last
> position
> using "ns_server active -checkforproxy". If it is used before the
> sub-command,
> "-checkforproxy" is applicable to all "ns_server" sub-commands (like e.g.
> "ns_server pagedir"), where the option does nothing but confusing a user.
> It was as well showing up in the error messages for invalid commands.
> By moving it to the end, it can be made specific to the three sub-commands
> where it makes sense ("active", "queued", and "all").
>
> all the best
> -gn
>
> Am 29.09.17 um 5:50 PM schrieb David Osborne:
>
> Hi,
>
> We have a need for ns_server active & queued commands to be proxy-aware
> in the IP addresses that they report.
>
> Similar to how the access log IP address can be configured to report the
> X-Forwarded-For value (if it exists)
> <https://bitbucket.org/naviserver/naviserver/src/5436d27a76802882b66d2e701a781a40693028db/nslog/nslog.c?at=default&fileviewer=file-view-default#nslog.c-588>,
> could we provide a command switch for ns_server which would do the same?
>
> I've created a pull request to show what I mean.
> https://bitbucket.org/naviserver/naviserver/pull-requests/
> 16/checkforproxy-switch-for-ns_server/diff
>
> Does this seem sensible?
>
> Regards,
>
> --
> David
>
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>
>

Re: [naviserver-devel] "ns_server active" request IP - proxy aware?

[Gustaf N. <ne...@wu...>]

Dear David,

this is in essence a good addition - i though about this as well in the 
past.

One alterative to the option would be a global configuration parameter
(maybe generalizing "checkforproy" for nslog), such that a webmaster
can say "i am running behind a proxy, i don't care about the physical
upstream peer". However, it might as well be that in some cases
the true peer address is needed, and sometimes the proxied one ...
which is possible with your modifications.

I have made one essential change by moving "-checkforproxy" to the last 
position
using "ns_server active -checkforproxy". If it is used before the 
sub-command,
"-checkforproxy" is applicable to all "ns_server" sub-commands (like e.g.
"ns_server pagedir"), where the option does nothing but confusing a user.
It was as well showing up in the error messages for invalid commands.
By moving it to the end, it can be made specific to the three sub-commands
where it makes sense ("active", "queued", and "all").

all the best
-gn

Am 29.09.17 um 5:50 PM schrieb David Osborne:
> Hi,
>
> We have a need for ns_server active & queued commands to be 
> proxy-aware in the IP addresses that they report.
>
> Similar to how the access log IP address can be configured to report 
> the X-Forwarded-For value (if it exists) 
> <https://bitbucket.org/naviserver/naviserver/src/5436d27a76802882b66d2e701a781a40693028db/nslog/nslog.c?at=default&fileviewer=file-view-default#nslog.c-588>, 
> could we provide a command switch for ns_server which would do the same?
>
> I've created a pull request to show what I mean.
> https://bitbucket.org/naviserver/naviserver/pull-requests/16/checkforproxy-switch-for-ns_server/diff 
> <https://bitbucket.org/naviserver/naviserver/pull-requests/16/checkforproxy-switch-for-ns_server/diff>
>
> Does this seem sensible?
>
> Regards,
>
> -- 
> David

[naviserver-devel] "ns_server active" request IP - proxy aware?

[David O. <da...@qc...>]

Hi,

We have a need for ns_server active & queued commands to be proxy-aware in
the IP addresses that they report.

Similar to how the access log IP address can be configured to report the
X-Forwarded-For value (if it exists)
<https://bitbucket.org/naviserver/naviserver/src/5436d27a76802882b66d2e701a781a40693028db/nslog/nslog.c?at=default&fileviewer=file-view-default#nslog.c-588>,
could we provide a command switch for ns_server which would do the same?

I've created a pull request to show what I mean.
https://bitbucket.org/naviserver/naviserver/pull-requests/16/checkforproxy-
switch-for-ns_server/diff

Does this seem sensible?

Regards,

-- 
David

Re: [naviserver-devel] letsencrypt client

[Gustaf N. <ne...@wu...>]

Am 13.09.17 um 2:40 PM schrieb Zoran Vasiljevic:
> On 13 Sep 2017, at 14:34, Gustaf Neumann <ne...@wu...> wrote:
>
>> You might be interested in the NaviServer letsencrypt client.
> Interesting... That would make self-signed certificates obsolete
> if I understand correctly?
yes, ...  these are nice certificates, checkout [1]

> So you wrote the client for NS?
> As an NS module?
yes... it is currently a single-page application, similar to nsstats
-g

[1] https://www.ssllabs.com/ssltest/analyze.html?d=openacs.org

Re: [naviserver-devel] letsencrypt client

[Zoran V. <zv...@ar...>]

On 13 Sep 2017, at 14:34, Gustaf Neumann <ne...@wu...> wrote:

> You might be interested in the NaviServer letsencrypt client.

Interesting... That would make self-signed certificates obsolete
if I understand correctly? So you wrote the client for NS?
As an NS module? 

[naviserver-devel] letsencrypt client

[Gustaf N. <ne...@wu...>]

Dear all,

You might be interested in the NaviServer letsencrypt client.
I have used this for the multi-domain certificate on openacs.org.

all the best
-gn

[naviserver-devel] Virtual server reform

[Gustaf N. <ne...@wu...>]

Dear all,

The virtual server setup of NaviServer was in part tricky
and buggy, so i did the following changes to improve the situation:

- One problem was the maintenance of the virtual server map:
   a) for drivers listening on a non-standard port, one has to
       list the entry with the port
   b) for drivers on the standard port (80/443), one has to list
      two entries, one with and one without the port.

  These rules are from [1]. since NaviServer used this entries
  as-is, one had to repeat the port for every host in the configuration,
  and to add entries without the port, when the driver should
  listen on the default port.

- When doing so, i found bugs, where it was possible that
   the start of NaviServer was aborted, since a server was
   not yet defined, or that a driver might have been initialized
   twice (when multiple servers use the same driver)

Up to my knowledge, everything continues to work, also the
more complex case with
- 2 drivers (nssock1 and nssock2)
- 2 servers (s1 and s2), and
- 3 "virtual hosts"

The new version is already running at OpenACS.org,
which has two servers and 8 virtual servers defined.

If you have complex server/virtual server setups, please test if
possible

all the best

-gn

###############################################################
# Define two servers
#
ns_section ns/servers
ns_param   s1      "Virtual Server 1"
ns_param   s2      "Virtual Server 2"

ns_section ns/server/s1/fastpath
ns_param   pagedir  /tmp/ns/s1

ns_section ns/server/s2/fastpath
ns_param   pagedir  /tmp/ns/s2

#
# Define two nssock drivers, listening on port 8001 and 8002
#
ns_section ns/modules
ns_param   nssock1   nssock.so
ns_param   nssock2   nssock.so

ns_section ns/module/nssock1
ns_param   port          8001
ns_param   defaultserver s1

ns_section ns/module/nssock2
ns_param   port          8002
ns_param   defaultserver s2

#
# Directing requests to three virtual servers (foo.com, bar.com,
# baz.com), which are mapped to the servers s1 and s2.
#
ns_section ns/module/nssock1/servers
ns_param   s1      foo.com
ns_param   s2      bar.com

ns_section ns/module/nssock2/servers
ns_param   s2      baz.com
###############################################################



Virtual server reform:
- new feature: Make mapping of host entries in the virtual
   server definition more intelligent to avoid newbie gotchas and
   hard to find mis-configurations (entries in e.g. nssock/servers)
   a) add automatically an entry with the port, when non is given
   b) complain, when driver is not listening on the specified port
   c) add automatically an entry without the port, when the driver
      listening on the default port
   Old configurations (doing a-z manually) should continue to work,
   non RFC compliant (test) cases have to be fixed (when a driver
   uses a non-default port, and the request contains no port)
- Added documentation
- bugfix: Avoid potential double-initialization of driver modules when
   multiple servers are used.
- bugfix: Postpone registration of virtual servers until all servers
   are defined. Before it was possible that NaviServer boot was
   terminated, when a default server of a driver was not yet defined.

[1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.23

Re: [naviserver-devel] Where is nspasswd utility?

[Gustaf N. <ne...@wu...>]

all of the installations i know better use NaviServer with OpenACS, 
which uses its own rights and user management system - so there is 
nsperm involved. So others might give you better informed answers.

What i can see is that the "passwdfile" is only used in htaccess.tcl, 
which is only used, when "htaccess" is true.
I'll try to improve the documentation.

-gn

[1] 
https://bitbucket.org/naviserver/naviserver/src/tip/nsperm/htaccess.tcl?at=default&fileviewer=file-view-default

Am 23.08.17 um 8:31 PM schrieb Roderick:
>
> To my preivious mail, I want to add the following experience.
>
> With passwd in the default place, auth with perm works, but not with
> htaccess (browser asks pass, but deny access).
>
> Defining a passwd not in the default place as at the end of
> doc/html/nsperm/files/nsperm.html, neither perm nor htaccess
> works (browser ask, but deny access in both cases). But
> "ns_perm checkpass user passwd" confirms that the pass was OK.
>
> I thank for any hint
> Rodrigo.
>
> ------------------------------------------------------------------------------ 
>
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel


-- 
Univ.Prof. Dr. Gustaf Neumann
WU Vienna
Institute of Information Systems and New Media
Welthandelsplatz 1, A-1020 Vienna, Austria

Re: [naviserver-devel] Global variables not cloned?!

[Gustaf N. <ne...@wu...>]

Dear Rodrigo,

Some of the man pages are unfortunately quite old and show many 
potential for improvements.
Please check the updated documentation at
https://bitbucket.org/naviserver/naviserver/commits/8c036c9c5b830d24c469eb1eb4a6c395fdf73226

in short: the blueprint contains just namespaced variables (global 
variables, which are not
in the topmost namespace "::"). If your application needs global 
variables in the interpreter,
define your own Tcl namespace and keep your variables there.

all the best
-gn


Am 28.08.17 um 9:39 AM schrieb Roderick:
>
> In naviserver-4.99.15/doc/html/manual/files/tcl-overview.html we read:
>
>>>>
> Tcl Interpreters
>
> During NaviServer initialization, only one interpreter exists. While 
> modules are loaded and initialized, they may add procedures to the 
> interpreter. When initialization is complete (all modules are loaded 
> and all Tcl libraries have been executed), the interpreter may no 
> longer be changed.
>
> Each connection thread that requires Tcl will create a copy of the 
> original interpreter.
> <<<
>
> That they may add procedures does not exclude that they also may
> add global variables.
>
> After some experiments, it seems to me that at some point global
> variables in the loaded modules *and* libraries are unset, I suppose
> immeadately after modules are loaded and libraries are executed.
> Is it true? When it happens? An explicit hint to it in the documentation
> would help beginners a lot.
>
> It is a question of taste: I would preffer to decide what variables
> to unset and not let the server do it automatically, but it could
> be a source of chaos.
>
> Thanks
> Rodrigo.
>
> ------------------------------------------------------------------------------ 
>
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel


-- 
Univ.Prof. Dr. Gustaf Neumann
WU Vienna
Institute of Information Systems and New Media
Welthandelsplatz 1, A-1020 Vienna, Austria

[naviserver-devel] does NsDbLogSql() do anything?

[Andrew P. <at...@pi...>]

NsDbLogSql() is defined in naviserver/nsdb/dbinit.c, and sometimes
calls Ns_Log(Ns_LogSqlDebug).  But at least with the nsoracle driver,
NsDbLogSql() apparently NEVER gets called at all!  (I temporarily put
in Ns_Log calls to be sure.)  Is it supposed to?

After learning about NaviServer's "ns_logctl severity Debug(sql) on"
approach to SQL logging, I fixed nsoracle so that SQL logging actually
works:

  https://bitbucket.org/naviserver/nsoracle/commits/cf5986c3b2e6ecd5cf9970d4c7ad37f667fc4c62
  http://openacs.org/forums/message-view?message_id=5374519
  https://bitbucket.org/naviserver/naviserver/src/tip/openacs-config.tcl?fileviewer=file-view-default

But I was surprised that the Ns_log statements that make the SQL
logging happen, are apparently all issued directly by the Oracle
driver.  The generic nsdb layer and NsDbLogSql() don't seem to be
involved at all.  Is that normal for NaviServer database drivers, or
is there something funny about nsoracle?

-- 
Andrew Piskorski <at...@pi...>

[naviserver-devel] Global variables not cloned?!

[Roderick <hr...@gm...>]

In naviserver-4.99.15/doc/html/manual/files/tcl-overview.html we read:

>>>
Tcl Interpreters

During NaviServer initialization, only one interpreter exists. While 
modules are loaded and initialized, they may add procedures to the 
interpreter. When initialization is complete (all modules are loaded and 
all Tcl libraries have been executed), the interpreter may no longer be 
changed.

Each connection thread that requires Tcl will create a copy of the 
original interpreter.
<<<

That they may add procedures does not exclude that they also may
add global variables.

After some experiments, it seems to me that at some point global
variables in the loaded modules *and* libraries are unset, I suppose
immeadately after modules are loaded and libraries are executed.
Is it true? When it happens? An explicit hint to it in the documentation
would help beginners a lot.

It is a question of taste: I would preffer to decide what variables
to unset and not let the server do it automatically, but it could
be a source of chaos.

Thanks
Rodrigo.