naviserver-devel — Developer discussion

[naviserver-devel] Fwd: NaviServer

[Sassy N. <sa...@gm...>]

I'm doing some work for a customer of mine, and he is still running
aolserver 3!!!
It is 15 years system running on fedora server! really old one with
postgres 8.

In my test I have manage to move the server to work with postgres 9.6 and
aolserver 4.5, but I still on the process for moving it to run under
NaviSever.

I want it to be run with the latest navisever and postgres 10.

The problem is with the ns_share command.

The server use some old tcl script from ArsDigita-Community-System-3.2.3
which use ns_share command many times. I would like to avoid fixing the
code here.

I saw there is some module name that implements some long-depreciated old
AOLserver commands
<https://bitbucket.org/naviserver/nsshare/src/099864af6816?at=default>.
However I never managed to compile it.

I get some error like this:

root@devplatfrom:/tmp/1# make NAVISERVER=/usr/lib/naviserver
DESTDIR=/tmp/121212/
gcc -O2 -fomit-frame-pointer -DNDEBUG -Wall -fPIC -g -O2
-fdebug-prefix-map=/build/tcl8.6-1Mwawt/tcl8.6-8.6.8+dfsg=.
-fstack-protector-strong -Wformat -Werror=format-security
-fno-unit-at-a-time -pipe -Wdate-time -D_FORTIFY_SOURCE=2
-I/usr/lib/naviserver/include -I"/usr/include/tcl8.6"  -DHAVE_CONFIG_H
-c -o nsshare.o nsshare.c
nsshare.c:93:1: error: conflicting types for ‘InitInterp’
 InitInterp(Tcl_Interp *interp, void *arg)
 ^~~~~~~~~~
nsshare.c:46:24: note: previous declaration of ‘InitInterp’ was here
 static Ns_TclTraceProc InitInterp;
                        ^~~~~~~~~~
nsshare.c:46:24: warning: ‘InitInterp’ used but never defined
nsshare.c:93:1: warning: ‘InitInterp’ defined but not used
[-Wunused-function]
 InitInterp(Tcl_Interp *interp, void *arg)
 ^~~~~~~~~~
<builtin>: recipe for target 'nsshare.o' failed
make: *** [nsshare.o] Error 1


Looking in the code, it seems to be like all other modules which I manage
to build, (nsaccess for example or nsdbpg etc..)

What do I missing? as to do  refactoring  to the
ArsDigita-Community-System-3.2.3
<https://github.com/MikeSisk/ArsDigita-Community-System-3.2.3/tree/master/tcl>
might be a huge risk.

I saw this https://openacs.org/forums/message-view?message_id=4263411, so I
guess there is a work around, and it might be even better to work with nsv
commands, But maybe we can make the module works?

Thank You!

Re: [naviserver-devel] Install NS on Debian

[Iuri S. <iu...@iu...>]

Hi Gustaf,

> Can it be, that you have an old version of the modules tar file?
> 
The directory /usr/local/src was clean. I removed everything before starting the upgrade process. 

> You mentioned in another post, that you have modified the install script.
> 
That was to install latest TCL (v.8.6) and tDOM (v.0.9) . As I noticed you got the scripts updated with those libraries, I’ve deleted my scripts and cloned yours. 

> Can your modification be the cause? 
> 

I've cloned the script from Github’s repository. I've done no modifications.

>> https://github.com/gustafn/install-ns.git <https://github.com/gustafn/install-ns.git>

Before, installing Naviserver, I ran apt-get update and apt-get upgrade. Afterwards,  If there were URLs pointed pointing to old version of the modules, they were downloaded automatically.

The box I ran the script is :
iuri@iurix:~$ echo $(uname) 
Linux



Thus, It probably got into the loop

  if [ -f "/etc/debian_version" ] ; then
	debian=1
	if [ $with_postgres = "1" ] ; then
	    pg_packages="postgresql libpq-dev"
	fi


However, lipq-dev was installed properly.   
Well, this server is very old. It’s a Debian Linux, from 2005. Since then, I always run upgrades and never rebuild the machine. Perhaps, there was a very old library running still. I can’t tell. Anyway, I resolved the problem, by installing pg9.5, via backports. 
 
Best wishes,
Iuri




> On Apr 2, 2018, at 04:51, Gustaf Neumann <ne...@wu...> wrote:
> 
> On 4/2/18 4:43 AM, Iuri Sampaio wrote:
>> In attempt to upgrade Naviserver to v.4.99.16 I’ve got the following error, running the current installation script install-ns.sh, available at GIT repository https://github.com/gustafn/install-ns.git <https://github.com/gustafn/install-ns.git>
>> 
>> Operating System: Debian 7 (wheezy)
>> Database: PostgreSQL 9.1 (installed via apt-get install)
>> 
>> Isn’t PGRES_SINGLE_TUPLE a processor MACRO on Postgres scripts?
> As far i can see, this problem was fixed on 2017-12-11 [1], which was before the release date of NaviServer 4.99.16 (on 2017-12-29). So, the released tar file of the modules (naviserver-4.99.16-modules.tar.gz) should contain this fix. If the install script is running successfully, it always gets the modules tar file and recompiles everything. 
> Can it be, that you have an old version of the modules tar file? You mentioned in another post, that you have modified the install script. Can your modification be the cause? 
> all the best
> 
> -gn
> [1] https://bitbucket.org/naviserver/nsdbpg/commits/5a5fa7547f5b64233891a3e4d3077527b37feffc <https://bitbucket.org/naviserver/nsdbpg/commits/5a5fa7547f5b64233891a3e4d3077527b37feffc>------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot_______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] Install NS on Debian

[Gustaf N. <ne...@wu...>]

On 4/2/18 4:43 AM, Iuri Sampaio wrote:
> In attempt to upgrade Naviserver to v.4.99.16 I’ve got the following 
> error, running the current installation script install-ns.sh, 
> available at GIT repository https://github.com/gustafn/install-ns.git
>
> Operating System: Debian 7 (wheezy)
> Database: PostgreSQL 9.1 (installed via apt-get install)
>
> Isn’t PGRES_SINGLE_TUPLE a processor MACRO on Postgres scripts?

As far i can see, this problem was fixed on 2017-12-11 [1], which was 
before the release date of NaviServer 4.99.16 (on 2017-12-29). So, the 
released tar file of the modules (naviserver-4.99.16-modules.tar.gz) 
should contain this fix. If the install script is running successfully, 
it always gets the modules tar file and recompiles everything.

Can it be, that you have an old version of the modules tar file? You 
mentioned in another post, that you have modified the install script. 
Can your modification be the cause?

all the best

-gn

[1] 
https://bitbucket.org/naviserver/nsdbpg/commits/5a5fa7547f5b64233891a3e4d3077527b37feffc

Re: [naviserver-devel] Install NS on Debian

[Iuri S. <iu...@iu...>]

Hi there,

The Problem was solved by installing PostgreSQL 95, using backports as I mentioned previously.
> https://www.tqhosting.com/kb/608/How-to-install-PostgreSQL-95-on-Debian-76-Wheezy.html <https://www.tqhosting.com/kb/608/How-to-install-PostgreSQL-95-on-Debian-76-Wheezy.html>



Best wishes,
Iuri

> On Apr 1, 2018, at 23:43, Iuri Sampaio <iu...@iu...> wrote:
> 
> Hi there,
> 
> 
> In attempt to upgrade Naviserver to v.4.99.16 I’ve got the following error, running the current installation script install-ns.sh, available at GIT repository https://github.com/gustafn/install-ns.git <https://github.com/gustafn/install-ns.git>
> 
> Operating System: Debian 7 (wheezy)
> Database: PostgreSQL 9.1 (installed via apt-get install)
> 
> Isn’t PGRES_SINGLE_TUPLE a processor MACRO on Postgres scripts?
> 
> Should I instal PostgreSQL 9.5 to Debian Wheezy, using backports.  I’m afraid of crashing my OS, if I install proceed 
> 
> https://www.tqhosting.com/kb/608/How-to-install-PostgreSQL-95-on-Debian-76-Wheezy.html <https://www.tqhosting.com/kb/608/How-to-install-PostgreSQL-95-on-Debian-76-Wheezy.html>
> 
> 
> modules/websocket/websocket-procs.tcl
> gcc -I/usr/include/postgresql -O2 -fomit-frame-pointer -DNDEBUG -Wall -fPIC  -pipe  -I/usr/local/ns/include -I"/usr/local/ns/include"  -DHAVE_CONFIG_H    -c -o nsdbpg.o nsdbpg.c
> In file included from /usr/local/ns/include/nsthread.h:45:0,
>                  from /usr/local/ns/include/ns.h:41,
>                  from /usr/local/ns/include/nsdb.h:40,
>                  from dbpg.h:54,
>                  from nsdbpg.c:37:
> /usr/local/ns/include/nsconfig.h:23:0: warning: "HAVE_CRYPT" redefined [enabled by default]
> In file included from dbpg.h:47:0,
>                  from nsdbpg.c:37:
> /usr/include/postgresql/pg_config.h:95:0: note: this is the location of the previous definition
> nsdbpg.c: In function 'Exec':
> nsdbpg.c:497:10: error: 'PGRES_SINGLE_TUPLE' undeclared (first use in this function)
> nsdbpg.c:497:10: note: each undeclared identifier is reported only once for each function it appears in
> make: *** [nsdbpg.o] Error 1
> 
> 
> 
> 
> I’ve noticed when I try to start Naviserver I get the following error related to TCL Libthread 2.8.2. However, I believe that’s secondary problem, because of the issue faced in the installation process.
> 
> 
> [01/Apr/2018:22:07:54][2833.b71076c0][-main-] Notice: modload: loading module libthread from file /usr/local/ns/lib/thread2.8.2/libthread2.8.2.so
> [01/Apr/2018:22:07:54][2833.b71076c0][-main-] Error: modload: /usr/local/ns/lib/thread2.8.2/libthread2.8.2.so: cannot find symbol "Ns_ModuleInit": /usr/local/ns/lib/thread2.8.2/libthread2.8.2.so: undefined symbol: _Ns_ModuleInit
> [01/Apr/2018:22:07:54][2833.b71076c0][-main-] Fatal: modload: failed to load module '/usr/local/ns/lib/thread2.8.2/libthread2.8.2.so'
> 
> 
> Best wishes,
> Iuri

[naviserver-devel] Install NS on Debian

[Iuri S. <iu...@iu...>]

Hi there,


In attempt to upgrade Naviserver to v.4.99.16 I’ve got the following error, running the current installation script install-ns.sh, available at GIT repository https://github.com/gustafn/install-ns.git <https://github.com/gustafn/install-ns.git>

Operating System: Debian 7 (wheezy)
Database: PostgreSQL 9.1 (installed via apt-get install)

Isn’t PGRES_SINGLE_TUPLE a processor MACRO on Postgres scripts?

Should I instal PostgreSQL 9.5 to Debian Wheezy, using backports.  I’m afraid of crashing my OS, if I install proceed 

https://www.tqhosting.com/kb/608/How-to-install-PostgreSQL-95-on-Debian-76-Wheezy.html <https://www.tqhosting.com/kb/608/How-to-install-PostgreSQL-95-on-Debian-76-Wheezy.html>


modules/websocket/websocket-procs.tcl
gcc -I/usr/include/postgresql -O2 -fomit-frame-pointer -DNDEBUG -Wall -fPIC  -pipe  -I/usr/local/ns/include -I"/usr/local/ns/include"  -DHAVE_CONFIG_H    -c -o nsdbpg.o nsdbpg.c
In file included from /usr/local/ns/include/nsthread.h:45:0,
                 from /usr/local/ns/include/ns.h:41,
                 from /usr/local/ns/include/nsdb.h:40,
                 from dbpg.h:54,
                 from nsdbpg.c:37:
/usr/local/ns/include/nsconfig.h:23:0: warning: "HAVE_CRYPT" redefined [enabled by default]
In file included from dbpg.h:47:0,
                 from nsdbpg.c:37:
/usr/include/postgresql/pg_config.h:95:0: note: this is the location of the previous definition
nsdbpg.c: In function 'Exec':
nsdbpg.c:497:10: error: 'PGRES_SINGLE_TUPLE' undeclared (first use in this function)
nsdbpg.c:497:10: note: each undeclared identifier is reported only once for each function it appears in
make: *** [nsdbpg.o] Error 1




I’ve noticed when I try to start Naviserver I get the following error related to TCL Libthread 2.8.2. However, I believe that’s secondary problem, because of the issue faced in the installation process.


[01/Apr/2018:22:07:54][2833.b71076c0][-main-] Notice: modload: loading module libthread from file /usr/local/ns/lib/thread2.8.2/libthread2.8.2.so
[01/Apr/2018:22:07:54][2833.b71076c0][-main-] Error: modload: /usr/local/ns/lib/thread2.8.2/libthread2.8.2.so: cannot find symbol "Ns_ModuleInit": /usr/local/ns/lib/thread2.8.2/libthread2.8.2.so: undefined symbol: _Ns_ModuleInit
[01/Apr/2018:22:07:54][2833.b71076c0][-main-] Fatal: modload: failed to load module '/usr/local/ns/lib/thread2.8.2/libthread2.8.2.so'


Best wishes,
Iuri

Re: [naviserver-devel] Long Accept Times

[Gustaf N. <ne...@wu...>]

On 3/14/18 4:30 PM, David Osborne wrote:
> Given that the smallest writersize available is 1024, it is impossible 
> (regardless of whether it's desirable) to have all requests serviced 
> by writer threads since small requests will always be delivered by the 
> connection thread.
this is true.
> Therefore, it is fair to say, that it is impossible to completely 
> eliminate the effect network delivery times has on cumulative runtimes?

When the write operation are such small, the potential danger of write 
requests blocking in connection threads is very little. Small write 
operations will be placed in a network buffer of the kernel and the 
connection thread will be released, no matter how slow the client reads. 
With the major operating systems and standard configurations, this is no 
issue. Maybe it is possible to (mis)configure a system with shuch small 
network buffers, but that would have bad consequences on the performance 
of the whole system.

As always, the best thing is to try it out on your system.

all the best

-g

Re: [naviserver-devel] Long Accept Times

[David O. <da...@qc...>]

Thank you very much for the detailed info.

Can I check something...

I'm understanding that runtimes are dependent on the client in cases where
there are no writer threads.
Given that the smallest writersize available is 1024, it is impossible
(regardless of whether it's desirable) to have all requests serviced by
writer threads since small requests will always be delivered by the
connection thread.

Therefore, it is fair to say, that it is impossible to completely eliminate
the effect network delivery times has on cumulative runtimes?


On 7 March 2018 at 18:21, Gustaf Neumann <ne...@wu...> wrote:

> On 3/7/18 5:28 PM, David Osborne wrote:
>
> Thank you very much for that info Gustaf.
> We'll have a look at whether we can adapt the ideas in boomerang to our
> environment.
>
> Yes we suspected the Accept timings might be outwith the control of the
> server.
> Am I right in thinking that Runtimes (if not using writer threads) are
> also affected by network conditions since they include the time taken to
> deliver content back to the client?
>
> Here are a few images for visualizing what happens: a typical request has
> a receiving, processing and delivery phase:
>
> if NaviServer processes a request without "read-ahead", and there are no
> writer threads involved, the full time is spent in a connection thread, all
> is accounted in the "runtime". The time of the "receive" and "delivery"
> phase is only under partial control of the server; a slow sending or
> receiving client can cause long blocking time of a connection thread, which
> is a limited resource. This can be as well the cause of potential "slow
> read/write" attacks to servers.
>
> Fortunately, NaviServer offers the means to delegate the receive and
> delivery phases to separate threads using async i/o, where every of these
> threads is able to handle thousands of sockets concurrently. The "driver"
> and "spooling" threads of NaviServer handle the receiving part, and the
> "writer" threads the delivery part. The http/tls drivers are configured
> with "read-ahead", other drivers are not, so that the receiving part is
> just the accept operation. With "readahead", one can define multiple driver
> threads listening on one port, and 0..n spooling threads for huge (file)
> uploads.
>
> With these i/o threads in place, the picture looks like the following for
> read-ahead threads:
>
> where only the orange time spans are tunes spent in a connection thread.
> With these in i/o threads every connection thread is able to handle much
> more requests per second than without those.  For application development,
> the times in the connection threads are the one which are influenced
> primarily by the application developers, so these are reported as runtimes
> of the requests. In reality, the picture is more complicated, since the
> server sends data back as soon it is available (it does not wait, until the
> whole reply is computed), such that the delivery starts actually already in
> the the "orange" phase, 2 threads can work pipelined on one one request.
> When writer threads are enabled, every writer operation is essentially just
> an enqueuing operation. When there are no writer threads enabled, the
> runtime will be dependent on the client (and the connection quality and
> conditions on the way to the client). So, writer threads are highly
> recommended.
>
> hope this helps, all the best
>
> -g
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>
>

Re: [naviserver-devel] Long Accept Times

[Gustaf N. <ne...@wu...>]

On 3/7/18 5:28 PM, David Osborne wrote:
> Thank you very much for that info Gustaf.
> We'll have a look at whether we can adapt the ideas in boomerang to 
> our environment.
>
> Yes we suspected the Accept timings might be outwith the control of 
> the server.
> Am I right in thinking that Runtimes (if not using writer threads) are 
> also affected by network conditions since they include the time taken 
> to deliver content back to the client?

Here are a few images for visualizing what happens: a typical request 
has a receiving, processing and delivery phase:

if NaviServer processes a request without "read-ahead", and there are no 
writer threads involved, the full time is spent in a connection thread, 
all is accounted in the "runtime". The time of the "receive" and 
"delivery" phase is only under partial control of the server; a slow 
sending or receiving client can cause long blocking time of a connection 
thread, which is a limited resource. This can be as well the cause of 
potential "slow read/write" attacks to servers.

Fortunately, NaviServer offers the means to delegate the receive and 
delivery phases to separate threads using async i/o, where every of 
these threads is able to handle thousands of sockets concurrently. The 
"driver" and "spooling" threads of NaviServer handle the receiving part, 
and the "writer" threads the delivery part. The http/tls drivers are 
configured with "read-ahead", other drivers are not, so that the 
receiving part is just the accept operation. With "readahead", one can 
define multiple driver threads listening on one port, and 0..n spooling 
threads for huge (file) uploads.

With these i/o threads in place, the picture looks like the following 
for read-ahead threads:

where only the orange time spans are tunes spent in a connection thread. 
With these in i/o threads every connection thread is able to handle much 
more requests per second than without those.  For application 
development, the times in the connection threads are the one which are 
influenced primarily by the application developers, so these are 
reported as runtimes of the requests. In reality, the picture is more 
complicated, since the server sends data back as soon it is available 
(it does not wait, until the whole reply is computed), such that the 
delivery starts actually already in the the "orange" phase, 2 threads 
can work pipelined on one one request. When writer threads are enabled, 
every writer operation is essentially just an enqueuing operation. When 
there are no writer threads enabled, the runtime will be dependent on 
the client (and the connection quality and conditions on the way to the 
client). So, writer threads are highly recommended.

hope this helps, all the best

-g

Re: [naviserver-devel] Long Accept Times

[David O. <da...@qc...>]

Thank you very much for that info Gustaf.
We'll have a look at whether we can adapt the ideas in boomerang to our
environment.

Yes we suspected the Accept timings might be outwith the control of the
server.
Am I right in thinking that Runtimes (if not using writer threads) are also
affected by network conditions since they include the time taken to deliver
content back to the client?

Thanks again,
--
David

On 5 March 2018 at 20:25, Gustaf Neumann <ne...@wu...> wrote:

> Dear David,
>
> i would not recommend to use the accept times are a measure of
> performance, since it measures the time between the accept of an incoming
> connection until the time, when the request is added to the queue, since
> for one "accept", there might be multiple requests added to the queue (on
> HTTP persistent connections).
>
> It is true, that this time contains for drivers with read-ahead configured
> (standard case) the time for receiving the request from the client. But
> note, that this time depends on the connection quality of the client. A
> client on a remote location connected via smart phone will lead to much
> worse values than a user on the local network. Increasing the bandwidth to
> your Internet provider might not change this picture. But note, that this
> high latency does not hit the server, since reading is performed with async
> i/o. Similarly, largish post requests will result in larger accept times.
>
> The interpretation of the accepttime is not easy, and depends also on the
> overall system architecture, e.g. whether or not NaviServer is running
> behind a proxy server. See below for two different systems: the first chart
> show the year statistics from a system running without a reverse proxy in
> front of it, the second chart is from a system is running behind nginx. The
> first system show high accept times (average 1.1 seconds), the second one
> shows low accept times (average 200 micro seconds). For the first system,
> one sees rather growing accept times (on the logarithmic scale), which says
> probably, that the number of clients with persistent connections is
> growing. On the second system, one can see a change in the accept time
> around mid January from typically below 100 microseconds (10^-4s) to the
> current value.  Interestingly, at that time, we fixed the spectre/meltdown
> vulnerabilities, and the kernel latencies increased.
>
> So, in short for some use cases, the accept-timings might be useful, but
> typically, these values can't be changed by faster soft- or hardware. We
> exclude the accept times often in our performance analysis.
>
> If you are interested in the real user experience, i would recommend a
> measurement tool like boomerang [1,2]. One lesson to learn from that is
> that only a relatively small fraction of the perceived performance is due
> direct server performance (in the example of openacs.org, total server
> runtime is 36ms, while the performance is around 2 secs; you can get from
> boomerang as well the "request time", which is the time after the TCP
> connect has finished until the request is received at the server)
>
> all the best
> -g
>
> [1] https://openacs.org/forums/message-view?message_id=5387444
> [2] https://github.com/openacs/boomerang
>
>
> [image: yearly graph]
>
>
>

Re: [naviserver-devel] Long Accept Times

[Gustaf N. <ne...@wu...>]

Dear David,

i would not recommend to use the accept times are a measure of 
performance, since it measures the time between the accept of an 
incoming connection until the time, when the request is added to the 
queue, since for one "accept", there might be multiple requests added to 
the queue (on HTTP persistent connections).

It is true, that this time contains for drivers with read-ahead 
configured (standard case) the time for receiving the request from the 
client. But note, that this time depends on the connection quality of 
the client. A client on a remote location connected via smart phone will 
lead to much worse values than a user on the local network. Increasing 
the bandwidth to your Internet provider might not change this picture. 
But note, that this high latency does not hit the server, since reading 
is performed with async i/o. Similarly, largish post requests will 
result in larger accept times.

The interpretation of the accepttime is not easy, and depends also on 
the overall system architecture, e.g. whether or not NaviServer is 
running behind a proxy server. See below for two different systems: the 
first chart show the year statistics from a system running without a 
reverse proxy in front of it, the second chart is from a system is 
running behind nginx. The first system show high accept times (average 
1.1 seconds), the second one shows low accept times (average 200 micro 
seconds). For the first system, one sees rather growing accept times (on 
the logarithmic scale), which says probably, that the number of clients 
with persistent connections is growing. On the second system, one can 
see a change in the accept time around mid January from typically below 
100 microseconds (10^-4s) to the current value.  Interestingly, at that 
time, we fixed the spectre/meltdown vulnerabilities, and the kernel 
latencies increased.

So, in short for some use cases, the accept-timings might be useful, but 
typically, these values can't be changed by faster soft- or hardware. We 
exclude the accept times often in our performance analysis.

If you are interested in the real user experience, i would recommend a 
measurement tool like boomerang [1,2]. One lesson to learn from that is 
that only a relatively small fraction of the perceived performance is 
due direct server performance (in the example of openacs.org, total 
server runtime is 36ms, while the performance is around 2 secs; you can 
get from boomerang as well the "request time", which is the time after 
the TCP connect has finished until the request is received at the server)

all the best
-g

[1] https://openacs.org/forums/message-view?message_id=5387444
[2] https://github.com/openacs/boomerang


yearly graph


On 3/5/18 5:31 PM, David Osborne wrote:
> Hi,
>
> One of our installations is appearing to randomly experience long 
> accept times.
>
> While all other requests are flowing nicely, suddenly one request will 
> have an accept time of between 2-20 seconds (as measured in the access 
> log with logpartialtimes).
>
> Our understanding is that this is the time it takes for the driver to 
> receive the entire request from the client, and long accept times can 
> be due to clients connecting with very low bandwidths.
>
> However, we're not really seeing any correlation between slow times 
> and any particular clients or request type. Plus there doesn't seem to 
> be any correlation to CPU/network/disk load.
>
> I was wondering if there are other factors apart from bandwidth we 
> should be looking at which could be affecting accept times?
>
> Regards,
> -- 
> David

[naviserver-devel] Long Accept Times

[David O. <da...@qc...>]

Hi,

One of our installations is appearing to randomly experience long accept
times.

While all other requests are flowing nicely, suddenly one request will have
an accept time of between 2-20 seconds (as measured in the access log with
logpartialtimes).

Our understanding is that this is the time it takes for the driver to
receive the entire request from the client, and long accept times can be
due to clients connecting with very low bandwidths.

However, we're not really seeing any correlation between slow times and any
particular clients or request type. Plus there doesn't seem to be any
correlation to CPU/network/disk load.

I was wondering if there are other factors apart from bandwidth we should
be looking at which could be affecting accept times?

Regards,
-- 
David

Re: [naviserver-devel] nsdbipg loading breaks next virtual server

[Gustaf N. <ne...@wu...>]

On 2/21/18 2:16 PM, Pavel Jurečka wrote:
> Maybe i have something misconfigured?

no.

there was a problem with the configuration of multiple servers
related with the initialization order. This problem should be fixed
with updated version of the nsdbi module on bitbucket.

please check

-g

[naviserver-devel] nsdbipg loading breaks next virtual server

[Pavel J. <jur...@gm...>]

Hi,

i encountering problem with nsdbipg module. 

When i load nsdbipg for first virtual server, then from every following virtual server i get following messages:

[21/Feb/2018:13:53:16][7640.b76bc700][-main-] Error: Ns_TclRegisterTrace: Invalid server: server2.local
[21/Feb/2018:13:53:16][7640.b76bc700][-main-] Error: dbi: error registering tcl commands for server 'server2.local'

This is how i load nsdbipg for first virutal server - it is ok and conection works:

ns_section "ns/server/${s1}/modules"
ns_param dbipg1 ${nsdir}/bin/nsdbipg.so

and for second server - this fails and page is without db connection:

ns_section "ns/server/${s2}/modules"
ns_param dbipg2 ${nsdir}/bin/nsdbipg.so

Maybe i have something misconfigured?

Thanks for any help!

Pj.

[naviserver-devel] NaviServer with TLS 1.3

[Gustaf N. <ne...@wu...>]

Dear all,

TLS 1.3 will be part of OpenSSL 1.1.1 [1] and is close before release [2].
TLS 1.3 is a major change (it should have been called better TLS 2.0),
so it might be worth testing your applications if possible beforehand,
such that the change can go smoothly. I've done some preliminary
testing on my local machine where everything looks ok.

When you install the version of [2] e.g. in /usr/local, you can
configure NaviServer with [3] to compile with this. It should
work out of the box with version 4.99.16. If you experience
troubles, please report back.

all the best

-g

[1] https://www.openssl.org/blog/blog/2018/02/08/tlsv1.3/
[2] https://www.openssl.org/source/
[3] configure ... --with-openssl=/usr/local/ ...

Re: [naviserver-devel] Profiling

[Gustaf N. <ne...@wu...>]

well. there are several options.

a) using Tcl traces (somewhat similar to what you are doing,
    see also [1]

b) when using nsf, one can profile oo methods and procs.
    nsf defines a value-added version of "proc" called "nsf::proc",
    that allows e.g. optional type checking of arguments, checking of
    return results ... and as well profiling (when compiled
    with "--enable-profile"

    From nsf profiling, one can get
    1) detailed profiling about a full run, or
    2) profiling about single calls ("nsf::proc -debug ... {args} { .. 
body ...}".

    See below the proc "profile" how profiling can be used in nsf,
    by collecting the profiling information in a Tcl list. We have
    some code that produces from the resulting  Tcl list a colored
    HTML rendering, but that can be quite large. It is also possible to
    log  the profiling information to the error.log of
    NaviServer (not shown here)

    There is as well a third option in nsf by using dtrace [2], which 
allows to get detailed
    profiling information from nsf (and Tcl as well).... but, last time 
i checked,
    this was only working well with Mac OS X (and maybe Solaris).

-gn

[1] 
https://github.com/openacs/openacs-core/blob/master/packages/acs-tcl/tcl/tcltrace-init.tcl
[2] https://github.com/gustafn/nsf/tree/master/dtrace

nsf::proc profile {cmd} {

     #
     # Clear all old profiling information and start tracing
     #
     nsf::__profile_clear
     nsf::__profile_trace -enable true

     #
     # Use a datafile for keeping the profiling information. You might
     # want to tailor this in case you are working with multiple
     # profiling runs.
     #
     set datafile [ns_mktemp]

     try {
	uplevel $cmd

     } on error {errorMsg} {
	error $errorMsg

     } finally {
	nsf::__profile_trace -enable false
     }

     #
     # Profiling is now turned off again, and we can the the profiling
     # information from nsf and save it to the trace file.
     #
     set f [open $datafile w]
     puts $f [::nsf::__profile_get]
     close $f
}




On 2/13/18 1:03 PM, David Osborne wrote:
> Hi,
>
> We've been looking at profiling the execution times of various 
> application-level procs running NaviServer.
>
> We want the ability to turn on profiling for selected commands, so we 
> started off using Tcl traces in a manner similar to:
> http://nikit.tcl.tk/page/Profiling+with+execution+traces
>
> So we'd have a proc to set up a Tcl trace on command execution 
> enter/leave in a list like so:
>
> proc instrument {args} {
>     #| Takes list of commands and adds Tcl execution trace for 
> entering and exit
>     set script ""
>     foreach cmd $args {
>             append script "trace add execution $cmd enter profile_begin\n"
>             append script "trace add execution $cmd leave profile_end\n"
>         }
>     }
>     eval $script
> }
>
>
> At the start of each proc in the list we'd push the start time to a stack:
>
> proc profile_begin args {
>     # Append current time to timer stack for this thread
>     variable ::timerStack
>     lappend ::timerStack [::tcl::clock::microseconds]
> }
>
> And then pop off the time when proc exits:
>
> proc profile_end {command_string code result op} {
>     #| Pop value off timer stack for this thread and record a subsegment
>     variable ::timerStack
>     set start_time [lindex $::timerStack end]
>     set end_time [::tcl::clock::microseconds]
>     set ::timerStack [lrange $::timerStack 0 end-1]
>     record_the_timing $command_string $start_time $end_time
> }
>
>
> It wasn't obvious to me how to set up these Tcl traces so they are 
> active in each NaviServer interpreter, but after some trial and error, 
> I could get a result by using ns_ictl to set up a callback on 
> interpreter allocation and use that to set up the Tcl trace:
>
> ns_ictl trace allocate "instrument $cmd_list"
>
> This seems to work ok but has the disadvantage that I think I need to 
> restart NaviServer in order to add or remove commands.
>
> So I wanted to ask:
> 1. Is there a more dynamic way of setting up Tcl traces across 
> NaviServer interpreters?
> 2. Is anyone aware of a different/better way of profiling applications 
> running under NaviServer?
>
> Regards,
> -- 

[naviserver-devel] Profiling

[David O. <da...@qc...>]

Hi,

We've been looking at profiling the execution times of various
application-level procs running NaviServer.

We want the ability to turn on profiling for selected commands, so we
started off using Tcl traces in a manner similar to:
http://nikit.tcl.tk/page/Profiling+with+execution+traces

So we'd have a proc to set up a Tcl trace on command execution enter/leave
in a list like so:

proc instrument {args} {
    #| Takes list of commands and adds Tcl execution trace for entering and
exit
    set script ""
    foreach cmd $args {
            append script "trace add execution $cmd enter profile_begin\n"
            append script "trace add execution $cmd leave profile_end\n"
        }
    }
    eval $script
}


At the start of each proc in the list we'd push the start time to a stack:

proc profile_begin args {
    # Append current time to timer stack for this thread
    variable ::timerStack
    lappend ::timerStack [::tcl::clock::microseconds]
}

And then pop off the time when proc exits:

proc profile_end {command_string code result op} {
    #| Pop value off timer stack for this thread and record a subsegment
    variable ::timerStack
    set start_time [lindex $::timerStack end]
    set end_time [::tcl::clock::microseconds]
    set ::timerStack [lrange $::timerStack 0 end-1]
    record_the_timing $command_string $start_time $end_time
}


It wasn't obvious to me how to set up these Tcl traces so they are active
in each NaviServer interpreter, but after some trial and error, I could get
a result by using ns_ictl to set up a callback on interpreter allocation
and use that to set up the Tcl trace:

ns_ictl trace allocate "instrument $cmd_list"

This seems to work ok but has the disadvantage that I think I need to
restart NaviServer in order to add or remove commands.

So I wanted to ask:
1. Is there a more dynamic way of setting up Tcl traces across NaviServer
interpreters?
2. Is anyone aware of a different/better way of profiling applications
running under NaviServer?

Regards,
-- 
David

Re: [naviserver-devel] TclOO in NaviServer, How to use?

[Anthony B. <to...@br...>]

Proc definitions are copied to other threads but things executed on init 
are not. Class definitions are created by running the 'class' proc and 
since execution does not happen in a copied thread we need trace to 
re-run the class proc.

On 12/22/17 8:56 AM, Pavel Jurečka wrote:
> Hi Tony,
>
> thanks for info and examples on garbage collecting. I played little with it and it works.
> And i agree with you. It looks like TclOO doesnt play nice with naviserver.
> I think, for now i stay with good old flat proc's. Anyway, as some wise-man said: "OO isnt cure for everything" :)
>
> Just out of curiosity... why must classes be inside "trace" and initializing them like proc in init.tcl isn't enough?
>

[naviserver-devel] NaviServer 4.99.16 available

[Gustaf N. <ne...@wu...>]

Dear friends of NaviServer,

the new release of NaviServer 4.99.16 is available from sourceforge [1]
and from bitbucket [2]. Below are the release notes (changes since 4.99.15)

I'll prepare as well an announcement for c.l.tcl and OpenACS.org.
best regards

-gustaf
[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.16/
[2] https://bitbucket.org/naviserver/naviserver/overview

=======================================
NaviServer 4.99.16, released 2017-12-29
=======================================

  345 files changed, 16488 insertions(+), 10401 deletions(-)

New Features:
-------------

   - ns_cache improvements:

     * Allow runtime reaction and re-configuration of caches
       (e.g. grow/shrink a cache via new cmd ns_cache_configure)

     * Added cache transaction semantics

       Background: When ns_cache_* commands are used within a database
       transaction (e.g. in OpenACS), it might occur, that partial
       results of the transaction are cached before the transaction is
       committed.  When the transaction is rolled back, invalid values
       might be kept in the stack leading to erroneous and hard to
       debug behavior.  Furthermore, information about changes might
       leak into other concurrent threads via the cache, even before
       the transaction is committed.

       The new cache transaction semantics is implemented via the three
       commands

	 - ns_cache_transaction_begin
	 - ns_cache_transaction_commit
	 - ns_cache_transaction_rollback

	When no ns_cache_transaction* commands are used, the behavior is
	exactly as before.

	When ns_cache_transaction* commands are in use, the following
	functionalities become available

	 - The ability to rollback of the values since the matching
	   ns_cache_transaction_begin

	 - Isolation of behavior: cached values from incomplete cache
	   transactions are just visible from the current thread, but
	   not from other threads.

	 - Nesting: transactions can be nested
	   (up to a compile time constant, per default: 16)

	 - Introspection: the statistics about cache commits and
	   rollbacks are included in the cache statistics.

     * Support caches sizes larger than 2GB

     * Summary of new ns_cache* commands
       . ns_cache_configure to change cache parameters at runtime
       .	ns_cache_exists: This command is is more than an order of
	magnitude faster than [expr {"foo" in [ns_cache_names]]
       . ns_cache_transaction_begin
       . ns_cache_transaction_commit
       . ns_cache_transaction_rollback


   - Virtual server improvements

     Make mapping of host entries in the virtual server definition more
     intelligent to avoid newbie gotchas and hard to find
     mis-configurations (entries in e.g. nssock/servers)
      a) add automatically an entry with the port, when non is given
      b) complain, when driver is not listening on the specified port
      c) add automatically an entry without the port, when the driver
        listening on the default port.
     Old configurations (doing a-z manually) should continue to work

   - Improved integration of NaviServer with e.g service files (systemd):
     When the server is starting in a forking mode, return a non-zero
     return code, when initialization of the server went wrong (e.g.
     error in the config file)


   - nsdb:
     provide interface for obtaining session_ids. When implementing
     e.g. prepared statements over the nsdbpg driver, the prepared
     statement is just valid for one session. by providing the
     session_id on the Tcl layer, prepared statements can be provided
     selectively on the Tcl level.


   - https:
     * Support OpenSSL 1.1.0 (configuration and runtime)
     * Support LibreSSL 2.6.3 and 2.6.4
     * Improved support for ancient versions of OpenSSL (e.g. in CentOS 5.0)
     * Added support for for client side Server Name Indication (SNI)
     * Report attempted TLS handshakes on plain connections to error.log


   - nscgi: new config parameter "allowstaticresources"

     New parameter "allowstaticresources": provide control on whether
     static resources can be delivered from a cgi-bin directory.

     Background: When CGI scripts are called from a cgi-bin directory
     without an explicit extension mapping to a script interpreter
     (e.g.  Perl, Bash, ...) the script has to be set executable. If
     this is not the case, the CGI script is delivered in source code,
     which might reveal unwanted information. The reason for this
     behavior was, that one can so deliver also static resources
     (e.g. images) from a cgi-bin directory. Now this behavior con be
     controlled with the configure parameter "allowstaticresources",
     which is by default off.  If an application depends on this
     behavior, please turn it on in the config file (in the nscgi
     section).


   - nsproxy:
     New subcommand "ns_proxy stats" to provide usage and runtime
     statistics per nsproxy pool.


   - logging:

     * Better error.log entries: Include the connection id in the
       thread name to ease debugging, when multiple requests of the
       same connection thread write to the error.log and it is not
       clear, where the first ends and the next starts.

     * Write notice message to error.log when entities are too large
       to provide information about possible misconfigurations or
       attacks in the error log

     * Added new config parameter for nslog: "logthreadname"

       When the parameter is specified, the thread name is placed as
       second field in the access.log. This eases debugging, since one
       can now link the lines in error.log caused by an request with
       the line in the access.log without the need of guessing via
       time-stamps. The thread name contains the the name of the
       connection pool and a connection id.

    *  When "mutexlocktrace" is activated, use thread name instead
       of id to ease debugging


   - Added parameter "defaultextension" for adp section in config file.
     Can be set e.g. to ".adp" to test for FILENAME.adp, when adp is
     mapped to FILENAME



Performance Improvements:
-------------------------

   * Improved performance of ns_urlencode/ns_urldecode in common
     cases by about a factor of 2

   * Improved performance by replacing the slow "snprintf(... %d ...)"
     function by a the new functions ns_uint32toa() and ns_uint64toa().
     This change improved the performance of EnterSet() by nearly a
     factor of 2.5. Before, snprintf() took in this function 3x (!) as
     long as Tcl_CreateHashEntry(); the new ns_u32toa() is about 6x
     faster than Tcl_CreateHashEntry().

   * Improved performance of ns_quotehtml:

     In real-world applications (with e.g. OpenACS) this function is
     one the the most frequently used functions of NaviServer. This
     change improves the performance of "ns_quotehtml" by about 40-60% by
     avoiding per-character calls to *DStringAppend and some more
     improvements.

   * Improved performance of server internal "ns_set" operations: remove
     the need of 3 malloc()/free() pairs per request

   * Use Tcl_CreateHashEntry() in frequently called functions instead
     of Tcl_FindHashEntry() since the second one uses the first.

   * Avoid strlen() operations on several occasions
   * Improved speed of line-counting in .adp files by a factor of 2
   * Reduced scope of mutex lock when registering filters


Bug Fixes:
----------

  - Allow fully qualified domain names in "Host" header field
    (as allowed in RFC 2976)

  - Avoid potential hangs of nxproxy in cases the evaltimeout is very
    small and the client is writing much data causing blocking write
    operations

  - urlencode reform:

    * The new code conforms to RFC3986 (2005) rather than RFC1738
      (1994) and RFC1808 (1995) vs. RFC2396 (1998).  The newer RFC3986
      has a more precise and differently structured definition (e.g. no
      'unwise' characters) of characters encoded in URLs. The coding of
      the query part is actually defined in the HTML 4.01
      definitions. Legacy sites can use the old encoding, when
      compiling NaviServer with RFC1738 defined.

    * Produce a warning, when a not properly encoded URL is passed to
      the location header field (e.g. ns_returnredirect). Some newer
      browsers might reject those redirects. Only characters, which
      have to be always coded, are flagged.

    * ns_urldecode: Make sure, that only valid percent codes are
      converted back.

    * ns_urledecode: Use URL encoding charset, when no charset is
      specified.  This fixes an old (at least 10 year old) bug which
      can allow to sneak in binary nulls into query variables, which
      provide a potential injection attack vector


   - ns_server active|all|queued: Better handling of querying data from
     concurrent threads. Handle now semi-parsed requests: these are not
     "queued", but not all information is available for querying it. We
     do now the best effort to report on fields that are trusted.

   - Complain on connection output operations when the connection is
     already closed. Previous version of NaviServer swallowed output
     operations on already-closed connections more or less silently,
     leading to hard to understand messages in the error log. This
     happens in particular often during error handling in OpenACS,
     where a "ad_script_obort" is missing. The new code raises now an
     error message, pointing to the erroneous code. When the old
     behavior is necessary for a while for legacy installations, the
     global configuration parameter "rejectalreadyclosedconn" can be
     set to "false".

   - Fix duplicate DriverClose calls in context of UDP drivers

   - Undo potential crlf-translations in the body of (e.g. POST)
     requests with Content-Type "www-form-urlencoded"

     Background: When a hidden form fields contains line-breaks, these
     are transformed pn transit into CRLF by current browsers in POST
     requests. In this step the value of a hidden form-field might
     become larger on every iteration, which might result in a
     quadratic growth on multiple edits of such a field. The new
     behavior undoes this effect.

   - Improved handling of requests in ancient (pre HTTP/1.0 1996) HTTP
     requests in combination with writer threads. Previous versions
     could run into a too restrictive sanity assert, assuming that
     reply headers must be always present (which is not the case in the
     rather informal HTTP versions before HTTP/1.0)

   - Virtual server fixes:
     * Avoid potential double-initialization of driver modules when
       multiple servers are used.
     * Postpone registration of virtual servers until all servers
       are defined. Before it was possible that NaviServer boot was
       terminated, when a default server of a driver was not yet defined.

   - Improved handling of binary data:
     NsTclObjIsByteArray() behaves now more like the (Tcl internal)
     TclIsPureByteArray() to figure out, when the bytearray data has
     to be treated as binary. Cases of pure and impure bytearrays
     are handled now by NaviServer.

   - Fixed old bug, were SSL_shutdown could hang (observed on Linux
     systems)

   - Fixed potential crashes as flagged by fb infer.

   - Improved robustness on invalid URLs: Don't Fatal on invalid URLs
     passed to NSDriverClientOpen(), but spit out a warning.

   - Improved Tcl code safety (guarding "file delete" against names
     starting with a "--")

   - Improved handling of invalid input from config file for rolling
     files

   - Make sure "ns_server serverdir" and "ns_server pagedir" return
     normalized paths to avoid potential confusions

   - Improved windows portability:
     * Cope with changes in Universal CRT in Visual Studio 2015 where e.g.
       vsnprintf() is no longer identical to _vsnprintf().

     * Improved Windows support for recent versions of visual studio
       (versions after visual studio 2010) by introducing NS_EINPROGRESS
       and NS_EINTR for abstracting from the raw constants (many thanks to
       Maurizio for the input)

     * Make nsproxy compile-able under windows

   - Improved Unix Portability:
     Support systems without RLIMIT_AS (e.g. OpenBSD 6.2)

   - Improved Tcl portability:

     * Starting with Tcl 8.7a1, Tcl has actually two different types
       for bytearrays, the old "tclByteArrayType" and a new
       "properByteArrayType", where both have the string name
       "bytearray".  NsTclObjIsByteArray() is now extended to handle
       both types.



Documentation improvements:
---------------------------

   - Updated several man pages
     * admin-maintenance.man
     * commandlist.man
     * ns_accesslog.man
     * ns_adp_exception.man
     * ns_adp_include.man
     * ns_base64decode.man
     * ns_buildsqldate.man
     * ns_cache.man
     * ns_cancel.man
     * ns_chan.man
     * ns_connchan.man
     * ns_db.man
     * ns_driver.man
     * ns_gmtime.man
     * ns_http.man
     * ns_info.man
     * ns_job.man
     * ns_locationproc.man
     * ns_parseheader.man
     * ns_perm.man
     * ns_proxy.man
     * ns_serverrootproc.man
     * ns_shutdown.man
     * ns_sockcallback.man
     * ns_urlencode.man
     * ns_write.man
     * nssock.man
     * nsssl.man
     * ns_setpriveleges.man
     * returnstatus-cmds.man
     * tcl-overview.man

   - Additional global documentation changes:
     * Added various cross references with "see also"
     * Added keywords for "global builtin" and "server builtin" to
       distinguish between per-server commands and global commands
     * Added for every module the name of the module as "keyword"
     * Make markup of options more consistent

   - Improved banner of online man pages:
     * fix markup
     * make URLs protocol agnostic
     * use new .io domain of sourceforge

   - Fixed spelling errors


Tcl API Changes:
   - ns_connchan: new option "-driver" for "ns_connchan open"
   - ns_cache_create returns now boolean to flag success
   - "ns_http queue|run" new parameter "-hostname" for handling SNI

   - Removed useless and undocumented subcommand "ns_set idelete"

    * ns_urlencode: New flag "-uppercase" added for supporting encoding
      for OAuth (RFC 5849); note that the "path" segment encoding has
      to be used to avoid coding space as "+".

    * ns_urlencode and ns_urldecode: Additional accepted value "cookies
      "for parameter "-part" to request cookie encoding/decoding.

    * ns_getcookie: New option "-all"

      Background: By using a cookie domain, it is possible that a
      cookie with a specified name is given two value, one from
      e.g. the parent domain, one from the current; this can lead to
      situation not easy to debug, when e.g. domain cookies are
      introduced from other applications in the same domain. The "-all"
      option provides infrastructure support to detect such situations
      by returning potentially multiple values for the cookie.

    * ns_connchan:
       New (experimental) subcommand
	 ns_connchan listen ?-driver d? ?-server s? ?-bind? address port script
       Start a listening connection for the ns_connchan interface;
       Still missing: https variant via specifying "-driver" +
       handling of "-server" (and obtaining defaultserver)

   * sendmail.tcl
     - For the deprecated warning, do not write entire body of email to
       the log
     - Use local time (with timezone) instead of UTC in Date header


C API Changes:
--------------

   - New API function NsConnRequire() to make handling of required
     connections and error results more regular.

   - New API functions
       . Ns_CookieEncode()
       . Ns_CookieDecode()
       . NSDriverSockNew(): create an initialized driver Sock structure
       . Ns_HttpMessageParse(): parse a (potentially incomplete) HTTP message
       . ns_uint32toa()
       . ns_uint64toa(): substantially faster conversion of integer to
	string than snprintf()

   - Ns_SockListenCallback: alter interface to (a) return the listening
     socket and (b) to be able to bind to the fresh socket. This is
     necessary for e.g. FTPD's passive server connections, where the
     server offers a freshly bound socket to the client to connect to
     (EPSV, extended passive mode).

   - driver.c:
     * Factored out LookupDriver() to improve reusability
     * New function NSDriverSockNew() to create an initialized driver Sock
       structure,


Configuration Changes:
----------------------

   - Removed need to specify port for virtual servers in */servers
     section of the config files

   - Extended sample config files:

     * nsd-config.tcl
        . Added new nslog parameter "logthreadname"
        . Added new global parameter "rejectalreadyclosedconn"

     * openacs-config.tcl
        . Added configuration for strict-transport-security (HSTS)
        . Added setting of OpenACS kernel parameter
	 "NsShutdownWithNonZeroExitCode" in sample config file
        . Eased configuration for nsssl by simply setting https port
        . Added new nslog parameter "logthreadname"
        . Added new global parameter "rejectalreadyclosedconn"

     * sample-config.tcl:
        . Included global parameter "shutdowntimeout" in sample config file
        . Included parameter "allowstaticresources" in nscgi section
        . Added new nslog parameter "logthreadname"
        . Added new global parameter "rejectalreadyclosedconn"
        . Make it work out-of-the box


Code Changes:
-------------

   - Reworked TLS cleanup (necessary for Tcl 8.7 and beyond); align it
     to the Tcl practices. Add compatibility for Tcl 8.5- w.r.t
     notifier-thread init after the fork.

   - Switched to the recommended way of creating detached threads via
     pthread attribute (see e.g.
     https://docs.oracle.com/cd/E19455-01/806-5257/6je9h032l/index.html)

  - Extended regression tests
     * ns_urlspace.test
     * cookies.test
     * ns_sls.test
     * tclresp.test
     * ns_urlencode.test
     * url2file.test
     * ns_conn_host.test
     * ns_pagepath.test
     * ns_serverpath.test
     * ns_server.test
     * ns_proxy.test
     * http.test

   - Code cleanup
     * Added const declarations
     * Added missing prototypes
     * Don't shadow local variables
     * Marked unused arguments as UNUSED
     * Removed old-style function definitions
     * Reduced implicit conversions (gcc7)
     * Prefer type use "bool" over "int" when applicable
     * Don't pass implementation-defined NULL after the last typed
       argument to a variadic function
     * Reduced number of return statements before end of function
     * Reduced variable scopes
     * Reduced size of huge switch statements

   - API modernization
     * Prefer usage of Ns_ParseObjv over manual parsing and
       error message generation
     * Prefer Tcl_SetObjResult over Tcl_SetResult
     * Stop using readdir_r() unless it is forced via compile flag
     * Aligned casts to Tcl 8.6
     * Replaced all occurrences of strcpy() with more safe variants
     * Replaced all occurrences of sprintf()  by snprintf()
       to protect better against buffer overflows
     * Prefer void* over legacy caddr_r
     * Replaced all calls to deprecated function Tcl_DStringTrunc()
       by Tcl_DStringSetLength()

  - Implemented deprecated commands as Tcl proc and complain on
    its usage
       . ns_puts
       . ns_returnadminnotice
       . env

  - Silenced static checker

  - Add CFLAGSs for convenient testing/cleanup
    (CFLAGS_TIDY, CFLAGS_DEFINITION, CFLAGS_CONVERSION)

  - Improved error messages


Modules:
--------

  - nsstats:
     . Format values in human readable form (unless &raw=1 is used in query)
     . Return proxy stats form all proxy pools
     . Include expire date of certificate in "process" page
     . Added "commit" and "rollback" to cache statistics

  - nsdbpg:
     . Report more detailed version numbers used during built process
     . Honor "logsqlerrors" settings from config file
     . Truncate SQL statement in error.log to 10.000 chars if longer
       (to avoid overlong log file entries)
     . Improved backwards compatibility with ancient versions of PostgreSQL

  - nsimap:
     . Added optional flag "-novalidatecert" to "ns_imap open"
       (name from the flag of the IMAP implementation)
     . Use NaviServer built-in config check for SSL libraries and use same libraries
     . Updated to meet engineering standards

  - revproxy:
     . Added url_rewrite_callback
     . Added timeout handling
     . Added support for handling requests with provided content (POST, PUT, ...)
     . Generalized error handling, reverse callback registration as suggested by David.
     . Close connection explicitly after every request

  - letsencrypt:
     . New module for managing server certificates via Let's Encrypt

  - websocket:
     . Varius small updates (links, spelling errors, ...)

Re: [naviserver-devel] Problems Compiling in OpenBSD

[Gustaf N. <ne...@wu...>]

There is now an updated version of the release candidate on sourceforge,
addressing as well the RLIMIT issue. I have installed OpenBSD 6.2 on a 
virtual machine and it
compiled nicely with "gmake LIBS=-lpthread". The regression tests are a 
well fine
(sometimes the regression test ends with a non-zero return code, but 
this is nothing
to worry, as long the test cases are OK).

all the best
-g

Am 12/22/17 um 6:51 PM schrieb Roderick:
>
> Thanks, Gustaf!
>
>> concerning 2:
>> build with LIBS specified should help, e.g.
>> gmake LIBS=-lpthread
>
> I tried with the variables and at the end, the only that worked was to
> give manually the cc command in the two cases mentioned. When doing
> gmake, flag -pthread is used, but not in theese two cases.
>
>> I've added just now an update to the repository at bitbucket,
>> that should address the compilation issue with LibreSSL;
>
> Thanks, I will try later.
>
> I forgot other compilation problem: in the file "nsd/tclmisc.c"
> I had to add "#define RLIMIT_AS RLIMIT_DATA". OpenBSD has not
> RLIMIT_AS defined in "/usr/include/sys/resource.h". I hope this
> is not problematic.
>
> Rodrigo 

Re: [naviserver-devel] TclOO in NaviServer, How to use?

[Andrew P. <at...@pi...>]

NaviServer (and OpenACS) include integration with NSF/XOTcl, so that's
what I'd try for object oriented work in NaviServer, not TclOO.

Btw, my understanding is that NSF/XOTcl is strictly more capable than
TclOO, and has certainly been in real-world use much longer, so it'd
be more first choice to try anyway.  The better NaviServer integration
just makes the choice more obvious.

-- 
Andrew Piskorski <at...@pi...>

Re: [naviserver-devel] Problems Compiling in OpenBSD

[Roderick <hr...@gm...>]

Thanks, Gustaf!

> concerning 2:
> build with LIBS specified should help, e.g.
> gmake LIBS=-lpthread

I tried with the variables and at the end, the only that worked was to
give manually the cc command in the two cases mentioned. When doing
gmake, flag -pthread is used, but not in theese two cases.

> I've added just now an update to the repository at bitbucket,
> that should address the compilation issue with LibreSSL;

Thanks, I will try later.

I forgot other compilation problem: in the file "nsd/tclmisc.c"
I had to add "#define RLIMIT_AS RLIMIT_DATA". OpenBSD has not
RLIMIT_AS defined in "/usr/include/sys/resource.h". I hope this
is not problematic.

Rodrigo

Re: [naviserver-devel] TclOO in NaviServer, How to use?

[Pavel J. <jur...@gm...>]

Hi Tony,

thanks for info and examples on garbage collecting. I played little with it and it works.
And i agree with you. It looks like TclOO doesnt play nice with naviserver. 
I think, for now i stay with good old flat proc's. Anyway, as some wise-man said: "OO isnt cure for everything" :) 

Just out of curiosity... why must classes be inside "trace" and initializing them like proc in init.tcl isn't enough? 

-- 
Pj.



On Fri, 15 Dec 2017 11:23:50 -0800
Anthony Bennett <to...@br...> wrote:

> I worked with TclOO for a little bit and made some libraries to get it 
> working.  Here's some of my old code.  Ultimately I found TclOO wasn't a 
> good fit for Naviserver.  Someone please chime in if things have changed 
> and the garbage collection is no longer needed.  Also note that class 
> creation needs to be in the ns_ictl trace create because it doesn't copy 
> over to other interpreters.
> 
> ns_ictl trace create {
> 	
> 	# ModelClass - Base class. Adds garbage collection for [ClassName new]
> 	# Use with:
> 	# superclass BaseClass
> 	::oo::class create BaseClass {
> 		
> 		construct {} {
> 			# Add to garbage collection.
> 			lappend ::garbage::objects [self namespace]
> 
> 			set className [info object class [self]]
> 
> 			# Don't allow [ModelClass new]
> 			if {![lindex [self call] 1]} {
> 				return -code error "Class '$className' is abstract. Use with 'superclass $className' inside '::oo::class create DerievedClass'."
> 			}
> 		}
> 	}
> 	
> }
> 
> #
> # garbage
> #
> # TclOO does not clear temporary objects when the interpreter is deallocated.
> # This namespace keeps track of temporary objects and destroys them when
> # the connection closes.
> #
> # If you are taking up a lot of space with temporary objects then you may need to manually
> # when they're no longer needed.  The garbage collector only destroys objects on connection
> # end and doesn't cleanup objects you lost references to. For instance
> # the following loop replaces the objet reference 1000 times. Those objects will persist
> # through the duration of the connection unless manually destroyed.
> #
> # set value ""
> # for {set i 0} {$i < 1000} {incr i} {
> #  set obj [MyClass new]
> #  append value [$obj myMethod]
> #  $obj destroy
> # }
> #
> # To see the memory leak without garbage collection on. Comment out
> # The deallocate garbage::collect and uncomment the ns_log in allocate.
> #
> # Note: Your constructor should have the line "lappend ::garbage::objects [self namespace]"
> # If the object is to be garbage collected. You should use "set obj [Classname new]" when
> # instantiating the object instead of [Classname create obj]. Then use it with
> # "$obj method" etc.
> #
> namespace eval garbage {
> 	variable objects [list]
> 	
> 	proc collect {} {
> 		
> 		set total 0
> 		set withError 0
> 		
> 		foreach {obj} $::garbage::objects {
> 
> 			if {[namespace exists $obj] && [catch {$obj destroy} err]} {
> 				# NOTE: on error the object is still destroyed.
> 				ns_log Error "destroy $obj.\n$err."
> 				incr withError
> 			}
> 			
> 			incr total
> 		}
> 		
> 		if {$withError > 0} {
> 			ns_log Notice  "Cleared '$total' objects. With error = '$withError'."
> 			ns_log Notice ":oo::Obj count is [llength [namespace children ::oo Obj*]]"
> 		}
> 		
> 		set ::garbage::objects [list]
> 	}
> }
> 
> ns_ictl trace deallocate {
> 	# Collect garbage when we leave.
> 	::garbage::collect
> }
> 
> ns_ictl trace allocate {
> 
> 	# Test for object destruction.
> 	#ns_log Notice ":oo::Obj count is [llength [namespace children ::oo Obj*]]"
> }
> 
> 
> - Tony
> 
> On 12/15/17 3:25 AM, Pavel Jurečka wrote:
> > Hi!
> >
> > How can i use TclOO in Naviserver?
> >
> > I have tcllib file: init.tcl, and this code:
> >
> > namespace eval ::test {
> >     oo::class create greeter {
> >         method say {} {
> >             ns_return 200 text/html "Hello"
> >         }
> >     }
> > }
> >
> > ns_register_proc -noinherit GET /ootest {
> >     set g [::test::greeter new]
> >     $g say
> > }
> >
> > Then, when i visit: 127.0.0.1/ootest <http://127.0.0.1/ootest>, i get 
> > error:
> >
> > Error: GET /ootest, PeerAddress: 127.0.0.1
> > invalid command name "::test::greeter"
> >     while executing
> > "::test::greeter new"
> >     invoked from within
> > "set g [::test::greeter new]"
> >     while executing callback
> > ns:tclrequest {
> >     set g [::test::greeter new]
> >     $g say
> > }
> >
> > So, where should i place classes?
> >
> >
> > Thanks for help!
> > Pj.
> >
> >
> >
> >
> > ------------------------------------------------------------------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> >
> >
> > _______________________________________________
> > naviserver-devel mailing list
> > nav...@li...
> > https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

Re: [naviserver-devel] Problems Compiling in OpenBSD

[Gustaf N. <ne...@wu...>]

Dear Rodrigo,

concerning 1:
    As it seems, OpenBSD uses LibreSSL and not OpenSSL,
    LibreSSL pretends to be OpenSSL 2.0.0 (!!!), which does not exist,
    so all version comparisons fail hopelessly

concerning 2:
      build with LIBS specified should help, e.g.

             gmake LIBS=-lpthread

      Not sure, what's needed to detect OpenBSD. AFAIK, FreeBSD
      does not have this issue

I've added just now an update to the repository at bitbucket,
that should address the compilation issue with LibreSSL; NaviServer should
compile now, but i am not sure, how compatible LibreSSL is with OpenSSL

-g

Am 12/21/17 um 9:52 PM schrieb Roderick:
>
> (1) I cannot compile with openssl, because it seems to need
> openssl-1.1.0 that changed the names of some funcions of openssl-1.0.2
> (EVP_MD_CTX_free for EVP_MD_CTX_destroy, etc). I failed to build
> openssl-1.1.0 and did not try too much.
>
> (2)
>
> Doing:
>
> cc -L../nsthread -L../nsd -L../nsdb -o nsd main.o libnsd.so -lz 
> -lnsthread -L/usr/opt/Tcl867/lib -ltcl86 -lm -Wl,-export-dynamic 
> -L/usr/opt/Tcl867/lib -Wl,-rpath,:/usr/opt/NaviServ/lib 
> -Wl,-rpath,:/usr/opt/Tcl867/lib
>
> in /usr/opt/NaviServ/src/naviserver-4.99.15/nsd
>
> needs the flag -pthread
>
> ----------------
>
> (3) The same doing
>
> cc -L../nsthread -L../nsd -L../nsdb -o nsproxy nsproxy.o libnsproxy.so 
> -lnsthread -lnsd -L/usr/opt/Tcl867/lib -ltcl86 -lm -Wl,-export-dynamic 
> -L/usr/opt/Tcl867/lib -Wl,-rpath,:/usr/opt/NaviServ/lib 
> -Wl,-rpath,:/usr/opt/Tcl867/lib
>
> in /usr/opt/NaviServ/src/naviserver-4.99.15/nsproxy
>
> -------------------
>
> (4)
>
> Doing "gmake test" I get gmake error:
>
> Tests ended at Thu Dec 21 20:50:38 GMT 2017
> all.tcl:        Total   1218    Passed  1194    Skipped 24 Failed  0
> Sourced 64 Test Files.
> Number of tests skipped for each constraint:
>         20      crypto
>         2       knownBug
>         1       notDarwin
>         1       stress
> Terminated
> gmake: *** [Makefile:222: test] Error 143
>
> --------
>
> Best regards
> Rodrigo
>

[naviserver-devel] Release Candidate for NaviServer 4.99.16

[Gustaf N. <ne...@wu...>]

Dear all,

On [1] is a release candidate for NaviServer 4.99.16. Below is a - 
preliminary -
summary of the changes since the last release. If you have urgent changes
for this release, please get in touch with me. If everything goes well, 
the release
should follow early next week. Please test if possible.

all the best

-gn

[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.16/


=======================================
NaviServer 4.99.16rc, released 2017-XX-XX
=======================================

  344 files changed, 16011 insertions(+), 10077 deletions(-)

New Features:
-------------

   - ns_cache improvements:

     * Allow runtime reaction and re-configuration of caches
       (e.g. grow/shrink a cache via new cmd ns_cache_configure)

     * Added cache transaction semantics

       Background: when ns_cache_* commands are used within a database
       transaction (e.g. in OpenACS), it might occur, that partial
       results of the transaction are cached before the transaction is
       committed.  When the transaction is rolled back, invalid values
       might be kept in the stack leading to erroneous and hard to
       debug behavior.  Furthermore, information about changes might
       leak into other concurrent threads via the cache, even before
       the transaction is committed.

       The new cache transaction semantics is implemented via the three
       commands

	 - ns_cache_transaction_begin
	 - ns_cache_transaction_commit
	 - ns_cache_transaction_rollback

	When no ns_cache_transaction* commands are used, the behavior is
	exactly as before.

	When ns_cache_transaction* commands are in use, the following
	functionalities become available

	 - The ability to rollback of the values since the matching
	   ns_cache_transaction_begin

	 - Isolation of behavior: cached values from incomplete cache
	   transactions are just visible from the current thread, but
	   not from other threads.

	 - Nesting: transactions can be nested
	   (up to a compile time constant, per default: 16)

	 - Introspection: the statistics about cache commits and
	   rollbacks are included in the cache statistics.

     * Support caches larger than 2GB

     * Summary of new ns_cache* commands
       . ns_cache_configure to change cache parameters at runtime
       .	ns_cache_exists: This is is more than an order of
	magnitude faster than [expr {"foo" in [ns_cache_names]]
       . ns_cache_transaction_begin
       . ns_cache_transaction_commit
       . ns_cache_transaction_rollback


   - Virtual server improvements

     Make mapping of host entries in the virtual server definition more
     intelligent to avoid newbie gotchas and hard to find
     mis-configurations (entries in e.g. nssock/servers)
      a) add automatically an entry with the port, when non is given
      b) complain, when driver is not listening on the specified port
      c) add automatically an entry without the port, when the driver
        listening on the default port.
     Old configurations (doing a-z manually) should continue to work

   - Improved integration of NaviServer with e.g service files (systemd):
     When the server is starting in a forking mode, return a non-zero
     return code, when initialization of the server went wrong (e.g.
     error in the config file)


   - nsdb:
     provide interface for obtaining session_ids. When implementing
     e.g.  prepared statements over the nsdbpg driver, the prepared
     statement is just valid for one session. by providing the
     session_id on the Tcl layer, prepared statements can be provided
     selectively on the Tcl level.


   - https:
     * Support OpenSSL 1.1.0
     * Improved support for ancient versions of OpenSSL (e.g. in CentOS 5.0)
     * Added support for for client side Server Name Indication (SNI)
     * Report attempted TLS handshakes on plain connections to error.log


   - nscgi: new config parameter "allowstaticresources"

     New parameter "allowstaticresources": provide control on whether
     static resources can be delivered from a cgi-bin directory.

     Background: when cgi scripts are called from a cgi-bin directory
     without an explicit extension mapping to a script interpreter
     (e.g.  Perl, Bash, ...) the script has to be set executable. If
     this is not the case, the cgi-script is delivered in source code,
     which might reveal unwanted information. The reason for this
     behavior was, that one can so deliver also static resources
     (e.g. images) from a cgi- bin directory. Now this behavior con be
     controlled with the configure parameter "allowstaticresources",
     which is by default off.  If an application depends on this
     behavior, please turn it on in the config file (in the nscgi
     section).


   - nsproxy:
     New subcommand "ns_proxy stats" to provide usage and runtime
     statistics per nsproxy pool.


   - logging:

     * Better error.log entries: Include the connection id in the
       thread name to ease debugging, when multiple requests of the
       same connection thread write to the error.log and it is not
       clear, where the first ends and the next starts.

     * Write notice message to error.log when entities are too large
       to provide information about possible misconfigurations or
       attacks in the error log

     * Added new config parameter for nslog: "logthreadname"

       When the parameter is specified, the thread name is placed as
       second field in the access.log. This eases debugging, since one
       can now link the lines in error.log caused by an request with
       the line in the access.log without the need of guessing via
       time-stamps. The thread name contains the the name of the
       connection pool and a connection id.
    *  When "mutexlocktrace" is activated, use thread name instead
       of id to ease debugging


   - Added parameter "defaultextension" for adp section in config file.
     Can be set e.g. to ".adp" to test for FILENAME.adp, when adp is
     mapped to FILENAME



Performance Improvements:
-------------------------

   * Improved performance of ns_urlencode/ns_urldecode in common
     cases by about a factor of 2

   * Improved performance by replacing the slow "snprintf(... %d ...)"
     function by a the new functions ns_uint32toa() and ns_uint64toa().
     This change improved the performance of EnterSet() by nearly a
     factor of 2.5. Before, snprintf() took in this function 3x (!) as
     long as Tcl_CreateHashEntry(); the new ns_u32toa() is about 6x
     faster than Tcl_CreateHashEntry().

   * Improved performance of ns_quotehtml:

     This function is in real-world applications (with e.g. OpenACS)
     one the the most frequently used functions of NaviServer. This
     change improves the performance of ns_quotehtml by about 40-60% by
     avoiding per-character calls to *DStringAppend; some more
     improvements added.

   * Improve performance of server-internal ns_set operations: remove
     the need of 3 malloc/free pairs per request

   * Use Tcl_CreateHashEntry() in frequently called functions instead
     of Tcl_FindHashEntry() since the second one uses the first.

   * Avoid strlen() operations on several occasions
   * Improve speed of line-counting in .adp files by a factor of 2
   * Reduce scope of lock when registering a filter


Bug Fixes:
----------

  - Allow fully qualified domain names in "Host" header field
    (as allowed in RFC 2976)

  - Avoid potential hangs of nxproxy in cases the evaltimeout is very
    small and the client is writing much data causing blocking write
    operations

  - urlencode reform:

    * The new code conforms to RFC3986 (2005) rather than
      RFC1738 (1994) and RFC1808 (1995) vs. RFC2396 (1998).
    , RFC3986 has a more precise and differently structured
      definition (e.g. no 'unwise' characters)
      of characters encoded in URLs. The coding of the query part is
      actually defined in the HTML 4.01 definitions. Legacy
      sites can use the old encoding, when compiling NaviServer with RFC1738 defined.

    * A warning is produced now, when an URL is passed to the location
      field (e.g. ns_returnredirect), which is not properly encoded. Only
      characters, which have to be always coded, are flagged.

    * make sure, that only valid percent codes are converted

    * ns_urledecode: Use URL encoding charset, when no charset is
      specified.  This fixes an old (at least 10 year old) bug which
      can allow to sneak in binary nulls into query variables, which
      provide a potential injection attack vector


   - ns_server active|all|queued: Better handling of querying data from
     concurrent threads. Handle now semi-parsed requests: these are not
     "queued", but not all information is available for querying it. We
     do now the best effort to report on fields that are trusted.

   - Complain on connection output operations when the connection is
     already closed. Previous version of NaviServer swallowed output
     operations on already-closed connections more or less silently,
     leading to hard to understand messages in the error log. This
     happens in particular often during error handling in OpenACS,
     where a "ad_script_obort" is missing. The new code raises now an
     error message, pointing to the erroneous code. When the old
     behavior is necessary for a while for legacy installations, the
     global configuration parameter "rejectalreadyclosedconn" can be
     set to "false".


   - Fix double DriverClose calls in context of UDP drivers

   - Undo potential crlf-translations in the body of requests with
     Content-Type "www-form-urlencoded"

     Background: when e.g. hidden form fields contain a line-breaks, these
     are transformed into CRLF by current browsers in POST requests. In
     this step the value of a hidden form-field might become larger on
     every iteration, which might result in a quadratic growth on multiple
     edits of sich a field. The new behavior undoes this effect.


   - Improve handling of requests in ancient (pre HTTP/1.0 1996) HTTP
	requests in combination with writer threads. Previous versions could
	run into a too restrictive sanity assert, assuming that reply
	headers must be always present (which is not the case in the rather
	informal HTTP versions before HTTP/1.0)

   - Virtual server fixes:
     * Avoid potential double-initialization of driver modules when
       multiple servers are used.
     * Postpone registration of virtual servers until all servers
       are defined. Before it was possible that NaviServer boot was
       terminated, when a default server of a driver was not yet defined.

   - NsTclObjIsByteArray() behaves now more like the (Tcl internal)
     TclIsPureByteArray() to figure out, when the bytearray data has
     to be treated as binary.

   - Fixed old bug, were SSL_shutdown could hang

   - Fixed potential crashes as flagged by fb infer.

   - Improve robustness on invalid URLs:
     don't Fatal on invalid URLs passed to NSDriverClientOpen(),
     but spit out a warning.

   - Improved code safety (guarding "file delete" against names
     starting with a "-")

   - Improve handling of invalid input from config file for rolling
     files

   - Make sure "ns_server serverdir" and "ns_server pagedir" return
     normalized paths to avoid potential confusions

   - Improved windows portability:
     * Cope with changes in Universal CRT in Visual Studio 2015 where e.g.
       vsnprintf() is no longer identical to _vsnprintf().

     * Improve Windows support for recent versions of visual studio
       (versions after visual studio 2010) by introducing NS_EINPROGRESS
       and NS_EINTR for abstracting from the raw constants (many thanks to
       Maurizio for the input)

     * make nsproxy compile-able under windows

   - Improved Tcl portability:

     * Starting with Tcl 8.7a1, Tcl has actually two different types
       for bytearrays, the old "tclByteArrayType" and a new
       "properByteArrayType", where both have the string name
       "bytearray".  NsTclObjIsByteArray() is now extended to handle
       both types.



Documentation improvements:
---------------------------

   - Updated several man pages
     * admin-maintenance.man
     * commandlist.man
     * ns_accesslog.man
     * ns_adp_exception.man
     * ns_adp_include.man
     * ns_base64decode.man
     * ns_buildsqldate.man
     * ns_cache.man
     * ns_cancel.man
     * ns_chan.man
     * ns_connchan.man
     * ns_db.man
     * ns_driver.man
     * ns_gmtime.man
     * ns_http.man
     * ns_info.man
     * ns_job.man
     * ns_locationproc.man
     * ns_parseheader.man
     * ns_perm.man
     * ns_proxy.man
     * ns_serverrootproc.man
     * ns_shutdown.man
     * ns_sockcallback.man
     * ns_urlencode.man
     * ns_write.man
     * nssock.man
     * nsssl.man
     * ns_setpriveleges.man
     * returnstatus-cmds.man
     * tcl-overview.man

   - Additional global documentation changes:
     * Added various cross references with "see also"
     * Added keywords for "global builtin" and "server builtin" to
       distinguish between per-server commands and global commands
     * Added for every module the name of the module as "keyword"
     * Make markup of options more consistent

   - Improved banner of online man pages:
     * fix markup
     * make URLs protocol agnostic
     * use new .io domain of sourceforge

   - Fix spelling errors


Tcl API Changes:
   - ns_connchan: new option "-driver" for "ns_connchan open"
   - ns_cache_create returns now boolean to flag success
   - "ns_http queue|run" new parameter "-hostname" for handling SNI

   - Removed useless and undocumented subcommand "ns_set idelete"

    * ns_urlencode: new flag "-uppercase" added for supporting encoding
      for OAuth (RFC 5849); note that the "path" segment encoding has
      to be used to avoid coding space as "+".

    * ns_urlencode and ns_urldecode: Additional accepted value "cookies
      "for parameter "-part" to request cookie encoding/decoding.

    * ns_getcookie: new option "-all"

      Background: By using a cookie domain, it is possible that a
      cookie with a specified name is given two value, one from
      e.g. the parent domain, one from the current; this can lead to
      situation not easy to debug, when e.g. domain cookies are
      introduced from other applications in the same domain. The "-all"
      option provides infrastructure support to detect such situations
      by returning potentially multiple values for the cookie.

    * ns_connchan:
       new (experimental) subcommand
	 ns_connchan listen ?-driver d? ?-server s? ?-bind? address port script
       Start a listening connection for the ns_connchan interface;
       Still missing: https variant via specifying "-driver" +
       handling of "-server" (and obtaining defaultserver)

   * sendmail.tcl
     - For the deprecated warning, do not write entire body of email to
       the log
     - Use local time (with timezone) instead of UTC in Date header


C API Changes:
--------------

   - New API function NsConnRequire() to make handling of required
     connections and error results more regular.

   - New API functions
       . Ns_CookieEncode()
       . Ns_CookieDecode()
       . NSDriverSockNew(): create an initialized driver Sock structure
       . Ns_HttpMessageParse(): parse a (potentially incomplete) HTTP message
       . ns_uint32toa()
       . ns_uint64toa(): substantially faster conversion of integer to
	string than snprintf()

   - Ns_SockListenCallback: alter interface to (a) return the listening
     socket and (b) to be able to bind to the fresh socket. This is
     necessary for e.g. FTPD's passive server connections, where the
     server offers a freshly bound socket to the client to connect to
     (EPSV, extended passive mode).

   - driver.c:
     * factor out LookupDriver() to improve reusability
     * new function NSDriverSockNew() to create an initialized driver Sock
       structure,


Configuration Changes:
----------------------

   - no need to specify port for virtual servers in */servers section

   - Extended sample config files

     * nsd-config.tcl
        . Added new nslog parameter "logthreadname"
        . Added new global parameter "rejectalreadyclosedconn"

     * openacs-config.tcl
        . Added configuration for strict-transport-security (HSTS)
        . Added setting of OpenACS kernel parameter
	 "NsShutdownWithNonZeroExitCode" in sample config file
        . Ease configuration for nsssl by simply setting https port
        . Added new nslog parameter "logthreadname"
        . Added new global parameter "rejectalreadyclosedconn"

     * sample-config.tcl:
        . Include global parameter "shutdowntimeout" in sample config file
        . Include parameter "allowstaticresources" in nscgi section
        . Added new nslog parameter "logthreadname"
        . Added new global parameter "rejectalreadyclosedconn"
        . Make it work out-of-the box


Code Changes:
-------------

   - Reworked TLS cleanup (necessary for Tcl 8.7 and beyond); align it
     to the Tcl practices. Add compatibility for Tcl 8.5- w.r.t
     notifier-thread init after the fork.

   - Switched to the recommended way of creating detached threads via
     pthread attribute (see e.g.
     https://docs.oracle.com/cd/E19455-01/806-5257/6je9h032l/index.html)

  - Extended regression tests
     * ns_urlspace.test
     * cookies.test
     * ns_sls.test
     * tclresp.test
     * ns_urlencode.test
     * url2file.test
     * ns_conn_host.test
     * ns_pagepath.test
     * ns_serverpath.test
     * ns_server.test
     * ns_proxy.test
     * http.test

   - Code cleanup
     * Added const declarations
     * Added missing prototypes
     * Don't shadow local variables
     * Marked unused arguments as UNUSED
     * Removed old-style function definitions
     * Reduced implicit conversions (gcc7)
     * Prefer type use "bool" over "int" when applicable
     * Don't pass implementation-defined NULL after the last typed
       argument to a variadic function
     * Reduced number of return statements before end of function
     * Reduced variable scopes
     * Reduced size of huge switch statements

   - API modernization
     * Prefer usage of Ns_ParseObjv over manual parsing and
       error message generation
     * Prefer Tcl_SetObjResult over Tcl_SetResult
     * Stop using readdir_r() unless it is forced via compile flag
     * Align casts to Tcl 8.6
     * Replaced all occurrences of strcpy() with more safe variants
     * Prefer void* over legacy caddr_r
     * Replaced all calls to deprecated function Tcl_DStringTrunc()
       by Tcl_DStringSetLength()

  - Mark deprecated procs explicitly as deprecated
    * Implemented deprecated commands as Tcl proc and complain on
      its usage
       . ns_puts
       . ns_returnadminnotice
       . env

  - Silenced static checker

  - Add CFLAGSs for convenient testing/cleanup
    (CFLAGS_TIDY, CFLAGS_DEFINITION, CFLAGS_CONVERSION)

  - Improved error messages

[naviserver-devel] Problems Compiling in OpenBSD

[Roderick <hr...@gm...>]

(1) I cannot compile with openssl, because it seems to need
openssl-1.1.0 that changed the names of some funcions of openssl-1.0.2
(EVP_MD_CTX_free for EVP_MD_CTX_destroy, etc). I failed to build
openssl-1.1.0 and did not try too much.

(2)

Doing:

cc -L../nsthread -L../nsd -L../nsdb -o nsd main.o libnsd.so -lz 
-lnsthread -L/usr/opt/Tcl867/lib -ltcl86 -lm -Wl,-export-dynamic 
-L/usr/opt/Tcl867/lib -Wl,-rpath,:/usr/opt/NaviServ/lib 
-Wl,-rpath,:/usr/opt/Tcl867/lib

in /usr/opt/NaviServ/src/naviserver-4.99.15/nsd

needs the flag -pthread

----------------

(3) The same doing

cc -L../nsthread -L../nsd -L../nsdb -o nsproxy nsproxy.o 
libnsproxy.so -lnsthread -lnsd -L/usr/opt/Tcl867/lib -ltcl86 -lm 
-Wl,-export-dynamic -L/usr/opt/Tcl867/lib 
-Wl,-rpath,:/usr/opt/NaviServ/lib -Wl,-rpath,:/usr/opt/Tcl867/lib

in /usr/opt/NaviServ/src/naviserver-4.99.15/nsproxy

-------------------

(4)

Doing "gmake test" I get gmake error:

Tests ended at Thu Dec 21 20:50:38 GMT 2017
all.tcl:        Total   1218    Passed  1194    Skipped 24      Failed  0
Sourced 64 Test Files.
Number of tests skipped for each constraint:
         20      crypto
         2       knownBug
         1       notDarwin
         1       stress
Terminated
gmake: *** [Makefile:222: test] Error 143

--------

Best regards
Rodrigo