You can subscribe to this list here.
2005 |
Jan
|
Feb
(53) |
Mar
(62) |
Apr
(88) |
May
(55) |
Jun
(204) |
Jul
(52) |
Aug
|
Sep
(1) |
Oct
(94) |
Nov
(15) |
Dec
(68) |
---|---|---|---|---|---|---|---|---|---|---|---|---|
2006 |
Jan
(130) |
Feb
(105) |
Mar
(34) |
Apr
(61) |
May
(41) |
Jun
(92) |
Jul
(176) |
Aug
(102) |
Sep
(247) |
Oct
(69) |
Nov
(32) |
Dec
(140) |
2007 |
Jan
(58) |
Feb
(51) |
Mar
(11) |
Apr
(20) |
May
(34) |
Jun
(37) |
Jul
(18) |
Aug
(60) |
Sep
(41) |
Oct
(105) |
Nov
(19) |
Dec
(14) |
2008 |
Jan
(3) |
Feb
|
Mar
(7) |
Apr
(5) |
May
(123) |
Jun
(5) |
Jul
(1) |
Aug
(29) |
Sep
(15) |
Oct
(21) |
Nov
(51) |
Dec
(3) |
2009 |
Jan
|
Feb
(36) |
Mar
(29) |
Apr
|
May
|
Jun
(7) |
Jul
(4) |
Aug
|
Sep
(4) |
Oct
|
Nov
(13) |
Dec
|
2010 |
Jan
|
Feb
|
Mar
(9) |
Apr
(11) |
May
(16) |
Jun
|
Jul
|
Aug
|
Sep
(1) |
Oct
|
Nov
|
Dec
|
2011 |
Jan
|
Feb
|
Mar
|
Apr
|
May
(1) |
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
2012 |
Jan
(7) |
Feb
(3) |
Mar
|
Apr
|
May
|
Jun
(3) |
Jul
|
Aug
|
Sep
|
Oct
(92) |
Nov
(28) |
Dec
(16) |
2013 |
Jan
(9) |
Feb
(2) |
Mar
|
Apr
(4) |
May
(4) |
Jun
(6) |
Jul
(14) |
Aug
(12) |
Sep
(4) |
Oct
(13) |
Nov
(1) |
Dec
(6) |
2014 |
Jan
(23) |
Feb
(19) |
Mar
(10) |
Apr
(14) |
May
(11) |
Jun
(6) |
Jul
(11) |
Aug
(15) |
Sep
(41) |
Oct
(95) |
Nov
(23) |
Dec
(11) |
2015 |
Jan
(3) |
Feb
(9) |
Mar
(19) |
Apr
(3) |
May
(1) |
Jun
(3) |
Jul
(11) |
Aug
(1) |
Sep
(15) |
Oct
(5) |
Nov
(2) |
Dec
|
2016 |
Jan
(7) |
Feb
(11) |
Mar
(8) |
Apr
(1) |
May
(3) |
Jun
(17) |
Jul
(12) |
Aug
(3) |
Sep
(5) |
Oct
(19) |
Nov
(12) |
Dec
(6) |
2017 |
Jan
(30) |
Feb
(23) |
Mar
(12) |
Apr
(32) |
May
(27) |
Jun
(7) |
Jul
(13) |
Aug
(16) |
Sep
(6) |
Oct
(11) |
Nov
|
Dec
(12) |
2018 |
Jan
(1) |
Feb
(5) |
Mar
(6) |
Apr
(7) |
May
(23) |
Jun
(3) |
Jul
(2) |
Aug
(1) |
Sep
(6) |
Oct
(6) |
Nov
(10) |
Dec
(3) |
2019 |
Jan
(26) |
Feb
(15) |
Mar
(9) |
Apr
|
May
(8) |
Jun
(14) |
Jul
(10) |
Aug
(10) |
Sep
(4) |
Oct
(2) |
Nov
(20) |
Dec
(10) |
2020 |
Jan
(10) |
Feb
(14) |
Mar
(29) |
Apr
(11) |
May
(25) |
Jun
(21) |
Jul
(23) |
Aug
(12) |
Sep
(19) |
Oct
(6) |
Nov
(8) |
Dec
(12) |
2021 |
Jan
(29) |
Feb
(9) |
Mar
(8) |
Apr
(8) |
May
(2) |
Jun
(2) |
Jul
(9) |
Aug
(9) |
Sep
(3) |
Oct
(4) |
Nov
(12) |
Dec
(13) |
2022 |
Jan
(4) |
Feb
|
Mar
(4) |
Apr
(12) |
May
(15) |
Jun
(7) |
Jul
(10) |
Aug
(2) |
Sep
|
Oct
(1) |
Nov
(8) |
Dec
|
2023 |
Jan
(15) |
Feb
|
Mar
(23) |
Apr
(1) |
May
(2) |
Jun
(10) |
Jul
|
Aug
(22) |
Sep
(19) |
Oct
(2) |
Nov
(20) |
Dec
|
2024 |
Jan
(1) |
Feb
|
Mar
(16) |
Apr
(15) |
May
(6) |
Jun
(4) |
Jul
(1) |
Aug
(1) |
Sep
|
Oct
(13) |
Nov
(18) |
Dec
(6) |
2025 |
Jan
(12) |
Feb
|
Mar
(2) |
Apr
(1) |
May
(11) |
Jun
(5) |
Jul
(4) |
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
From: Gustaf N. <ne...@wu...> - 2021-01-25 20:40:05
|
Dear all, some of you might be interested in the following: I have just committed a change for the nsstats module that visualizes for a cache its reuse statistics via a histogram. In the chart below we see that only 30% of the particular cache are actually reused (the rest was just loaded but never reused). One can reduce the size of this cache to a third without losing any performance (for the current load and usage pattern) all the best -gn |
From: David O. <da...@qc...> - 2021-01-25 16:42:57
|
This is part of the commit which seems to be changing the behaviour in our case in SockSetServer in driver.c.. sockPtr->location would have previously been set to drvPtr->location here and passed on to connPtr->location in NsQueueConn I think. However, now sockPtr->location is set to NULL, and the "if (sockPtr->servPtr == NULL)" check fails in this case and sockPtr->location remains unset. [image: image.png] On Mon, 25 Jan 2021 at 11:45, David Osborne <da...@qc...> wrote: > Hi again, > > We're also having issues with the use of the "location" param to specify > the location for ns_returnredirect in the latest version of naviserver. > > We used to be able to have something like: > > ns_section ns/server/default/module/nssock > ns_param location https://domain:port > ... > > And a "ns_returnredirect /" from that server would expand "/" to " > https://domain:port/". > > However, in the latest version this doesn't seem to be the case anymore. > The redirect is being expanded to the server address (0.0.0.0 in > this case) rather than the defined location. > > > *Reproduction steps (on debian stretch)*: > > 1. Build and install Naviserver. > > 2. Changes to /usr/local/ns/conf/nsd-config.tcl > > Change ns_section "ns/fastpath" > to > > ns_section "ns/server/default/fastpath" > > Remove: > ns_section "ns/modules" > > Rename ns_section "ns/module/nssock" to > "ns/server/default/module/nssock" > Add > > ns_param location http://192.168.56.10:8080 > > Add to "ns/server/default/modules" > > ns_param nssock nssock > > 3. Add file /usr/local/ns/pages/test.tcl > containing > "ns_returnredirect /" > > 4. sudo /usr/local/ns/bin/nsd -u nsadmin -t > /usr/local/ns/conf/nsd-config.tcl -c > > 5. At naviserver 4.99.19 when visiting /test.tcl we get a 302 to the > location defined: > "Location: http://192.168.56.10:8080/" > > 6. At naviserver 4.99.20 with the same config we get redirected to the > driver address instead: > "Location: http://0.0.0.0:8080/" > > Is this a case of the new code exposing a faulty config, or has something > else changed? > > Seems to be down to *Ns_ConnLocationAppend *now having a null value in > connPtr->location at the point where the redirection is happening. So the > procedure drops back to the final option which is to query the driver > protocol, address and port. > But I'm failing to work out what change has stopped connPtr->location > being populated at this point. > > Any help would be appreciated! > > Regards, > > -- > > *David Osborne | Software Engineer* > Qcode Software, Castle House, Fairways Business Park, Inverness, IV2 6AA > *Email:* da...@qc... | *Phone:* 01463 896 484 > www.qcode.co.uk > > > -- *David Osborne | Software Engineer* Qcode Software, Castle House, Fairways Business Park, Inverness, IV2 6AA *Email:* da...@qc... | *Phone:* 01463 896 484 www.qcode.co.uk |
From: David O. <da...@qc...> - 2021-01-25 11:47:04
|
Thank you - working perfectly now. On Sun, 24 Jan 2021 at 18:23, Gustaf Neumann <ne...@wu...> wrote: > Dear all, > > The 4.99 branch (bug-fix branch for the 4.99 family) is now updated > with the change of David. Additionally, i have added basic test cases via > introducing a new command "nscp users". > > all the best -g > > > https://bitbucket.org/naviserver/naviserver/commits/782e8cbd4529d548a9667a2eb0f2e370b8841632 > > > > > _______________________________________________ > naviserver-devel mailing list > nav...@li... > https://lists.sourceforge.net/lists/listinfo/naviserver-devel > -- *David Osborne | Software Engineer* Qcode Software, Castle House, Fairways Business Park, Inverness, IV2 6AA *Email:* da...@qc... | *Phone:* 01463 896 484 www.qcode.co.uk |
From: David O. <da...@qc...> - 2021-01-25 11:45:35
|
Hi again, We're also having issues with the use of the "location" param to specify the location for ns_returnredirect in the latest version of naviserver. We used to be able to have something like: ns_section ns/server/default/module/nssock ns_param location https://domain:port ... And a "ns_returnredirect /" from that server would expand "/" to " https://domain:port/". However, in the latest version this doesn't seem to be the case anymore. The redirect is being expanded to the server address (0.0.0.0 in this case) rather than the defined location. *Reproduction steps (on debian stretch)*: 1. Build and install Naviserver. 2. Changes to /usr/local/ns/conf/nsd-config.tcl Change ns_section "ns/fastpath" to > ns_section "ns/server/default/fastpath" Remove: ns_section "ns/modules" Rename ns_section "ns/module/nssock" to "ns/server/default/module/nssock" Add > ns_param location http://192.168.56.10:8080 Add to "ns/server/default/modules" > ns_param nssock nssock 3. Add file /usr/local/ns/pages/test.tcl containing "ns_returnredirect /" 4. sudo /usr/local/ns/bin/nsd -u nsadmin -t /usr/local/ns/conf/nsd-config.tcl -c 5. At naviserver 4.99.19 when visiting /test.tcl we get a 302 to the location defined: "Location: http://192.168.56.10:8080/" 6. At naviserver 4.99.20 with the same config we get redirected to the driver address instead: "Location: http://0.0.0.0:8080/" Is this a case of the new code exposing a faulty config, or has something else changed? Seems to be down to *Ns_ConnLocationAppend *now having a null value in connPtr->location at the point where the redirection is happening. So the procedure drops back to the final option which is to query the driver protocol, address and port. But I'm failing to work out what change has stopped connPtr->location being populated at this point. Any help would be appreciated! Regards, -- *David Osborne | Software Engineer* Qcode Software, Castle House, Fairways Business Park, Inverness, IV2 6AA *Email:* da...@qc... | *Phone:* 01463 896 484 www.qcode.co.uk |
From: Gustaf N. <ne...@wu...> - 2021-01-24 18:22:50
|
Dear all, The 4.99 branch (bug-fix branch for the 4.99 family) is now updated with the change of David. Additionally, i have added basic test cases via introducing a new command "nscp users". all the best -g https://bitbucket.org/naviserver/naviserver/commits/782e8cbd4529d548a9667a2eb0f2e370b8841632 |
From: Gustaf N. <ne...@wu...> - 2021-01-21 16:01:08
|
Hi David, Many thanks for that details, this helps a lot. I will look into this, but probably, the weekend is the earliest time. Now, i understand the problem at least. all the best -g On 21.01.21 16:15, David Osborne wrote: > > I dug a bit more into why NaviServer at master might not be allowing > us to configure any nscp users. > > I created a pull request to show you which changes got it working for us. |
From: David O. <da...@qc...> - 2021-01-21 15:16:17
|
Hi again, I dug a bit more into why NaviServer at master might not be allowing us to configure any nscp users. I created a pull request to show you which changes got it working for us. At master, for us, LoadUsers ignores any nscp users defined in the "ns/server/default/module/nscp/users" section unless we make this type of change: https://bitbucket.org/naviserver/naviserver/pull-requests/10/illustrative-fix-for-nscp-loadusers Hope this might help. Regards, David On Tue, 19 Jan 2021 at 16:58, David Osborne <da...@qc...> wrote: > Hi again, > > One issue I'm noticing since these changes is that setting up users for > the control port has changed. > Using the default config of nsd-config.tcl from master (the same basic > test case from the top of this thread): > > sudo /usr/local/ns/bin/nsd -u nsadmin -t /usr/local/ns/conf/nsd-config.tcl > ... > [19/Jan/2021:16:50:46][19522.7fcb4ff43700][-main:default-] Notice: > modload: loading module nscp from file nscp > [19/Jan/2021:16:50:46][19522.7fcb4ff43700][-main:default-] Notice: > nscp[default]: listening on [127.0.0.1]:4080 > [19/Jan/2021:16:50:46][19522.7fcb4ff43700][-main:default-] Warning: > nscp[default]: no authorized users > > And then obviously, no access of the cp is possible: > > $ nc 127.0.0.1 4080 > login: > Password: > Access denied! > > Previously, the default config would allow logging into the control port > with blank user and password from the config: > > ns_section "ns/server/default/module/nscp" { > ns_param port 4080 > #ns_param address 0.0.0.0 > } > > ns_section "ns/server/default/module/nscp/users" { > ns_param user "::" > } > > Regards, > David > > On Tue, 12 Jan 2021 at 11:07, David Osborne <da...@qc...> wrote: > >> Thank you for a very quick turnaround Gustaf! I'll do a build at rc4 and >> continue testing. >> Regards, >> >> On Tue, 12 Jan 2021 at 10:16, Gustaf Neumann <ne...@wu...> wrote: >> >>> On 11.01.21 16:45, Gustaf Neumann wrote: >>> > Will go through these more carefully and produce a new rc when done. >>> -gn >>> >>> The updated version is running since yesterday on openacs.org (4 >>> virtual >>> servers [1]), i have tested also with macOS 10.14.6, Ubuntu 20.04, >>> Fedora Core 32, OpenBSD 6.8 (LibreSSL 3.1.1), FreeBSD 12.2. >>> >>> There is also an updated release candidate on sourceforge [2]. My plan >>> would be to release on Friday. >>> >>> all the best and many thanks for your help >>> >>> -g >>> >>> [1] https://openacs.org/forums/message-view?message_id=5468281 >>> [2] >>> https://sourceforge.net/projects/naviserver/files/naviserver/4.99.20/ >>> >>> >>> >>> -- *David Osborne | Software Engineer* Qcode Software, Castle House, Fairways Business Park, Inverness, IV2 6AA *Email:* da...@qc... | *Phone:* 01463 896 484 www.qcode.co.uk |
From: David O. <da...@qc...> - 2021-01-19 16:58:50
|
Hi again, One issue I'm noticing since these changes is that setting up users for the control port has changed. Using the default config of nsd-config.tcl from master (the same basic test case from the top of this thread): sudo /usr/local/ns/bin/nsd -u nsadmin -t /usr/local/ns/conf/nsd-config.tcl ... [19/Jan/2021:16:50:46][19522.7fcb4ff43700][-main:default-] Notice: modload: loading module nscp from file nscp [19/Jan/2021:16:50:46][19522.7fcb4ff43700][-main:default-] Notice: nscp[default]: listening on [127.0.0.1]:4080 [19/Jan/2021:16:50:46][19522.7fcb4ff43700][-main:default-] Warning: nscp[default]: no authorized users And then obviously, no access of the cp is possible: $ nc 127.0.0.1 4080 login: Password: Access denied! Previously, the default config would allow logging into the control port with blank user and password from the config: ns_section "ns/server/default/module/nscp" { ns_param port 4080 #ns_param address 0.0.0.0 } ns_section "ns/server/default/module/nscp/users" { ns_param user "::" } Regards, David On Tue, 12 Jan 2021 at 11:07, David Osborne <da...@qc...> wrote: > Thank you for a very quick turnaround Gustaf! I'll do a build at rc4 and > continue testing. > Regards, > > On Tue, 12 Jan 2021 at 10:16, Gustaf Neumann <ne...@wu...> wrote: > >> On 11.01.21 16:45, Gustaf Neumann wrote: >> > Will go through these more carefully and produce a new rc when done. -gn >> >> The updated version is running since yesterday on openacs.org (4 virtual >> servers [1]), i have tested also with macOS 10.14.6, Ubuntu 20.04, >> Fedora Core 32, OpenBSD 6.8 (LibreSSL 3.1.1), FreeBSD 12.2. >> >> There is also an updated release candidate on sourceforge [2]. My plan >> would be to release on Friday. >> >> all the best and many thanks for your help >> >> -g >> >> [1] https://openacs.org/forums/message-view?message_id=5468281 >> [2] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.20/ >> >> >> >> |
From: oleg <oo...@ua...> - 2021-01-18 10:00:46
|
On Sun, 17 Jan 2021 19:17:51 +0100 Gustaf Neumann <ne...@wu...> wrote: > Under windows, the regression tests reports now 1611 passes > and 65 fails.... we are improving. I have 23 failed tests for mingw build and 24 - for msvc32/64 (with ~1660 successed). It looks like the mentioned test were executed by Naviserver w/o zlib. Oleg. PS. To fix I use attached Makefile.win32 in the nsd/ folder. This fix may use zlib sources from then Tcl8.6 distribution. |
From: Maksym Z. <siq...@gm...> - 2021-01-17 22:25:55
|
Thanks Gustaf, I've seen it already in the release changelog announcement. As always amazing job. On Sun, Jan 17, 2021, 21:06 Gustaf Neumann <ne...@wu...> wrote: > I forgot to mention here: the updated version of "nsshell" performs now > proper html-quoting, such that confusing result interpretations won't > happen anymore. > > all the best -gn > On 10.01.21 11:40, Maksym Zinchenko wrote: > > :) Sorry false alert, my bad. Due to lack of coffee and some good sleep, I > was testing some procs in "nsshell", and of course browser rendering > escaped characters back to normal, and my zombie brain couldn't process > why. It works fine, at least it has a regression test now ;) > > On Sat, Jan 9, 2021 at 5:44 PM Gustaf Neumann <ne...@wu...> wrote: > >> >> On 09.01.21 19:01, Maksym Zinchenko wrote: >> >> Hello everyone, may someone confirm that ns_quotehtml works as it should, >> because on my system its not: >> >> Command: ns_quotehtml {<span class="foo">} >> Result: <span class="foo"> >> >> What is your system? It works for me. What is your OS? >> Interestingly, ns_quotehtml was not covered by the regression tests. >> >> -g >> >> >> % ./nsd/nsd -c >> [09/Jan/2021:19:36:16][93167.10d7eb5c0][-main:conf-] Notice: OpenSSL 1.1.1i 8 Dec 2020 initialized >> [09/Jan/2021:19:36:16][93167.10d7eb5c0][-main:conf-] Notice: nsmain: NaviServer/4.99.20rc2 (c407fafe7b3f+) starting >> ... >> % >> % ns_quotehtml {<span class="foo">} >> <span class="foo"> >> >> >> _______________________________________________ >> naviserver-devel mailing list >> nav...@li... >> https://lists.sourceforge.net/lists/listinfo/naviserver-devel >> > > > _______________________________________________ > naviserver-devel mailing lis...@li...https://lists.sourceforge.net/lists/listinfo/naviserver-devel > > _______________________________________________ > naviserver-devel mailing list > nav...@li... > https://lists.sourceforge.net/lists/listinfo/naviserver-devel > |
From: Gustaf N. <ne...@wu...> - 2021-01-17 22:06:23
|
I forgot to mention here: the updated version of "nsshell" performs now proper html-quoting, such that confusing result interpretations won't happen anymore. all the best -gn On 10.01.21 11:40, Maksym Zinchenko wrote: > :) Sorry false alert, my bad. Due to lack of coffee and some good > sleep, I was testing some procs in "nsshell", and of course browser > rendering escaped characters back to normal, and my zombie brain > couldn't process why. It works fine, at least it has a regression test > now ;) > > On Sat, Jan 9, 2021 at 5:44 PM Gustaf Neumann <ne...@wu... > <mailto:ne...@wu...>> wrote: > > > On 09.01.21 19:01, Maksym Zinchenko wrote: >> Hello everyone, may someone confirm that ns_quotehtml works as it >> should, because on my system its not: >> >> Command: ns_quotehtml {<span class="foo">} >> Result: <span class="foo"> > > What is your system? It works for me. What is your OS? > Interestingly, ns_quotehtml was not covered by the regression tests. > > -g > > > % ./nsd/nsd -c > [09/Jan/2021:19:36:16][93167.10d7eb5c0][-main:conf-] Notice: OpenSSL 1.1.1i 8 Dec 2020 initialized > [09/Jan/2021:19:36:16][93167.10d7eb5c0][-main:conf-] Notice: nsmain: NaviServer/4.99.20rc2 (c407fafe7b3f+) starting > ... > % > % ns_quotehtml {<span class="foo">} > <span class="foo"> > > > _______________________________________________ > naviserver-devel mailing list > nav...@li... > <mailto:nav...@li...> > https://lists.sourceforge.net/lists/listinfo/naviserver-devel > <https://lists.sourceforge.net/lists/listinfo/naviserver-devel> > > > > _______________________________________________ > naviserver-devel mailing list > nav...@li... > https://lists.sourceforge.net/lists/listinfo/naviserver-devel |
From: Gustaf N. <ne...@wu...> - 2021-01-17 18:18:10
|
Dear all, The release of NaviServer 4.99.20 is now available on sourceforge [1]. This release was originally planned to be a small release (essentially the movement from mercurial to git, as forced by bitbucket), but turned out to be the release with the highest number of changes so far. The short summary of changes is (see below for more details): - Support for time units (specify time units in minutes, seconds, milliseconds...) - Subsecond granularity for all NaviServer commands - HTTP client request logging - Added support for "100 CONTINUE" - Improved server liveliness (management of high loads and request queue overruns) - Support for running behind a reverse proxy server - Driver support for listening on multiple ports - Improved HTTPS support (OCSP Stapling, server-side SNI) - Improved support for client-side HTTP requests via ns_http (timeout handling, binary hindling, log file for outgoing requests) - Improved logging for detecting long running background jobs - Dict support for nsvs - More complete set of subcommands for "ns_set" - Improved form-processig (especially for REST processing) - C-level WebSocket support - Improved server statistics - Improved scalablility (reduced lock contention, higher usage of rwlocks, significanly faster nsv_*appends, ...) - Improved portability: many improvements for windows, compiled with macOS 10.14.6, Ubuntu 20.04, Fedora Core 32, OpenBSD 6.8 (LibreSSL 3.1.1), FreeBSD 12.2 The following people have contributed to this release: Andrew Piskorski David Osborne Gustaf Neumann Hector Romojaro Maksym Zinchenko Oleg Oleinick Zoran Vasiljevic Below is a summary of changes. The major changes since RC1 are improved robustness in case sections are missing in the configuration file and improved windows support (also addressing the problem with ns_sockselect under windows 64-bit that Oleg reported on Jan 10). Under windows, the regression tests reports now 1611 passes and 65 fails.... we are improving. Future goals: - Move towards release 5.0 - Upgrade license from Mozilla 1.1/GPL -> Mozilla 2.0 (MPL allows these upgrades, change is necessary for debian packaging with OpenSSL) - Change to multiple branches (make current master to the 4.99 branch, provide in future for every release an own branch to ease providing fixes for older releases; the last digit of the version number should be just the patch level; tags are unchanged on the patch level) best regards -gustaf [1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.20/ ======================================= NaviServer 4.99.20, released 2021-01-17 ======================================= 380 files changed, 27639 insertions(+), 12103 deletions(-) New Features: ------------- - Ns_Time reform * Support for time units: . arguments of functions accepting time durations can be provided now with a time unit suffix (such as ms, s, m, d). These values can be written e.g. as 1.5ms, 3h and the like. . units can be used as configuration values wherever appropriate . time values wherever appropriate in documentation . commands using Ns_ObjvTime can be specified now with time units . This change is fully backward compatible for all commands and configuration options accepting seconds, which was normally the case. . All sample configuration files have been updated with time units . nsproxy: The only exception to the precious sentence is nsproxy, where durations we specified in earlier releases as milliseconds. This is a POTENTIAL INCOMPATIBILITY with previous versions. These parameters are * config parameters of the nsproxy module "evaltimeout", "gettimeout", "idletimeout", "recvtimeout", "sendtimeout", "waittimeout" * ns_proxy get: changed "-timeout" parameter to time unit value * ns_proxy wait: changed optional "timeout" parameter to time unit value * Change granularity from seconds to Ns_Time (internally and in Tcl API) . Added proper handling of negative values in Ns_Time (was broken/not implemented) . Upgraded scheduling machinery from seconds to Ns_Time values . Extended Ns_Time arithmetic (Ns_DiffTime()) to handle positive and negative Ns_Time values . Extended regression test for ns_time significantly . Ns_After(), Ns_ScheduleProc(), Ns_ScheduleProcEx(): change argument from int secs to Ns_Time . Marked Ns_ScheduleProc() as deprecated, Ns_ScheduleProcEx() should be used instead. . ns_schedule_proc, ns_after: support time specs with time units (instead of only full seconds) . Ns_TimeToMilliseconds(): new API call for converting positive and negative Ns_Time values to ms . Ns_DStringAppendTime(): format times in a more compact form (no trailing zeros) . Use consistently Ns_TimeToMilliseconds() to convert Ns_Time to ms . Ns_ObjvTimeRange, Ns_CheckTimeRange(): support to specify time ranges in argument handling . Trim trailing zeros after comma point when printing Ns_Time, represent as int when possible . "ns_info scheduled" returns now timestamps in sub-second resolution (with a second faction). By this, the runtime of the scheduled procedure can be determined on the sub-second level. (potential incompatibility) . Added handling of "schedmaxelapsed" on the sub-second level. - HTTP client request logging NaviServer can now optionally write a logfile for outgoing HTTP requests initiated via "ns_http". This is important for webservers that are dependent on other web services for the healthiness and performance. The logfile is similar to the access.log and suitable for automated analysis. Previously, NaviServer was "blind" on that data. Now it is easier to provide data about the performance of external services (e.g. cloud services such as authentication, Google Firebase Cloud Messaging, Office 365, ...) To activate HTTP client request logging, a new config-file section "ns/server/$server/httpclient" was created where logroll, logrollfmt and the other usual logging parameter can be specified. - New command line option for NaviServer: "-T" When this option is used, NaviServer just checks the configuration file (specified as usual by "-t") for syntax errors. This option should be used on production sites to reduce the downtime of a server in case of typos. - Added support for "100 CONTINUE" This feature is in HTTP since a very long time, but not widely supported although useful (best support is currently in curl). When the client sends "Expect: 100-continue", the server can check the header fields and provide an early feedback, whether it is useful to transmit the full body. This way, e.g. content which is too large can be rejected quickly. - Improved server liveliness under high load (improved and configurable handling of connection queue overrun) When NaviServer processes an HTTP request, it is received (head and body) and associated with a connection thread pool. If the thread pool has a thread free, the request is directly assigned to this thread. Otherwise, the request is added to a queue of the connection pool, waiting for a connection thread. Since the queue has a limited size (limited by "maxconnections"), this queue might overrun. In this situation, the received request is added to a waiting list. Here is, where the behavior is now improved. 1) previously, as soon as there were entries in the waiting list, no more fresh requests were accepted by the driver (just working on waiting requests). While this behavior is sensible if only one connection pool is available, it causes a blocking behavior in cases, when a single connection pool overruns but others could continue. Now, more fresh requests are still accepted in this situation. 2) In addition, it is now possible to configure the overrun behavior via two pool parameters: a) pool parameter "rejectoverrun": when set to true, NaviServer sends on queue overruns an error "503 service unavailable". The overflowing request is not added to the waiting list. b) when (a) is set, it is possible to specify parameter "retryafter" with a time interval to add reply header field "Retry-After" to the 503 response. (see https://tools.ietf.org/html/rfc7231#section-7.1.3) When "rejectoverrun" is false (default) then behavior is similar to before, accept the server does not fully block. The request is kept in the waiting list, which is checked, whenever a request finishes. For the client, there is no need to resubmit the request, which will be proceed eventually. Technically, this means an arbitrary large queue. When "rejectoverrun" is true, then the request causing the overrun is rejected, and will have to be resubmitted by the client (maybe automatically, when "Retry-After" is specified. However, the client support for "Retry-After" is rather bad. For busy sites, "rejectoverrun" is probably the better behavior, since no background queuing in the waiting list is performed. For now, we keep the old behavior as default for better backwards compatibility. The default might change in the future. - General support for running NaviServer behind a reverse proxy server This change provides general support for running NaviServer behind a reverse proxy. Previously, the support was rather punctual (e.g. for nslog). Reverse proxy mode is activated by setting the global parameter (in section "ns/parameters") named "reverseproxymode" to true (default false). Current limitation: In the current implementation, NaviServer supports only the "X-Forwarded-For" header field, which is supported by most proxy servers, treating the first provided IP-address as the client IP address (see https://en.wikipedia.org/wiki/X-Forwarded-For). The newer "Forwarded" header field is currently not supported. In the reverse proxy mode, the client address is taken as forwarded by the reverse proxy. This has the following consequences: * Reported IP address by the following commands. ns_conn peeraddr ns_server all|running ns_writer list ns_connchan list * Reported IP address in the access.log * Context filters (used e.g. for mapping requests to pools) In reverse proxy mode, all IP-address specific context filters refer to the x-forwarded-for addresses. This makes it possible to use context filters (of URLspace commands) based on IP-addresses (IPv4, IPv6, with masks) also behind reverse proxy servers. Previously the IP-based mapping was just possible when no reverse proxy was in use. - Allow a single driver to listen on multiple ports: This new feature simplifies setups, where a single server is listening on multiple ports using the same driver. Previously, it was necessary to define separate drivers for such cases, which need different names but which are often configured identically. Together with the change introduced in 4.99.19 (support for multiple listen addresses) this means that when multiple addresses and multiple ports are specified for a single driver, it will listen on the cross product of all combinations. The maximum number of listening ports of a driver is limited by a compile time constant to MAX_LISTEN_ADDR_PER_DRIVER (per default 16). To use this feature, specify in the configuration file for "port" in the network driver section a space separated list of multiple ports. This change is fully backwards compatible, old configuration files will continue to work. - SSL/TLS improvements: * Support for OCSP Stapling . NaviServer obtains and provides information about the validity of the used certificates. For proving this information, the server performs two level of caching: in-memory caching and disk caching. When the server receives the first TLS request with OCSP stapling turned on, it checks for an already retrieved OCSP response. The disk cache file is saved in the "log" directory of the server and uses the serial number of the certificate to be checked as filename with ".der" as extension. When the disk cache file does not exist, an http/https request is made to the server issuing the servers certificate as defined by the Authority Information Access (AIA) Extension. The names of the file and the http/https request for the OCSP response can be obtained from the system log of the server . OCSP Stapling can be turned on in the configuration files (see sample configuration files) . OCSP Stapling requires OpenSSL 1.1.0 or newer * Support for server-side SNI (Server Name Indication (SNI) Server-side SNI can be used to use different certificates for virtual hosts. In addition to the top-level certificate, different certificates can be defined per virtual server (below: srv2). The certificate will be registered for all alternate names specified for the server. In addition to the certificate, also all other OpenSSL specific parameters (ciphers, protocols, ...) will be loaded form the same place as the certificate. The following example shows the setup for two virtual servers "srv1" and "srv2", where the first one is the default server. ns_section ns/modules { # ... ns_param nsssl ${bindir}/nsssl.so } ns_section ns/module/nsssl { # ... ns_param defaultserver srv1 ns_param certificate /usr/local/ns/modules/nsssl/srv1.pem } ns_section "ns/module/nsssl/servers" { ns_param srv1 test.org ns_param srv1 www.test.org ns_param srv2 someotherhost.org } ns_section "ns/server/srv2/module/nsssl" { # ... ns_param certificate /usr/local/ns/modules/nsssl/srv2.pem ns_param ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 } SNI support includes OCSP, DH ephemeral key exchange, reuse of SSLcontexts. * Support for OpenSSL "ciphersuites" (TLSv1.3 feature) * Support for client-side SNI: new parameter "hostname" for "ns_connchan open" * Support for removing TLSv1.0, TLSv1.1, TLSv1.2, and TLSv1.3 via configuration file - ns_http: * Clear separation of "-timeout" and "-expire": . "-timeout" is used for single communication acts (e.g. connect, read write) . "-expire" is used for limiting the full duration of the request (also useful for slow read/write attacks). Scripts already using the -timeout will most probably not see any change unless they are handling huge up/downloads. In such cases we will now continue with the request, instead of terminating it in the middle. * added "-binary" flag to transmit data binary and perform no automatic encoding transformations. - moved code repository on BitBucket from mercurial to git. * adjusted documentation * adjusted build system (Makefiles) - Logging improvements: * New log severity "Debug(timeout)" to force NaviServer to write an entry to the systemlog, whenever a timeout occurs. Therefore, the command ns_logctl severity Debug(timeout) on can be used to debug cases, where NaviServer runs into HTTP status code 503 (Service Unavailable). * Logging of commands with high latency: Provide a consistent interface and naming convention for logging slow commands for nsproxy, ns_jobs, scheduled procedures and SQL (nsdb drivers). . nsproxy and nsjobs have a new configuration option "logminduration" This configuration option specifies a time-limit for logging (long) eval operations to the system log (similar to "logminduration" in the database drivers). Set it to a high value to avoid logging (e.g. 1d). The default is 1s. . scheduled procedures: Changed "schedmaxelapsed" to "schedlogminduration" for consistency. This is a potential incompatibility for old configurations. - New commands: * "nsv_dict": The new command "nsv_dict" provides thread-safe support for setting, getting, updating, and unsetting dict values in nsv array. Motivation: Some larger OpenACS installations have currently more than 1 mio (!) nsv arrays, which comes mostly from creating nsv arrays per packages. Most of these arrays have only a few keys with very short values. The number of arrays can be greatly reduced by supporting complex values (dictionaries), similar to REDIS hash values. . "nsv_dict" supports nested dicts like in Tcl . The following dict commands: nsv_dict append array key dictkey dictvalue:0..n nsv_dict exists array key dictkey:1..n nsv_dict get array key dictkey:0..n nsv_dict getdef array key dictkey:0..n default nsv_dict incr array key dictkey ?increment? nsv_dict keys array key ?pattern? nsv_dict lappend array key dictkey dictvalue:0..n nsv_dict set array key dictkey:1..n dictvalue nsv_dict size array key nsv_dict unset array key dictkey:1..n . added 41 test cases in regression test for nsv_dict * "ns_dbquotelist": similar to "ns_dbquotevalue", but provides quoting for Tcl lists. * "ns_trim": multiline trim command: ns_trim ?-subst? ?-delimiter delimiter? text - New features for existing commands/subcommands * New subcommands "keys" and "values" for "ns_set": "ns_set keys /setId/ ?pattern?" "ns_set values /setId/ ?pattern?" The commands are similar to the Tcl command "dict keys" and "dict values" and can reduce the necessity for loops. * Form processing: . "ns_getformfile": return a list of filenames when "multiple" attribute is provided in the input field and multiple files are submitted . Decouple content-type processing from HTTP methods: Previously, parsing of form data in these two formats happened only on POST requests. However, several REST services use often also PUT with form data and distinguish POST and PUT methods mostly by "add item to a collection" (POST) and process single item (PUT) but not by their content-type processing. This change eases implementation of REST services. . Perform more strict parsing based on "ns_parsefieldvalue" and "ns_parseheader" in file-based form parser (for spooled content). Previously, there was a bug with the regexp-based parsing, when e.g. a filename of and upload file contained e.g. a semicolon. * "ns_conn": . new subcommand "headerlength" . "ns_conn auth" returns now Information about "Bearer" authentication (RFC 6750) when applicable, used e.g. in OAuth2 . "ns_conn peeraddr": New flag "-source" for specifying detailed information about the peer, especially useful in combination with reverse proxy mode; allowed values are "configured", "direct", "forwarded". * "ns_cookie": Support explicit setting of "same-site=none" In previous version of browsers, no explicit setting of the samesite flag was exactly the same as explicit setting. Since some browsers switched to a default of "lax", explicit setting became necessary. * "ns_dbquotevalue": provided a C implementation; was scripted before. * "ns_connchan": . Added option "-server" to "ns_connchan close" to be able to cleaning stale handles of every server. . Added C-level WebSocket support and buffered output handling. The new code is about 200x faster than the Tcl version (especially when bytewise XORs are needed), simpler to use (no Tcl-leval handling of incomplete frames, split frames, remaining leftover from partial writes) and more complete (handling of segmented WebSocket messages). . "ns_connchan read -websocket ....": returns a dict containing the WebSocket data (including payload, WebSocket status bits, buffering state, etc.; see documentation and NaviServer websocket module for details) . "ns_connchan write -buffered ..." handles partial writes via an internal buffer (Tcl developer has just to take care about writable callbacks when necessary) . "ns_connchan wsencode ...": WebSocket frame generation, returns binary data based on input parameters suitable for sending on a WebSocket channel. . "ns_connchan status ..." returns a dict containing the buffering status of a channel. * "ns_urelencode": added one more option "oauth1" for the "-part" argument. The "oauth1" encoding is defined in RFC 5849 Section 3.6 for the construction of the signature base string and the "Authorization" header field. * "ns_ictl": . "ns_ictl trace idle ...": new tracetype "idle", which fires, when a connection thread is idle for the threadtimeout period, but minthreads is already reached, and the thread won't terminate. . "ns_ictl maxconcurrentupdates ?max?": Configuration option for specifying the maximum number of concurrent updates for interpreter reloads (when epoch increases, e.g. reloads on OpenACS). Background: For large (say 50-100 connection threads) and busy sites (sustained 800 requests per second) with large blueprints (e.g. OpenACS) blueprint updates can cause delays when all these operations are concurrently performed. The blueprint updates happen when the epoch is incremented (e.g. in OpenACS: reloads of packages). A too high number of concurrent blueprint updates cases (a) delays in request processing (b) increased update times per thread (e.g. up to 10 seconds). As a consequence, queueing will happen. This new option limits the number of concurrent updates (e.g. to 10) leading to smooth operations in such situations. When the maximum number of concurrent blueprints is reached, the pending updates are delayed until the limit is sufficiently low. By default, the value is unlimited. * Improved statistics: . "ns_server stats" returns now as well dropped requests from queue overruns. . "ns_info locks": report number of read and write operations for rwlocks. . "ns_proxy stats": report also the number of running worker processes. * "ns_getcsv": . provide compatibility with RFC 4180 . accept multi-line input . handle quotes in quotes and empty elements . optionally stripping unquoted elements (not covered by RFC) . optional support for a different quote character (option "-quotechar"). Although RFC 4180 specifies double quotes as quoting chars, sometimes also different quoting characters, which can be now successfully decoded. * "ns_parseurl": Support for fragment parsing. When provided URL contains a tail element, this is now parsed info a "query" and/or "fragment" component in the result. Previously, the query and fragment part was included in the "tail" element. * Improved thread naming for log-files: previously, it was not clear, from which server some output of the "main" thread originated. The new version adds always a server name specific suffix. * "ns_driver info": return also "port" and "defaultport" Bug Fixes: ---------- - Improved support for newer versions of OpenSSL (handling of SSL_R_UNEXPECTED_EOF_WHILE_READING) - ns_cache: Base cache size calculation on the full size of a cache entry. In previous versions, the cache size calculation was based only on the value size, leaving the size of the key and the size of the overhead (size of cache entry, about 126 bytes on a 64-bit machine) aside. This is misleading, when the cache value size is 0 bytes (just using the cache key as a flag), leaving admins wondering about the memory consumption. - Report to log file, when ns_ictl traces are not executed. Previously, these were silently ignored (e.g. in some error cases). - Fixed a bug with PUT methods, when query-parameters were passed in addition to the put content. Previously, these parameters were only available via [ns_queryget ...] when the content file was not spooled. - Fixed bug for custom error pages for HTTP methods other than GET (Custom error handles are defined as before via redirect sections): * Always use GET in the redirection not the original HTTP method. When e.g. "PUT /somepage" is requested, where no PUT handler but a custom error page /405.tcl, the previous code tried a PUT /405.tcl, leading the same error. Now, the error page us delivered like with a GET request. * Provided a hint as a warning, where NaviServer searches for non-existing pages. * Extended regression test: cover HEAD requests for existing and non-existing pages, and various forms of redirects. - Fixed issue with tDOM as noted by Maksym Zinchenko on the naviserver developer list: A recent change of tDOM 0.9.2 introduced a namespace local command "namespace" which was picked up by the blueprint generator incorrectly instead of the global ::namespace command. As a fix, all "namespace" invocations in the blueprint generator were replaced by "::namespace" to avoid considering invocation context for every occurrence. - Fixed potential abortion of regression test runs under macOS: Under macOS, tests runs were sometimes abruptly ended with a broken pipe signal. The call "NsBlockSignal(NS_SIGPIPE);" does not avoid these signals. So, we ignore bluntly all such signals by using "NsBlockSignal(NS_SIGPIPE);" - Fixed charset handling for spooled requests. - Fixed several cases of invalid chunked data (potential buffer overflows). - Don't swallow errors in braced config sections in the configuration file. - Fixed a potential crash, when certain configuration section were missing, but code tried to obtain configuration values from such sections. - Blueprint generation: * Fixed bug with interp aliases defined without fully qualified namespace path * Improve robustness when "rename" is used (when source exists and target does not exist) - Support calling "ns_conn status" (with the argument setting the HTTP return status) in traces (without open connection). - Make code more robust in case Tcl and NaviServer were compiled with different memory allocators. The mix if memory allocators could crash NaviServer. - Race Conditions: * Fixed several race conditions in "ns_http" * Fixed race conditions in "ns_connchan list" and "ns_connchan create" - Use time_t for secs in Ns_Time, since long is just 32-bit on 64-bit windows, but time_t is 64 bits. - "ns_writefp": fixed to act as advertised (read to end of file when no length is provided) - "ns_sendmail": Convert leading "." in message bodies to ".." as required by RFC 5321 section 4.5.2 - "ns_cache" transactions: in certain situations, where during concurrent entry creates, and one of the create attempts failed and the other succeeded, and the one of the cache transitions is rolled back, updates could lead to an error. - "ns_connchan": Fixed registry of connchans from all servers to the default server (not the best scalable solution, but useful as long we have only one "socks" thread (for more details see: https://sourceforge.net/p/naviserver/mailman/message/37179476/) - Fixed a 20-year-old time bomb in HTTP time parsing. Server was rejecting to parse HTTP time strings for 2021. Performance improvements: ------------------------- - nsv_lappend: The old implementation showed a linear slowdown for nsv_lappend operations depending on the number of elements in the list. The new implementation avoids frequent string to list and vice versa operations by using Tcl_DStrings. It is essentially independent on the number of elements in the nsv. For a list with 10K elements, the performance improvement is about a factor of 400. Test: nsv_set d 1 1 time {nsv_lappend d 1 1} 10000 OLD: 403.7255 microseconds per iteration NEW: 0.860547 microseconds per iteration This change makes it possible to use the OpenACS developer support on busy sites (before, this led to a quick standstill). - nsv_append: The performance gain is much less than with nsv_lappend (see previous commit), but still noticeable (about 35% faster, 10K ns_append operations went from 0.94 microseconds to 0.62 - Locking: * Reduced mutex-locked regions in driver. * Use ptrhreads interface for Ns_RWLock* when available So far, Ns_RWLock* used an implementation based on mutexes and cond vars. This implementation has the advantage of being portable. However, since on all *nix platforms, NaviServer uses the pthread library, using the POSIX rwlock implementation poses no additional dependency. Performance comparison fur using mutex/old rwlocks/pthread rwlocks on ns_urlspace; macOS (8 cores) rwlock (POSIX) 0.3105 2,27 % rwlock (ns) 11.5263 84,23 % mutex (ns) 13.6839 100,00 % Linux (6 cores) rwlock (POSIX) 0.6081 43,38 % rwlock (ns) 3.6259 258,66 % mutex (ns) 1.4018 100,00 % This data shows that the POSIX rwlocks implementation (pthread_rwlock_rdlock etc.) is always better than the classical NaviServer rwlock implementation, and that significant improvements can be achieved by using rwlocks. A more detailed performance comparison was provided on the developer mailing list. This change introduces: a) Ns_RWLock* based on pthreads when available b) switch to recommended configure setup for pthreads (AX_PTHREAD) c) New API call: Ns_RWLockSetName2() modeled after Ns_MutexSetName2() d) New API call: Ns_RWLockList() modeld after Ns_MutexList() (removed dependency of rwlocks to mutex statistics) * Support rwlocks for nsvs: Per default, rwlocks are used for all nsv* commands. The default behavior can be altered by setting the configuration variable "nsvrwlocks" to false in the section ns/server/${server}/tcl of the configuration file. This change can lead to improved scalability especially on multi-core machines since in read cases, the variables can be accessd unblocked for parallel executions. In most applications nsv are typically more often read than written rwlocks offer better scalability for large sites. The following table compares the busy locks with mutex operations vs the number of busy locks with rwlocks for openacs.org: With mutex (24h) locks busy nsv:3:openacs.org 27 4.71M 1.3K nsv:6:openacs.org 24 4.88M 1.03K nsv:2:openacs.org 28 3.37M 784 nsv:7:openacs.org 23 9.11M 755 nsv:5:openacs.org 25 2.88M 460 With rwlocks (24h) nsv:7:openacs.org 1 7.22M 0 nsv:6:openacs.org 2 3.92M 143 nsv:3:openacs.org 5 3.31M 1 nsv:2:openacs.org 6 2.23M 16 nsv:5:openacs.org 3 2.16M 0 On more busy sites, the improved scalability si much higher. * rwlocks are better in cases, where the number of readers is significantly higher than the number of writers. So, it has to be carefully considered in which cases rwlocks should be preferred. The new version uses rwlocks for nsv, ns_connchan, filters and tcl.cache (rwlocks for nsv and filters can be deactivated via configuration file). Earlier version used rwlocks already for adp tags, epoch handling and permissions. - Avoid multiple library initializations for OpenSSL 1.1.0 or newer. Documentation improvements: --------------------------- - Update references to Tcl website (replace http://tcl.tk by https://core.tcl-lang.org) - Added hints about required OpenSSL version to man pages - Made manual pages more consistent - Consistent indentation of program examples in documentation - Made source code documentation more consistent - Improved source code documentation (provide more details, remove obsolete comments) - Removed deprecated man pages - Removed obsolete sections in manual pages - Follow the recommended spelling convention of the Linux documentation project - Improved spelling, typesetting, and cross references - Reduced technical jargon - Made code listings in documentation more consistent (e.g. indenting) - Added missing commands in command list (overview page) - Updated listing of deprecated commands and suggested alternatives - Replaced several "slave process" by "child process" - 161 manual pages were updated since the last release Configuration Changes: ---------------------- - New configuration options: * Section "ns/parameters" . "reverseproxymode": Activate reverse proxy mode. The former configuration parameter of "nslog" named "checkforproxy" is obsolete now. . "joblogminduration": report jobs taking longer in systemlog . "schedlogminduration": report scheduled jobs taking longer in systemlog * Section "ns/server/${server}" . "rejectoverrun": when true send 503 when queue overruns; default "false" . "retryafter": time for Retry-After in 503 cases * Section "ns/server/${server}" . "filterrwlocks": use RWlocks for filters (default: true) * Section "ns/server/${server}/tcl" . "nsvrwlocks": use RWlocks for nsv (default: true) * Section "ns/server/$server/httpclient" . "logging": (default: off) . "logfile": name of http client logfile . "logrollfmt": format appended to log filename . "logmaxbackup": max number of backup log files (default: 10) . "logroll": perform automatic rolling (default: true) . "logrollonsignal": (default: false) . "logrollhour": specify at which hour to roll (default: 0) - Sample configurations: * Added values with time units where appropriate * Added new values mentioned above with short comments * For multi-server configuration files: use always the first (and not the last) server as default server. * openacs-config.tcl: * sample-config.tcl: . fixed config parameter: use "dnscachetimeout" instead of "keepwaittimeout" (name was changed in NaviServer ages ago, but change was not reflected in config files * nsd-config.tcl: * win32/test/nsd.tcl modernized test config file * nsproxy: Changed term "maxslaves" to "maxworkers" (the old term is deprecated) Code Changes: ------------- - Fixed potential access to an out-of-scope stack variable. - Added more declarations for PURE and CONST functions - Replaced usage of reserved identifier [cert-dcl37-c,cert-dcl51-cpp] - Replaced sscanf() by strtol*() to increase code-safety - Improved structure packing - Marked "ns_adp_mime" explicitly as deprecated, and provide deprecated messages for it. The command was deprecated more than 10 years ago, ns_adp_mimetype should be used - Added mime types for "heif" and "heic" which have become a preferred image type for iOS and macOS. - Regressions testing: * Reduced usages of old-style nstest::http-0.9 (socket and not ns_http-based tests) * Make test more robust concerning CRLF results (Windows) * Extended tests: 367 tests were added since the last release (changes in 38 test files) - Improved logging: * Added metering for long wait times of Ns_CsEnter locks (e.g. Ns_MasterLock) * Added concurrency level to log message of interp updates * Added warnings, when an NsTclRequestProc or an ADP request runs into a timeout. * Reduced time sensitivity on time when rolling log files_ Rolling log files happens often at midnight, using often a day precision. When e.g. a scheduled procedure the time when this function is called might be slightly after the scheduled time, which might lead to a day jump. The problem aggravates, when multiple log files are rotated. The new code identifies day wraps and uses in such cases the earlier day. - Improved portability: * Improved compatibility with the forthcoming OpenSSL 3.0 * Improved compatibility with various versions of LibreSSL * Improved Windows (MSVC) compatibility . Simplified Makefile.win32 for nmake . Fixed thread exit result handling for 64-bit Windows . Fixed thread handle access . Fixed default temp directory for windows . Turned on OpenSSL by default . Minimal support for running regression tests under windows . Fixed windows size incompatibility for range requests (64-bit Windows uses a 32-bit sized offset type off_t) . Fixed ns_sockselect for 64-bit windows installations . Fixes for cross-compiling for Windows 64-bit (using gcc/mingw) * Use consequently NS_EAGAIN (can be used in *nix and Windows) * Fixed potential problem with GNUC_PURE for older version of gcc * Removed dependency of tests from an explicit "/tmp" directory * Fixed regression test warnings with older versions of curl for IPv6 addresses * Fixed m4 configuration for "crypt_r" (broken since many years) * Compiled with macOS 10.14.6, Ubuntu 20.04, Fedora Core 32, OpenBSD 6.8 (LibreSSL 3.1.1), FreeBSD 12.2 - More OpenSSL changes: * Reactivate CRYPTO_set_mem_functions() for OpenSSL 1.1.0 or newer (interface om OpenSSL has changed) * Switched to SSL_CTX_set_dh_auto() from low-level functions: In newer versions of OpenSSL, all low-level DH* functions were deprecated. - Build-system * Compute version tag from git similarly like to the mercurial version. - improved support for MS windows * Windows math.h does not include round(), add simple implementation. - Added cross-platform function ns_localtime_r (localtime with user-provided storage): The previously defined function ns_localtime() uses a single per-thread local storage, so, it was not possible to have in one thread two time buffers with different values. - Improved error code handling on driver receive operations Previously, higher level of API calls were not able to access realiably the error information detected on the lower-level (async I/O-based or OpenSSL based) driver operations. This made it impossible to provide good error messages e.g. in "ns_connchan" and we could not provide error processing based on Tcl error codes. Furthermore, the new interface works for POSIX errnos and OpenSSL errors and improves the situation (currently just for driver read operations). * Ns_Sock* based setting of Tcl error codes: NsSockSetRecvErrorCode(const Sock *sockPtr, Tcl_Interp *interp) Set the Tcl error code in the interpreter based on the information provided by Sock* (actually by recvErrno); the function works for OpenSSL and plain POSIX style error codes. * Setting Tcl error codes for POSIX errno: Ns_PosixSetErrorCode(Tcl_Interp *interp, int errorNum) Set the Tcl error code in the interpreter based on the POSIX errorNum and return the error message. * Setting Tcl error codes for OpenSSL error values Ns_SSLSetErrorCode(Tcl_Interp *interp, unsigned long sslERRcode) Set the Tcl error code in the interpreter based on the OpenSSL error value and return the error message. * Return the last error from a socket. Ns_SockErrorCode(Tcl_Interp *interp, NS_SOCKET sock) The function Ns_SockErrorCode() returns the last error from a socket. When the provided interp is not NULL it sets the Tcl error code based on the error code. Modules: -------- 18 files changed, 1858 insertions(+), 1195 deletions(-) nsdb: * Added support for time units for nsdb LogMinDuration nsdbpg: * Force bindvars to start with alnum or underscore (otherwise, surprises with e.g. lines ending with a colon might arise). * Align spelling with LDP recommendations. * Avoid shadowing of symbol "bind". nssmtpd: * Escape all leading dots according to RFC 5321 section 4.5.2 in mail bodies * Fixed check, if relay host announced STARTTLS * Provide compatibility for versions before 4.99.19, when NS_FALL_THROUGH was not defined yet * Subsecond time resolution (Handled interface changes in post 4.99.19) * Support for 'ciphersuites' and 'protocols' parameters of 'Ns_TLS_CtxServerCreate' * I/O overhaul - Replaced select() call by poll() to ensure scalability - Pushed handling of EAGAIN etc. to the lowest level - Removed one layer of partial write handling - Replaced huge send buffer attempts, since these seem to be rejected sometimes - Improved robustness (never go into an infinite retry loop) - Handle potential "resource is not available" while sending SMTP data * Improved cleanness of compilation * Improved documentation * Align spelling with LDP * Code cleanup (C and Tcl code) * Extended regression test nsdns: * Code Cleanup - Added NS_EXPORT to Ns_ModuleInit - Improved cleanness of compilation nsudp: * Minor code cleanup (type cleanness, documentation) nsphp: * Adjust to changed parameter types in PHP 7 * Improved cleanness of compilation * Improved documentation * Brought test environment in sync with naviserver nsstats: * Improved XSS protection for potentially user supplied strings * Improved statistics: - Added calculated maximum number of reqs/pool - Added "maxconnections" to the per-pool statistics, since it determines the queue length - Improved nsproxy output in "process" page (include seconds, don't truncate everything to 0 digits after the dot) - Added write-percentage for RWLock in lock statistics - Add statistics for dropped requests (queue overruns) - Added nsv usage statistics * Added web-interface for toggling between raw and human centric values * Aligned with fractional seconds interface in NaviServer 4.99.20 * Improve help messages nsoracle: * Changed the include order to avoid bad interactions between oci.h and zlib. with the previous include order, we run with recent oracle versions (19.5) into /usr/include/zlib.h:1677:4: error: unknown type name ‘off64_t’ on 64-bit Linux systems (centos 7) * Added ORA_CFLAGS to allow developers to pass arbitrary include and library flags. Background: for some (newer?) version of the oracle client library, ORACLE_HOME (see README) seems to be gone. * Changes to build nsoracle on Windows with new Makefile.win32 file. * Improved cleanness of compilation websocket: * Major rewrite, using C-level implementing in ns_connchan This version make uses of the new C-level support for WebSockets. The new version is several orders of magnitudes faster than the previous version, it is more robust (C-level handling of partial read and write operations) and it is more complete than previous versions (handling segmented messages). * Added deprecated function ::ws::build_msg for providing better backward compatibility * Provide name for mutex lock to ease tracing of long locks * Added client-side SNI (requires NaviServer 4.99.20) * Added simple WebSocket client interface revproxy: * Improved exception handling (use try interface) * Preserve existing X-Forwarded-For IPs in cases where nsproxy is itself behind a proxy or load balancer * Switch to "ns_connchan write -buffered" to simplify leftover management for partial writes (requires NaviServer 4.99.20) * Fixed bug fix for potential stale handles * Code cleanup - Fine-tuning logging messages - Breaking overlong lines, reduce verbosity letsencrypt: * Minor changes in documentation nsldap: * Fixed result setting of "connected" subcommand * Updated modules interface to usual conventions * Added config parameter "schema" to specify other schemata * Improved documentation in README file * Reduced number of deprecated calls nsshell: * Added HTML-escaping on results of Tcl commands before these are displayed in the shell. |
From: Maksym Z. <siq...@gm...> - 2021-01-17 13:23:02
|
Hello, all config files are TCL scripts, so I have in my main "conf" folder a folder called "servers". My main nsd configuration file like this: ~~~tcl ... set servers [glob -directory [file join /opt/ns/conf/ servers] *] ... ns_section "ns/module/nsssl" { ns_param defaultserver $hostname ns_param certificate /opt/ns/modules/nsssl/main.pem ns_param address $ip_addr ns_param port $ssl_port ns_param hostname $hostname ns_param ciphers "ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!RC4" ns_param protocols "!SSLv2:!SSLv3" ns_param verify 0 ns_param extraheaders { Strict-Transport-Security "max-age=31536000; includeSubDomains" X-Frame-Options SAMEORIGIN X-Content-Type-Options nosniff } ns_param maxinput [expr {$max_file_upload_mb * 1024 * 1024}] ns_param recvwait [expr {$max_file_upload_min * 60}] } ... ns_section "ns/servers" { foreach server $servers { ns_param [file tail $server] "[file tail $server]" } } ns_section "ns/module/nsssl/servers" { foreach server $servers { ns_param [file tail $server] [file tail $server] } } ns_section "ns/module/nssock/servers" { foreach server $servers { ns_param [file tail $server] [file tail $server] } } ... ~~~ For each server I want to add, I create a config file inside servers folder, and name it as my domain or subdomain. For example: myfirstserver.com, secondserver.com ~~~tcl set server "myfirstserver.com" set db_name $server ... ns_section "ns/server/${server}/module/nsssl" { ns_param certificate ${homedir}/modules/nsssl/${server}.pem ns_param ciphersuites "TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256" } ... ~~~ Basically I'm just cloning one file and setting up domain name in the beginning of it. My certificates have the same name as domain they are used for So when Naviserver starts it will read my main config file, read config files from the "server" directory with certificates, database connection settings etc. So it's really easy to setup new virtual server: clone one of conf/server files, change name, setup server var, get and save certificate with the same name as server, reboot nsd and ready to go. Cheers On Sun, Jan 17, 2021 at 10:30 AM Jeff R <dv...@di...> wrote: > Hi all, > > Looking at the docs and recent list postings Naviserver supports virtual > hosting over ssl using SNI by defining multiple servers in the config and > specifying the certificate for each server. The downside of this is that > in order to add a new host you need to modify the config, rather than just > adding new certificates. > > I'm thinking it would allow a simpler config to support having a scheme > matching the host name to the certificate file name implicitly, rather than > explicitly. That is, instead of looking for the certificate configured > for a matching nsssl server, look for a file matching > "directory/${host}.pem" > > This could be mostly accomplished by having the config file create the > sections at runtime (i.e., at config load time) by scanning the desired > directory, but doing so would still require the server to be restarted if a > new certificate was added. > > A config supporting this might look like > > ns_section ns/module/nsssl { > ns_param port 8433 > ns_param defaultserver s1 > ns_param certificate /usr/local/ns/modules/nsssl/server.pem > ns_param sni_certificate /usr/local/ns/modules/nsssl/%host%.pem > } > with whatever placeholder syntax makes sense for "host" there. ($host > obviously can't work) > > I don't know how this approach would work for wildcard certificates - > maybe it would need to look for "domain" and then "*.domain", or if looking > for "foo.domain.xyz" if could try "domain.xzy" if it exists on the > assumption it's a wildcard? > > This is not critical for me as a letsencrypt SAN certificate will probably > work fine, but I like to keep my options open :) > > Cheers, > -J > > _______________________________________________ > naviserver-devel mailing list > nav...@li... > https://lists.sourceforge.net/lists/listinfo/naviserver-devel > |
From: Jeff R <dv...@di...> - 2021-01-17 11:30:25
|
Hi all, Looking at the docs and recent list postings Naviserver supports virtual hosting over ssl using SNI by defining multiple servers in the config and specifying the certificate for each server. The downside of this is that in order to add a new host you need to modify the config, rather than just adding new certificates. I'm thinking it would allow a simpler config to support having a scheme matching the host name to the certificate file name implicitly, rather than explicitly. That is, instead of looking for the certificate configured for a matching nsssl server, look for a file matching "directory/${host}.pem" This could be mostly accomplished by having the config file create the sections at runtime (i.e., at config load time) by scanning the desired directory, but doing so would still require the server to be restarted if a new certificate was added. A config supporting this might look like ns_section ns/module/nsssl { ns_param port 8433 ns_param defaultserver s1 ns_param certificate /usr/local/ns/modules/nsssl/server.pem ns_param sni_certificate /usr/local/ns/modules/nsssl/%host%.pem } with whatever placeholder syntax makes sense for "host" there. ($host obviously can't work) I don't know how this approach would work for wildcard certificates - maybe it would need to look for "domain" and then "*.domain", or if looking for "foo.domain.xyz" if could try "domain.xzy" if it exists on the assumption it's a wildcard? This is not critical for me as a letsencrypt SAN certificate will probably work fine, but I like to keep my options open :) Cheers, -J |
From: David O. <da...@qc...> - 2021-01-12 11:08:24
|
Thank you for a very quick turnaround Gustaf! I'll do a build at rc4 and continue testing. Regards, On Tue, 12 Jan 2021 at 10:16, Gustaf Neumann <ne...@wu...> wrote: > On 11.01.21 16:45, Gustaf Neumann wrote: > > Will go through these more carefully and produce a new rc when done. -gn > > The updated version is running since yesterday on openacs.org (4 virtual > servers [1]), i have tested also with macOS 10.14.6, Ubuntu 20.04, > Fedora Core 32, OpenBSD 6.8 (LibreSSL 3.1.1), FreeBSD 12.2. > > There is also an updated release candidate on sourceforge [2]. My plan > would be to release on Friday. > > all the best and many thanks for your help > > -g > > [1] https://openacs.org/forums/message-view?message_id=5468281 > [2] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.20/ > > > > > _______________________________________________ > naviserver-devel mailing list > nav...@li... > https://lists.sourceforge.net/lists/listinfo/naviserver-devel > -- *David Osborne | Software Engineer* Qcode Software, Castle House, Fairways Business Park, Inverness, IV2 6AA *Email:* da...@qc... | *Phone:* 01463 896 484 www.qcode.co.uk |
From: Gustaf N. <ne...@wu...> - 2021-01-12 10:15:59
|
On 11.01.21 16:45, Gustaf Neumann wrote: > Will go through these more carefully and produce a new rc when done. -gn The updated version is running since yesterday on openacs.org (4 virtual servers [1]), i have tested also with macOS 10.14.6, Ubuntu 20.04, Fedora Core 32, OpenBSD 6.8 (LibreSSL 3.1.1), FreeBSD 12.2. There is also an updated release candidate on sourceforge [2]. My plan would be to release on Friday. all the best and many thanks for your help -g [1] https://openacs.org/forums/message-view?message_id=5468281 [2] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.20/ |
From: Gustaf N. <ne...@wu...> - 2021-01-11 15:46:05
|
Dear alll, i found a couple of more places, where non-existing config sections might cause troubles. Will go through these more carefully and produce a new rc when done. -gn On 11.01.21 15:13, Gustaf Neumann wrote: > Dear David, > > I could reproduce the behavior. The problem was due to a change about > a year ago, which reduced code duplication in cases, where no section > for nsproxy was provided in the configuration file. In reported case, > it was looking for "ns/server/default/module/nsproxy". This explains > as well, while this was not noted earlier. > > The version on bitbucket has this problem now fixed. I have extended > the API slightly to reduce complexity. > > All the best and many thanks for the report. > > -gn > > > > _______________________________________________ > naviserver-devel mailing list > nav...@li... > https://lists.sourceforge.net/lists/listinfo/naviserver-devel |
From: Gustaf N. <ne...@wu...> - 2021-01-11 14:13:51
|
Dear David, I could reproduce the behavior. The problem was due to a change about a year ago, which reduced code duplication in cases, where no section for nsproxy was provided in the configuration file. In reported case, it was looking for "ns/server/default/module/nsproxy". This explains as well, while this was not noted earlier. The version on bitbucket has this problem now fixed. I have extended the API slightly to reduce complexity. All the best and many thanks for the report. -gn |
From: David O. <da...@qc...> - 2021-01-11 11:49:50
|
Hi, I've done a build of NaviServer 4.99.20 rc1 and am hitting issues with the ns_proxy command seg faulting. I have a full backtrace of the seg fault here from our development environment: https://gist.github.com/davidqc/e41f7852de831de4b4b49f5d03f165b7 I can reproduce what looks like the same segfault from a basic build (at the brand new rc2 level) also. Fairly stumped at this - any ideas what's causing this? The recreation steps are (from a fresh Debian Stretch install): $ uname -a Linux stretch 4.9.0-14-amd64 #1 SMP Debian 4.9.246-2 (2020-12-17) x86_64 GNU/Linux $ sudo apt-cache policy tcl8.6 tcl8.6: Installed: 8.6.6+dfsg-1+b1 Candidate: 8.6.6+dfsg-1+b1 Version table: *** 8.6.6+dfsg-1+b1 500 500 http://cdn-aws.deb.debian.org/debian stretch/main amd64 Packages 100 /var/lib/dpkg/status $ sudo apt-cache policy openssl openssl: Installed: 1.1.0l-1~deb9u2 Candidate: 1.1.0l-1~deb9u2 Version table: *** 1.1.0l-1~deb9u2 500 500 http://security.debian.org/debian-security stretch/updates/main amd64 Packages 100 /var/lib/dpkg/status 1.1.0l-1~deb9u1 500 500 http://cdn-aws.deb.debian.org/debian stretch/main amd64 Packages # apt-get install build-essential git automake tcl8.6-dev libssl-dev # git clone https://bitbucket.org/naviserver/naviserver.git # cd naviserver # ./autogen.sh --disable-ipv6 --with-tcl=/usr/lib/tcl8.6 --enable-rpath --enable-threads --enable-symbols # make # sudo make install # sudo useradd nsadmin # sudo chown -R nsadmin:nsadmin /usr/local/ns/logs Add nsproxy to /usr/local/ns/conf/nsd-config.tcl: ns_section ns/server/default/modules { ns_param nscp nscp ns_param nslog nslog ns_param nscgi nscgi ns_param nsproxy nsproxy.so } Then.. # sudo /usr/local/ns/bin/nsd -u nsadmin -t /usr/local/ns/conf/nsd-config.tcl -c % ns_proxy get test_pool Segmentation fault -- *David Osborne | Software Engineer* Qcode Software, Castle House, Fairways Business Park, Inverness, IV2 6AA *Email:* da...@qc... | *Phone:* 01463 896 484 www.qcode.co.uk |
From: Gustaf N. <ne...@wu...> - 2021-01-10 18:35:24
|
On 10.01.21 15:58, oleg wrote: > > We have tested Naviserver RC. > No problem found on Ubuntu 20.04 and Windows10/MSVC2019/32bit. > Great! > > Have an issue for Windows10/MSVC2019/64bit build. > > lassign [ns_sockopen -nonblock httpbin.org 80] rfd wfd > > ns_sockselect -timeout 1000 {} $wfd {} > This is an old and known issue. "ns_sockopen" is a somewhat deprecated call, which was used mostly for the regression test (most NaviServer applications will work happily without it), so there was seemingly never a high urgency to fix this. I am surprised that this works with MSVC2019/32bit - but maybe this was the "lucky" part. I would not see this as showstopper for the upcoming release. > Crosscompiled (gcc/mingw) for Windows 64bit Naviserver works perfectly > but there are difficulties in configure/make/install: Cool! I have added the mingw changes to the code base and added your instructions to the README file ... and placed rc3 to sourceforge https://sourceforge.net/projects/naviserver/files/naviserver/4.99.20/ If nothing else shows up, i'll finish the release in the next few days. Oleg, many thanks for testing and the good suggestions. -gn |
From: oleg <oo...@ua...> - 2021-01-10 16:05:56
|
Hello. We have tested Naviserver RC. No problem found on Ubuntu 20.04 and Windows10/MSVC2019/32bit. Have an issue for Windows10/MSVC2019/64bit build. Deprecated ns_httpget/post does not work with the error "select failed: no such file or directory". Code that causes an error: lassign [ns_sockopen -nonblock httpbin.org 80] rfd wfd ns_sockselect -timeout 1000 {} $wfd {} It looks like the problem is using DuplicateHandle on the socket handle and passing duplicated handle to the select call. >From DuplicateHandle docs: "Sockets. No error is returned, but the duplicate handle may not be recognized by Winsock at the target process." >From googling: should work in most cases. I was just unlucky. Crosscompiled (gcc/mingw) for Windows 64bit Naviserver works perfectly but there are difficulties in configure/make/install: We had to make changes to the code (patch attached): nsthread.h - some variables are not defined, connchan.c - definition of beXXtoh/htobeXX, as for MSVC, dstring.c - realloc dstrings as for older MSVC, exec.c - lowercased header. Configure magic does not detect some settings for MINGW, they need to be specified explicitly ./configure --host=x86_64-w64-mingw32 --enable-64bit \ --prefix=<path> --with-zlib=<path> --with-openssl=<path> --with-tcl=<path>/lib CFLAGS="-DHAVE_INET_PTON -DHAVE_INET_NTON -DHAVE_GETADDRINFO -D_WIN32_WINNT=0x600" \ LDFLAGS="-static-libgcc" \ make LIBLIBS="-Wl,-Bstatic -lpthread -Wl,-Bdynamic" Installation script does not expect .exe extension, workaround cp nsthread/nsthreadtest.exe nsthread/nsthreadtest cp nsd/nsd.exe nsd/nsd cp nsproxy/nsproxy.exe nsproxy/nsproxy make install Regards, Oleg. |
From: Gustaf N. <ne...@wu...> - 2021-01-10 14:23:59
|
> :) Sorry false alert, my bad. Due to lack of coffee and some good > sleep, I was testing some procs in "nsshell", and of course browser > rendering escaped characters back to normal, and my zombie brain > couldn't process why. It works fine, at least it has a regression test > now ;) Many thanks for the quick feedback! -g |
From: Maksym Z. <siq...@gm...> - 2021-01-10 10:41:13
|
:) Sorry false alert, my bad. Due to lack of coffee and some good sleep, I was testing some procs in "nsshell", and of course browser rendering escaped characters back to normal, and my zombie brain couldn't process why. It works fine, at least it has a regression test now ;) On Sat, Jan 9, 2021 at 5:44 PM Gustaf Neumann <ne...@wu...> wrote: > > On 09.01.21 19:01, Maksym Zinchenko wrote: > > Hello everyone, may someone confirm that ns_quotehtml works as it should, > because on my system its not: > > Command: ns_quotehtml {<span class="foo">} > Result: <span class="foo"> > > What is your system? It works for me. What is your OS? > Interestingly, ns_quotehtml was not covered by the regression tests. > > -g > > > % ./nsd/nsd -c > [09/Jan/2021:19:36:16][93167.10d7eb5c0][-main:conf-] Notice: OpenSSL 1.1.1i 8 Dec 2020 initialized > [09/Jan/2021:19:36:16][93167.10d7eb5c0][-main:conf-] Notice: nsmain: NaviServer/4.99.20rc2 (c407fafe7b3f+) starting > ... > % > % ns_quotehtml {<span class="foo">} > <span class="foo"> > > > _______________________________________________ > naviserver-devel mailing list > nav...@li... > https://lists.sourceforge.net/lists/listinfo/naviserver-devel > |
From: Gustaf N. <ne...@wu...> - 2021-01-09 18:44:15
|
On 09.01.21 19:01, Maksym Zinchenko wrote: > Hello everyone, may someone confirm that ns_quotehtml works as it > should, because on my system its not: > > Command: ns_quotehtml {<span class="foo">} > Result: <span class="foo"> What is your system? It works for me. What is your OS? Interestingly, ns_quotehtml was not covered by the regression tests. -g % ./nsd/nsd -c [09/Jan/2021:19:36:16][93167.10d7eb5c0][-main:conf-] Notice: OpenSSL 1.1.1i 8 Dec 2020 initialized [09/Jan/2021:19:36:16][93167.10d7eb5c0][-main:conf-] Notice: nsmain: NaviServer/4.99.20rc2 (c407fafe7b3f+) starting ... % % ns_quotehtml {<span class="foo">} <span class="foo"> |
From: Maksym Z. <siq...@gm...> - 2021-01-09 18:02:06
|
Hello everyone, may someone confirm that ns_quotehtml works as it should, because on my system its not: Command: ns_quotehtml {<span class="foo">} Result: <span class="foo"> Thank you in advance |