naviserver-devel — Developer discussion

Re: [naviserver-devel] Does Naviserver support OCSP stapling?

[Gustaf N. <ne...@wu...>]

Brian,


Current versions of NaviServer have no such option.
Are you asking out of curiosity or do you have a strong need for this?


-g


On 18.02.20 13:25, Brian Fenton wrote:
> Hi all
>
> does Naviserver support OCSP stapling, and if so, how to configure it?
> https://en.wikipedia.org/wiki/OCSP_stapling
>
> thanks
> Brian

[naviserver-devel] Does Naviserver support OCSP stapling?

[Brian F. <Bri...@qu...>]

Hi all

does Naviserver support OCSP stapling, and if so, how to configure it?
https://en.wikipedia.org/wiki/OCSP_stapling

thanks
Brian

Re: [naviserver-devel] Mercurial at Bitbucket is ending. What next?

[Gustaf N. <ne...@wu...>]

In case you are wondering about the changed modification
dates of the NaviServer repositories at bitbucket: i've updated
the "description" fields of the git repositories, since these are
used for search on bitbucket. These had to be updated manually...

-g

Re: [naviserver-devel] Mercurial at Bitbucket is ending. What next?

[Zoran V. <zv...@ar...>]

On Sun, 9 Feb 2020 10:51:55 +0100
Gustaf Neumann <ne...@wu...> wrote:

> all NaviServer repositories are now converted on bitbucket to git.

You are a gentleman and a scholar!

Re: [naviserver-devel] Mercurial at Bitbucket is ending. What next?

[Gustaf N. <ne...@wu...>]

Dear all,

all NaviServer repositories are now converted on bitbucket to git.

The "old" mercurial repositories are still on bitbucket, but named
with the suffix "-hg". For example the mercurial repository of
main naviserver is called "naviserver-hg", while "naviserver"
is the git repository.

In case, you still want to keep your local repository in mercurial
as long allowed by bitbucket (until May 31, 2020), just add
the suffix "-hg" to the .hg/hgrc file:

====================================================================
--- .hg/hgrc-orig	2020-02-09 10:10:32.000000000 +0100
+++ .hg/hgrc	2020-02-09 10:05:58.000000000 +0100
@@ -1,6 +1,6 @@
  # example repository config (see "hg help config" for more info)
  [paths]
-default = https://bitbucket.org/naviserver/naviserver
+default = https://bitbucket.org/naviserver/naviserver-hg
  
  # path aliases to other clones of this repo in URLs or filesystem paths
  # (see "hg help config.paths" for more info)
====================================================================

Otherwise, just delete your mercurial repositories and make
fresh git clones from bitbucket.

Please commit from now on to the git repositories.

All the best
-gn


On 07.02.20 17:43, Gustaf Neumann wrote:
> Dear all,
>
> So far, i've received only positive feedback!
>
> I'll start with the migration on the weekend, but try to make
> the migration phase as short as possible to reduce documentation
> and communication work for differences in the migration phase.
>
> all the best
>
> -gn
>
>
> On 10.01.20 09:55, Gustaf Neumann wrote:
>> Dear all,
>>
>> Bitbucket has defined the following dates for shutting down their
>> mercurial support, which are in the not too distant future:
>>
>>  - February 1, 2020: users will no longer be able to create new 
>> Mercurial repositories
>>  - June 1, 2020: users will not be able to use Mercurial features in
>>     Bitbucket or via its API and all Mercurial repositories will be 
>> removed.
>>  - All current Mercurial functionality in Bitbucket will be available 
>> through May 31, 2020.
>>
>> These dates are approaching quickly. I fully agree with Zoran about git,
>> although there are as well merits in other systems. Keep in mind that we
>> have more than 70 naviserver+modules repositories, and we have to
>> carry the accounts and permissions to some target system.
>>
>> From the effort point of view, a migration from bitbucket+hg to 
>> bitbucket+git
>> would be my preferred way, in case, i am doing the conversions. So 
>> please
>> speak up now, if someone objects.
>>
>> As a proof of concept, i've transformed two repositories to
>> bitbucket+git by adding a "-git" suffix to the repository name:
>>
>>    https://bitbucket.org/naviserver/naviserver-git
>>
>> I would be glad, if some of our developers with git experience
>> can checkout  this git repo of naviserver to see, whether this
>> looks ok to you. In the future, we should change from a tagging
>> model to a branching model, but currently this is just a 1:1 conversion.
>>
>> Once we are convinced this is fine, then we can do the same on all 70+
>> other repos, and make some tests to check, if everything looks fine.
>> Then in the last step do a final renaming game
>> to swap to the original repository names:
>>
>>   1) rename hg-repo from NAME to NAME-hg
>>   2) rename git-repo from NAME-git to NAME
>>   3) set hg repo to READ-only
>>
>> ... and certainly change the documentation/READMEs
>> from hg to git.
>>
>> ... please do not commit to the git repo, continue to
>> commit to hg for the time being. Part of the conversion has
>> to be done manually (creating the repos, renaming the repos,
>> transferring individual permissions), part can be done scripted.
>>
>> all the best
>>
>> -gn
>>
>> On 29.08.19 12:06, Zoran Vasiljevic wrote:
>>> On Thu, 29 Aug 2019 09:13:17 +0000 (UTC)
>>> Roderick <hr...@gm...> wrote:
>>>
>>>> After reading something about mercurial only for cloning Naviservers
>>>> Repo, I read this:
>>>>
>>>> https://bitbucket.org/blog/sunsetting-mercurial-support-in-bitbucket
>>>>
>>>> It would be nice to have NaviServer as fossil repo. :)
>>> I believe most logical choice is git. I'm not fan of any of
>>> the systems, to be honest. Just, the chance that we will have
>>> to switch again (cvs -> mercurial -> ?) is less if the ? = git.

Re: [naviserver-devel] Mercurial at Bitbucket is ending. What next?

[Gustaf N. <ne...@wu...>]

Dear all,

So far, i've received only positive feedback!

I'll start with the migration on the weekend, but try to make
the migration phase as short as possible to reduce documentation
and communication work for differences in the migration phase.

all the best

-gn


On 10.01.20 09:55, Gustaf Neumann wrote:
> Dear all,
>
> Bitbucket has defined the following dates for shutting down their
> mercurial support, which are in the not too distant future:
>
>  - February 1, 2020: users will no longer be able to create new 
> Mercurial repositories
>  - June 1, 2020: users will not be able to use Mercurial features in
>     Bitbucket or via its API and all Mercurial repositories will be 
> removed.
>  - All current Mercurial functionality in Bitbucket will be available 
> through May 31, 2020.
>
> These dates are approaching quickly. I fully agree with Zoran about git,
> although there are as well merits in other systems. Keep in mind that we
> have more than 70 naviserver+modules repositories, and we have to
> carry the accounts and permissions to some target system.
>
> From the effort point of view, a migration from bitbucket+hg to 
> bitbucket+git
> would be my preferred way, in case, i am doing the conversions. So please
> speak up now, if someone objects.
>
> As a proof of concept, i've transformed two repositories to
> bitbucket+git by adding a "-git" suffix to the repository name:
>
>    https://bitbucket.org/naviserver/naviserver-git
>
> I would be glad, if some of our developers with git experience
> can checkout  this git repo of naviserver to see, whether this
> looks ok to you. In the future, we should change from a tagging
> model to a branching model, but currently this is just a 1:1 conversion.
>
> Once we are convinced this is fine, then we can do the same on all 70+
> other repos, and make some tests to check, if everything looks fine.
> Then in the last step do a final renaming game
> to swap to the original repository names:
>
>   1) rename hg-repo from NAME to NAME-hg
>   2) rename git-repo from NAME-git to NAME
>   3) set hg repo to READ-only
>
> ... and certainly change the documentation/READMEs
> from hg to git.
>
> ... please do not commit to the git repo, continue to
> commit to hg for the time being. Part of the conversion has
> to be done manually (creating the repos, renaming the repos,
> transferring individual permissions), part can be done scripted.
>
> all the best
>
> -gn
>
> On 29.08.19 12:06, Zoran Vasiljevic wrote:
>> On Thu, 29 Aug 2019 09:13:17 +0000 (UTC)
>> Roderick <hr...@gm...> wrote:
>>
>>> After reading something about mercurial only for cloning Naviservers
>>> Repo, I read this:
>>>
>>> https://bitbucket.org/blog/sunsetting-mercurial-support-in-bitbucket
>>>
>>> It would be nice to have NaviServer as fossil repo. :)
>> I believe most logical choice is git. I'm not fan of any of
>> the systems, to be honest. Just, the chance that we will have
>> to switch again (cvs -> mercurial -> ?) is less if the ? = git.
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] Mercurial at Bitbucket is ending. What next?

[Wolfgang W. <wol...@di...>]

Dear Gustaf!

I just cloned the new git repository and compiled it without any 
problems. The history dates back to the first revision in 2006, the tags 
are up-to-date.

I haven't found any problems.

Regards,

Wolfgang

Am 10.01.20 um 09:55 schrieb Gustaf Neumann:
> Dear all,
>
> Bitbucket has defined the following dates for shutting down their
> mercurial support, which are in the not too distant future:
>
>  - February 1, 2020: users will no longer be able to create new 
> Mercurial repositories
>  - June 1, 2020: users will not be able to use Mercurial features in
>     Bitbucket or via its API and all Mercurial repositories will be 
> removed.
>  - All current Mercurial functionality in Bitbucket will be available 
> through May 31, 2020.
>
> These dates are approaching quickly. I fully agree with Zoran about git,
> although there are as well merits in other systems. Keep in mind that we
> have more than 70 naviserver+modules repositories, and we have to
> carry the accounts and permissions to some target system.
>
> From the effort point of view, a migration from bitbucket+hg to 
> bitbucket+git
> would be my preferred way, in case, i am doing the conversions. So please
> speak up now, if someone objects.
>
> As a proof of concept, i've transformed two repositories to
> bitbucket+git by adding a "-git" suffix to the repository name:
>
>    https://bitbucket.org/naviserver/naviserver-git
>
> I would be glad, if some of our developers with git experience
> can checkout  this git repo of naviserver to see, whether this
> looks ok to you. In the future, we should change from a tagging
> model to a branching model, but currently this is just a 1:1 conversion.
>
> Once we are convinced this is fine, then we can do the same on all 70+
> other repos, and make some tests to check, if everything looks fine.
> Then in the last step do a final renaming game
> to swap to the original repository names:
>
>   1) rename hg-repo from NAME to NAME-hg
>   2) rename git-repo from NAME-git to NAME
>   3) set hg repo to READ-only
>
> ... and certainly change the documentation/READMEs
> from hg to git.
>
> ... please do not commit to the git repo, continue to
> commit to hg for the time being. Part of the conversion has
> to be done manually (creating the repos, renaming the repos,
> transferring individual permissions), part can be done scripted.
>
> all the best
>
> -gn
>
> On 29.08.19 12:06, Zoran Vasiljevic wrote:
>> On Thu, 29 Aug 2019 09:13:17 +0000 (UTC)
>> Roderick <hr...@gm...> wrote:
>>
>>> After reading something about mercurial only for cloning Naviservers
>>> Repo, I read this:
>>>
>>> https://bitbucket.org/blog/sunsetting-mercurial-support-in-bitbucket
>>>
>>> It would be nice to have NaviServer as fossil repo. :)
>> I believe most logical choice is git. I'm not fan of any of
>> the systems, to be honest. Just, the chance that we will have
>> to switch again (cvs -> mercurial -> ?) is less if the ? = git.
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel


-- 

*Wolfgang Winkler*
Geschäftsführung
wol...@di...
mobil +43.699.19971172

dc:*büro*
digital concepts Novak Winkler OG
Software & Design
Landstraße 68, 5. Stock, 4020 Linz
www.digital-concepts.com <http://www.digital-concepts.com>
tel +43.732.997117.72
tel +43.699.1997117.2

Firmenbuchnummer: 192003h
Firmenbuchgericht: Landesgericht Linz

Re: [naviserver-devel] Mercurial at Bitbucket is ending. What next?

[Gustaf N. <ne...@wu...>]

Dear all,

Bitbucket has defined the following dates for shutting down their
mercurial support, which are in the not too distant future:

  - February 1, 2020: users will no longer be able to create new 
Mercurial repositories
  - June 1, 2020: users will not be able to use Mercurial features in
     Bitbucket or via its API and all Mercurial repositories will be 
removed.
  - All current Mercurial functionality in Bitbucket will be available 
through May 31, 2020.

These dates are approaching quickly. I fully agree with Zoran about git,
although there are as well merits in other systems. Keep in mind that we
have more than 70 naviserver+modules repositories, and we have to
carry the accounts and permissions to some target system.

 From the effort point of view, a migration from bitbucket+hg to 
bitbucket+git
would be my preferred way, in case, i am doing the conversions. So please
speak up now, if someone objects.

As a proof of concept, i've transformed two repositories to
bitbucket+git by adding a "-git" suffix to the repository name:

    https://bitbucket.org/naviserver/naviserver-git

I would be glad, if some of our developers with git experience
can checkout  this git repo of naviserver to see, whether this
looks ok to you. In the future, we should change from a tagging
model to a branching model, but currently this is just a 1:1 conversion.

Once we are convinced this is fine, then we can do the same on all 70+
other repos, and make some tests to check, if everything looks fine.
Then in the last step do a final renaming game
to swap to the original repository names:

   1) rename hg-repo from NAME to NAME-hg
   2) rename git-repo from NAME-git to NAME
   3) set hg repo to READ-only

... and certainly change the documentation/READMEs
from hg to git.

... please do not commit to the git repo, continue to
commit to hg for the time being. Part of the conversion has
to be done manually (creating the repos, renaming the repos,
transferring individual permissions), part can be done scripted.

all the best

-gn

On 29.08.19 12:06, Zoran Vasiljevic wrote:
> On Thu, 29 Aug 2019 09:13:17 +0000 (UTC)
> Roderick <hr...@gm...> wrote:
>
>> After reading something about mercurial only for cloning Naviservers
>> Repo, I read this:
>>
>> https://bitbucket.org/blog/sunsetting-mercurial-support-in-bitbucket
>>
>> It would be nice to have NaviServer as fossil repo. :)
> I believe most logical choice is git. I'm not fan of any of
> the systems, to be honest. Just, the chance that we will have
> to switch again (cvs -> mercurial -> ?) is less if the ? = git.

Re: [naviserver-devel] Naviserver 4.99.19

[Gustaf N. <ne...@wu...>]

When one starts naviserver in watchdog mode [1], this is normal.
all the best

-g

[1] https://naviserver.sourceforge.io/n/manual/files/admin-maintenance.html

On 08.01.20 16:05, Maksym Zinchenko wrote:
>
> Hello everyone, I have a question about latest "Naviserver".
>
> After updating to 4.99.19, I have 2 process running, one from root and 
> another my "Naviserver" user nsadmin for example:
>
> ps -aux | grepnsd
>
> root14410.00.0142722960 ?Ss13:480:00 /opt/ns/bin/nsd -w -u nsadmin -t 
> /opt/ns/conf/dz_nsd.tcl -b 0.0.0.0:80 <http://0.0.0.0:80>,0.0.0.0:443 
> <http://0.0.0.0:443>
>
> nsadmin14420.00.1 565044 65456 ?Sl13:480:04 /opt/ns/bin/nsd -w -u 
> nsadmin -t /opt/ns/conf/dz_nsd.tcl -b 0.0.0.0:80 
> <http://0.0.0.0:80>,0.0.0.0:443 <http://0.0.0.0:443>
>
> root26070.00.06640980 pts/1S+15:530:00 grep --color=auto nsd
>
> Is it normal behavior? Or I did something wrong?
>
> One more time thank you for your great work.
>
>

Re: [naviserver-devel] Naviserver 4.99.19

[Iuri S. <iu...@iu...>]

One can always kill a process and leave the desired one.
Then, it must confirm that nsd isn't set to start on boot, as user nsadmin, at /etc/init.d/, neither systemctl is enabled.
systemctl oacs-5.9.1.service disable
systemctl daemon-reload
Visit Automate NaviServer Startup at:
http://www.project-open.com/en/install-debian-stretch <http://www.project-open.com/en/install-debian-stretch>
Best wishes,
I
 

> On Jan 8, 2020, at 12:05, Maksym Zinchenko <siq...@gm...> wrote:
> 
> Hello everyone, I have a question about latest "Naviserver".
> 
> After updating to 4.99.19, I have 2 process running, one from root and another my "Naviserver" user nsadmin for example:
> 
>  ps -aux | grep  nsd
> 
> root      1441  0.0  0.0  14272  2960 ?        Ss   13:48   0:00 /opt/ns/bin/nsd -w -u nsadmin -t /opt/ns/conf/dz_nsd.tcl -b 0.0.0.0:80 <http://0.0.0.0/>,0.0.0.0:443 <http://0.0.0.0:443/>
> nsadmin   1442  0.0  0.1 565044 65456 ?        Sl   13:48   0:04 /opt/ns/bin/nsd -w -u nsadmin -t /opt/ns/conf/dz_nsd.tcl -b 0.0.0.0:80 <http://0.0.0.0/>,0.0.0.0:443 <http://0.0.0.0:443/>
> root      2607  0.0  0.0   6640   980 pts/1    S+   15:53   0:00 grep --color=auto nsd
> 
>  
> Is it normal behavior? Or I did something wrong?
> 
> One more time thank you for your great work.
> 
> 
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

[naviserver-devel] Naviserver 4.99.19

[Maksym Z. <siq...@gm...>]

Hello everyone, I have a question about latest "Naviserver".

After updating to 4.99.19, I have 2 process running, one from root and
another my "Naviserver" user nsadmin for example:

 ps -aux | grep  nsd

root      1441  0.0  0.0  14272  2960 ?        Ss   13:48   0:00
/opt/ns/bin/nsd -w -u nsadmin -t /opt/ns/conf/dz_nsd.tcl -b 0.0.0.0:80,
0.0.0.0:443

nsadmin   1442  0.0  0.1 565044 65456 ?        Sl   13:48   0:04
/opt/ns/bin/nsd -w -u nsadmin -t /opt/ns/conf/dz_nsd.tcl -b 0.0.0.0:80,
0.0.0.0:443

root      2607  0.0  0.0   6640   980 pts/1    S+   15:53   0:00 grep
--color=auto nsd



Is it normal behavior? Or I did something wrong?

One more time thank you for your great work.

[naviserver-devel] NaviServer 4.99.19 available

[Gustaf N. <ne...@wu...>]

Dear all,

on sourceforge is now the release of NaviServer 4.99.19 [1].
This release contains the fixes and enhancements of the work
of the last year. Many thanks to all contributors!

The upgrade to 4.99.19 is recommended.
Below is a summary of changes.

all the best, and a happy new year!

-gustaf
[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.19/

=======================================
NaviServer 4.99.19, released 2020-01-06
=======================================

  247 files changed, 19527 insertions(+), 8275 deletions(-)

New Features:
-------------

  - Bandwidth management: limit download rate to a maximum (specified
    in kilobytes per second)

    On servers with bad internet connections, it is possible that a few
    eager downloaders can use up all available bandwidth.  NaviServer
    allows now admins and developers to specify the maximum outgoing
    bandwidth per connection or the total rate per connection pool. The
    rate limit can be specified per connection (highest weight) or per
    connection thread pool (second highest weight) or per network
    driver. The rate limitation is only active, when writer threads are
    used. Per default, the rate limitation is turned off (value 0).

    By using this option together with the context filters, one can now
    define e.g. a pool for bots and limit the bandwidth for all bots.

    The query the per-connection limit, ns_conn has a new option:
       ns_conn ratelimit ?limit?
    Introspecton "ns_writer list" returns the current
    transfer rates per writer job.

    Configuration option for connection pools:
       ns_param connectionratelimit ...
       ns_param poolratelimit ...
    Configuration option for network drivers:
       ns_param writerratelimit ...

    Commands for changing the rate limits dynamically:
        ns_server ?-server s? ?-pool p? connectionratelimit ?value?
        ns_server ?-server s? ?-pool p? poolratelimit ?value?


  - Context filter for urlspace:

    NaviServer's urlspace trie data structure used for various purposes
    such as registering methods/URL pairs for procs or for mapping of
    requests to connection thread pools. The classical trie mapping for
    e.g. "GET /foo/bar/*.html" is based on method + path + string match
    pattern, where the last part "*.html" is used as a "filter".

    The new version of NaviServer allows in addition to the string
    based glob pattern so-called "context filter", which includes the
    IP-address or header fields in the matching process. The context
    filters are applied after the classical urlspace processing, such
    that the behavior is fully backwards compatible. The context
    filters are kept in a stable order (IP before header-based filters,
    specific before less specific) such that the behavior is not order
    dependent.

    The new feature can be used map e.g. bots or certain IP ranges to
    the specified connection pools. When connection pools named "bots"
    and "local" are defined, one can use the following mapping commands
    to map incoming requests to the specified pools. Examples for
    adding urlspace mappings at runtime:

	 ns_server -pool bots map "GET /* {user-agent *bot*}"
	 ns_server -pool bots map "GET /* {user-agent *crawl*}"
	 ns_server -pool bots map "GET /* {user-agent *baidu*}"
	 ns_server -pool bots map "GET /* {X-NS-ip 2a03:2880::/29}"

	 ns_server -pool local map "GET /* {X-NS-ip 127.0.0.1}"
	 ns_server -pool local map "GET /* {X-NS-ip 137.208.1.0/16}"

    Note that arbitrary header fields can be used for the mapping,
    containing potentially some glob style match characters. The
    special header field "X-NS-ip" denotes IP based filters, which can
    be provided fully qualified or in CIDR notation (denoting
    significant bits) for IPv4 and IPv6.


  - Allow a single driver to listen on multiple IP addresses:

    This feature greatly simplifies setups, where a single server is
    listening on multiple IP addresses (e.g. IPv4 and
    IPv6). Previously, it was necessary to define separate drivers for
    these, which need different names but which are often
    configured identically.  Furthermore, when the address is omitted,
    the server performs a lookup from the hostname to determine the IP
    address. Previously, it used the first one, now it can work with
    all returned addresses.

    To use this feature, simply specify the configure values of
    "address" as a Tcl list containing multiple IP addresses. The
    change is fully backwards compatible, old configuration files will
    continue to work.


  - Added support for sending of multiple file chunks in a
    single connection:

       ns_writer sendfiles /list of filespecs/

    Every "filespec" is a dict which must contain a "filename" element
    and can contain additionally an "-offset" and/or a "-size" element.

       ns_writer submitfiles {{filename /tmp/f1} {filename /tmp/f2 -offset 10}}

    This function is e.g. useful for video streaming applications,
    where multiple video file chunks should be transferred (sometimes)
    concatenated in a single chunk. Therefore, this function avoids the
    necessity to concatenate these files in advance (causing increased
    latency and disk usage).


  - Added per-server "extraheaders". These extra headers fields are
    merged with the following precedence (the earlier ones have more
    weight)
      1) application specific headers
      2) per-server extra headers
      3) per-driver extra headers

    This change makes it possible to provide in the config files
    defaults, which can be overwritten by certain pages in the
    applications. Furthermore, one can e.g. common header fields to all
    requests of a virtual server (e.g.  Strict-Transport-Security, ...)


  - Logging improvements:

    * Support separate access logs for requests sent via
      different drivers.

      The nslog module accepts now an additional parameter named
      "driver".  This option can be used to produce different access
      logs for requests submitted via different drivers to sort out
      e.g. local server talk.  Per default, every request is
      logged. When a value is provided for "driver", only
      requests are logged in this log file when these come from a
      driver matching the provided glob pattern.

    * log-file sanitizer:
      When data entered into a log file contains special characters
      (e.g.  end line characters), these could alter the appearance of
      content within the log file. Single entries may appear as
      multiple entries.  Attackers may leverage log forging to insert
      fake entries to that obfuscate malicious acts, or these can
      confuse log-file analyzers to avoid analysis.

      The new global parameter "sanitizelogfiles" can control this
      behavior (values 0: none, 1: full, 2: human-friendly; default:
      2);

    * Additional debugging flag "Debug(access)": When activated, entries
      of the access log are mirrored in the system log
      (error.log). This option eases tracking the end of requests in
      the system log.

    * Additional debugging flag "Debug(writer)" for debugging of writer.

    * Provided access to request AND reply header fields via
      "extendedheaders" specification (in "ns_accesslog" and "nslog"
      configuration section):

      It is now optionally possible to specify the header fields in the
      provided "extendedheaders" list with a prefix "request:" or
      "response:" to denote request and reply header fields. In case,
      no tag is provided, field names are interpreted as request header
      fields (like before). The change is fully backwards compatible,


   - ns_set:

     * Added sucommand "ns_set imerge", a is the case insignificant
       version of "ns_set merge". This option is useful especially for
       working with header fields.

     * Added subcommand "ns_set iupdate": this command is the case
       insenstive counterpart of "ns_set update".

     * Fixed "nsv_set a b" when neither a key nor the array exists.

  - ns_http:
    * Revamp of ns_http and related supportive code.
    * Added chunked-encoding parser
    * Tcl channel support for body (PUT, POST) and for result.
      This makes it possible to use e.g. reflected Tcl channels
      in "ns_http".
      New Options for "ns_http run"
        - ?-body_size size?
        - ?-body_chan chan?
        - ?-outputfile fn?
        - ?-outputchan chan?
    * "ns_http run" is now the preferred interface
      (rather than "queue" + "wait")
    * Deprecated output variables of "ns_http wait" (since everything
      is included in the resulting dict)


  - New commands:

    * ns_asynclogfile:
      Facility for thread-safe writing async
      log files of various kinds using the AsyncWriterThread.
      Usage example:
	# ... at startup
	set fd [ns_asynclogfile open /tmp/test.log]
	# ... during run
	ns_asynclogfile write $fd hello\n
	# ... at shutdown
	ns_asynclogfile close $fd

    * ns_parsefieldvalue:

      This function parses the provided field value (from an HTTP
      request or reply header field) into its parts and returns these
      in the form of a list of Tcl dicts or a single Tcl dict depending
      on parameters. The syntax of the contents of these header fields
      is specified in RFC 7230 section 3.2.6.

  - New features for existing commands/subcommands

     * Binary reform: Some NaviServer commands accepted previously
       implicitly binary or non-binary input, some commands used
       the flag "-binary" to denote the differences (e.g. ns_return).
       For a more uniform and expectable behavior binary input the
       explicit option "-binary" flag to the following commands:

	ns_base64decode
	ns_base64encode
	ns_base64urldecode
	ns_base64urlencode
	ns_md5
	ns_sha1
	ns_uudecode
	ns_uuencode
	ns_crypto::aead::decrypt string (for "-key", "-aad", "-iv", "input")
	ns_crypto::aead::encrypt string (for "-key", "-aad", "-iv", "input")
	ns_crypto::eckey import ("-string")
	ns_crypto::eckey sharedsecret ("pubkey")
	ns_crypto::hmac add (for "message")
	ns_crypto::hmac new (for "key")
	ns_crypto::hmac string (for "key", "message")
	ns_crypto::md add (for "message")
	ns_crypto::md hkdf (for "-salt" "-secret" "-info")
	ns_crypto::md string (for "message")
	ns_crypto::md vapidsign (for "message")
	ns_crypto::scrypt (for "-salt" "-secret")

     Note that this option is not fully backward compatible. Early
     versions of "ns_sha1" were expecting always non-binary input, some
     new version assumed binary input. The new version is now more
     compatible with AOLserver and older NaviServer versions.

     * "ns_server threads" reports now how often a thread of this pool
       was started.  This statistic can help to identify
       installations, where threads are started and stopped in a too
       eager fashion.
     * "ns_server.... stats": added "sendbodysize" and "replybodysize"
       to the dict of the per-pool statistics.
     * "ns_server ... connectionratelimit ?value?": Query or set
        the default per-connection rate limit.
     * "ns_server ... poolratelimit ?value?": Query or set
        the pool connection rate limit.
     * "ns_writer list" returns now the current transferrates per
       writer job.

     * "ns_writer submitfile -offset X -size Y" and "ns_writer size X"
       accept now memory units, the option ?-driver driver? was added
       to the "ns_writer" subcommands "size", and "streaming"
       (defaulting to the current driver).

       One can now write
	   ns_writer size 150KB
        instead of
	   ns_writer size nssock [expr 150*1024]

Bug Fixes:
----------

   - Blueprint serializer: moved serialized objects in the blueprint
     after the namespace imports and ensemble recreators since
     constructors can call functions depending on it.

   - Include *xml* and *json* in non-binary MIME types. Note that
     there is no exact definition, what MIME types are exactly.

   - nslog:
     * Abort in error situation with an error message instead of crashing
     * Added server name in log entries to make it easier to distinguish
       messages per-server during bootup

   - Range requests:
     * Fixed potential problem with too many non-contiguous byte ranges
     * Don't silently ignore invalid syntax of range requests

   - Improved handling of already closed connections (data delivery was
     already delegated to writer threads, but driver still wants to
     return data directly; this could happen in error situations)

   - connchan:
     * Fixed potential invalid reads in (error) cases, where
       LogConnchanDebug is enabled and the callback is deleted during a
       Tcl_Eval().
     * Fixed potential problem, where call to Tcl_Eval() might clean
       structures used in the callback handler

   - Made sure, memory allocated by Tcl is returned by Tcl (important,
     when NaviServer is compiled with -DSYSTEM_MALLOC)

   - Fixed Tcl argument parsing of "ns_roll", "ns_fmttime",
     "nsv_bucket" and "ns_critsec eval".

   - Fixed bug in "nsv_set -default" overwriting pre-existing values

   - Fixed potential crashes in (desperate) error situations, where the
     connection was already closed.

   - Fixed leaking tmp file, when writerstraming is activated and
     streaming HTML output is used. This bug could appear, when
     "writerstreaming" is turned on in the config file (default off).
     See also: https://sourceforge.net/p/naviserver/mailman/naviserver-
	devel/thread/dd508519-1262-bd91-288a-fa30a7634224%40digital-
	concepts.com/#msg36794472

   - Made table used for enumeration values for Ns_ObjvIndex static.
     Background: This is a fix for a tricky Tcl_Obj sharing bug
     triggered by the usage of C-level IndexObjs based on volatile
     tables. Consider the following example:

        proc foo {x} {
	  return [bar -value x ...]
        }

     The Tcl_Obj "x" is shared as name of argument and as a value of
     the non-positional parameter "-value". When bar
     (e.g. C-implemented) uses Tcl_GetIndexFromObj*() to lookup "x" in
     a table of options, which is volatile, the involved Tcl_Obj will
     be converted to an indexObj.

     On a call with wrong number of arguments( e.g."foo 1 2 3"), Tcl
     will try to give a nice error message, saying that "foo x" can be
     called only with one argument. When printing argument "x", it sees
     that "x" is an indexObj, and for these kinds of objects, "x" might
     be an abbreviated version of a full name. Since the table behind
     the indexObj is in the case above volatile, a crash might happen.

   - Added API call Ns_SockInErrorState() since SSL_shutdown() must not
     be called if a previous fatal error has occurred on a connection
     i.e. if SSL_get_error() has returned SSL_ERROR_SYSCALL or
     SSL_ERROR_SSL.

   - Provided local buffers for OpenSSL ERR_error_string handling to
     avoid potential race conditions (OpenSSL keeps otherwise error
     messages in static memory).

   - Crypto support:
     * New feature: scrypt Password-Based Key Derivation Function (RFC 7914)

       The scrypt function is a modern replacement for crypt and bcrypt
       and derives secret keys from a secret string. It is based on
       memory- hard functions, which offer added protection against
       attacks using custom hardware and GPU arrays.

       The function requires the compilation of NaviServer against
       OpenSSL 3.0 or newer (not yet released)

       Example from RFC 7914:
	% ::ns_crypto::scrypt -secret "password" -salt NaCl -n 1024 -r 8 -p 16
	fdbabe1c9d3472007856e7190d01e9fe7c6ad7cbc8237830e77376634b373162
	2eaf30d92e22a3886ff109279d9830dac727afb94a83ee6d8360cbdfa2cc0640

      * "ns_md" and "ns_hmac": added flag "-encoding"

      * Added options "-passphrase" to all "ns_crypto" commands
        accepting a PEM file. In principle, every PEM file can be
        protected by a pass phrase.

      * Signing and verifying signatures: "::ns_crypto::md string" can
        be used for signing and verifying of cryptgraphic signatures:

	% set sig [::ns_crypto::md string \
		     -digest sha1 \
		     -encoding binary \
		     -sign /usr/local/src/naviserver/myprivate.pem \
		     "abcdefghijklmnopqrstuxvwxyz\n"]

	% set vfy [::ns_crypto::md string \
		     -digest sha1 \
		     -verify /usr/local/src/naviserver/myprivate.pem \
		     -signature $sig \
		     "abcdefghijklmnopqrstuxvwxyz\n"]


Documentation improvements:
---------------------------

  - Improved cross references between man pages
  - Added documentation to undocumented flags
  - Added more example
  - Improved spelling

  - The following 48 man pages were updated since the last release:

	admin-db.man, admin-install.man, admin-maintenance.man,
	admin-tuning.man, commandlist.man, ns_adp.man, ns_adp_ctl.man,
	ns_adp_info.man, ns_adp_parse.man, ns_asynclogfile.man,
	ns_base64.man, ns_conn.man, ns_connchan.man, ns_crypt.man,
	ns_crypto.man, ns_driver.man, ns_filestat.man, ns_hmac.man,
	ns_hotp.man, ns_http.man, ns_job.man, ns_locationproc.man,
	ns_log.man, ns_md.man, ns_md5.man, ns_parsefieldvalue.man,
	ns_parseheader.man, ns_parseurl.man, ns_rand.man,
	ns_register.man, ns_return.man, ns_server.man, ns_set.man,
	ns_sha1.man, ns_sls.man, ns_sockcallback.man,
	ns_sockselect.man, ns_tmpnam.man, ns_totp.man,
	ns_urlspace.man, ns_uudecode.man, ns_uuencode.man,
	ns_writer.man, nsv.man, returnstatus-cmds.man,
	tcl-lib-file.man, tcl-libraries.man, tcl-overview.man


Configuration Changes:
----------------------

  - Output warning to system log, when NaviServer was compiled without
    zlib support, but the config file request compression.  Before, the
    compress enable request was silently ignored, giving no hint, why
    compression attempts of a developer were ignored.

  - Sample configurations:

     * Documented usage of connection pools in sample config files.

     * Added sample entries for new features (e.g. rate limits, log
       file sanitizer)

     * openacs-config.tcl:

        . Changed driver installation to "global" in sample configuration
	 file. This change makes it easier for site admins to add
	 further servers (e.g. for virtual hosting).

        . Simplified sample config file by using multiple IP addresses
	 per driver

    * nsd-config.tcl:

        . Changed driver installation to "global" in sample configuration
	 file. This change makes it easier for site admins to add


Code Changes:
-------------

  - Improved scalability: provided different mutex locks variables per
    urlspace IDs.

  - Improved range checking for Ns_ObjvInt, Ns_ObjvLong, Ns_ObjvWideInt
    and MemUnits:

    * built-in range checkingsupport for objv parser: This eases
      specification of the C API, makes accepted ranges clear and
      provides consistent error messages. These change fixes several
      potential crashes, when e.g.  passed-in values are used as
      array-indices (crash on negative numbers) or where the values
      were casted to unsigned types (very high unsigned values,
      potential memory alloc problems).

     * Fixed all cases, where Tcl_GetIntFromObj(), Tcl_GetLongFromObj,
       and Tcl_GetWideintFromObj was used without a range check.

  - Argument parser: skip processing of optional arguments when the
    number of provided arguments is equal to the number of required
    parameters.

    Background: This change allows a fully backward compatible version
    of e.g. "ns_md5 $foo", no matter what the value of "$foo" is,
    although the new version of "ns_md5" has now a "-binary" option.


  - Refactored low-level socket I/O:
    * All driver operations are now strictly non-blocking
    * Use same infrastructure for all network I/O operations
      (driver, connchan, ns_http, ...)
    * Rewritten socketfile/sendfile handling (including *BSD support)
    * Improved handling of closed and detached channels, e.g.delegated
      to writers (new macros NS_CONN_REQUIRE_CONNECTED,
      NS_CONN_REQUIRE_OPEN, NS_CONN_REQUIRE_CONFIGURED,
      NS_CONN_REQUIRE_ALL)
    * Improved protection for SIGPIPE signals.

  - New API infrastructure: Ns_DList

    The Ns_DList is similar to Tcl_DString, but operates on pointers
    instead of characters. Like Tcl_DString, Ns_DList pre-allocates
    some data and extends it when necessary. Ns_DList structures are
    especially useful alternative to linked lists, when usually data
    is added to the end. Since the data is kept as a dense
    (potentially growing) array, the memory locality is much better
    than with linked lists, resulting in better CPU cache hit rates.

  - Distinguish between Ns_NormalizePath() and Ns_NormalizeUrl(),
    where the first is for the file system and the latter for URLs.
    Previously, both cases were handled identically.

   - Function pointer handling:
     * Created a Tcl hash type for function pointers, since ANSI/ISO C
       forbids casting/comparison of data and function pointers. The
       new type is used whenever a hash lookup is performed for a
       function pointer.
     * Fixed all other occurrences of such comparisons.

  - Regressions testing:
    * Removed false positive
    * Switch to testing via ns_http in regression tests
      (this makes it as well possible for testing https connections)
    * Extended tests:
      + Added regression tests for "ns_rand"
      + Added nsssl tests
      + Added automatic certificate generation for test-server

     * The following 35 tests were added or extended since the last release

	adp.test, encoding.test, http.test, http_byteranges.test,
	http_chunked.test, http_keep.test, https.test, misc.test,
	ns_adp_compress.test, ns_base64.test, ns_cache.test,
	ns_conn.test, ns_conn_host.test, ns_crypto.test,
	ns_driver.test, ns_hashpath.test, ns_hostbyaddr.test,
	ns_info.test, ns_limits.test, ns_md5.test, ns_nsv.test,
	ns_pagepath.test, ns_parsefieldvalue.test, ns_proxy.test,
	ns_reflow_text.test, ns_schedule.test, ns_server.test,
	ns_serverpath.test, ns_set.test, ns_sha1.test,
	ns_urlencode.test, ns_urlspace.test, ns_uuencode.test,
	ns_writer.test, tclconnio.test, tclresp.test

  - Improved portability:
    * improved compatibility with LibreSSL (2.7 and 2.9)
    * Windows changes (Many thanks to Andrew Piskorski):
      + Fixed Ns_LogRoll() to work on Windows.
      + WSASend receives as 5th argument flags, but not a pointer to flags.
      + Improved makefiles
      + Updated _MSC_VER version numbers to include Visual Studio 2019
      + Added macro NS_INLINE to achieve higher portability with (older?)
        Microsoft compilers.

  - Added warnings about potential misconfiguration of NaviServer "tcllib"
    setting (Tcl written NaviServer modules)

  - Marked compatibility wrappers Ns_SetThreadServer() and
    Ns_SetThreadServer() explicitly as deprecated. Use
    Ns_ThreadSetName() and Ns_ThreadGetName() instead.

  - Build-system
    * Added -DSYSTEM_MALLOC to default compile flags.
    * Added nsssl to the default test target.
    * Require RSA keys of size 2048: newer versions of OpenSSL refuse to work
      with RSA keys of size 1024.
    * Improved alignment with Tcl's current .m4 file to get rid of
      "-prebind" deprecated message under macOS.
    * Improved handling of posix thread library for FreeBSD and OpenBSD

  - Use native thread_local storage for log handling when available
    (experimental).

  - Improved type cleanness for function pointers
  - Improved code locality
  - Improved error messages
  - Improved structure packing
  - Reduced variable scopes
  - Added missing "extern" declarations
  - Aligned function prototypes
  - Dropped potentially dangerous call to alloca()
  - Added typedefs for commonly used functions
  - Reduce potential dangling pointer dereferences
  - Aligned names of arguments in prototype with function definition
  - Introduced use attribute-based approach for denoting
    fall through in case statements
  - Added more declarations for PURE and CONST functions
  - Improved configurability for clang-tidy
  - Fixed macro name-clash with PostgreSQL
  - Improved spelling



Modules:
--------
  24 files changed, 1638 insertions(+), 2395 deletions(-)

nsdbpg:
- Fixed loading when multiple servers are used
- C Code cleanup (reduced warnings)

nsdbmysql:
- Fixed compatibility with current versions of NaviServer
- C Code cleanup (reduced warnings, improved spelling)

nssmtpd:
- Removed deprecated calls
- fix compilation with TCL_NO_DEPRECATED
- C Code cleanup (reduced implicit conversions, improved spelling)

nsdns:
- C Code cleanup (reduced warnings)

nsudp:
- Made code compilable with TCL_NO_DEPRECATED
- C Code cleanup (reduced warnings, improved spelling)

nszlib:
- Made code compile cleanly

nsimap:
- C Code cleanup (improved spelling)

nsphp:
- Upgraded module to PHP 7
- Fixed memory leaks
- extended regression test
- C Code cleanup (improved spelling)
- This is a major overhaul of nsphp, which makes it possible to use
   PHP 7 (tested with PHP 7.3.5). PHP 5 reached it EOL by jan 2019. PHP
   7 has substantial changes e.g. in memory and thread management
   compared to PHP 5, several interfaces and API calls have changed,
   such that it is not possible to compile the new version of nsphp
   with still PHP 5. In case, someone wants to use nsphp with PHP 5,
   please checkout versions before this commit from the repository.

nsstats:
- Added link to process page from mecurial hash to version
   details on Bitbucket
- Included "ns_driver info" in the reported values
- Include number of started thread per pools in reported statistics

nsdbi:
- C Code cleanup (don't shadow variables, clean compilation, improved spelling)

nsdbipg:
- C Code cleanup (clean compilation, improved spelling)

nsoracle:
- C Code cleanup (clean compilation)

nswebsocket:
- Compatibility with Tcl 8.7, which has no "identity" encoding anymore
- fixed startup messages
- C Code cleanup (improved spelling)

revproxy:
- Support for request cancellation via url_rewrite_callback (when it
   returns an empty URL)
- Don't raise error on ECONNRESET during spooling
- Improved logging
- Improved documentation
- Made channelCleanup more robust

letsencrypt:
- Major overhaul of the letsencrypt module to support ACME v2

   ACME (the Automated Certificate Management Environment, [1]) is the
   protocol used for certificate mamagement on letsencrypt.org. The API
   version v was released on 2016 but was updated in 2018 by ACME v2,
   is not backwards compatible with v1. Letsencrypt announced in march
   2018 to drop the support of ACME v1 in several steps:

    - Nov 2019: End of account registrations via ACME v1
    - Jun 2020: End of new domain registrations via ACME v1
    - Jun 2021: EOL ACME v1 certificate issuing

   The new version is based on its crypto requirements solely on
   OpenSSL, it uses the NaviServer builtins and as well the "openssl"
   binary (the usage of tcllib pki was dropped). To avoid potential
   troubles, use this with a recent version of NaviServer (currently
   the tip version form BitBucket) or with NaviServer 4.99.19 when this
   is released.
- Improved logging and documentation
- Add documentation for obtaining multi-domain attributes (SAN).

nswebpush:
  - Added parameter "dopadding" to improve compatibility with Firefox on
    Android. It does not support padding. Many thanks to Wolfgang
    Winkler for reporting.
  - Added compatibility with current version of NaviServer
  - Always use binary format for appending (it does not seem necessary
    here, but it is used for consistency)
  - Code cleanup (improved spelling)

nsladp:
  - Adapted changes from Malte Sussdorf for the AOLserver version
    (from before 2008)
    https://www.mail-	archive.com/aol...@li.../msg12073.html
   - Added compile flag to force LDAP v3
   - Rebind with original credentials after bind auth try
   - Incorporated changes from Andreas Parschalk (Univ. Innsbruck)
     including better documentation
   - Provided clean compilation on recent versions of NaviServer and Tcl

Re: [naviserver-devel] nswebpush and Firefox on Android

[Gustaf N. <ne...@wu...>]

Dear Wolfgang,

i've added a config option to nswebpush (for a default setting)
and a switch "-nopadding" to webpush::send for per-message control.
The default is still the (in general recommended) padding mode. The
version passed the regression test against the to-be-released version
of NaviServer on bitbucket.

A short feedback, if this works for you is welcome, i am abroad
and have no good testing environment.

-g

On 02.01.20 12:38, Wolfgang Winkler wrote:

> Hello!
>
> When testing webpush with various clients, we encountered a problem 
> with Firefox on Android.
>
>     Webpush failed with status 413!
>
>     {{"code": 413, "errno": 104, "error": "Payload Too Large",
>     "more_info":
>     "http://autopush.readthedocs.io/en/latest/http.html#error-codes",
>     "message": "This message is intended for a constrained device and
>     is limited to 3070 bytes. Converted buffer too long by 1366 bytes"}
>
> This is a known problem and Mozilla is obviously not going to fix it:
>
> https://github.com/mozilla-services/autopush/issues/748
>
> With padding switched off (we just set the paddingLength in padData to 
> 0), the messages are delivered.
>
> Maybe a switch in webpush::send for disabling padding could be used to 
> circumvent this problem.
>
> regards,
>
> Wolfgang
>
> -- 
>
> *Wolfgang Winkler*
> Geschäftsführung
> wol...@di...
> mobil +43.699.19971172
>
> dc:*büro*
> digital concepts Novak Winkler OG
> Software & Design
> Landstraße 68, 5. Stock, 4020 Linz
> www.digital-concepts.com <http://www.digital-concepts.com>
> tel +43.732.997117.72
> tel +43.699.1997117.2
>
> Firmenbuchnummer: 192003h
> Firmenbuchgericht: Landesgericht Linz
>
>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] Towards NaviServer 4.99.19

[Wolfgang W. <wol...@di...>]

The warning is gone now.

Happy new year!

Wolfgang

Am 01.01.20 um 08:33 schrieb Gustaf Neumann:
>
> This is probably still not perfect. i've made one more change to make
> the handling of closed connection inside filters more general. If this
> is still shows the warning, we have to get into more details.
>
> Please test the updated version if possible.
>
> Happy new year!
>
> all the best
>
> -g
>
>
> On 31.12.19 11:51, Wolfgang Winkler wrote:
>> Dear Gustaf!
>>
>> NaviServer does not crash anymore, we get a "connection socket is 
>> detached" warning.
>>
>> Thanks for the fix and a happy new year,
>>
>> Wolfgang
>>
>> Am 30.12.19 um 20:50 schrieb Gustaf Neumann:
>>> > ... To use "filter_break" in this situation is an error on our 
>>> side ...
>>>
>>> Dear Wolfgang,
>>>
>>> I could reproduce the problem with a filter returning 
>>> "filter_break", when
>>> the filter has already closed the connection to the client. In such 
>>> cases,
>>> no further replies are possible to this client.
>>>
>>> i've added a fix + regression test to bitbucket.
>>> Please check, of this behaves not well in your cases...
>>>
>>> all the best
>>>
>>> -g
>>>
>>
>> -- 
>>
>> *Wolfgang Winkler*
>> Geschäftsführung
>> wol...@di...
>> mobil +43.699.19971172
>>
>> dc:*büro*
>> digital concepts Novak Winkler OG
>> Software & Design
>> Landstraße 68, 5. Stock, 4020 Linz
>> www.digital-concepts.com <http://www.digital-concepts.com>
>> tel +43.732.997117.72
>> tel +43.699.1997117.2
>>
>> Firmenbuchnummer: 192003h
>> Firmenbuchgericht: Landesgericht Linz
>>
>>

-- 

*Wolfgang Winkler*
Geschäftsführung
wol...@di...
mobil +43.699.19971172

dc:*büro*
digital concepts Novak Winkler OG
Software & Design
Landstraße 68, 5. Stock, 4020 Linz
www.digital-concepts.com <http://www.digital-concepts.com>
tel +43.732.997117.72
tel +43.699.1997117.2

Firmenbuchnummer: 192003h
Firmenbuchgericht: Landesgericht Linz

[naviserver-devel] nswebpush and Firefox on Android

[Wolfgang W. <wol...@di...>]

Hello!

When testing webpush with various clients, we encountered a problem with 
Firefox on Android.

    Webpush failed with status 413!

    {{"code": 413, "errno": 104, "error": "Payload Too Large",
    "more_info":
    "http://autopush.readthedocs.io/en/latest/http.html#error-codes",
    "message": "This message is intended for a constrained device and is
    limited to 3070 bytes. Converted buffer too long by 1366 bytes"}

This is a known problem and Mozilla is obviously not going to fix it:

https://github.com/mozilla-services/autopush/issues/748

With padding switched off (we just set the paddingLength in padData to 
0), the messages are delivered.

Maybe a switch in webpush::send for disabling padding could be used to 
circumvent this problem.

regards,

Wolfgang

-- 

*Wolfgang Winkler*
Geschäftsführung
wol...@di...
mobil +43.699.19971172

dc:*büro*
digital concepts Novak Winkler OG
Software & Design
Landstraße 68, 5. Stock, 4020 Linz
www.digital-concepts.com <http://www.digital-concepts.com>
tel +43.732.997117.72
tel +43.699.1997117.2

Firmenbuchnummer: 192003h
Firmenbuchgericht: Landesgericht Linz

Re: [naviserver-devel] Towards NaviServer 4.99.19

[Gustaf N. <ne...@wu...>]

This is probably still not perfect. i've made one more change to make
the handling of closed connection inside filters more general. If this
is still shows the warning, we have to get into more details.

Please test the updated version if possible.

Happy new year!

all the best

-g


On 31.12.19 11:51, Wolfgang Winkler wrote:
> Dear Gustaf!
>
> NaviServer does not crash anymore, we get a "connection socket is 
> detached" warning.
>
> Thanks for the fix and a happy new year,
>
> Wolfgang
>
> Am 30.12.19 um 20:50 schrieb Gustaf Neumann:
>> > ... To use "filter_break" in this situation is an error on our side 
>> ...
>>
>> Dear Wolfgang,
>>
>> I could reproduce the problem with a filter returning "filter_break", 
>> when
>> the filter has already closed the connection to the client. In such 
>> cases,
>> no further replies are possible to this client.
>>
>> i've added a fix + regression test to bitbucket.
>> Please check, of this behaves not well in your cases...
>>
>> all the best
>>
>> -g
>>
>
> -- 
>
> *Wolfgang Winkler*
> Geschäftsführung
> wol...@di...
> mobil +43.699.19971172
>
> dc:*büro*
> digital concepts Novak Winkler OG
> Software & Design
> Landstraße 68, 5. Stock, 4020 Linz
> www.digital-concepts.com <http://www.digital-concepts.com>
> tel +43.732.997117.72
> tel +43.699.1997117.2
>
> Firmenbuchnummer: 192003h
> Firmenbuchgericht: Landesgericht Linz
>
>

Re: [naviserver-devel] Towards NaviServer 4.99.19

[Wolfgang W. <wol...@di...>]

Dear Gustaf!

NaviServer does not crash anymore, we get a "connection socket is 
detached" warning.

Thanks for the fix and a happy new year,

Wolfgang

Am 30.12.19 um 20:50 schrieb Gustaf Neumann:
> > ... To use "filter_break" in this situation is an error on our side ...
>
> Dear Wolfgang,
>
> I could reproduce the problem with a filter returning "filter_break", 
> when
> the filter has already closed the connection to the client. In such 
> cases,
> no further replies are possible to this client.
>
> i've added a fix + regression test to bitbucket.
> Please check, of this behaves not well in your cases...
>
> all the best
>
> -g
>

-- 

*Wolfgang Winkler*
Geschäftsführung
wol...@di...
mobil +43.699.19971172

dc:*büro*
digital concepts Novak Winkler OG
Software & Design
Landstraße 68, 5. Stock, 4020 Linz
www.digital-concepts.com <http://www.digital-concepts.com>
tel +43.732.997117.72
tel +43.699.1997117.2

Firmenbuchnummer: 192003h
Firmenbuchgericht: Landesgericht Linz

Re: [naviserver-devel] Towards NaviServer 4.99.19

[Gustaf N. <ne...@wu...>]

 > ... To use "filter_break" in this situation is an error on our side ...

Dear Wolfgang,

I could reproduce the problem with a filter returning "filter_break", when
the filter has already closed the connection to the client. In such cases,
no further replies are possible to this client.

i've added a fix + regression test to bitbucket.
Please check, of this behaves not well in your cases...

all the best

-g

Re: [naviserver-devel] Towards NaviServer 4.99.19

[Wolfgang W. <wol...@di...>]

This is good news!

At the current tip version we encountered a signal 11 error. It happens 
in a filter, when we raise an error and return "filter_break". With 
"filter_return", there is no problem. To use "filter_break" in this 
situation is an error on our side but maybe the crash can happen in 
other situations as well. This is what we see in our log:

Warning: redirect status 404: connection already closed
Warning: NsWriterQueue: called without sockPtr size 534 bufs 1 flags 
103031 stream 000000 chan (nil) fd -1
Fatal: received fatal signal 11

Regards,

Wolfgang

Am 27.12.19 um 00:22 schrieb Gustaf Neumann:
>
> Dear all,
>
> on sourceforge is a release candidate for NaviServer 4.99.19 [1].
>
> Below is a - preliminary - summary of changes; especially the
> summary of changes in the modules is missing. In case you have more
> changes for this release, please commit or get in touch with me.
> If everything goes well, the release should follow in about one week.
>
> Please test if possible.
>
> best regards
>
> -gustaf
> [1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.19/
>
> =======================================
> NaviServer 4.99.19, released 2019-XX-XX
> =======================================
>
>   244 files changed, 18221 insertions(+), 7899 deletions(-)
>
> New Features:
> -------------
>
>   - Bandwidth management: Limit download rate to a maximum (specified
>     as kilobytes per second)
>
>     On servers with bad internet connections, it is possible that a few
>     eager downloaders can use up all available bandwidth.  NaviServer
>     allows now admins and developers to specify the maximum outgoing
>     bandwidth per connection or per connection pool. The rate limit can
>     be specified per connection (highest weight) or per connection
>     thread pool (second highest weight) or per network driver. The rate
>     limitation is only active, when writer threads are used. Per
>     default, the rate limitation is turned off (value 0).
>
>     New option for ns_conn to query set the per-connection limit
>        ns_conn ratelimit ?limit?
>     Introspecton "ns_writer list" returns the current
>     transferrates per writer job.
>
>     Configuration option for connection pools:
>        ns_param connectionratelimit ...
>        ns_param poolratelimit ...
>     Configuration option for network drivers:
>        ns_param writerratelimit ...
>
>     Commands for changing the rate limits dynamically:
>         ns_server ?-server s? ?-pool p? connectionratelimit ?value?
>         ns_server ?-server s? ?-pool p? poolratelimit ?value?
>
>
>   - Context filter for urlspace:
>
>     NaviServer's urlspace trie data structure used for various purposes
>     such as registering methods/URL pairs for procs or for mapping of
>     requests to connection thread pools. The classical trie mapping for
>     e.g. "GET /foo/bar/*.html" is based on method + path + string match
>     pattern, where the last part "*.html" is used as a "filter".
>
>     The new version of NaviServer allows in addition to the string
>     based glob pattern so-called "context filter", which includes the
>     IP-address or header fields in the matching process. The context
>     filters are applied after the classical urlspace processing, such
>     that the behavior is fully backwards compatible. The context
>     filters are kept in a stable order (IP before header-based filters,
>     specific before less specific) such that the behavior is not order
>     dependent.
>
>     The new feature can be used map e.g. bots or certain IP ranges to
>     the specified connection pools. When connection pools named "bots"
>     and "local" are defined, one can use the following mapping commands
>     to map incoming requests to the specified pools. Examples for
>     adding urlspace mappings at runtime:
>
> 	 ns_server -pool bots map "GET /* {user-agent *bot*}"
> 	 ns_server -pool bots map "GET /* {user-agent *crawl*}"
> 	 ns_server -pool bots map "GET /* {user-agent *baidu*}"
> 	 ns_server -pool bots map "GET /* {X-NS-ip 2a03:2880::/29}"
>
> 	 ns_server -pool local map "GET /* {X-NS-ip 127.0.0.1}"
> 	 ns_server -pool local map "GET /* {X-NS-ip 137.208.1.0/16}"
>
>     Note that arbitrary header fields can be used for the mapping,
>     containing potentially some glob style match characters. The
>     special header field "X-NS-ip" denotes IP based filters, which can
>     be provided fully qualified or in CIDR notation (denoting
>     significant bits) for IPv4 and IPv6.
>
>
>   - Allow a single driver to listen on multiple IP addresses:
>
>     This feature greatly simplifies setups, where a single server is
>     listening on multiple IP addresses (e.g. IPv4 and
>     IPv6). Previously, it was necessary to define separate drivers for
>     these, which need different names but which are often
>     configured identically.  Furthermore, when the address is omitted,
>     the server performs a lookup from the hostname to determine the IP
>     address. Previously, it used the first one, now it can work with
>     all returned addresses.
>
>     To use this feature, simply specify the configure values of
>     "address" as a Tcl list containing multiple IP addresses. The
>     change is fully backwards compatible, old configuration files will
>     continue to work.
>
>
>   - Added sending of multiple file chunks with a single command:
>   
>        ns_writer sendfiles /list of filespecs/
>
>     Every filespec is a dict which must contain a "filename" element
>     and can contain additionally an "-offset" and/or a "-size" element.
>
>        ns_writer submitfiles {{filename /tmp/f1} {filename /tmp/f2 -offset 10}}
>
>     This function is e.g. useful for video streaming applications,
>     where multiple video file chunks should be transferred (sometimes)
>     concatenated in a single chunk. Therefore, this function avoids the
>     necessity to concatenate these files in advance (causing increased
>     latency and disk usage).
>    
>
>   - Added per-server "extraheaders". These extra headers fields are
>     merged with the following precedence (the earlier ones have more
>     weight)
>       1) application specific headers
>       2) per-server extra headers
>       3) per-driver extra headers
>
>     This change makes it possible to provide in the config files
>     defaults, which can be overwritten by certain pages in the
>     applications. Furthermore, this change makes it easy to add
>     header fields to all requests of a virtual server (e.g.
>     Strict-Transport-Security, ...)
>
>
>   - Logging improvements:
>
>     * Support separate access logs for requests sent via
>       different drivers.
>
>       The nslog module accepts now an additional parameter named
>       "driver".  This option can be used to produce different access
>       logs for requests submitted via different drivers to sort out
>       e.g. local server talk.  Per default, every request is
>       logged. When a value is provided for this parameter, only
>       requests are logged in this log file when these come from a
>       driver matching the provided glob pattern.
>       
>     * log-file sanitizer:
>       When data entered into a log file contains special characters
>       (e.g.  end line characters), these could alter the appearance of
>       content within the log file. Single entries may appear as
>       multiple entries.  Attackers may leverage log forging to insert
>       fake entries to that obfuscate malicious acts, or these can
>       confuse log-file analyzers to avoid analysis.
>
>       The new global parameter "sanitizelogfiles" can control this
>       behavior (values 0: none, 1: full, 2: human-friendly; default:
>       2);
>
>     * Additional debugging flag "Debug(access)": When activated, entries
>       of the access log are mirrored in the system log
>       (error.log). This option eases tracking the end of requests in
>       the system log.
>
>     * Additional debugging flag "Debug(writer)" for debugging of writer.
>
>     * Provided access to request AND reply header fields via
>       "extendedheaders" specification (in "ns_accesslog" and "nslog"
>       configuration section):
>
>       It is now optionally possible to specify the header fields in the
>       provided "extendedheaders" list with a prefix "request:" or
>       "response:" to denote request and reply header fields. In case,
>       no tag is provided, field names are interpreted as request header
>       fields (like before). The change is fully backwards compatible,
>
>
>    - ns_set:
>
>      * Added sucommand "ns_set imerge", a is the case insignificant
>        version of "ns_set merge". This option is useful especially for
>        working with header fields.
>
>      * Added subcommand "ns_set iupdate": this command is the case
>        insenstive counterpart of "ns_set update".
>
>      * Fixed "nsv_set a b" when no key nor array exist
>
>   - ns_http:
>     * Revamp of ns_http and related supportive code.
>     * Added chunked-encoding parser
>     * Tcl channel support for body (PUT, POST) and for result.
>       This makes it possible to use e.g. reflected Tcl channels
>       in "ns_http".
>       New Options for "ns_http run"
>         - ?-body_size size?
>         - ?-body_chan chan?
>         - ?-outputfile fn?
>         - ?-outputchan chan?
>     * Make clear that "ns_http run" is the preferred interface
>       (rather than "queue" + "wait")
>     * Deprecated output variables of "ns_http wait" (since everything
>       is included in the resulting dict)
>
>
>   - New commands:
>
>     * ns_asynclogfile:
>       Facility for thread-safe writing async
>       log files of various kinds using the AsyncWriterThread.
>       Usage example:
> 	# ... at startup
> 	set fd [ns_asynclogfile open /tmp/test.log]
> 	# ... during run
> 	ns_asynclogfile write $fd hello\n
> 	# ... at shutdown
> 	ns_asynclogfile close $fd
>
>     * ns_parsefieldvalue:
>
>       This function parses the provided field value (from an HTTP
>       request or reply header field) into its parts and returns these
>       in the form of a list of Tcl dicts or a single Tcl dict depending
>       on parameters. The syntax of the contents of these header fields
>       is specified in RFC 7230 section 3.2.6.
>
>   - New features for existing commands/subcommands
>   
>      * Binary reform: Some NaviServer commands accepted previously
>        implicitly binary or non-binary input, some commands used
>        the flag "-binary" to denote the differences (e.g. ns_return).
>        For a more uniform and expectable behavior binary input the
>        explicit option "-binary" flag to the following commands:
>
> 	ns_base64decode
> 	ns_base64encode
> 	ns_base64urldecode
> 	ns_base64urlencode
> 	ns_md5
> 	ns_sha1
> 	ns_uudecode
> 	ns_uuencode
> 	ns_crypto::aead::decrypt string (for "-key", "-aad", "-iv", "input")
> 	ns_crypto::aead::encrypt string (for "-key", "-aad", "-iv", "input")
> 	ns_crypto::eckey import ("-string")
> 	ns_crypto::eckey sharedsecret ("pubkey")
> 	ns_crypto::hmac add (for "message")
> 	ns_crypto::hmac new (for "key")
> 	ns_crypto::hmac string (for "key", "message")
> 	ns_crypto::md add (for "message")
> 	ns_crypto::md hkdf (for "-salt" "-secret" "-info")
> 	ns_crypto::md string (for "message")
> 	ns_crypto::md vapidsign (for "message")
> 	ns_crypto::scrypt (for "-salt" "-secret")
>
>      Note that this option is not fully backward compatible. Early
>      versions of "ns_sha1" were expecting always non-binary input, some
>      new version assumed binary input. The new version is now more
>      compatible with AOLserver and older NaviServer versions.
>
>      * "ns_server threads" reports now how often a thread of this pool
>        was started.  This statistic can help to identify
>        installations, where threads are started and stopped in a too
>        eager fashion.
>      * "ns_server.... stats": added "sendbodysize" and "replybodysize"
>        to the dict of the per-pool statistics.
>      * "ns_server ... connectionratelimit ?value?": Query or set
>         the default per-connection rate limit.
>      * "ns_server ... poolratelimit ?value?": Query or set
>         the pool connection rate limit.
>      * "ns_writer list" returns now the current transferrates per
>        writer job.
>        
>      * "ns_writer submitfile -offset X -size Y" and "ns_writer size X"
>        accept now memory units, the option ?-driver driver? was added
>        to the "ns_writer" subcommands "size", and "streaming"
>        (defaulting to the current driver).
>        
>        One can now write
>             ns_writer size 150KB
>         instead of
>             ns_writer size nssock [expr 150*1024]
>
>
>
>
> Performance Improvements:
> -------------------------
>
>
>
>
>
> Bug Fixes:
> ----------
>
>    - Blueprint serializer: moved serialized objects in the blueprint
>      after the namespace imports and ensemble recreators since
>      constructors can call functions depending on it.
>
>    - Include *xml* and *json* in non-binary MIME types. Note that
>      there is no exact definition, what MIME types are exactly.
>
>    - nslog:
>      * Abort in error situation with an error message instead of crashing
>      * Added server name in log entries to make it easier to distinguish
>        messages per-server during bootup
>
>    - Range requests:
>      * Fixed potential problem with too many non-contiguous byte ranges
>      * Don't silently ignore invalid syntax of range requests
>
>    - Improved handling of already closed connections (data delivery was
>      already delegated to writer threads, but driver still wants to
>      return data directly; this could happen in error situations)
>
>    - connchan:
>      * Fixed potential invalid reads in (error) cases, where
>        LogConnchanDebug is enabled and the callback is deleted during a
>        Tcl_Eval().
>      * Fixed potential problem, where call to Tcl_Eval() might clean
>        structures used in the callback handler
>
>    - Make sure, memory allocated by Tcl is returned by Tcl (important,
>      when NaviServer is compiled with -DSYSTEM_MALLOC)
>
>    - Fixed Tcl argument parsing of "ns_roll", "ns_fmttime",
>      "nsv_bucket" and "ns_critsec eval".
>
>    - Fixed bug in "nsv_set -default" overwriting pre-existing values
>
>    - Fixed potential crashes in (desperate) error situations, where the
>      connection was already closed.
>   
>    - Fixed leaking tmp file, when writerstraming is activated and
>      streaming HTML output is used. This bug could appear, when
>      "writerstreaming" is turned on in the config file (default off).
>      See also:https://sourceforge.net/p/naviserver/mailman/naviserver-
> 	devel/thread/dd508519-1262-bd91-288a-fa30a7634224%40digital-
> 	concepts.com/#msg36794472
>
>    - Made table used for enumeration values for Ns_ObjvIndex static.
>      Background: This is a fix for a tricky Tcl_Obj sharing bug
>      triggered by the usage of C-level IndexObjs based on volatile
>      tables. Consider the following example:
>
>         proc foo {x} {
>            return [bar -value x ...]
>         }
>
>      The Tcl_Obj "x" is shared as name of argument and as a value of
>      the non-positional parameter "-value". When bar
>      (e.g. C-implemented) uses Tcl_GetIndexFromObj*() to lookup "x" in
>      a table of options, which is volatile, the involved Tcl_Obj will
>      be converted to an indexObj.
>
>      On a call with wrong number of arguments( e.g."foo 1 2 3"), Tcl
>      will try to give a nice error message, saying that "foo x" can be
>      called only with one argument. When printing argument "x", it sees
>      that "x" is an indexObj, and for these kinds of objects, "x" might
>      be an abbreviated version of a full name. Since the table behind
>      the indexObj is in the case above volatile, a crash might happen.
>
>    - Added API call Ns_SockInErrorState() since SSL_shutdown() must not
>      be called if a previous fatal error has occurred on a connection
>      i.e. if SSL_get_error() has returned SSL_ERROR_SYSCALL or
>      SSL_ERROR_SSL.
>
>    - Provided local buffers for OpenSSL ERR_error_string handling to
>      avoid potential race conditions (OpenSSL keeps otherwise error
>      messages in static memory).
>
>    - Crypto support:
>      * New feature: scrypt Password-Based Key Derivation Function (RFC 7914)
>
>        The scrypt function is a modern replacement for crypt and bcrypt
>        and derives secret keys from a secret string. It is based on
>        memory- hard functions, which offer added protection against
>        attacks using custom hardware and GPU arrays.
>
>        The function requires the compilation of NaviServer against
>        OpenSSL 3.0 or newer (not yet released)
>
>        Example from RFC 7914:
> 	% ::ns_crypto::scrypt -secret "password" -salt NaCl -n 1024 -r 8 -p 16
> 	fdbabe1c9d3472007856e7190d01e9fe7c6ad7cbc8237830e77376634b373162
> 	2eaf30d92e22a3886ff109279d9830dac727afb94a83ee6d8360cbdfa2cc0640
>
>       * "ns_md" and "ns_hmac": added flag "-encoding"
>
>       * Added options "-passphrase" to all "ns_crypto" commands
>         accepting a PEM file. In principle, every PEM file can be
>         protected by a pass phrase.
>
>       * Signing and verifying signatures: "::ns_crypto::md string" can
>         be used for signing and verifying of cryptgraphic signatures:
>
> 	% set sig [::ns_crypto::md string \
> 		     -digest sha1 \
> 		     -encoding binary \
> 		     -sign /usr/local/src/naviserver/myprivate.pem \
> 	             "abcdefghijklmnopqrstuxvwxyz\n"]
>
> 	% set vfy [::ns_crypto::md string \
> 		     -digest sha1 \
> 		     -verify /usr/local/src/naviserver/myprivate.pem \
> 		     -signature $sig \
> 		     "abcdefghijklmnopqrstuxvwxyz\n"]
> 		
>    - nscgi:
>      * ...
>
>
> Documentation improvements:
> ---------------------------
>
>   - Improved cross references between man pages
>   - Added documentation to undocumented flags
>   - Added more example
>   - Improved spelling
>   
>   - The following 48 man pages were updated since the last release:
>   
>          admin-db.man, admin-install.man, admin-maintenance.man,
>          admin-tuning.man, commandlist.man, ns_adp.man, ns_adp_ctl.man,
>          ns_adp_info.man, ns_adp_parse.man, ns_asynclogfile.man,
>          ns_base64.man, ns_conn.man, ns_connchan.man, ns_crypt.man,
>          ns_crypto.man, ns_driver.man, ns_filestat.man, ns_hmac.man,
>          ns_hotp.man, ns_http.man, ns_job.man, ns_locationproc.man,
>          ns_log.man, ns_md.man, ns_md5.man, ns_parsefieldvalue.man,
>          ns_parseheader.man, ns_parseurl.man, ns_rand.man,
>          ns_register.man, ns_return.man, ns_server.man, ns_set.man,
>          ns_sha1.man, ns_sls.man, ns_sockcallback.man,
>          ns_sockselect.man, ns_tmpnam.man, ns_totp.man,
>          ns_urlspace.man, ns_uudecode.man, ns_uuencode.man,
>          ns_writer.man, nsv.man, returnstatus-cmds.man,
>          tcl-lib-file.man, tcl-libraries.man, tcl-overview.man
>
>
> Configuration Changes:
> ----------------------
>
>   - Output warning to system log, when NaviServer was compiled without
>     zlib support, but the config file request compression.  Before, the
>     compress enable request was silently ignored, giving no hint, why
>     compression attempts of a developer were ignored.
>
>   - Sample configurations:
>
>      * Documented usage of connection pools in sample config files.
>
>      * Added sample entries for new features (e.g. rate limits, log
>        file sanitizer)
>        
>      * openacs-config.tcl:
>
>         . Changed driver installation to "global" in sample configuration
> 	 file. This change makes it easier for site admins to add
> 	 further servers (e.g. for virtual hosting).
>
>         . Simplified sample config file by using multiple IP addresses
> 	 per driver
>
>     * nsd-config.tcl:
>
>         . Changed driver installation to "global" in sample configuration
> 	 file. This change makes it easier for site admins to add
>
>
> Code Changes:
> -------------
>
>   - Improved scalability: provide different locks per urlspace IDs.
>    
>   - Improved range checking for Ns_ObjvInt, Ns_ObjvLong,
>     Ns_ObjvWideInt and MemUnits:
>
>     * built-in support for objv parser This eases specification of the
>       C API, makes accepted ranges clear and provides consistent error
>       messages. These change fixes several potential crashes, when e.g.
>       passed-in values are used as array-indices (crash on negative
>       numbers) or where the values were casted to unsigned types (very
>       high unsigned values, potential memory alloc problems).
>
>      * Fixed all cases, where Tcl_GetIntFromObj(), Tcl_GetLongFromObj,
>        and Tcl_GetWideintFromObj was used without a range check.
>
>
>   - Argument parser: skip processing of optional arguments when the
>     number of provided arguments is equal to the number of required
>     parameters.
>
>     Background: This change allows a fully backward compatible version
>     of e.g. "ns_md5 $foo", no matter what the value of "$foo" is,
>     although the new version of "ns_md5" has now a "-binary" option.
>
>
>   - Refactored low-level socket I/O:
>     * All driver operations are now strictly non-blocking
>     * Use same infrastructure for all network I/O operations
>       (driver, connchan, ns_http, ...)
>     * Rewritten socketfile/sendfile handling (including *BSD support)
>     * Improved handling of closed and detached channels, e.g.delegated
>       to writers (new macros NS_CONN_REQUIRE_CONNECTED,
>       NS_CONN_REQUIRE_OPEN, NS_CONN_REQUIRE_CONFIGURED,
>       NS_CONN_REQUIRE_ALL)
>     * Improved protection for SIGPIPE signals.
>
>   - New API infrastructure: Ns_DList
>    
>     The Ns_DList is similar to Tcl_DString, but operates on pointers
>     instead of characters. Like Tcl_DString, Ns_DList pre-allocates
>     some data and extends it when necessary. Ns_DList structures are
>     especially useful alternative to linked lists, when usually data
>     is added to the end. Since the data is kept as a dense
>     (potentially growing) array, the memory locality is much better
>     than with linked lists, resulting in better CPU cache hit rates.
>
>   - Distinguish between Ns_NormalizePath() and Ns_NormalizeUrl(),
>     where the first is for the file system and the latter for URLs.
>     Previously, both cases were handled identically.
>
>    - Function pointer handling:
>      * Created a Tcl hash type for function pointers, since ANSI/ISO C
>        forbids casting/comparison of data and function pointers. The
>        new type is used whenever a hash lookup is performed for a
>        function pointer.
>      * Fixed all other occurrences of such comparisons.
>
>   - Regressions testing:
>     * Removed false positive
>     * Switch to testing via ns_http in regression tests
>       (this makes it as well possible for testing https connections)
>     * Extended tests:
>       + Added regression tests for "ns_rand"
>       + Added nsssl tests
>       + Added automatic certificate generation for test-server
>       
>      * The following 35 tests were added or extended since the last release
>      
>          adp.test, encoding.test, http.test, http_byteranges.test,
>          http_chunked.test, http_keep.test, https.test, misc.test,
>          ns_adp_compress.test, ns_base64.test, ns_cache.test,
>          ns_conn.test, ns_conn_host.test, ns_crypto.test,
>          ns_driver.test, ns_hashpath.test, ns_hostbyaddr.test,
>          ns_info.test, ns_limits.test, ns_md5.test, ns_nsv.test,
>          ns_pagepath.test, ns_parsefieldvalue.test, ns_proxy.test,
>          ns_reflow_text.test, ns_schedule.test, ns_server.test,
>          ns_serverpath.test, ns_set.test, ns_sha1.test,
>          ns_urlencode.test, ns_urlspace.test, ns_uuencode.test,
>          ns_writer.test, tclconnio.test, tclresp.test
>
>   - Improved portability:
>     * improved compatibility with LibreSSL (2.7 and 2.9)
>     * Windows changes (Many thanks to Andrew Piskorski):
>       + Fixed Ns_LogRoll() to work on Windows.
>       + WSASend receives as 5th argument flags, but not a pointer to flags.
>       + Improved makefiles
>       + Updated _MSC_VER version numbers to include Visual Studio 2019
>       + Added macro NS_INLINE to achieve higher portability with (older?)
>         Microsoft compilers.
>
>   - Added warnings about potential misconfiguration of NaviServer "tcllib"
>     setting (Tcl written NaviServer modules)
>
>   - Marked compatibility wrappers Ns_SetThreadServer() and
>     Ns_SetThreadServer() explicitly as deprecated. Use
>     Ns_ThreadSetName() and Ns_ThreadGetName() instead.
>
>   - Build-system
>     * Added -DSYSTEM_MALLOC to default compile flags.
>     * Added nsssl to the default test target.
>     * Require RSA keys of size 2048: newer versions of OpenSSL refuse to work
>       with RSA keys of size 1024.
>     * Improved alignment with Tcl's current .m4 file to get rid of
>       "-prebind" deprecated message under macOS.
>     * Improved handling of posix thread library for FreeBSD and OpenBSD
>
>   - Use native thread_local storage for log handling when available
>     (experimental).
>
>   - Improved type cleanness for function pointers
>   - Improved code locality
>   - Improved error messages
>   - Improved structure packing
>   - Reduced variable scopes
>   - Added missing "extern" declarations
>   - Aligned function prototypes
>   - Dropped potentially dangerous call to alloca()
>   - Added typedefs for commonly used functions
>   - Reduce potential dangling pointer dereferences
>   - Aligned names of arguments in prototype with function definition
>   - Introduced use attribute-based approach for denoting
>     fall through in case statements
>   - Adding more declarations for PURE and CONST functions
>   - Improved configurability for clang-tidy
>
>   - Fixed macro name-clash with PostgreSQL
>   - Improved spelling
>
>
>
> Modules:
> --------
>
> ....
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel


-- 

*Wolfgang Winkler*
Geschäftsführung
wol...@di...
mobil +43.699.19971172

dc:*büro*
digital concepts Novak Winkler OG
Software & Design
Landstraße 68, 5. Stock, 4020 Linz
www.digital-concepts.com <http://www.digital-concepts.com>
tel +43.732.997117.72
tel +43.699.1997117.2

Firmenbuchnummer: 192003h
Firmenbuchgericht: Landesgericht Linz

Re: [naviserver-devel] nswebpush and NaviServer version

[Wolfgang W. <wol...@di...>]

OK, thanks for the clarification.

regards,

Wolfgang


Am 23.12.19 um 13:28 schrieb Gustaf Neumann:
> The rules are as follows:
>
> - the releases version of NaviServer and the modules tar files pm 
> sourceforge
>   work together
> - the head version on bitbucket are likely to match for the major modules
>   (not all modules are adjusted immediately to every change).
>
> The same is also true for nswebpush: the released version from sourceforge
> should work (was running e.g. a while on openacs.org), the tip 
> versions are as
> well working together (currently running on openacs.org). The changes 
> are due
> to changes in the relatively young ns_crypto support in NaviServer.
>
> The best solution is to produce a new release of NaviServer (just a 
> matter of
> work, not sure, when this can sneak in.
>
> all the best
> .gn
>
> On 23.12.19 08:46, Wolfgang Winkler wrote:
>>
>> Hello!
>>
>> I think the tip nswebpush version in the repository does not work 
>> with NaviServer 4.99.18.
>>
>> ns_crypto::eckey sharedsecret
>>
>> and
>>
>> ns_base64urldecode -binary
>>
>> return an error.
>>
>> I think you could just change the minimal NaviServer version in the 
>> Readme to 4.99.19.
>>
>> Regards,
>>
>> Wolfgang
>>
>> -- 
>>
>> *Wolfgang Winkler*
>> Geschäftsführung
>> wol...@di...
>> mobil +43.699.19971172
>>
>> dc:*büro*
>> digital concepts Novak Winkler OG
>> Software & Design
>> Landstraße 68, 5. Stock, 4020 Linz
>> www.digital-concepts.com <http://www.digital-concepts.com>
>> tel +43.732.997117.72
>> tel +43.699.1997117.2
>>
>> Firmenbuchnummer: 192003h
>> Firmenbuchgericht: Landesgericht Linz
>>
>>
>>
>>
>> _______________________________________________
>> naviserver-devel mailing list
>> nav...@li...
>> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>
>
> -- 
> Univ.Prof. Dr. Gustaf Neumann
> WU Vienna
> Institute of Information Systems and New Media
> Welthandelsplatz 1, A-1020 Vienna, Austria
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel


-- 

*Wolfgang Winkler*
Geschäftsführung
wol...@di...
mobil +43.699.19971172

dc:*büro*
digital concepts Novak Winkler OG
Software & Design
Landstraße 68, 5. Stock, 4020 Linz
www.digital-concepts.com <http://www.digital-concepts.com>
tel +43.732.997117.72
tel +43.699.1997117.2

Firmenbuchnummer: 192003h
Firmenbuchgericht: Landesgericht Linz

[naviserver-devel] Towards NaviServer 4.99.19

[Gustaf N. <ne...@wu...>]

Dear all,

on sourceforge is a release candidate for NaviServer 4.99.19 [1].

Below is a - preliminary - summary of changes; especially the
summary of changes in the modules is missing. In case you have more
changes for this release, please commit or get in touch with me.
If everything goes well, the release should follow in about one week.

Please test if possible.

best regards

-gustaf
[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.19/

=======================================
NaviServer 4.99.19, released 2019-XX-XX
=======================================

  244 files changed, 18221 insertions(+), 7899 deletions(-)

New Features:
-------------

  - Bandwidth management: Limit download rate to a maximum (specified
    as kilobytes per second)

    On servers with bad internet connections, it is possible that a few
    eager downloaders can use up all available bandwidth.  NaviServer
    allows now admins and developers to specify the maximum outgoing
    bandwidth per connection or per connection pool. The rate limit can
    be specified per connection (highest weight) or per connection
    thread pool (second highest weight) or per network driver. The rate
    limitation is only active, when writer threads are used. Per
    default, the rate limitation is turned off (value 0).

    New option for ns_conn to query set the per-connection limit
       ns_conn ratelimit ?limit?
    Introspecton "ns_writer list" returns the current
    transferrates per writer job.

    Configuration option for connection pools:
       ns_param connectionratelimit ...
       ns_param poolratelimit ...
    Configuration option for network drivers:
       ns_param writerratelimit ...

    Commands for changing the rate limits dynamically:
        ns_server ?-server s? ?-pool p? connectionratelimit ?value?
        ns_server ?-server s? ?-pool p? poolratelimit ?value?


  - Context filter for urlspace:

    NaviServer's urlspace trie data structure used for various purposes
    such as registering methods/URL pairs for procs or for mapping of
    requests to connection thread pools. The classical trie mapping for
    e.g. "GET /foo/bar/*.html" is based on method + path + string match
    pattern, where the last part "*.html" is used as a "filter".

    The new version of NaviServer allows in addition to the string
    based glob pattern so-called "context filter", which includes the
    IP-address or header fields in the matching process. The context
    filters are applied after the classical urlspace processing, such
    that the behavior is fully backwards compatible. The context
    filters are kept in a stable order (IP before header-based filters,
    specific before less specific) such that the behavior is not order
    dependent.

    The new feature can be used map e.g. bots or certain IP ranges to
    the specified connection pools. When connection pools named "bots"
    and "local" are defined, one can use the following mapping commands
    to map incoming requests to the specified pools. Examples for
    adding urlspace mappings at runtime:

	 ns_server -pool bots map "GET /* {user-agent *bot*}"
	 ns_server -pool bots map "GET /* {user-agent *crawl*}"
	 ns_server -pool bots map "GET /* {user-agent *baidu*}"
	 ns_server -pool bots map "GET /* {X-NS-ip 2a03:2880::/29}"

	 ns_server -pool local map "GET /* {X-NS-ip 127.0.0.1}"
	 ns_server -pool local map "GET /* {X-NS-ip 137.208.1.0/16}"

    Note that arbitrary header fields can be used for the mapping,
    containing potentially some glob style match characters. The
    special header field "X-NS-ip" denotes IP based filters, which can
    be provided fully qualified or in CIDR notation (denoting
    significant bits) for IPv4 and IPv6.


  - Allow a single driver to listen on multiple IP addresses:

    This feature greatly simplifies setups, where a single server is
    listening on multiple IP addresses (e.g. IPv4 and
    IPv6). Previously, it was necessary to define separate drivers for
    these, which need different names but which are often
    configured identically.  Furthermore, when the address is omitted,
    the server performs a lookup from the hostname to determine the IP
    address. Previously, it used the first one, now it can work with
    all returned addresses.

    To use this feature, simply specify the configure values of
    "address" as a Tcl list containing multiple IP addresses. The
    change is fully backwards compatible, old configuration files will
    continue to work.


  - Added sending of multiple file chunks with a single command:
  
       ns_writer sendfiles /list of filespecs/

    Every filespec is a dict which must contain a "filename" element
    and can contain additionally an "-offset" and/or a "-size" element.

       ns_writer submitfiles {{filename /tmp/f1} {filename /tmp/f2 -offset 10}}

    This function is e.g. useful for video streaming applications,
    where multiple video file chunks should be transferred (sometimes)
    concatenated in a single chunk. Therefore, this function avoids the
    necessity to concatenate these files in advance (causing increased
    latency and disk usage).
   

  - Added per-server "extraheaders". These extra headers fields are
    merged with the following precedence (the earlier ones have more
    weight)
      1) application specific headers
      2) per-server extra headers
      3) per-driver extra headers

    This change makes it possible to provide in the config files
    defaults, which can be overwritten by certain pages in the
    applications. Furthermore, this change makes it easy to add
    header fields to all requests of a virtual server (e.g.
    Strict-Transport-Security, ...)


  - Logging improvements:

    * Support separate access logs for requests sent via
      different drivers.

      The nslog module accepts now an additional parameter named
      "driver".  This option can be used to produce different access
      logs for requests submitted via different drivers to sort out
      e.g. local server talk.  Per default, every request is
      logged. When a value is provided for this parameter, only
      requests are logged in this log file when these come from a
      driver matching the provided glob pattern.
      
    * log-file sanitizer:
      When data entered into a log file contains special characters
      (e.g.  end line characters), these could alter the appearance of
      content within the log file. Single entries may appear as
      multiple entries.  Attackers may leverage log forging to insert
      fake entries to that obfuscate malicious acts, or these can
      confuse log-file analyzers to avoid analysis.

      The new global parameter "sanitizelogfiles" can control this
      behavior (values 0: none, 1: full, 2: human-friendly; default:
      2);

    * Additional debugging flag "Debug(access)": When activated, entries
      of the access log are mirrored in the system log
      (error.log). This option eases tracking the end of requests in
      the system log.

    * Additional debugging flag "Debug(writer)" for debugging of writer.

    * Provided access to request AND reply header fields via
      "extendedheaders" specification (in "ns_accesslog" and "nslog"
      configuration section):

      It is now optionally possible to specify the header fields in the
      provided "extendedheaders" list with a prefix "request:" or
      "response:" to denote request and reply header fields. In case,
      no tag is provided, field names are interpreted as request header
      fields (like before). The change is fully backwards compatible,


   - ns_set:

     * Added sucommand "ns_set imerge", a is the case insignificant
       version of "ns_set merge". This option is useful especially for
       working with header fields.

     * Added subcommand "ns_set iupdate": this command is the case
       insenstive counterpart of "ns_set update".

     * Fixed "nsv_set a b" when no key nor array exist

  - ns_http:
    * Revamp of ns_http and related supportive code.
    * Added chunked-encoding parser
    * Tcl channel support for body (PUT, POST) and for result.
      This makes it possible to use e.g. reflected Tcl channels
      in "ns_http".
      New Options for "ns_http run"
        - ?-body_size size?
        - ?-body_chan chan?
        - ?-outputfile fn?
        - ?-outputchan chan?
    * Make clear that "ns_http run" is the preferred interface
      (rather than "queue" + "wait")
    * Deprecated output variables of "ns_http wait" (since everything
      is included in the resulting dict)


  - New commands:

    * ns_asynclogfile:
      Facility for thread-safe writing async
      log files of various kinds using the AsyncWriterThread.
      Usage example:
	# ... at startup
	set fd [ns_asynclogfile open /tmp/test.log]
	# ... during run
	ns_asynclogfile write $fd hello\n
	# ... at shutdown
	ns_asynclogfile close $fd

    * ns_parsefieldvalue:

      This function parses the provided field value (from an HTTP
      request or reply header field) into its parts and returns these
      in the form of a list of Tcl dicts or a single Tcl dict depending
      on parameters. The syntax of the contents of these header fields
      is specified in RFC 7230 section 3.2.6.

  - New features for existing commands/subcommands
  
     * Binary reform: Some NaviServer commands accepted previously
       implicitly binary or non-binary input, some commands used
       the flag "-binary" to denote the differences (e.g. ns_return).
       For a more uniform and expectable behavior binary input the
       explicit option "-binary" flag to the following commands:

	ns_base64decode
	ns_base64encode
	ns_base64urldecode
	ns_base64urlencode
	ns_md5
	ns_sha1
	ns_uudecode
	ns_uuencode
	ns_crypto::aead::decrypt string (for "-key", "-aad", "-iv", "input")
	ns_crypto::aead::encrypt string (for "-key", "-aad", "-iv", "input")
	ns_crypto::eckey import ("-string")
	ns_crypto::eckey sharedsecret ("pubkey")
	ns_crypto::hmac add (for "message")
	ns_crypto::hmac new (for "key")
	ns_crypto::hmac string (for "key", "message")
	ns_crypto::md add (for "message")
	ns_crypto::md hkdf (for "-salt" "-secret" "-info")
	ns_crypto::md string (for "message")
	ns_crypto::md vapidsign (for "message")
	ns_crypto::scrypt (for "-salt" "-secret")

     Note that this option is not fully backward compatible. Early
     versions of "ns_sha1" were expecting always non-binary input, some
     new version assumed binary input. The new version is now more
     compatible with AOLserver and older NaviServer versions.

     * "ns_server threads" reports now how often a thread of this pool
       was started.  This statistic can help to identify
       installations, where threads are started and stopped in a too
       eager fashion.
     * "ns_server.... stats": added "sendbodysize" and "replybodysize"
       to the dict of the per-pool statistics.
     * "ns_server ... connectionratelimit ?value?": Query or set
        the default per-connection rate limit.
     * "ns_server ... poolratelimit ?value?": Query or set
        the pool connection rate limit.
     * "ns_writer list" returns now the current transferrates per
       writer job.
       
     * "ns_writer submitfile -offset X -size Y" and "ns_writer size X"
       accept now memory units, the option ?-driver driver? was added
       to the "ns_writer" subcommands "size", and "streaming"
       (defaulting to the current driver).
       
       One can now write
            ns_writer size 150KB
        instead of
            ns_writer size nssock [expr 150*1024]




Performance Improvements:
-------------------------





Bug Fixes:
----------

   - Blueprint serializer: moved serialized objects in the blueprint
     after the namespace imports and ensemble recreators since
     constructors can call functions depending on it.

   - Include *xml* and *json* in non-binary MIME types. Note that
     there is no exact definition, what MIME types are exactly.

   - nslog:
     * Abort in error situation with an error message instead of crashing
     * Added server name in log entries to make it easier to distinguish
       messages per-server during bootup

   - Range requests:
     * Fixed potential problem with too many non-contiguous byte ranges
     * Don't silently ignore invalid syntax of range requests

   - Improved handling of already closed connections (data delivery was
     already delegated to writer threads, but driver still wants to
     return data directly; this could happen in error situations)

   - connchan:
     * Fixed potential invalid reads in (error) cases, where
       LogConnchanDebug is enabled and the callback is deleted during a
       Tcl_Eval().
     * Fixed potential problem, where call to Tcl_Eval() might clean
       structures used in the callback handler

   - Make sure, memory allocated by Tcl is returned by Tcl (important,
     when NaviServer is compiled with -DSYSTEM_MALLOC)

   - Fixed Tcl argument parsing of "ns_roll", "ns_fmttime",
     "nsv_bucket" and "ns_critsec eval".

   - Fixed bug in "nsv_set -default" overwriting pre-existing values

   - Fixed potential crashes in (desperate) error situations, where the
     connection was already closed.
  
   - Fixed leaking tmp file, when writerstraming is activated and
     streaming HTML output is used. This bug could appear, when
     "writerstreaming" is turned on in the config file (default off).
     See also: https://sourceforge.net/p/naviserver/mailman/naviserver-
	devel/thread/dd508519-1262-bd91-288a-fa30a7634224%40digital-
	concepts.com/#msg36794472

   - Made table used for enumeration values for Ns_ObjvIndex static.
     Background: This is a fix for a tricky Tcl_Obj sharing bug
     triggered by the usage of C-level IndexObjs based on volatile
     tables. Consider the following example:

        proc foo {x} {
           return [bar -value x ...]
        }

     The Tcl_Obj "x" is shared as name of argument and as a value of
     the non-positional parameter "-value". When bar
     (e.g. C-implemented) uses Tcl_GetIndexFromObj*() to lookup "x" in
     a table of options, which is volatile, the involved Tcl_Obj will
     be converted to an indexObj.

     On a call with wrong number of arguments( e.g."foo 1 2 3"), Tcl
     will try to give a nice error message, saying that "foo x" can be
     called only with one argument. When printing argument "x", it sees
     that "x" is an indexObj, and for these kinds of objects, "x" might
     be an abbreviated version of a full name. Since the table behind
     the indexObj is in the case above volatile, a crash might happen.

   - Added API call Ns_SockInErrorState() since SSL_shutdown() must not
     be called if a previous fatal error has occurred on a connection
     i.e. if SSL_get_error() has returned SSL_ERROR_SYSCALL or
     SSL_ERROR_SSL.

   - Provided local buffers for OpenSSL ERR_error_string handling to
     avoid potential race conditions (OpenSSL keeps otherwise error
     messages in static memory).

   - Crypto support:
     * New feature: scrypt Password-Based Key Derivation Function (RFC 7914)

       The scrypt function is a modern replacement for crypt and bcrypt
       and derives secret keys from a secret string. It is based on
       memory- hard functions, which offer added protection against
       attacks using custom hardware and GPU arrays.

       The function requires the compilation of NaviServer against
       OpenSSL 3.0 or newer (not yet released)

       Example from RFC 7914:
	% ::ns_crypto::scrypt -secret "password" -salt NaCl -n 1024 -r 8 -p 16
	fdbabe1c9d3472007856e7190d01e9fe7c6ad7cbc8237830e77376634b373162
	2eaf30d92e22a3886ff109279d9830dac727afb94a83ee6d8360cbdfa2cc0640

      * "ns_md" and "ns_hmac": added flag "-encoding"

      * Added options "-passphrase" to all "ns_crypto" commands
        accepting a PEM file. In principle, every PEM file can be
        protected by a pass phrase.

      * Signing and verifying signatures: "::ns_crypto::md string" can
        be used for signing and verifying of cryptgraphic signatures:

	% set sig [::ns_crypto::md string \
		     -digest sha1 \
		     -encoding binary \
		     -sign /usr/local/src/naviserver/myprivate.pem \
	             "abcdefghijklmnopqrstuxvwxyz\n"]

	% set vfy [::ns_crypto::md string \
		     -digest sha1 \
		     -verify /usr/local/src/naviserver/myprivate.pem \
		     -signature $sig \
		     "abcdefghijklmnopqrstuxvwxyz\n"]
		
   - nscgi:
     * ...


Documentation improvements:
---------------------------

  - Improved cross references between man pages
  - Added documentation to undocumented flags
  - Added more example
  - Improved spelling
  
  - The following 48 man pages were updated since the last release:
  
         admin-db.man, admin-install.man, admin-maintenance.man,
         admin-tuning.man, commandlist.man, ns_adp.man, ns_adp_ctl.man,
         ns_adp_info.man, ns_adp_parse.man, ns_asynclogfile.man,
         ns_base64.man, ns_conn.man, ns_connchan.man, ns_crypt.man,
         ns_crypto.man, ns_driver.man, ns_filestat.man, ns_hmac.man,
         ns_hotp.man, ns_http.man, ns_job.man, ns_locationproc.man,
         ns_log.man, ns_md.man, ns_md5.man, ns_parsefieldvalue.man,
         ns_parseheader.man, ns_parseurl.man, ns_rand.man,
         ns_register.man, ns_return.man, ns_server.man, ns_set.man,
         ns_sha1.man, ns_sls.man, ns_sockcallback.man,
         ns_sockselect.man, ns_tmpnam.man, ns_totp.man,
         ns_urlspace.man, ns_uudecode.man, ns_uuencode.man,
         ns_writer.man, nsv.man, returnstatus-cmds.man,
         tcl-lib-file.man, tcl-libraries.man, tcl-overview.man


Configuration Changes:
----------------------

  - Output warning to system log, when NaviServer was compiled without
    zlib support, but the config file request compression.  Before, the
    compress enable request was silently ignored, giving no hint, why
    compression attempts of a developer were ignored.

  - Sample configurations:

     * Documented usage of connection pools in sample config files.

     * Added sample entries for new features (e.g. rate limits, log
       file sanitizer)
       
     * openacs-config.tcl:

        . Changed driver installation to "global" in sample configuration
	 file. This change makes it easier for site admins to add
	 further servers (e.g. for virtual hosting).

        . Simplified sample config file by using multiple IP addresses
	 per driver

    * nsd-config.tcl:

        . Changed driver installation to "global" in sample configuration
	 file. This change makes it easier for site admins to add


Code Changes:
-------------

  - Improved scalability: provide different locks per urlspace IDs.
   
  - Improved range checking for Ns_ObjvInt, Ns_ObjvLong,
    Ns_ObjvWideInt and MemUnits:

    * built-in support for objv parser This eases specification of the
      C API, makes accepted ranges clear and provides consistent error
      messages. These change fixes several potential crashes, when e.g.
      passed-in values are used as array-indices (crash on negative
      numbers) or where the values were casted to unsigned types (very
      high unsigned values, potential memory alloc problems).

     * Fixed all cases, where Tcl_GetIntFromObj(), Tcl_GetLongFromObj,
       and Tcl_GetWideintFromObj was used without a range check.


  - Argument parser: skip processing of optional arguments when the
    number of provided arguments is equal to the number of required
    parameters.

    Background: This change allows a fully backward compatible version
    of e.g. "ns_md5 $foo", no matter what the value of "$foo" is,
    although the new version of "ns_md5" has now a "-binary" option.


  - Refactored low-level socket I/O:
    * All driver operations are now strictly non-blocking
    * Use same infrastructure for all network I/O operations
      (driver, connchan, ns_http, ...)
    * Rewritten socketfile/sendfile handling (including *BSD support)
    * Improved handling of closed and detached channels, e.g.delegated
      to writers (new macros NS_CONN_REQUIRE_CONNECTED,
      NS_CONN_REQUIRE_OPEN, NS_CONN_REQUIRE_CONFIGURED,
      NS_CONN_REQUIRE_ALL)
    * Improved protection for SIGPIPE signals.

  - New API infrastructure: Ns_DList
   
    The Ns_DList is similar to Tcl_DString, but operates on pointers
    instead of characters. Like Tcl_DString, Ns_DList pre-allocates
    some data and extends it when necessary. Ns_DList structures are
    especially useful alternative to linked lists, when usually data
    is added to the end. Since the data is kept as a dense
    (potentially growing) array, the memory locality is much better
    than with linked lists, resulting in better CPU cache hit rates.

  - Distinguish between Ns_NormalizePath() and Ns_NormalizeUrl(),
    where the first is for the file system and the latter for URLs.
    Previously, both cases were handled identically.

   - Function pointer handling:
     * Created a Tcl hash type for function pointers, since ANSI/ISO C
       forbids casting/comparison of data and function pointers. The
       new type is used whenever a hash lookup is performed for a
       function pointer.
     * Fixed all other occurrences of such comparisons.

  - Regressions testing:
    * Removed false positive
    * Switch to testing via ns_http in regression tests
      (this makes it as well possible for testing https connections)
    * Extended tests:
      + Added regression tests for "ns_rand"
      + Added nsssl tests
      + Added automatic certificate generation for test-server
      
     * The following 35 tests were added or extended since the last release
     
         adp.test, encoding.test, http.test, http_byteranges.test,
         http_chunked.test, http_keep.test, https.test, misc.test,
         ns_adp_compress.test, ns_base64.test, ns_cache.test,
         ns_conn.test, ns_conn_host.test, ns_crypto.test,
         ns_driver.test, ns_hashpath.test, ns_hostbyaddr.test,
         ns_info.test, ns_limits.test, ns_md5.test, ns_nsv.test,
         ns_pagepath.test, ns_parsefieldvalue.test, ns_proxy.test,
         ns_reflow_text.test, ns_schedule.test, ns_server.test,
         ns_serverpath.test, ns_set.test, ns_sha1.test,
         ns_urlencode.test, ns_urlspace.test, ns_uuencode.test,
         ns_writer.test, tclconnio.test, tclresp.test

  - Improved portability:
    * improved compatibility with LibreSSL (2.7 and 2.9)
    * Windows changes (Many thanks to Andrew Piskorski):
      + Fixed Ns_LogRoll() to work on Windows.
      + WSASend receives as 5th argument flags, but not a pointer to flags.
      + Improved makefiles
      + Updated _MSC_VER version numbers to include Visual Studio 2019
      + Added macro NS_INLINE to achieve higher portability with (older?)
        Microsoft compilers.

  - Added warnings about potential misconfiguration of NaviServer "tcllib"
    setting (Tcl written NaviServer modules)

  - Marked compatibility wrappers Ns_SetThreadServer() and
    Ns_SetThreadServer() explicitly as deprecated. Use
    Ns_ThreadSetName() and Ns_ThreadGetName() instead.

  - Build-system
    * Added -DSYSTEM_MALLOC to default compile flags.
    * Added nsssl to the default test target.
    * Require RSA keys of size 2048: newer versions of OpenSSL refuse to work
      with RSA keys of size 1024.
    * Improved alignment with Tcl's current .m4 file to get rid of
      "-prebind" deprecated message under macOS.
    * Improved handling of posix thread library for FreeBSD and OpenBSD

  - Use native thread_local storage for log handling when available
    (experimental).

  - Improved type cleanness for function pointers
  - Improved code locality
  - Improved error messages
  - Improved structure packing
  - Reduced variable scopes
  - Added missing "extern" declarations
  - Aligned function prototypes
  - Dropped potentially dangerous call to alloca()
  - Added typedefs for commonly used functions
  - Reduce potential dangling pointer dereferences
  - Aligned names of arguments in prototype with function definition
  - Introduced use attribute-based approach for denoting
    fall through in case statements
  - Adding more declarations for PURE and CONST functions
  - Improved configurability for clang-tidy

  - Fixed macro name-clash with PostgreSQL
  - Improved spelling



Modules:
--------

....

Re: [naviserver-devel] Naviserver crashing while loading chilkat library

[Gustaf N. <ne...@wu...>]

On 25.12.19 21:03, Maksym Zinchenko wrote:
> Hello, merry Christmas.
> Im trying to load chilkat (https://www.chilkatsoft.com/tcl.asp 
> <https://www.chilkatsoft.com/tcl.asp>) extention to my naviserver, 
> from global tcl libraries folder. Basically I have a simple tcl file 
> in tcllib folder (/opt/ns/tcl) which says load 
> /usr/share/tcltk/chilkat/chilkat.so, but its crashing my server at 
> start up without even saying why in logs. Ive tried to load it from 
> adp page and its working.
> Any idea why i cant load it globally, i want use it across all virtual 
> servers im gonna run, and dont want to load it one by one.

Dear Maksym,

I can't reproduce the problem. I've tried this with the released 
NaviServer 4.99.18
and from the tip version from bitbucket, all with Tcl 8.6. I've added 
the following file
to /usr/local/ns/tcl

$ cat  /usr/local/ns/tcl/chil.tcl
ns_log notice "============================ load /vagrant/chilkat.so "
load  /vagrant/chilkat.so
ns_log notice "============================ load /vagrant/chilkat.so DONE"

... and i see no crash. This is with Ubuntu 18.04.3 LTS. Same with my macOS.
Can it be that your installation has some Tcl version mismatch (wrong 
chilcat)?
Maybe chilkat loads some library, which conflicts with what's already 
loaded on your OS.

Below are the log-lines from the server startup with the two debug lines
from above are included.

all the best

gn


$ /usr/local/ns/bin/nsd -u nsadmin -t /usr/local/ns/conf/nsd-config.tcl -f
[26/Dec/2019:10:42:50][20504.7f7d44ebf740][-main-] Notice: OpenSSL 1.1.1  11 Sep 2018 initialized
[26/Dec/2019:10:42:50][20504.7f7d44ebf740][-main-] Notice: nsmain: enable progress statistics for uploads >= 1048576 bytes
[26/Dec/2019:10:42:50][20504.7f7d44ebf740][-main-] Notice: nsmain: NaviServer/4.99.18 (a3c8903860b5 default tip) starting
[26/Dec/2019:10:42:50][20504.7f7d44ebf740][-main-] Notice: nsmain: security info: uid=1001, euid=1001, gid=1001, egid=1001
[26/Dec/2019:10:42:50][20504.7f7d44ebf740][-main-] Notice: nsmain: Tcl version: 8.6.9
[26/Dec/2019:10:42:50][20504.7f7d44ebf740][-main-] Notice: nsmain: max files: soft limit 1048576, hard limit 1048576
[26/Dec/2019:10:42:50][20504.7f7d44ebf740][-main-] Warning: nsmain: rl_cur (1048576) > FD_SETSIZE (1024), select() calls should not be used
[26/Dec/2019:10:42:50][20504.7f7d44ebf740][-main:default-] Notice: pool default: queueLength 0 low water 0 high water 0
[26/Dec/2019:10:42:50][20504.7f7d44ebf740][-main:default-] Notice: nsd/init.tcl[default]: booting virtual server:  Tcl system encoding: "utf-8"
[26/Dec/2019:10:42:50][20504.7f7d44ebf740][-main:default-] Notice: modload: loading module nscp from file nscp
[26/Dec/2019:10:42:50][20504.7f7d44ebf740][-main-] Notice: binder: started
[26/Dec/2019:10:42:50][20504.7f7d44ebf740][-main:default-] Notice: nscp[default]: listening on [::1]:4080
[26/Dec/2019:10:42:50][20504.7f7d44ebf740][-main:default-] Notice: nscp[default]: added user: ""
[26/Dec/2019:10:42:50][20504.7f7d44ebf740][-main:default-] Notice: modload: loading module nslog from file nslog
[26/Dec/2019:10:42:50][20504.7f7d44ebf740][-main:default-] Notice: nslog: opened '/usr/local/ns/logs/access.log'
[26/Dec/2019:10:42:50][20504.7f7d44ebf740][-main:default-] Notice: modload: loading module nscgi from file nscgi
[26/Dec/2019:10:42:50][20504.7f7d44ebf740][-main:default-] Notice: nscgi: GET /cgi-bin -> /usr/local/ns/cgi-bin
[26/Dec/2019:10:42:50][20504.7f7d44ebf740][-main:default-] Notice: nscgi: POST /cgi-bin -> /usr/local/ns/cgi-bin
[26/Dec/2019:10:42:50][20504.7f7d44ebf740][-main:default-] Notice: nx::serializer version 2.3
[26/Dec/2019:10:42:50][20504.7f7d44ebf740][-main:default-] Notice: XOTcl 2.3 loaded featuring: memcount 0 profile 0 memtrace 0 assertions 1 dtrace 0 development 0
[26/Dec/2019:10:42:50][20504.7f7d44ebf740][-main:default-] Notice: Using ns_cache based on NX 2.3
[26/Dec/2019:10:42:50][20504.7f7d44ebf740][-main:default-] Notice: ============================ load /vagrant/chilkat.so
[26/Dec/2019:10:42:50][20504.7f7d44ebf740][-main:default-] Notice: ============================ load /vagrant/chilkat.so DONE
[26/Dec/2019:10:42:50][20504.7f7d44ebf740][-main:default-] Notice: adp[default]: mapped {GET HEAD POST} /*.adp
[26/Dec/2019:10:42:50][20504.7f7d44ebf740][-main:default-] Notice: tcl[default]: enabletclpages for {GET HEAD POST} requests
...

[naviserver-devel] Naviserver crashing while loading chilkat library

[Maksym Z. <siq...@gm...>]

Hello, merry Christmas.
Im trying to load chilkat (https://www.chilkatsoft.com/tcl.asp) extention
to my naviserver, from global tcl libraries folder. Basically I have a
simple tcl file in tcllib folder (/opt/ns/tcl) which says load
/usr/share/tcltk/chilkat/chilkat.so, but its crashing my server at start up
without even saying why in logs. Ive tried to load it from adp page and its
working.
Any idea why i cant load it globally, i want use it across all virtual
servers im gonna run, and dont want to load it one by one.

Thank you

Re: [naviserver-devel] nswebpush and NaviServer version

[Gustaf N. <ne...@wu...>]

The rules are as follows:

- the releases version of NaviServer and the modules tar files pm 
sourceforge
   work together
- the head version on bitbucket are likely to match for the major modules
   (not all modules are adjusted immediately to every change).

The same is also true for nswebpush: the released version from sourceforge
should work (was running e.g. a while on openacs.org), the tip versions 
are as
well working together (currently running on openacs.org). The changes 
are due
to changes in the relatively young ns_crypto support in NaviServer.

The best solution is to produce a new release of NaviServer (just a 
matter of
work, not sure, when this can sneak in.

all the best
.gn

On 23.12.19 08:46, Wolfgang Winkler wrote:
>
> Hello!
>
> I think the tip nswebpush version in the repository does not work with 
> NaviServer 4.99.18.
>
> ns_crypto::eckey sharedsecret
>
> and
>
> ns_base64urldecode -binary
>
> return an error.
>
> I think you could just change the minimal NaviServer version in the 
> Readme to 4.99.19.
>
> Regards,
>
> Wolfgang
>
> -- 
>
> *Wolfgang Winkler*
> Geschäftsführung
> wol...@di...
> mobil +43.699.19971172
>
> dc:*büro*
> digital concepts Novak Winkler OG
> Software & Design
> Landstraße 68, 5. Stock, 4020 Linz
> www.digital-concepts.com <http://www.digital-concepts.com>
> tel +43.732.997117.72
> tel +43.699.1997117.2
>
> Firmenbuchnummer: 192003h
> Firmenbuchgericht: Landesgericht Linz
>
>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel


-- 
Univ.Prof. Dr. Gustaf Neumann
WU Vienna
Institute of Information Systems and New Media
Welthandelsplatz 1, A-1020 Vienna, Austria

[naviserver-devel] nswebpush and NaviServer version

[Wolfgang W. <wol...@di...>]

Hello!

I think the tip nswebpush version in the repository does not work with 
NaviServer 4.99.18.

ns_crypto::eckey sharedsecret

and

ns_base64urldecode -binary

return an error.

I think you could just change the minimal NaviServer version in the 
Readme to 4.99.19.

Regards,

Wolfgang

-- 

*Wolfgang Winkler*
Geschäftsführung
wol...@di...
mobil +43.699.19971172

dc:*büro*
digital concepts Novak Winkler OG
Software & Design
Landstraße 68, 5. Stock, 4020 Linz
www.digital-concepts.com <http://www.digital-concepts.com>
tel +43.732.997117.72
tel +43.699.1997117.2

Firmenbuchnummer: 192003h
Firmenbuchgericht: Landesgericht Linz