naviserver-devel Mailing List for NaviServer (Page 17)

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

Dear all,

on sourceforge is a release candidate for NaviServer 4.99.20 [1].

This release was originally planned to be a small release (essentially
the movement from mercurial to git, as forced by bitbucket), but turned 
out to be
the release with the highest number of changes so far.

The short summary of changes is (see below for more details):

- Support for time units
- Subsecond granularity for all NaviServer commands
  - HTTP client request logging
  - Added support for "100 CONTINUE"
  - Improved server liveliness (management of high loads and request 
queue overruns)
  - Support for running behind a reverse proxy server
  - Driver support for listening on multiple ports
  - Improved HTTPS support (OCSP Stapling, server-side SNI)
  - Improved support for client-side HTTP requests via ns_http
    (timeout handling, binary hindling, log file for outgoing requests)
  - Improved logging for detecting long running background jobs
  - Dict support for nsvs
  - More complete set of subcommands for "ns_set"
  - Improved form-processig (especially for REST processing)
  - C-level WebSocket support
  - Improved server statistics
  - Improved scalablility (reduced lock contention, higher usage of 
rwlocks,
    significanly faster nsv_*appends, ...)
  - Improved portability: many improvements for windows, compiled with
    macOS 10.14.6, Ubuntu 20.04, Fedora Core 32, OpenBSD 6.8 (LibreSSL
    3.1.1), FreeBSD 12.2

The following people have contributed to this release:

     Andrew Piskorski
     David Osborne
     Gustaf Neumann
     Maksym Zinchenko
     Oleg Oleinick
     Zoran Vasiljevic

Below is a preliminary summary of changes.
Please test if possible. The release should be in about
one week.

Future goals:
- move towards release 5.0
- upgrade license from Mozilla 1.1/GPL -> Mozilla 2.0
   (MPL allows these upgrades, change is necessary for debian
   packaging with with OpenSSL
- change to multiple branches
   (make current master to the 4.99 branch, provide in future
   for every release an own branch to ease providing fixes
   for older releases)

best regards

-gustaf
[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.19/

=======================================
NaviServer 4.99.20, released 2021-01-XX
=======================================

  379 files changed, 25411 insertions(+), 11455 deletions(-)

New Features:
-------------

  - Ns_Time reform

    * Support for time units:
      . arguments of functions accepting time durations can be provided
        now with a time unit suffix (such as ms, s, m, d). These values
        can be written e.g. as 1.5ms, 3h and the like.
      . units can be used as configuration values wherever appropriate
      . time values wherever appropriate in documentation
      . commands using Ns_ObjvTime can be specified now with time units
      . This change is fully backward compatible for all commands
        and configuration options accepting seconds, which was normally the case.
      . All sample configuration files have been updated with time units

      . nsproxy: The only exception to the precious sentence is nsproxy,
        where durations we specified in earlier releases as
        milliseconds. This is a POTENTIAL INCOMPATIBILITY with previous
        versions. These parameters are
         * config parameters of the nsproxy module
           "evaltimeout", "gettimeout", "idletimeout",
           "recvtimeout", "sendtimeout", "waittimeout"
         * ns_proxy get: changed "-timeout" parameter to time unit value
         * ns_proxy wait: changed optional "timeout" parameter to time unit value

     * Change granularity from seconds to Ns_Time (internally and in Tcl API)
       . Added proper handling of negative values in Ns_Time (was
         broken/not implemented)
       . Upgraded scheduling machinery from seconds to Ns_Time values
       . Extended Ns_Time arithmetic (Ns_DiffTime()) to handle positive
         and negative Ns_Time values
       . Extended regression test for ns_time significantly
       . Ns_After(), Ns_ScheduleProc(), Ns_ScheduleProcEx(): change
         argument from int secs to Ns_Time
       . Marked Ns_ScheduleProc() as deprecated, Ns_ScheduleProcEx()
         should be used instead.
       . ns_schedule_proc, ns_after: support time specs with time units
         (instead of only full seconds)
       . Ns_TimeToMilliseconds(): new API call for converting positive
         and negative Ns_Time values to ms
       . Ns_DStringAppendTime(): format times in a more compact form (no
         trailing zeros)
       . Use consistently Ns_TimeToMilliseconds() to convert Ns_Time to ms
       . Ns_ObjvTimeRange, Ns_CheckTimeRange(): support to specify time
         ranges in argument handling
       . Trim trailing zeros after comma point when printing Ns_Time,
         represent as int when possible
       . "ns_info scheduled" returns now timestamps in sub-second
         resolution (with a second faction).  By this, the runtime of the
         scheduled procedure can be determined on the sub-second level.
         (potential incompatibility)
       . Added handling of "schedmaxelapsed" on the sub-second level.

  - HTTP client request logging

    NaviServer can now optionally write a logfile for outgoing HTTP
    requests initiated via "ns_http". This is important for webservers
    that are dependent on other web services for the heathiness and
    performance. The logfile is similar to the access.log and suitable
    for automated analysis. Previously, NaviServer was "blind" on that
    data. Now it is easier to provide data about the performance of
    external services (e.g. cloud services such as authentication, Google
    Firebase Cloud Messaging, Office 365, ...)

    To activate HTTP client request logging, a new config-file section
    "ns/server/$server/httpclient" was created where logroll, logrollfmt
    and the other usual logging parameter can be specified.

   - New command line option for NaviServer: "-T"

     When this option is used, NaviServer just checks the configuration
     file (specified as usual by "-t") for syntax errors.  This option
     should be used on production sites to reduce the downtime of a
     server in case of typos.

  - Added support for "100 CONTINUE"

    This feature is in HTTP since a very long time, but not widely
    supported although useful (best support is currently in curl). When
    the client sends "Expect: 100-continue", the server can check the
    header fields and provide an early feedback, whether it is useful to
    transmit the full body. This way, e.g. content which is too large can
    be rejected quickly.

  - Improved server liveliness under high load (improved and configurable
    handling of connection queue overrun)

    When NaviServer processes an HTTP request, it is received (head and
    body) and associated with a connection thread pool. If the thread
    pool has a thread free, the request is directly assigned to this
    thread. Otherwise, the request is added to a queue of the connection
    pool, waiting for a connection thread.  Since the queue has a limited
    size (limited by "maxconnections"), this queue might overrun. In this
    situation, the received request is added to a waiting list. Here is,
    where the behavior is now improved.

     1) previously, as soon as there were entries in the waiting list, no
        more fresh requests were accepted by the driver (just working on
        waiting requests). While this behavior is sensible if only one
        connection pool is available, it causes a blocking behavior in
        cases, when a single connection pool overruns but others could
        continue. Now, more fresh requests are still accepted in this
        situation.

     2) In addition, it is now possible to configure the overrun behavior
        via two pool parameters:
        a) pool parameter "rejectoverrun": when set to true, NaviServer sends on
           queue overruns an error "503 service unavailable". The overflowing
           request is not added to the waiting list.
        b) when (a) is set, it is possible to specify parameter "retryafter"
           with a seconds interval to add reply header field "Retry-After" to the
           503 response. (see https://tools.ietf.org/html/rfc7231#section-7.1.3)

    When "rejectoverrun" is false (default) then behavior is similar to
    before, accept the server does not fully block. The request is kept in
    the waiting list, which is checked, whenever a request finishes. For
    the client, there is no need to resubmit the request, which will be
    proceed eventually. Technically, this means an arbirary large queue.

    When "rejectoverrun" is true, then the request causing the overrun is
    rejected, and will have to be resubmitted by the client (maybe
    automatically, when "Retry-After" is specified. However, the client
    support for "Retry-After" is rather bad.

    For busy sites, "rejectoverrun" is probably the better behavior,
    since no background queuing in the waiting list is performed. For
    now, we keep the old behavior as default for better backwards
    compatibility.  The default might change in the future.

  - General support for running NaviServer behind a reverse proxy server

    This change provides general support for running NaviServer behind a
    reverse proxy.  Previously, the support was rather punctual (e.g. for
    nslog). Reverse proxy mode is activated by setting the global
    parameter (in section "ns/parameters") named "reverseproxymode" to
    true (default false).

    Current limitation: In the current implementation, NaviServer supports
    only the "X-Forwarded-For" header field, which is supported by most
    proxy servers, treating the first provided IP-address as the client
    IP address (see https://en.wikipedia.org/wiki/X-Forwarded-For). The
    newer "Forwarded" header field is currently not supported.

    In the reverse proxy mode, the client address is taken as forwarded
    by the reverse proxy- This effects:

    * Reported IP address by the following commands.

           ns_conn peeraddr
           ns_server all|running
           ns_writer list
           ns_connchan list

    * Reported IP address in the access.log

    * Context filters (used e.g. for mapping requests to pools)

      In reverse proxy mode, all IP-address specific context filters
      refer to the x-forwarded-for addresses. This makes it possible to
      use context filters (of URLspace commands) based on IP-addresses
      (IPv4, IPv6, with masks) also behind reverse proxy
      servers. Previously the IP-based mapping was just possible when no
      reverse proxy was in use.

  - Allow a single driver to listen on multiple ports:

    This new feature simplifies setups, where a single server is
    listening on multiple ports using the same driver. Previously, it was
    necessary to define separate drivers for such cases, which need
    different names but which are often configured identically.  Together
    with the change introduced in 4.99.19 (support for multiple listen
    addresses) this means that when multiple addresses and multiple ports
    are specified for a single driver, it will listen on the cross
    product of all combinations. The maximum number of listening ports of
    a driver is limited by a compile time constant to
    MAX_LISTEN_ADDR_PER_DRIVER (per default 16).

    To use this feature, specify in the configuration file for "port" in
    the network driver section a space separated list of multiple
    ports. This change is fully backwards compatible, old configuration
    files will continue to work.

  - SSL/TLS improvements:

    * Support for OCSP Stapling

      . NaviServer obtains and provides information about the validity
        of the used certificates. For proving this information the
        server performs two level of caching: in-memory caching and
        disk caching. When the server receives the first TLS request
        with OCSP stapling turned on, it checks for an already
        retrieved OCSP response. The disk cache file is saved in the
        "log" directory of the server and uses the serial number of the
        certificate to be checked as filename with ".der" as
        extension. When the disk cache file does not exist, an
        http/https request is made to the server issuing the servers
        certificate as defined by the Authority Information Access
        (AIA) Extension. The names of the file and the http/https
        request for the OCSP response can be obtained from the system
        log of the server

      . OCSP Stapling can be turned on in the configuration files (see
        sample configuration files)

      . OCSP Stapling requires OpenSSL 1.1.0 or newer

   * Support for server-side SNI (Server Name Indication (SNI)

      Server-side SNI can be used to use different certificates for
      virtual hosts. In addition to the top-level certificate, different
      certificates can be defined per virtual server (below: srv2). The
      certificate will be registered for all alternate names specified
      for the server.

      In addition to the certificate, also all other OpenSSL specific
      parameters (ciphers, protocols, ...) will be loaded form the same
      place as the certificate. The following example shows the setup for
      two virtual servers "srv1" and "srv2", where the first one is the
      default server.

         ns_section ns/modules {
           # ...
           ns_param        nsssl           ${bindir}/nsssl.so
         }
         ns_section ns/module/nsssl {
           # ...
           ns_param   defaultserver   srv1
           ns_param   certificate     /usr/local/ns/modules/nsssl/srv1.pem
         }
         ns_section "ns/module/nsssl/servers" {
            ns_param   srv1            test.org
            ns_param   srv1            www.test.org
            ns_param   srv2            someotherhost.org
         }
         ns_section "ns/server/srv2/module/nsssl" {
           # ...
           ns_param   certificate     /usr/local/ns/modules/nsssl/srv2.pem
           ns_param   ciphersuites    TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
         }

      SNI support includes OCSP, DH ephemeral key exchange, reuse of SSLcontexts.

    * Support for OpenSSL "ciphersuites" (TLSv1.3 feature)

    * Support for client-side SNI: new parameter "hostname" for
      "ns_connchan open"

    * Support for removing TLSv1.0, TLSv1.1, TLSv1.2, and TLSv1.3 via
      configuration file

  - ns_http:

     * Clear separation of "-timeout" and "-expire":
        . "-timeout" is used for single communication acts (e.g. connect, read write)
        . "-expire" is used for limiting the full duration of the
          request (also useful for slow read/write attacks).

       Scripts already using the -timeout will most probably not see
       any change unless they are handling huge up/downloads. In such
       cases we will now continue with the request, instead of
       terminating it in the middle.

     * added "-binary" flag to transmit data binary and perform no
       automatic encoding transformations.

  - moved code repository on BitBucket from mercurial to git.
    * adjusted documentation
    * adjusted build system (Makefiles)

  - Logging improvements:

    * New log severity "Debug(timeout)" to force NaviServer to write an entry to the
      systemlog, whenever a timeout occurs. Therefore, the command

         ns_logctl severity Debug(timeout) on

      can be used to debug cases, where NaviServer runs into HTTP status code 503
      (Service Unavailable).

    * Logging of commands with high latency:

      Provide a consistent interface and naming convention for logging
      slow commands for nsproxy, ns_jobs, scheduled procedures and SQL
      (nsdb drivers).

      . nsproxy and nsjobs have a new configuration option "logminduration"

        This configuration option specifies a time-limit for logging
        (long) eval operations to the system log (similar to
        "logminduration" in the database drivers). Set it to a high value
        to avoid logging (e.g. 1d). The default is 1s.

      . scheduled procedures:
        Changed "schedmaxelapsed" to "schedlogminduration" for consistency.
        This is a potential incompatibility for old configurations.

  - New commands:

    * "nsv_dict":

       The new command "nsv_dict" provides thread-safe support for
       setting, getting, updating, and unsetting dict values in nsv
       array.

       Motivation: Some larger OpenACS installations have currently more
       than 1 mio (!) nsv arrays, which comes mostly from creating nsv
       arrays per packages. Most of these arrays have only a few keys
       with very short values. The number of arrays can be greatly
       reduced by supporting complex values (dictionaries), similar to
       REDIS hash values.

       . "nsv_dict" supports nested dicts like in Tcl

       . The following dict commands:

           nsv_dict append  array key dictkey dictvalue:0..n
           nsv_dict exists  array key dictkey:1..n
           nsv_dict get     array key dictkey:0..n
           nsv_dict getdef  array key dictkey:0..n default
           nsv_dict incr    array key dictkey ?increment?
           nsv_dict keys    array key ?pattern?
           nsv_dict lappend array key dictkey dictvalue:0..n
           nsv_dict set     array key dictkey:1..n dictvalue
           nsv_dict size    array key
           nsv_dict unset   array key dictkey:1..n

      . added 41 test cases in regression test for nsv_dict

    * "ns_dbquotelist": similar to "ns_dbquotevalue", but provides
      quoting for Tcl lists.

    * "ns_trim": multiline trim command:
       ns_trim ?-subst? ?-delimiter delimiter? text

  - New features for existing commands/subcommands

    * New subcommands "keys" and "values" for "ns_set":

         "ns_set keys /setId/ ?pattern?"
         "ns_set values /setId/ ?pattern?"

      The commands are similar to the Tcl command "dict keys" and "dict
      values" and can reduce the necessity for loops.

    * Form processing:

       . "ns_getformfile": return a list of filenames when "multiple"
         attribute is provided in the input field and multiple files are
         submitted

       . Decouple content-type processing from HTTP methods:

         Previously, parsing of form data in these two formats happened
         only on POST requests. However, several REST services use often
         also PUT with form data and distinguish POST and PUT methods
         mostly by "add item to a collection" (POST) and process single
         item (PUT) but not by their content-type processing. This change
         eases implementation of REST services.

       . Perform more strict parsing based on "ns_parsefieldvalue" and
         "ns_parseheader" in file-based form parser (for spooled
         content).  Previously, there was a bug with the regexp-based
         parsing, when e.g. a filename of and upload file contained
         e.g. a semicolon.

    * "ns_conn":
       . new subcommand "headerlength"
       . "ns_conn auth" returns now Information about "Bearer"
         authentication (RFC 6750) when applicable, used e.g. in OAuth2
       . "ns_conn peeraddr": New flag "-source" for specifying detailed
         information about the peer, especially useful in combination
         with reverse proxy mode; allowed values are "configured",
         "direct", "forwarded".

    * "ns_cookie": Support explicit setting of "same-site=none"

      In previous version of browsers, no explicit setting
      of the samesite flag was exactly the same as explicit setting.
      Since some browsers switched to a default of "lax", explicit
      setting became necessary.

    * "ns_dbquotevalue": provided a C implementation; was scripted
      before.

    * "ns_connchan":
       . Added option "-server" to "ns_connchan close" to be able
         to cleaning stale handles of every server.

       . Added C-level WebSocket support and buffered output handling.
         The new code is about 200x faster than the Tcl version
         (especially when bytewise XORs are needed), simpler to use (no
         Tcl-leval handling of incomplete frames, split frames, remaining
         leftover from partial writes) and more complete (handling of
         segmented WebSocket messages).

       . "ns_connchan read -websocket ....": returns a dict containing
         the websocket data (including payload, websocket status bits,
         buffering state, etc.; see documentation and NaviServer
         websocket module for details)

       . "ns_connchan write -buffered ..." handles partial writes via an
         internal buffer (Tcl developer has just to take care about
         writable callbacks when necessary)

       . "ns_connchan wsencode ...": WebSocket frame generation, returns
         binary data based on input parameters suitable for sending on a
         WebSocket channel.

       . "ns_connchan status ..." returns a dict containing the buffering
         status of a channel.

    * "ns_urelencode": added one more option "oauth1" for the "-part"
      argument. The "oauth1" encoding is defined in RFC 5849 Section 3.6
      for the construction of the signature base string and the
      "Authorization" header field.

    * "ns_ictl":
       . "ns_ictl trace idle ...": new tracetype "idle", which fires,
         when a connection thread is idle for the threadtimeout period,
         but minthreads is already reached, and the thread won't
         terminate.

       . "ns_ictl maxconcurrentupdates ?max?": Configuration option
         for specifying the maximum number of concurrent updates
         for interpreter releads (when epoch incrases, e.g. reloads
         on OpenACS).

         Background: For large (say 50-100 connection threads) and busy
         sites (sustained 800 requests per second) with large blueprints
         (e.g. OpenACS) blueprint updates can cause delays when all
         blueprint are currently updates. The blueprint updates happen
         when the epoch is incremented (e.g. in OpenACS: reloads of
         packages).  A too high number of concurrent blueprint updates
         cases (a) delays in request processing (b) increased update
         times per thread (e.g. up to 10 seconds). As a consequnce,
         queueing will happen.  This new option limits the number of
         concurrent updates (e.g. to 10) leading to smooth operations in
         such situations. When the maximum number of concurrent blueprint
         is reached, the pending updates are delayed until the limit is
         sufficiently low. By default, the value is unlimited.

    * Improved statistics:
      . "ns_server stats" returns now as well dropped requests from queue
        overruns.

      . "ns_info locks": report number of read and write operations for
        rwlocks.

      . "ns_proxy stats": report also the number of running worker
         processes.

    * "ns_getcsv":
      . provide compatibility with RFC 4180
      . accept multi-line input
      . handle quotes in quotes and empty elements
      . optionally stripping unquoted elements (not covered by RFC)

      . optional support for a different quote character (option
        "-quotechar"). Although RFC 4180 specifies double quotes as
        quoting chars, sometimes also different quoting characters,
        which can be now successfully decoded.

    * "ns_parseurl": Support for fragment parsing.  When provided URL
      contains a tail element, this is now parsed info a "query" and/or
      "fragment" component in the result. Previously, the query and
      fragmet part was included in the "tail" element.

    * Improved thread naming for log-files: previously, it was not clear,
      from which server some output of the "main" thread originated.
      The new version adds always a server name specific suffix.

    * "ns_driver info": return also "port" and "defaultport"

Bug Fixes:
----------

  - Improved support for newer versions of OpenSSL (handling of
    SSL_R_UNEXPECTED_EOF_WHILE_READING)

  - ns_cache: Base base cache size calculation on the full size of a
    cache entry.

    In previous versions, the cache size calculation was based only on
    the value size, leaving the size of the key and the size of the
    overhead (size of cache entry, about 126 bytes on a 64-bit machine)
    aside. This lead to misleading interpretations, when the cache value
    size is 0 bytes (just using the cache key as a flag), leaving admins
    wondering about the memory consumption.

  - Report to log file, when ns_ictl traces are not executed. Previously,
    these were silently ignored (e.g. in some error cases).

  - Fixed a bug with PUT methods, when query-parameters were passed in
    addition to the put content.  Previously, these parameters were only
    available via [ns_queryget ...] when the content file was not
    spooled.

  - Fixed bug for custom error pages for HTTP methods other than GET
    (Custom error handles are defined as before via redirect sections):
     * Always use GET in the redirection not the original HTTP method.
       When e.g. "PUT /somepage" is requested, where no PUT handler
       but a custom error page /405.tcl, the previoud code tried a
       PUT /405.tcl, leading the same error. Now, the error
       page us delivered like with a GET request.
     * Provided a hint as a warning, where NaviServer searches
       for non-existing pages.
     * Extended regression test: cover HEAD requests for existing
       and non-existing pages, and various forms of redirects

  - Fixed issue noted by Maksym Zinchenko on the naviserver developer list

    A recent change of tDOM 0.9.2 introduced a namespace local command
    "namespace" which was picked up by the blueprint generator
    incorrectly instead of the global ::namespace command.  As a fix, all
    "namespace" invocations in the blueprint generator were replaced by
    "::namespace" to avoid considering invocation context for every
    occurrence.

  - Fixed potential abortion of regression testruns under macOS

    Under macOS, tests runs were sometimes abruptly ended with a
    broken pipe signal. The call "NsBlockSignal(NS_SIGPIPE);" does
    not avoid these signals. So we ignore bluntly all such
    signals by using "NsBlockSignal(NS_SIGPIPE);"

  - Fixed charset handling for spooled requests.

  - Fixed several cases of invalid chunked data (potential buffer
    overflows).

  - Don't swallow errors in braced config sections.

  - Blueprint generation:
     * Fixed bug with interp aliases defined without fully qualified
       namespace path
     * Improve robustness when "rename" is used (when source exists and
       target does not exist)

  - Support calling "ns_conn status" (with the argument setting the HTTP
    return status) in traces (without open connection).

  - Make code more robust in case Tcl and NaviServer were compiled
    with different memory allocators.

  - Race Conditions:
     * Fixed several race conditions in "ns_http"
     * Fixed race conditions in "ns_connchan list" and "ns_connchan create"

  - Use time_t for secs in Ns_Time, since long is just 32-bit on 64-bit
    windows, but time_t is 64 bits.

  - "ns_writefp": fixed to act as advertised (read to end of file when no
    length is provided)

  - "ns_sendmail": Convert leading "." in message bodies to ".." as
     required by RFC 5321 section 4.5.2

  - "ns_cache" transactions: in certain situations, where during
     concurrent entry creates, and one of the create attempts failed and
     the other succeeded, and the one of the cache transitions is rolled
     back, updates could lead to an error.

  - "ns_connchan": Fixed registry of connchans from all servers to the
     default server (not the best scalable solution, but useful as long
     we have only one "socks" thread (for more details see:
     https://sourceforge.net/p/naviserver/mailman/message/37179476/)

Performance improvements:
-------------------------

   - nsv_lappend:

     The old implementation showed a linear slowdown for nsv_lappend
     operations depending on the number of elements in the list. The
     new implementation avoids frequent string to list and vice versa
     operations by using Tcl_DStrings. It is essentially independent on
     the number of elements in the nsv.  For a list with 10K elements,
     the performance improvement is about a factor of 400.

     Test:
        nsv_set d 1 1
        time {nsv_lappend d 1 1} 10000

     OLD:
        403.7255 microseconds per iteration

     NEW:
        0.860547 microseconds per iteration

    This changes makes it possible to use the OpenACS developer support
    on busy sites (before, this lead to a quick standstill).

   - nsv_append:

     The performance gain is much less than with nsv_lappend (see
     previous commit), but still noticeable (about 35% faster, 10K
     ns_append operatiopns went from 0.94 microseconds to 0.62

   - Locking:
     * Reduced mutex-locked regions in driver.

     * Use ptrhreads interface for Ns_RWLock* when available

       So far, Ns_RWLock* used an implementation based on mutexes and
       cond vars. This implementation has the advantage of being
       portable. However, since on all *nix platforms, NaviServer uses
       the pthread library, using the POSIX rwlock implementation poses
       no additional dependency.

       Performance comparison fur using mutex/old rwlocks/pther rwlocks
       on ns_urlspace;

             macOS (8 cores) rwlock (POSIX)  0.3105    2,27 %
                             rwlock (ns)     11.5263  84,23 %
                             mutex (ns)      13.6839 100,00 %

             Linux (6 cores) rwlock (POSIX)  0.6081   43,38 %
                             rwlock (ns)     3.6259  258,66 %
                             mutex (ns)      1.4018  100,00 %

       This data shows that the POSIX rwlocks implementation
       (pthread_rwlock_rdlock etc.)  is always better than the classical
       NavisSrver rwlock implementation, and that significant
       improvements can be achieved by using rwlocks. A more detailed
       performance comparison was provided on the developer mailing list.

       This change introduces:
       a) Ns_RWLock* based on pthreads when available
       b) switch to recommended configure setup for pthreads (AX_PTHREAD)
       c) New API call: Ns_RWLockSetName2() modeled after Ns_MutexSetName2()
       d) New API call: Ns_RWLockList() modeld after Ns_MutexList()
          (removed dependency of rwlocks to mutex statistics)

    * Support rwlocks for nsvs:

      Per default, rwlocks are used for all nsv* commands.  The default
      behavior can be altered by setting the configuration variable
      "nsvrwlocks" to false in the section ns/server/${server}/tcl of
      the configuration file.

      This change can lead to improved scalability especially on
      multi-core machines since in read cases, the variables can be
      accessd unblocked for parallel executions. In most applications nsv
      are typically more often read than written rwlocks offer better
      scalability for large sites.

      The following table compares the busy locks with mutex operations vs
      the number of busy locks with rwlocks for openacs.org:

        With mutex (24h)
                                     locks   busy
        nsv:3:openacs.org    27      4.71M   1.3K
        nsv:6:openacs.org    24      4.88M   1.03K
        nsv:2:openacs.org    28      3.37M   784
        nsv:7:openacs.org    23      9.11M   755
        nsv:5:openacs.org    25      2.88M   460

        With rwlocks (24h)
        nsv:7:openacs.org    1       7.22M   0
        nsv:6:openacs.org    2       3.92M   143
        nsv:3:openacs.org    5       3.31M   1
        nsv:2:openacs.org    6       2.23M   16
        nsv:5:openacs.org    3       2.16M   0

      On more busy sites, the improved scalability si much higher.

    * rwlocks are better in cases, where the number of readers is
      significantly higher than the number of writers.  So, it has to be
      carefully considered in which cases rwlocks should be preferred.
      The new version uses rwlocks for ns_connchan, filters and tcl.cache.

  - Avoid multiple library initializations for OpenSSL 1.1.0 or newer.

Documentation improvements:
---------------------------

  - Update references to Tcl website (replace http://tcl.tk by
    https://core.tcl-lang.org)

  - Added hints about required OpenSSL version to man pages

  - Made man pages more consistent
  - Consistent indentation of program examples in documentation
  - Made source code documentation more consistent

  - Improved source code documentation (provide more details, remove
    obsolete comments)

  - Removed deprecated man pages
  - Removed obsolete sections in man pages
  - Follow the recommended spelling convention of the Linux documentation
    project
  - Improved spelling, typesetting, and cross references
  - Reduced technical jargon
  - Made code listings in documentation more consistent (e.g. indenting)
  - Added missing commands in command list (overview page)
  - Updated listing of deprecated commands and suggested alternatives
  - Replaced several "slave process" by "child process"

  - 161 man pages were updated since the last release

Configuration Changes:
----------------------

  - New configuration options:

     * Section "ns/parameters"

       . "reverseproxymode": Activate reverse proxy mode.  The former
          configuration parameter of "nslog" named "checkforproxy" is
          obsolete now.
       . "joblogminduration": report jobs taking longer in systemlog
       . "schedlogminduration": report scheduled jobs taking longer in systemlog

     * Section "ns/server/${server}"

       . "rejectoverrun": when true send 503 when queue overruns;
          default "false"
       . "retryafter": time for Retry-After in 503 cases

     * Section "ns/server/${server}"

       . "nsvrwlocks": use RWloccks for nsv (default: true)

     * Section "ns/server/$server/httpclient"
       . "logging": (default: off)
       . "logfile": name of http client logfile
       . "logrollfmt": format appended to log filename
       . "logmaxbackup": max number of backup log files (default: 10)
       . "logroll": perform automatic rolling (default: true)
       . "logrollonsignal": (default: false)
       . "logrollhour": specify at which hour to roll (default: 0)

  - Sample configurations:

     * Added values with time units where appropriate
     * Added new values mentioned above with short comments

     * For multi-server configuration files: use always the first (and
       not the last) server as default server.

     * openacs-config.tcl:
     * sample-config.tcl:

        . fixed config parameter: use "dnscachetimeout" instead of "keepwaittimeout"
          (name was changed in Naviserver ages ago, but change was not
          reflected in config files

    * nsd-config.tcl:

    * win32/test/nsd.tcl
      modernized test config file

    * nsproxy: Changed term "maxslaves" to "maxworkers" (the old term
      is deprecated)

Code Changes:
-------------

  - Fixed potential access to a out-of-scope stack variable.
  - Added more declarations for PURE and CONST functions
  - Replaced usage of reserved identifier [cert-dcl37-c,cert-dcl51-cpp]
  - Replaced sscanf() by strtol*() to increase code-safety
  - Improved structure packing

  - Marked "ns_adp_mime" explicitly as deprecated, and provide deprecated
    messages for it.  The command was deprecated more than 10 years ago,
    ns_adp_mimetype should be used

  - Added mime types for "heif" and "heic" which have become a preferred
    image type for iOS and macOS.

  - Regressions testing:

    * Reduced usages of old-style nstest::http-0.9 (socket and not
      ns_http based tests)

    * Make test more robust concerning crlf results (Windows)

    * Extended tests:
      367 tests were added since the last release
      (changes in 38 test files)

  - Improved logging:
     * Added metering for long wait times of Ns_CsEnter locks
       (e.g. Ns_MasterLock)
     * Added concurrency level to log message of interp updates
     * Added warnings, when an NsTclRequestProc or an ADP request runs
       into a timeout.

     * Reduced time sensitivity on time when rolling log files_ Rolling
       log files happens often at midnight, using often a day
       precision. When e.g. a scheduled procedure the time when this
       function is called might be slightly after the scheduled time,
       which might lead to a day jump. The problem aggrevates, when
       multiple log files are rotated.  The new code identifies day wraps
       and uses in such cases the earlier day.

  - Improved portability:
    * Improved compatibility with the forthcoming OpenSSL 3.0
    * Improved compatibility with various versions of LibreSSL
    * Improved Windows (MSVC) compatibility
      . simplified Makefile.win32 for nmake
      . fixed thread exit result handling for 64-bit Windows
      . fixed thread handle access
      . turned on OpenSSL by default
      . minimal support for running regression tests under windows
      . Fixed windows size incompatibility for range requests (64-bit
        windows uses a 32-bit sized offset type off_t)
    * Use consequently NS_EAGAIN (can be used in *nix and Windows)
    * Fixed potential problem with GNUC_PURE for older version of gcc
    * Fixed regression test warnings with older versions of curl for IPv6 addresses
    * Fixed m4 configuration for "crypt_r" (broken since many years)
    * Compiled with macOS 10.14.6, Ubuntu 20.04, Fedora Core 32, OpenBSD
      6.8 (LibreSSL 3.1.1), FreeBSD 12.2

  - More OpenSSL changes:

    * Reactivate CRYPTO_set_mem_functions() for OpenSSL 1.1.0 or newer
      (interface om OpenSSL has changed)

    * Switched to SSL_CTX_set_dh_auto() from low-level functions: In
      newer versions of OpenSSL, all low-level DH* functions were
      deprecated.

  - Build-system
    * compute version tag from git slimilarly like to the mercurial version

  - improved support for MS windows
    * Windows math.h does not include round(), add simple implementation.

  - Added cross-platform function ns_localtime_r (localtime with
    user-provided storage): The previously defined function
    ns_localtime() uses a single per-thread local storage, so, it was not
    possible to have in one thread two time buffers with different
    values.

- Improved error code handling on driver receive operations

    Previously, higher level of API calls were not able to access
    realiably the error information detected on the lower-level (async
    I/O-based or OpenSSL based) driver operations. This made it
    impossible to provide good error messages e.g. in "ns_connchan" and
    we could not provide error processing based on Tcl error
    codes. Furthermore, the new interface works for POSIX errnos and
    OpenSSL errors and improves the situation (currently just for driver
    read operations).

    * Ns_Sock* based setting of Tcl error codes:

        NsSockSetRecvErrorCode(const Sock *sockPtr, Tcl_Interp *interp)

      Set the Tcl error code in the interpreter based on the
      information provided by Sock* (actually by recvErrno); the
      function works for OpenSSL and plain POSIX style error codes.

    * Setting Tcl error codes for POSIX errno:

         Ns_PosixSetErrorCode(Tcl_Interp *interp, int errorNum)

      Set the Tcl error code in the interpreter based on the POSIX
      errorNum and return the error message.

    * Setting Tcl error codes for OpenSSL error values

        Ns_SSLSetErrorCode(Tcl_Interp *interp, unsigned long sslERRcode)

      Set the Tcl error code in the interpreter based on the OpenSSL
      error value and return the error message.

    * Return the last error from a socket.

         Ns_SockErrorCode(Tcl_Interp *interp, NS_SOCKET sock)

      The function Ns_SockErrorCode() returns the last error from a
      socket. When the provided interp is not NULL it sets the Tcl error
      code based on the error code.

Modules:
--------
### AAA files changed, BBBB insertions(+), CCCC deletions(-)

nsdb:
   added support for time units for nsdb LogMinDuration

nsdbmysql:

nssmtpd:

nsdns:

nsudp:

nszlib:

nsimap:

nsphp:

nsstats:

nsdbi:

nsdbipg:

nsoracle:

nswebsocket:

revproxy:

letsencrypt:

nswebpush:

nsladp:

2005	Jan	Feb (53)	Mar (62)	Apr (88)	May (55)	Jun (204)	Jul (52)	Aug	Sep (1)	Oct (94)	Nov (15)	Dec (68)
2006	Jan (130)	Feb (105)	Mar (34)	Apr (61)	May (41)	Jun (92)	Jul (176)	Aug (102)	Sep (247)	Oct (69)	Nov (32)	Dec (140)
2007	Jan (58)	Feb (51)	Mar (11)	Apr (20)	May (34)	Jun (37)	Jul (18)	Aug (60)	Sep (41)	Oct (105)	Nov (19)	Dec (14)
2008	Jan (3)	Feb	Mar (7)	Apr (5)	May (123)	Jun (5)	Jul (1)	Aug (29)	Sep (15)	Oct (21)	Nov (51)	Dec (3)
2009	Jan	Feb (36)	Mar (29)	Apr	May	Jun (7)	Jul (4)	Aug	Sep (4)	Oct	Nov (13)	Dec
2010	Jan	Feb	Mar (9)	Apr (11)	May (16)	Jun	Jul	Aug	Sep (1)	Oct	Nov	Dec
2011	Jan	Feb	Mar	Apr	May (1)	Jun	Jul	Aug	Sep	Oct	Nov	Dec
2012	Jan (7)	Feb (3)	Mar	Apr	May	Jun (3)	Jul	Aug	Sep	Oct (92)	Nov (28)	Dec (16)
2013	Jan (9)	Feb (2)	Mar	Apr (4)	May (4)	Jun (6)	Jul (14)	Aug (12)	Sep (4)	Oct (13)	Nov (1)	Dec (6)
2014	Jan (23)	Feb (19)	Mar (10)	Apr (14)	May (11)	Jun (6)	Jul (11)	Aug (15)	Sep (41)	Oct (95)	Nov (23)	Dec (11)
2015	Jan (3)	Feb (9)	Mar (19)	Apr (3)	May (1)	Jun (3)	Jul (11)	Aug (1)	Sep (15)	Oct (5)	Nov (2)	Dec
2016	Jan (7)	Feb (11)	Mar (8)	Apr (1)	May (3)	Jun (17)	Jul (12)	Aug (3)	Sep (5)	Oct (19)	Nov (12)	Dec (6)
2017	Jan (30)	Feb (23)	Mar (12)	Apr (32)	May (27)	Jun (7)	Jul (13)	Aug (16)	Sep (6)	Oct (11)	Nov	Dec (12)
2018	Jan (1)	Feb (5)	Mar (6)	Apr (7)	May (23)	Jun (3)	Jul (2)	Aug (1)	Sep (6)	Oct (6)	Nov (10)	Dec (3)
2019	Jan (26)	Feb (15)	Mar (9)	Apr	May (8)	Jun (14)	Jul (10)	Aug (10)	Sep (4)	Oct (2)	Nov (20)	Dec (10)
2020	Jan (10)	Feb (14)	Mar (29)	Apr (11)	May (25)	Jun (21)	Jul (23)	Aug (12)	Sep (19)	Oct (6)	Nov (8)	Dec (12)
2021	Jan (29)	Feb (9)	Mar (8)	Apr (8)	May (2)	Jun (2)	Jul (9)	Aug (9)	Sep (3)	Oct (4)	Nov (12)	Dec (13)
2022	Jan (4)	Feb	Mar (4)	Apr (12)	May (15)	Jun (7)	Jul (10)	Aug (2)	Sep	Oct (1)	Nov (8)	Dec
2023	Jan (15)	Feb	Mar (23)	Apr (1)	May (2)	Jun (10)	Jul	Aug (22)	Sep (19)	Oct (2)	Nov (20)	Dec
2024	Jan (1)	Feb	Mar (16)	Apr (15)	May (6)	Jun (4)	Jul (1)	Aug (1)	Sep	Oct (13)	Nov (18)	Dec (6)
2025	Jan (12)	Feb	Mar (2)	Apr (1)	May (11)	Jun (5)	Jul (4)	Aug	Sep	Oct	Nov	Dec

naviserver-devel Mailing List for NaviServer (Page 17)

NaviServer, a high performance web server written in C and Tcl

naviserver-devel — Developer discussion