[naviserver-devel] Towards NaviServer 4.99.24

[Gustaf N. <ne...@wu...>]

Dear all,

on sourceforge is a release candidate for NaviServer 4.99.24 [1].

Please test if possible.  The release should be in the near future.
Below is a preliminary summary of changes.

All the best, and have a nice easter weekend!
-g

[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.24/


=======================================
NaviServer 4.99.24, released 2022-04-XX
=======================================

  57 files changed, 1840 insertions(+), 824 deletions(-)

New Features:

  - Improved security

    * Added protection against certain attacks in ns_dbquotevalue
     
      Due to the corrected conversion to external UTF-8 in db-output,
      new potential attack vectors appeared that were protected earlier
      via the Tcl-internal 'modified UTF-8'. E.g., the binary null
      character is stored as an overlong (two-byte) encoding of null
      (0xc0 0x80), so that an actual (embedded) null byte (0x00) never
      appears in the string. Due to the conversion, the internal
      representation is translated back to the binary null
      character. Embedded null byte characters can lead to
      non-terminated string literals via ns_dbquotevalue.
     
      In the updated version of NaviServer, ns_dbquotevalue raises an
      exception when this occurs. Therefore, the function can be used
      as well as an input checker (together with "try").

    * Raise an exception when trying to use "ns_urldecode" to
      produce invalid UTF-8
     
      Background: several (external) functions expect valid UTF-8 to be
      passed in and crash if this is not the case. One such example is
      tDOM. These nasty byte sequences are used more intensively by
      vulnerability scanners.
     
      Therefore, ns_urldecode raises now an exception, when it tries to
      convert to invalid UTF-8. It is still possible to use
      ns_urldecode to convert to other charsets.
     
         ns_urldecode -charset iso8859-1 -part path "/mot%C3or"
     
      When urldecode() is called internally and would produce invalid UTF-8,
      it truncates the string (and writes a warning to the system log).
     
  - Provide a hint when cache-entry was too large for caching
     
    Background: the size of the entry is typically determined after the
    execution of a potentially expensive query. During the eval of the
    command, the cache entry is locked and forces a
    serialization. However, this means that in these cases the
    situation is worse than without a cache, where some queries can be
    executed in parallel. We faced the situation of an expected
    slowdown of the server with many "create entry collision", where
    due to application matters, an entry was becoming too large. This
    situation is not easy to debug, especially in stress
    situations. The log message would have helped a log to identify the
    cause.

  - Added support for multibyte numeric entities
     
    This change supports conversion of numeric entities representing
    multibyte characters into HTML in "ns_striphtml" and
    "ns_unquotehtml". Technically, the numeric entities represent
    Unicode code points, which are transformed into UTF-8
    serialization.  Every entity represents a single code point; The
    values can be provided in decimal or hexadecimal notation. Before
    this change, only single byte numeric entities were
    supported. ASCII control characters (decimal 0-31) are ignored as
    before.

  - New and extended commands:

    * ns_unquotehtml /text/
     
      This command is the inverse operation of "ns_quotehtml". It
      replaces the named and numeric entities in the provided string
      with the native values. The command is similar to "ns_striphtml",
      but "ns_striphtml" removes as well other HTML markup which might
      not be desired in all cases.
     
      This change fixes as well a bug with numeric entities (the old
      code assumed, these are starting directly with a number after the
      ampersand) and it adds support for numeric entities with
      hexadecimal values (so far with the same value range as for
      decimal numeric entities).

    * ns_subnetmatch /subnet/ /ipaddr/
     
      Determine, if a provided IP address (IPv4 or IPv6) is included in
      a subnet specification, which is provided in CIDR notation. The
      command makes internal NaviServer functionality available at the
      Tcl level.  The regression test was extended to cover the
      functionality.
     
      The command ns_subnetmatch validates the provided subnet
      specification (IPv4 or IPv6 address followed by slash and number
      of significant bits) and the provided IP address and tests whether
      the IP address is in the implied range. The command returns a
      boolean value as the result. When comparing an IPv4 and IPv6
      address/CIDR specification or vice versa, the result is always
      false.

      The function can be use when e.g. restricting access to certain
      functionality to some subnets. The function can be used as well
      to check, whether an IP address is an IPv4 or IPv6 address.
     
       Examples:
         % ns_subnetmatch 137.208.0.0/16 137.208.116.31
         1
     
         % ns_subnetmatch 137.208.0.0/16 112.207.16.33
         0
     
         % ns_subnetmatch 2001:628:404:74::31/64 [ns_conn peeraddr]
         ...
     
         # Is IP address a valid IPv6 address?
         % ns_subnetmatch ::/0 $ip
     
         # Is IP address a valid IPv4 address?
         % ns_subnetmatch 0.0.0.0/0 $ip

    * ns_connchan: Added new subcommand "ns_connchan connect"
     
      "ns_connchan connect" is similar to "ns_connchan open", except
      that it does not send an HTTP request (HTTP method, URL, and
      header fields) but just opens the connection. It can be used for
      some non-HTTP communication over TCP and TLS over the ns_connchan
      infrastructure.

    * ns_parseheader, Ns_ParseHeader(): return the field
      number (index) of the parsed entry
     
      Previously, there was no explicit feedback, what field of an
      "ns_set" has been parsed by "ns_parseheader". Now, in success
      cases, the function returns the index of the new/modified
      entry. This function made it possible to generalize and simplify
      the Tcl-level parsing of "multipart/form-data"
      significantly. Additionally, a new optional argument "-prefix"
      was added. When specified, it adds the specified prefix to the
      key.


API changes:

  - ns_parsequery: added option "-charset" and raise exception on failure
     
    The new option "-charset" can be used to add a charset for the
    result encoding of the passed-in HTTP query. In case the charset is
    UTF8 (default on most platforms), and the content is invalid UTF-8,
    an exception is raised (similar to ns_urldecode).


  - Better Unicode support, including emojis requiring 4-byte UTF-8 characters.

  - ns_trim enhancements:
    The new option "-prefix ..." can be used to strip a string
    (such as ">> ") from every line starting with it.

     


Performance Improvements:



Bug Fixes:

   - Improved robustness of "ns_parseurl" for handling query parameters
     and fragments for partial URLs
     
     * fix over-eager collecting of URL components in tail
     * extended regression test

   - Fixed Ns_ResetFileVec NOT to invalidate residual Ns_FileVec
     buffer.q (caused problems under Windows).

   - ns_striphtml: Fixed probably very old bug for markup immediately after an entity
     
     This bug fix handles cases, where e.g. two entities are in a text
     right next to each other, like e.g. in the string
     "hello<>world". The old code was correctly decoding the
     first entity, but output the second one literally.

   - Fixed compilation for C++, which was introduced in 4.99.23
     to avoid usage of reserved C identifiers
     
     Many thanks to Brendan Graves for reporting the problem.

   - Added missing named entities "apos" and "quote". These have been
     missing since ages.

   - Provide an error message when the configured locale is not
     installed on the host.
     
     This change causes NaviServer to abort, when the configured locale
     is not installed on the host. Typically, this locale is e.g. used
     by "ns_strcoll" for determining the default collating order. The
     configuration file for the regression testing sets the environment
     variable LANG to "en_US.UTF-8". This means that for running the
     stock regression test, this locale must be installed on the
     system. Before this change, NaviServer could crash at runtime when
     trying to access the default locale (as e.g. in "ns_strcoll")
     
     
Documentation improvements:
---------------------------

  - Improved the following man pages:

        doc/src/manual/admin-install.man
        doc/src/naviserver/ns_conn.man
        doc/src/naviserver/ns_connchan.man
        doc/src/naviserver/ns_crypto.man
        doc/src/naviserver/ns_http.man
        doc/src/naviserver/ns_httptime.man
        doc/src/naviserver/ns_log.man
        doc/src/naviserver/ns_parseheader.man
        doc/src/naviserver/ns_parsequery.man
        doc/src/naviserver/ns_parseurl.man
        doc/src/naviserver/ns_rlimit.man
        doc/src/naviserver/ns_subnetmatch.man
        doc/src/naviserver/ns_urldecode.man
        doc/src/naviserver/ns_urlencode.man
        doc/src/naviserver/textutil-cmds.man
        nsdb/doc/mann/ns_db.man


Configuration Changes:
----------------------

  - Updated OpenACS sample configuration file
     * reflect recent Oracle (tested with Oracle 19c)
     
     * added documentation for "StaticCSP", "CookieNamespace",
       "NsShutdownWithNonZeroExitCode", "LogIncludeUserId"


Code Changes:
-------------
  - Extended regression test

  - Improve Tcl version compatibility
  
    * Removed -DTCL_NO_DEPRECATED from default CFLAGS to cope with
      recent deprecation in Tcl 8.7a5
      
  - Code Cleanup
      . Do not declare reserved C identifiers
      . Improved type cleanness

  - Improved comments, fixed typos

  - Marked "ns_set_precision" as deprecated, since there is no
    reason why not setting the Tcl variable ::tcl_precision directly.

  - Don't hard-wire port for https testing to 8443
     
    The setup code looks now for a free port for HTTPS connections
    starting with 8443, and remembers the free port in the
    configuration value "tls_listenport" and "tls_listenurl". This is
    now fully analogous to the setup of the plain HTTP testing (setting
    "listenport" and "listenurl")

   - Silence warning with recent versions of gcc when certain values of
     _FORTIFY_SOURCE/-Wstringop-overflow are set.
     
     

Changes in NaviServer Modules:
==============================

...