From: Gustaf N. <ne...@wu...> - 2019-02-16 19:09:27
|
Dear all, on sourceforge is a release candidate for NaviServer 4.99.18 [1]. This release is primarily a bug-fix release for installations still using Tcl 8.5, or installations having a half-working IPv6 installations. Below is a - preliminary - summary of changes; especially the summary of changes in the modules is missing. In case you have more changes for this release, please commit or get in touch with me. If everything goes well, the release should follow in about one week. Please test if possible. best regards -gustaf [1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.18/ ======================================= NaviServer 4.99.18, released 2019-XX-XX ======================================= 87 files changed, 2615 insertions(+), 1476 deletions(-) New Features: - Added ability to provide memory sizes via memory units (kB, MB, GB, KiB, MiB, GiB) in the configuration files and commands. We follow the approach taken by e.g. PostgreSQL and use 1024 as multiplicator, which is e.g. useful for buffer sizes. All memory sizes in the config files can be specified in memory units. Currently, the only commands supporting memory units are ns_cache_create, ns_cache_configure and ns_http. The configuration based on integers without memory units continues to work. - Portable config files (windows compatibility) Rationale: unixoid platforms use traditionally ".so" as extension of binary modules, no matter what "info sharedlibextension" is. On macOS, the sharedlibextension returns ".dylib", while the module is named ".so". Under windows, it seems that the extension of the module has to be ".dll", so we can't write portable config files without extra effort. The new code tries now first to load the library as specified for backward compatibility (maybe containing a file extension). When this fails, it tries to append either ".so" in unixoid platforms or ".dll" under windows. - new API call: ns_db currenthandles Return information about handles currently allocated to the thread performing this call. The result contains a dict with the pools, from which handles were allocated, the names of the handles and their "active" state (a handle is active between a "ns_db select" and the last "ns_db getrow" statement). If no handles are allocated by this thread, empty is returned. For example, a possible result is "main {nsdb0 1} subquery {nsdb1 0}" indicating that the current thread has allocated handles from pool "main" and "subquery", but the handle from pool "main" (i.e. "nsdb0") is currently active (during a select). This feature requires Tcl 8.5 or newer. Performance Improvements: - nsproxy: Fixed block in reaper thread that led to excessive time spent in the main reaper loop, thus blocking the whole proxy interface. This change improves concurrency and reduces potentially long locks of the full nsproxy machinery. Bug Fixes: - improved compatility with Tcl 8.5 * make sure package "try" is used in regression test cases where needed - ns_http * Fixed potential problem with partial writes on HTTP requests with large bodies (e.g. POST of larger files) * Aligned implementation with documentation (if "wait" operation has no timeout, use timeout of queue operation) * Improved client connect() operations with half-configured IPv6 installations. This change addresses especially a problem, where IPv6 is activated on a server, but it cannot be used to connect to the outside world. In this situations, connect operations to hosts with domain names having both IPv4 and IPv6 addresses would fail, although IPv4 connections are possible. In this cases, one should actually either deactivate IPv6 on the machine or probably deactivate IPv6 name lookups in bind (option "-4", filter-aaaa-on-v4 yes;). With this change, this should be be necessary. This change fixes also a memory and socket leak in this situation. - Cache transactions: make "ns_cache flush $cache" behave like "ns_cache_flush -glob $cache *" on "ns_cache_transaction_rollback". Previously, when the full cache was flushed, the rollback was not performed properly. - tcl_crypto: * align code with documentation (provide a default for option "-digest") - nscgi: * fixed potential memory leak - ns_sourceproc: fix deprecated function in cases in error cases, make clear in documentation that ns_register_procns_register_tcl is the preferred approach (many thanks to Russell Sorensen for reporting) - tcl/nstrace.tcl: undo change introduced with 4.99.17 to exclude content of ::tcl namespace since this causes problems with Tcl 8.5 (problems with clock or "package require try" when executed during initial load phase. - Ns_VarUnset(): Deleting the entire nsv array when last value is deleted (many thanks to Andrew Piskorski). Documentation improvements: - all documentation: use memory units when appropriate - doc/src/naviserver/ns_job.man: aligned documentation with implementaton; use itemized list for describing entries in the result lists. - doc/src/manual/admin-maintenance.man: Improved sample service file - nsssl/doc/mann/nsssl.man: Improved sample service file - doc/src/naviserver/ns_log.man Added sample configuration for configuring logging behavior - doc/src/naviserver/ns_register.man, doc/src/naviserver/ns_shortcut_filter.man: provide links (and hints) between "ns_shortcut_filter" and "ns_register_filter" - made markup in documentation more consistent doc/src/naviserver/commandlist.man, doc/src/naviserver/ns_adp_ctl.man, doc/src/naviserver/ns_cond.man, doc/src/naviserver/ns_critsec.man, doc/src/naviserver/ns_env.man, doc/src/naviserver/ns_event.man, doc/src/naviserver/ns_info.man, doc/src/naviserver/ns_rwlock.man, doc/src/naviserver/ns_sema.man, doc/src/naviserver/ns_writer.man, nslog/doc/mann/ns_accesslog.man - various improvements in the following man pages nsssl/doc/mann/nsssl.man doc/src/manual/tcl-lib-file.man Tcl API Changes: C API Changes: Incompatible API Changes: Configuration Changes: - Turn off Nagle algorithm (parameter nodelay) per default since this leads in current Linux systems to bad performance (e.g. seeing with Apache bench ~40ms per request latencency when keepalive is used). The default is now set to fasel in the C code as wel in nsd-config.tcl and openacs-config.tcl - New parameter "logusecdiff": include time difference since last log entry with microsecond (usec) resolution. This option is useful for determine latencies with minimal effort. - All sample config files: * Remove file extensions for binary modules * Improved documentation - sample-config.tcl: * Added sample for DNS configuration * Added sample server log configuration - openacs-config.tcl: * Deactivated SSLv3 by default * Adjusted comments to values from source code * Add sorting to use always pick the library with the highest version number Command Line Changes: Code Changes: - Regressions testing: * removed false positive * extended tests: http.test, ns_striphtml.test, ns_cache.test - Improved portability: * fixed compilation issues for versions of OpenSSL before 1.0.2 * Windows changes (Many thanks to Andrew Piskorski): + Fixed compilation issues with nsssl under windows + Improved makefiles + use "I64u" or "I32u" instead of c99 conventions, since these seem to be broken in many (all?) MSC versions. - Reduced code security smells: add *CLOEXEC to file descriptor open operation. This fixes no real issue (forks are very limited in nsd), but silences static analyzers. - Replaced UTF-8 string literals by hexadecimal-escape-sequences to avoid potential encoding limitations from older compilers - Improved code locality - Removed unused macro - Adding more declarations for PURE and CONST functions - Eased live of static analysis programs - Added "const" declarations, made symbol table static - Reduced variable scopes - Deduced dead assignments |
From: Gustaf N. <ne...@wu...> - 2019-02-17 12:09:46
|
The version *b2 uploaded to sourceforge contains also support for the "SameSite" flag for cookies, a means to mitigate CSRF attacks. all the best -gn ns_setcookie: add flag "-samesite" with values "strict|lax|none" When the flag is set it prevents the browser from sending this cookie along with cross-site requests to mitigate cross site scripting attacks. Permissible values are [term strict], [term lax], or [term none] (default). While the value [term strict] prevents sending the cookie to the target site in all cross-site browsing context, the value of [term lax] allows sending the cookie when the user clicks on regular links. For details, see https://www.owasp.org/index.php/SameSite This cookie flag is not yet part of an RFC, but most major browsers support it. Browsers that do not support it, ignore the flag silently (see https://caniuse.com/#search=samesite). Although most cookies should probably use the flags, in order to provide backward compatibility, the flag can't be activated by default on all cookies. On 16.02.19 20:08, Gustaf Neumann wrote: > Dear all, > > on sourceforge is a release candidate for NaviServer 4.99.18 [1]. > > This release is primarily a bug-fix release for installations still > using Tcl 8.5, or installations having a half-working IPv6 installations. > > Below is a - preliminary - summary of changes; especially the > summary of changes in the modules is missing. In case you have more > changes for this release, please commit or get in touch with me. > If everything goes well, the release should follow in about one week. > > Please test if possible. > > best regards > > -gustaf > [1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.18/ > > ======================================= > NaviServer 4.99.18, released 2019-XX-XX > ======================================= > > 87 files changed, 2615 insertions(+), 1476 deletions(-) > > New Features: > > - Added ability to provide memory sizes via memory units (kB, MB, GB, > KiB, MiB, GiB) in the configuration files and commands. We follow > the approach taken by e.g. PostgreSQL and use 1024 as > multiplicator, which is e.g. useful for buffer sizes. All memory > sizes in the config files can be specified in memory > units. Currently, the only commands supporting memory units are > ns_cache_create, ns_cache_configure and ns_http. The configuration > based on integers without memory units continues to work. > > > - Portable config files (windows compatibility) > > Rationale: unixoid platforms use traditionally ".so" as extension > of binary modules, no matter what "info sharedlibextension" is. On > macOS, the sharedlibextension returns ".dylib", while the module is > named ".so". Under windows, it seems that the extension of the module > has to be ".dll", so we can't write portable config > files without extra effort. > > The new code tries now first to load the library as specified for > backward compatibility (maybe containing a file extension). When > this fails, it tries to append either ".so" in unixoid platforms or > ".dll" under windows. > > > - new API call: ns_db currenthandles > > Return information about handles currently allocated to the thread > performing this call. The result contains a dict with the pools, > from which handles were allocated, the names of the handles and > their "active" state (a handle is active between a "ns_db select" > and the last "ns_db getrow" statement). If no handles are allocated > by this thread, empty is returned. > > For example, a possible result is "main {nsdb0 1} subquery {nsdb1 > 0}" indicating that the current thread has allocated handles from > pool "main" and "subquery", but the handle from pool "main" (i.e. > "nsdb0") is currently active (during a select). > > This feature requires Tcl 8.5 or newer. > > > Performance Improvements: > > - nsproxy: Fixed block in reaper thread that led to excessive time > spent in the main reaper loop, thus blocking the whole proxy > interface. This change improves concurrency and reduces > potentially long > locks of the full nsproxy machinery. > > > Bug Fixes: > > - improved compatility with Tcl 8.5 > * make sure package "try" is used in regression test cases where > needed > > - ns_http > * Fixed potential problem with partial writes on HTTP requests > with large bodies (e.g. POST of larger files) > * Aligned implementation with documentation (if "wait" > operation has no > timeout, use timeout of queue operation) > * Improved client connect() operations with half-configured > IPv6 > installations. This change addresses especially a problem, where > IPv6 is activated on a server, but it cannot be used to connect > to the outside world. In this situations, connect operations to > hosts with domain names having both IPv4 and IPv6 addresses > would fail, although IPv4 connections are possible. In this > cases, one should actually either deactivate IPv6 on the machine > or probably deactivate IPv6 name lookups in bind (option "-4", > filter-aaaa-on-v4 yes;). With this change, this should be be > necessary. This change fixes also a memory and socket leak in > this situation. > > - Cache transactions: make "ns_cache flush $cache" behave like > "ns_cache_flush -glob $cache *" on "ns_cache_transaction_rollback". > Previously, when the full cache was flushed, the rollback was not > performed properly. > - tcl_crypto: > * align code with documentation (provide a default for option > "-digest") > > - nscgi: > * fixed potential memory leak > > - ns_sourceproc: fix deprecated function in cases in error cases, > make clear in documentation that ns_register_procns_register_tcl > is the preferred approach (many thanks to Russell Sorensen for > reporting) > > - tcl/nstrace.tcl: undo change introduced with 4.99.17 to exclude > content of ::tcl namespace since this causes problems with Tcl 8.5 > (problems with clock or "package require try" when executed during > initial load phase. > > - Ns_VarUnset(): Deleting the entire nsv array when last value is > deleted (many thanks to Andrew Piskorski). > > > Documentation improvements: > > - all documentation: use memory units when appropriate > - doc/src/naviserver/ns_job.man: aligned documentation with > implementaton; use itemized list for describing entries in the > result lists. > - doc/src/manual/admin-maintenance.man: > Improved sample service file > - nsssl/doc/mann/nsssl.man: > Improved sample service file > - doc/src/naviserver/ns_log.man > Added sample configuration for configuring logging behavior > > - doc/src/naviserver/ns_register.man, > doc/src/naviserver/ns_shortcut_filter.man: provide links (and > hints) between "ns_shortcut_filter" and "ns_register_filter" > > - made markup in documentation more consistent > doc/src/naviserver/commandlist.man, > doc/src/naviserver/ns_adp_ctl.man, doc/src/naviserver/ns_cond.man, > doc/src/naviserver/ns_critsec.man, doc/src/naviserver/ns_env.man, > doc/src/naviserver/ns_event.man, doc/src/naviserver/ns_info.man, > doc/src/naviserver/ns_rwlock.man, doc/src/naviserver/ns_sema.man, > doc/src/naviserver/ns_writer.man, nslog/doc/mann/ns_accesslog.man > - various improvements in the following man pages > nsssl/doc/mann/nsssl.man > doc/src/manual/tcl-lib-file.man > > Tcl API Changes: > > C API Changes: > > Incompatible API Changes: > > Configuration Changes: > - Turn off Nagle algorithm (parameter nodelay) per default > since this leads in current Linux systems to bad performance > (e.g. seeing with Apache bench ~40ms per request latencency > when keepalive is used). The default is now set to fasel > in the C code as wel in nsd-config.tcl and openacs-config.tcl > > - New parameter "logusecdiff": include time difference since last > log entry with microsecond (usec) resolution. This option is useful > for determine latencies with minimal effort. > > - All sample config files: > * Remove file extensions for binary modules > * Improved documentation > > - sample-config.tcl: > * Added sample for DNS configuration > * Added sample server log configuration > - openacs-config.tcl: > * Deactivated SSLv3 by default > * Adjusted comments to values from source code > * Add sorting to use always pick the library with the highest > version number > > > Command Line Changes: > > Code Changes: > - Regressions testing: > * removed false positive > * extended tests: http.test, ns_striphtml.test, ns_cache.test > - Improved portability: > * fixed compilation issues for versions of OpenSSL before 1.0.2 > * Windows changes (Many thanks to Andrew Piskorski): > + Fixed compilation issues with nsssl under windows > + Improved makefiles > + use "I64u" or "I32u" instead of c99 conventions, since > these seem to be broken in many (all?) MSC versions. > - Reduced code security smells: add *CLOEXEC to file descriptor > open > operation. This fixes no real issue (forks are very limited in > nsd), but silences static analyzers. > > - Replaced UTF-8 string literals by hexadecimal-escape-sequences to > avoid potential encoding limitations from older compilers > > - Improved code locality > - Removed unused macro > - Adding more declarations for PURE and CONST functions > - Eased live of static analysis programs > - Added "const" declarations, made symbol table static > - Reduced variable scopes > - Reduced dead assignments |
From: Gustaf N. <ne...@wu...> - 2019-12-26 23:22:50
|
Dear all, on sourceforge is a release candidate for NaviServer 4.99.19 [1]. Below is a - preliminary - summary of changes; especially the summary of changes in the modules is missing. In case you have more changes for this release, please commit or get in touch with me. If everything goes well, the release should follow in about one week. Please test if possible. best regards -gustaf [1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.19/ ======================================= NaviServer 4.99.19, released 2019-XX-XX ======================================= 244 files changed, 18221 insertions(+), 7899 deletions(-) New Features: ------------- - Bandwidth management: Limit download rate to a maximum (specified as kilobytes per second) On servers with bad internet connections, it is possible that a few eager downloaders can use up all available bandwidth. NaviServer allows now admins and developers to specify the maximum outgoing bandwidth per connection or per connection pool. The rate limit can be specified per connection (highest weight) or per connection thread pool (second highest weight) or per network driver. The rate limitation is only active, when writer threads are used. Per default, the rate limitation is turned off (value 0). New option for ns_conn to query set the per-connection limit ns_conn ratelimit ?limit? Introspecton "ns_writer list" returns the current transferrates per writer job. Configuration option for connection pools: ns_param connectionratelimit ... ns_param poolratelimit ... Configuration option for network drivers: ns_param writerratelimit ... Commands for changing the rate limits dynamically: ns_server ?-server s? ?-pool p? connectionratelimit ?value? ns_server ?-server s? ?-pool p? poolratelimit ?value? - Context filter for urlspace: NaviServer's urlspace trie data structure used for various purposes such as registering methods/URL pairs for procs or for mapping of requests to connection thread pools. The classical trie mapping for e.g. "GET /foo/bar/*.html" is based on method + path + string match pattern, where the last part "*.html" is used as a "filter". The new version of NaviServer allows in addition to the string based glob pattern so-called "context filter", which includes the IP-address or header fields in the matching process. The context filters are applied after the classical urlspace processing, such that the behavior is fully backwards compatible. The context filters are kept in a stable order (IP before header-based filters, specific before less specific) such that the behavior is not order dependent. The new feature can be used map e.g. bots or certain IP ranges to the specified connection pools. When connection pools named "bots" and "local" are defined, one can use the following mapping commands to map incoming requests to the specified pools. Examples for adding urlspace mappings at runtime: ns_server -pool bots map "GET /* {user-agent *bot*}" ns_server -pool bots map "GET /* {user-agent *crawl*}" ns_server -pool bots map "GET /* {user-agent *baidu*}" ns_server -pool bots map "GET /* {X-NS-ip 2a03:2880::/29}" ns_server -pool local map "GET /* {X-NS-ip 127.0.0.1}" ns_server -pool local map "GET /* {X-NS-ip 137.208.1.0/16}" Note that arbitrary header fields can be used for the mapping, containing potentially some glob style match characters. The special header field "X-NS-ip" denotes IP based filters, which can be provided fully qualified or in CIDR notation (denoting significant bits) for IPv4 and IPv6. - Allow a single driver to listen on multiple IP addresses: This feature greatly simplifies setups, where a single server is listening on multiple IP addresses (e.g. IPv4 and IPv6). Previously, it was necessary to define separate drivers for these, which need different names but which are often configured identically. Furthermore, when the address is omitted, the server performs a lookup from the hostname to determine the IP address. Previously, it used the first one, now it can work with all returned addresses. To use this feature, simply specify the configure values of "address" as a Tcl list containing multiple IP addresses. The change is fully backwards compatible, old configuration files will continue to work. - Added sending of multiple file chunks with a single command: ns_writer sendfiles /list of filespecs/ Every filespec is a dict which must contain a "filename" element and can contain additionally an "-offset" and/or a "-size" element. ns_writer submitfiles {{filename /tmp/f1} {filename /tmp/f2 -offset 10}} This function is e.g. useful for video streaming applications, where multiple video file chunks should be transferred (sometimes) concatenated in a single chunk. Therefore, this function avoids the necessity to concatenate these files in advance (causing increased latency and disk usage). - Added per-server "extraheaders". These extra headers fields are merged with the following precedence (the earlier ones have more weight) 1) application specific headers 2) per-server extra headers 3) per-driver extra headers This change makes it possible to provide in the config files defaults, which can be overwritten by certain pages in the applications. Furthermore, this change makes it easy to add header fields to all requests of a virtual server (e.g. Strict-Transport-Security, ...) - Logging improvements: * Support separate access logs for requests sent via different drivers. The nslog module accepts now an additional parameter named "driver". This option can be used to produce different access logs for requests submitted via different drivers to sort out e.g. local server talk. Per default, every request is logged. When a value is provided for this parameter, only requests are logged in this log file when these come from a driver matching the provided glob pattern. * log-file sanitizer: When data entered into a log file contains special characters (e.g. end line characters), these could alter the appearance of content within the log file. Single entries may appear as multiple entries. Attackers may leverage log forging to insert fake entries to that obfuscate malicious acts, or these can confuse log-file analyzers to avoid analysis. The new global parameter "sanitizelogfiles" can control this behavior (values 0: none, 1: full, 2: human-friendly; default: 2); * Additional debugging flag "Debug(access)": When activated, entries of the access log are mirrored in the system log (error.log). This option eases tracking the end of requests in the system log. * Additional debugging flag "Debug(writer)" for debugging of writer. * Provided access to request AND reply header fields via "extendedheaders" specification (in "ns_accesslog" and "nslog" configuration section): It is now optionally possible to specify the header fields in the provided "extendedheaders" list with a prefix "request:" or "response:" to denote request and reply header fields. In case, no tag is provided, field names are interpreted as request header fields (like before). The change is fully backwards compatible, - ns_set: * Added sucommand "ns_set imerge", a is the case insignificant version of "ns_set merge". This option is useful especially for working with header fields. * Added subcommand "ns_set iupdate": this command is the case insenstive counterpart of "ns_set update". * Fixed "nsv_set a b" when no key nor array exist - ns_http: * Revamp of ns_http and related supportive code. * Added chunked-encoding parser * Tcl channel support for body (PUT, POST) and for result. This makes it possible to use e.g. reflected Tcl channels in "ns_http". New Options for "ns_http run" - ?-body_size size? - ?-body_chan chan? - ?-outputfile fn? - ?-outputchan chan? * Make clear that "ns_http run" is the preferred interface (rather than "queue" + "wait") * Deprecated output variables of "ns_http wait" (since everything is included in the resulting dict) - New commands: * ns_asynclogfile: Facility for thread-safe writing async log files of various kinds using the AsyncWriterThread. Usage example: # ... at startup set fd [ns_asynclogfile open /tmp/test.log] # ... during run ns_asynclogfile write $fd hello\n # ... at shutdown ns_asynclogfile close $fd * ns_parsefieldvalue: This function parses the provided field value (from an HTTP request or reply header field) into its parts and returns these in the form of a list of Tcl dicts or a single Tcl dict depending on parameters. The syntax of the contents of these header fields is specified in RFC 7230 section 3.2.6. - New features for existing commands/subcommands * Binary reform: Some NaviServer commands accepted previously implicitly binary or non-binary input, some commands used the flag "-binary" to denote the differences (e.g. ns_return). For a more uniform and expectable behavior binary input the explicit option "-binary" flag to the following commands: ns_base64decode ns_base64encode ns_base64urldecode ns_base64urlencode ns_md5 ns_sha1 ns_uudecode ns_uuencode ns_crypto::aead::decrypt string (for "-key", "-aad", "-iv", "input") ns_crypto::aead::encrypt string (for "-key", "-aad", "-iv", "input") ns_crypto::eckey import ("-string") ns_crypto::eckey sharedsecret ("pubkey") ns_crypto::hmac add (for "message") ns_crypto::hmac new (for "key") ns_crypto::hmac string (for "key", "message") ns_crypto::md add (for "message") ns_crypto::md hkdf (for "-salt" "-secret" "-info") ns_crypto::md string (for "message") ns_crypto::md vapidsign (for "message") ns_crypto::scrypt (for "-salt" "-secret") Note that this option is not fully backward compatible. Early versions of "ns_sha1" were expecting always non-binary input, some new version assumed binary input. The new version is now more compatible with AOLserver and older NaviServer versions. * "ns_server threads" reports now how often a thread of this pool was started. This statistic can help to identify installations, where threads are started and stopped in a too eager fashion. * "ns_server.... stats": added "sendbodysize" and "replybodysize" to the dict of the per-pool statistics. * "ns_server ... connectionratelimit ?value?": Query or set the default per-connection rate limit. * "ns_server ... poolratelimit ?value?": Query or set the pool connection rate limit. * "ns_writer list" returns now the current transferrates per writer job. * "ns_writer submitfile -offset X -size Y" and "ns_writer size X" accept now memory units, the option ?-driver driver? was added to the "ns_writer" subcommands "size", and "streaming" (defaulting to the current driver). One can now write ns_writer size 150KB instead of ns_writer size nssock [expr 150*1024] Performance Improvements: ------------------------- Bug Fixes: ---------- - Blueprint serializer: moved serialized objects in the blueprint after the namespace imports and ensemble recreators since constructors can call functions depending on it. - Include *xml* and *json* in non-binary MIME types. Note that there is no exact definition, what MIME types are exactly. - nslog: * Abort in error situation with an error message instead of crashing * Added server name in log entries to make it easier to distinguish messages per-server during bootup - Range requests: * Fixed potential problem with too many non-contiguous byte ranges * Don't silently ignore invalid syntax of range requests - Improved handling of already closed connections (data delivery was already delegated to writer threads, but driver still wants to return data directly; this could happen in error situations) - connchan: * Fixed potential invalid reads in (error) cases, where LogConnchanDebug is enabled and the callback is deleted during a Tcl_Eval(). * Fixed potential problem, where call to Tcl_Eval() might clean structures used in the callback handler - Make sure, memory allocated by Tcl is returned by Tcl (important, when NaviServer is compiled with -DSYSTEM_MALLOC) - Fixed Tcl argument parsing of "ns_roll", "ns_fmttime", "nsv_bucket" and "ns_critsec eval". - Fixed bug in "nsv_set -default" overwriting pre-existing values - Fixed potential crashes in (desperate) error situations, where the connection was already closed. - Fixed leaking tmp file, when writerstraming is activated and streaming HTML output is used. This bug could appear, when "writerstreaming" is turned on in the config file (default off). See also: https://sourceforge.net/p/naviserver/mailman/naviserver- devel/thread/dd508519-1262-bd91-288a-fa30a7634224%40digital- concepts.com/#msg36794472 - Made table used for enumeration values for Ns_ObjvIndex static. Background: This is a fix for a tricky Tcl_Obj sharing bug triggered by the usage of C-level IndexObjs based on volatile tables. Consider the following example: proc foo {x} { return [bar -value x ...] } The Tcl_Obj "x" is shared as name of argument and as a value of the non-positional parameter "-value". When bar (e.g. C-implemented) uses Tcl_GetIndexFromObj*() to lookup "x" in a table of options, which is volatile, the involved Tcl_Obj will be converted to an indexObj. On a call with wrong number of arguments( e.g."foo 1 2 3"), Tcl will try to give a nice error message, saying that "foo x" can be called only with one argument. When printing argument "x", it sees that "x" is an indexObj, and for these kinds of objects, "x" might be an abbreviated version of a full name. Since the table behind the indexObj is in the case above volatile, a crash might happen. - Added API call Ns_SockInErrorState() since SSL_shutdown() must not be called if a previous fatal error has occurred on a connection i.e. if SSL_get_error() has returned SSL_ERROR_SYSCALL or SSL_ERROR_SSL. - Provided local buffers for OpenSSL ERR_error_string handling to avoid potential race conditions (OpenSSL keeps otherwise error messages in static memory). - Crypto support: * New feature: scrypt Password-Based Key Derivation Function (RFC 7914) The scrypt function is a modern replacement for crypt and bcrypt and derives secret keys from a secret string. It is based on memory- hard functions, which offer added protection against attacks using custom hardware and GPU arrays. The function requires the compilation of NaviServer against OpenSSL 3.0 or newer (not yet released) Example from RFC 7914: % ::ns_crypto::scrypt -secret "password" -salt NaCl -n 1024 -r 8 -p 16 fdbabe1c9d3472007856e7190d01e9fe7c6ad7cbc8237830e77376634b373162 2eaf30d92e22a3886ff109279d9830dac727afb94a83ee6d8360cbdfa2cc0640 * "ns_md" and "ns_hmac": added flag "-encoding" * Added options "-passphrase" to all "ns_crypto" commands accepting a PEM file. In principle, every PEM file can be protected by a pass phrase. * Signing and verifying signatures: "::ns_crypto::md string" can be used for signing and verifying of cryptgraphic signatures: % set sig [::ns_crypto::md string \ -digest sha1 \ -encoding binary \ -sign /usr/local/src/naviserver/myprivate.pem \ "abcdefghijklmnopqrstuxvwxyz\n"] % set vfy [::ns_crypto::md string \ -digest sha1 \ -verify /usr/local/src/naviserver/myprivate.pem \ -signature $sig \ "abcdefghijklmnopqrstuxvwxyz\n"] - nscgi: * ... Documentation improvements: --------------------------- - Improved cross references between man pages - Added documentation to undocumented flags - Added more example - Improved spelling - The following 48 man pages were updated since the last release: admin-db.man, admin-install.man, admin-maintenance.man, admin-tuning.man, commandlist.man, ns_adp.man, ns_adp_ctl.man, ns_adp_info.man, ns_adp_parse.man, ns_asynclogfile.man, ns_base64.man, ns_conn.man, ns_connchan.man, ns_crypt.man, ns_crypto.man, ns_driver.man, ns_filestat.man, ns_hmac.man, ns_hotp.man, ns_http.man, ns_job.man, ns_locationproc.man, ns_log.man, ns_md.man, ns_md5.man, ns_parsefieldvalue.man, ns_parseheader.man, ns_parseurl.man, ns_rand.man, ns_register.man, ns_return.man, ns_server.man, ns_set.man, ns_sha1.man, ns_sls.man, ns_sockcallback.man, ns_sockselect.man, ns_tmpnam.man, ns_totp.man, ns_urlspace.man, ns_uudecode.man, ns_uuencode.man, ns_writer.man, nsv.man, returnstatus-cmds.man, tcl-lib-file.man, tcl-libraries.man, tcl-overview.man Configuration Changes: ---------------------- - Output warning to system log, when NaviServer was compiled without zlib support, but the config file request compression. Before, the compress enable request was silently ignored, giving no hint, why compression attempts of a developer were ignored. - Sample configurations: * Documented usage of connection pools in sample config files. * Added sample entries for new features (e.g. rate limits, log file sanitizer) * openacs-config.tcl: . Changed driver installation to "global" in sample configuration file. This change makes it easier for site admins to add further servers (e.g. for virtual hosting). . Simplified sample config file by using multiple IP addresses per driver * nsd-config.tcl: . Changed driver installation to "global" in sample configuration file. This change makes it easier for site admins to add Code Changes: ------------- - Improved scalability: provide different locks per urlspace IDs. - Improved range checking for Ns_ObjvInt, Ns_ObjvLong, Ns_ObjvWideInt and MemUnits: * built-in support for objv parser This eases specification of the C API, makes accepted ranges clear and provides consistent error messages. These change fixes several potential crashes, when e.g. passed-in values are used as array-indices (crash on negative numbers) or where the values were casted to unsigned types (very high unsigned values, potential memory alloc problems). * Fixed all cases, where Tcl_GetIntFromObj(), Tcl_GetLongFromObj, and Tcl_GetWideintFromObj was used without a range check. - Argument parser: skip processing of optional arguments when the number of provided arguments is equal to the number of required parameters. Background: This change allows a fully backward compatible version of e.g. "ns_md5 $foo", no matter what the value of "$foo" is, although the new version of "ns_md5" has now a "-binary" option. - Refactored low-level socket I/O: * All driver operations are now strictly non-blocking * Use same infrastructure for all network I/O operations (driver, connchan, ns_http, ...) * Rewritten socketfile/sendfile handling (including *BSD support) * Improved handling of closed and detached channels, e.g.delegated to writers (new macros NS_CONN_REQUIRE_CONNECTED, NS_CONN_REQUIRE_OPEN, NS_CONN_REQUIRE_CONFIGURED, NS_CONN_REQUIRE_ALL) * Improved protection for SIGPIPE signals. - New API infrastructure: Ns_DList The Ns_DList is similar to Tcl_DString, but operates on pointers instead of characters. Like Tcl_DString, Ns_DList pre-allocates some data and extends it when necessary. Ns_DList structures are especially useful alternative to linked lists, when usually data is added to the end. Since the data is kept as a dense (potentially growing) array, the memory locality is much better than with linked lists, resulting in better CPU cache hit rates. - Distinguish between Ns_NormalizePath() and Ns_NormalizeUrl(), where the first is for the file system and the latter for URLs. Previously, both cases were handled identically. - Function pointer handling: * Created a Tcl hash type for function pointers, since ANSI/ISO C forbids casting/comparison of data and function pointers. The new type is used whenever a hash lookup is performed for a function pointer. * Fixed all other occurrences of such comparisons. - Regressions testing: * Removed false positive * Switch to testing via ns_http in regression tests (this makes it as well possible for testing https connections) * Extended tests: + Added regression tests for "ns_rand" + Added nsssl tests + Added automatic certificate generation for test-server * The following 35 tests were added or extended since the last release adp.test, encoding.test, http.test, http_byteranges.test, http_chunked.test, http_keep.test, https.test, misc.test, ns_adp_compress.test, ns_base64.test, ns_cache.test, ns_conn.test, ns_conn_host.test, ns_crypto.test, ns_driver.test, ns_hashpath.test, ns_hostbyaddr.test, ns_info.test, ns_limits.test, ns_md5.test, ns_nsv.test, ns_pagepath.test, ns_parsefieldvalue.test, ns_proxy.test, ns_reflow_text.test, ns_schedule.test, ns_server.test, ns_serverpath.test, ns_set.test, ns_sha1.test, ns_urlencode.test, ns_urlspace.test, ns_uuencode.test, ns_writer.test, tclconnio.test, tclresp.test - Improved portability: * improved compatibility with LibreSSL (2.7 and 2.9) * Windows changes (Many thanks to Andrew Piskorski): + Fixed Ns_LogRoll() to work on Windows. + WSASend receives as 5th argument flags, but not a pointer to flags. + Improved makefiles + Updated _MSC_VER version numbers to include Visual Studio 2019 + Added macro NS_INLINE to achieve higher portability with (older?) Microsoft compilers. - Added warnings about potential misconfiguration of NaviServer "tcllib" setting (Tcl written NaviServer modules) - Marked compatibility wrappers Ns_SetThreadServer() and Ns_SetThreadServer() explicitly as deprecated. Use Ns_ThreadSetName() and Ns_ThreadGetName() instead. - Build-system * Added -DSYSTEM_MALLOC to default compile flags. * Added nsssl to the default test target. * Require RSA keys of size 2048: newer versions of OpenSSL refuse to work with RSA keys of size 1024. * Improved alignment with Tcl's current .m4 file to get rid of "-prebind" deprecated message under macOS. * Improved handling of posix thread library for FreeBSD and OpenBSD - Use native thread_local storage for log handling when available (experimental). - Improved type cleanness for function pointers - Improved code locality - Improved error messages - Improved structure packing - Reduced variable scopes - Added missing "extern" declarations - Aligned function prototypes - Dropped potentially dangerous call to alloca() - Added typedefs for commonly used functions - Reduce potential dangling pointer dereferences - Aligned names of arguments in prototype with function definition - Introduced use attribute-based approach for denoting fall through in case statements - Adding more declarations for PURE and CONST functions - Improved configurability for clang-tidy - Fixed macro name-clash with PostgreSQL - Improved spelling Modules: -------- .... |
From: Wolfgang W. <wol...@di...> - 2019-12-27 09:56:43
|
This is good news! At the current tip version we encountered a signal 11 error. It happens in a filter, when we raise an error and return "filter_break". With "filter_return", there is no problem. To use "filter_break" in this situation is an error on our side but maybe the crash can happen in other situations as well. This is what we see in our log: Warning: redirect status 404: connection already closed Warning: NsWriterQueue: called without sockPtr size 534 bufs 1 flags 103031 stream 000000 chan (nil) fd -1 Fatal: received fatal signal 11 Regards, Wolfgang Am 27.12.19 um 00:22 schrieb Gustaf Neumann: > > Dear all, > > on sourceforge is a release candidate for NaviServer 4.99.19 [1]. > > Below is a - preliminary - summary of changes; especially the > summary of changes in the modules is missing. In case you have more > changes for this release, please commit or get in touch with me. > If everything goes well, the release should follow in about one week. > > Please test if possible. > > best regards > > -gustaf > [1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.19/ > > ======================================= > NaviServer 4.99.19, released 2019-XX-XX > ======================================= > > 244 files changed, 18221 insertions(+), 7899 deletions(-) > > New Features: > ------------- > > - Bandwidth management: Limit download rate to a maximum (specified > as kilobytes per second) > > On servers with bad internet connections, it is possible that a few > eager downloaders can use up all available bandwidth. NaviServer > allows now admins and developers to specify the maximum outgoing > bandwidth per connection or per connection pool. The rate limit can > be specified per connection (highest weight) or per connection > thread pool (second highest weight) or per network driver. The rate > limitation is only active, when writer threads are used. Per > default, the rate limitation is turned off (value 0). > > New option for ns_conn to query set the per-connection limit > ns_conn ratelimit ?limit? > Introspecton "ns_writer list" returns the current > transferrates per writer job. > > Configuration option for connection pools: > ns_param connectionratelimit ... > ns_param poolratelimit ... > Configuration option for network drivers: > ns_param writerratelimit ... > > Commands for changing the rate limits dynamically: > ns_server ?-server s? ?-pool p? connectionratelimit ?value? > ns_server ?-server s? ?-pool p? poolratelimit ?value? > > > - Context filter for urlspace: > > NaviServer's urlspace trie data structure used for various purposes > such as registering methods/URL pairs for procs or for mapping of > requests to connection thread pools. The classical trie mapping for > e.g. "GET /foo/bar/*.html" is based on method + path + string match > pattern, where the last part "*.html" is used as a "filter". > > The new version of NaviServer allows in addition to the string > based glob pattern so-called "context filter", which includes the > IP-address or header fields in the matching process. The context > filters are applied after the classical urlspace processing, such > that the behavior is fully backwards compatible. The context > filters are kept in a stable order (IP before header-based filters, > specific before less specific) such that the behavior is not order > dependent. > > The new feature can be used map e.g. bots or certain IP ranges to > the specified connection pools. When connection pools named "bots" > and "local" are defined, one can use the following mapping commands > to map incoming requests to the specified pools. Examples for > adding urlspace mappings at runtime: > > ns_server -pool bots map "GET /* {user-agent *bot*}" > ns_server -pool bots map "GET /* {user-agent *crawl*}" > ns_server -pool bots map "GET /* {user-agent *baidu*}" > ns_server -pool bots map "GET /* {X-NS-ip 2a03:2880::/29}" > > ns_server -pool local map "GET /* {X-NS-ip 127.0.0.1}" > ns_server -pool local map "GET /* {X-NS-ip 137.208.1.0/16}" > > Note that arbitrary header fields can be used for the mapping, > containing potentially some glob style match characters. The > special header field "X-NS-ip" denotes IP based filters, which can > be provided fully qualified or in CIDR notation (denoting > significant bits) for IPv4 and IPv6. > > > - Allow a single driver to listen on multiple IP addresses: > > This feature greatly simplifies setups, where a single server is > listening on multiple IP addresses (e.g. IPv4 and > IPv6). Previously, it was necessary to define separate drivers for > these, which need different names but which are often > configured identically. Furthermore, when the address is omitted, > the server performs a lookup from the hostname to determine the IP > address. Previously, it used the first one, now it can work with > all returned addresses. > > To use this feature, simply specify the configure values of > "address" as a Tcl list containing multiple IP addresses. The > change is fully backwards compatible, old configuration files will > continue to work. > > > - Added sending of multiple file chunks with a single command: > > ns_writer sendfiles /list of filespecs/ > > Every filespec is a dict which must contain a "filename" element > and can contain additionally an "-offset" and/or a "-size" element. > > ns_writer submitfiles {{filename /tmp/f1} {filename /tmp/f2 -offset 10}} > > This function is e.g. useful for video streaming applications, > where multiple video file chunks should be transferred (sometimes) > concatenated in a single chunk. Therefore, this function avoids the > necessity to concatenate these files in advance (causing increased > latency and disk usage). > > > - Added per-server "extraheaders". These extra headers fields are > merged with the following precedence (the earlier ones have more > weight) > 1) application specific headers > 2) per-server extra headers > 3) per-driver extra headers > > This change makes it possible to provide in the config files > defaults, which can be overwritten by certain pages in the > applications. Furthermore, this change makes it easy to add > header fields to all requests of a virtual server (e.g. > Strict-Transport-Security, ...) > > > - Logging improvements: > > * Support separate access logs for requests sent via > different drivers. > > The nslog module accepts now an additional parameter named > "driver". This option can be used to produce different access > logs for requests submitted via different drivers to sort out > e.g. local server talk. Per default, every request is > logged. When a value is provided for this parameter, only > requests are logged in this log file when these come from a > driver matching the provided glob pattern. > > * log-file sanitizer: > When data entered into a log file contains special characters > (e.g. end line characters), these could alter the appearance of > content within the log file. Single entries may appear as > multiple entries. Attackers may leverage log forging to insert > fake entries to that obfuscate malicious acts, or these can > confuse log-file analyzers to avoid analysis. > > The new global parameter "sanitizelogfiles" can control this > behavior (values 0: none, 1: full, 2: human-friendly; default: > 2); > > * Additional debugging flag "Debug(access)": When activated, entries > of the access log are mirrored in the system log > (error.log). This option eases tracking the end of requests in > the system log. > > * Additional debugging flag "Debug(writer)" for debugging of writer. > > * Provided access to request AND reply header fields via > "extendedheaders" specification (in "ns_accesslog" and "nslog" > configuration section): > > It is now optionally possible to specify the header fields in the > provided "extendedheaders" list with a prefix "request:" or > "response:" to denote request and reply header fields. In case, > no tag is provided, field names are interpreted as request header > fields (like before). The change is fully backwards compatible, > > > - ns_set: > > * Added sucommand "ns_set imerge", a is the case insignificant > version of "ns_set merge". This option is useful especially for > working with header fields. > > * Added subcommand "ns_set iupdate": this command is the case > insenstive counterpart of "ns_set update". > > * Fixed "nsv_set a b" when no key nor array exist > > - ns_http: > * Revamp of ns_http and related supportive code. > * Added chunked-encoding parser > * Tcl channel support for body (PUT, POST) and for result. > This makes it possible to use e.g. reflected Tcl channels > in "ns_http". > New Options for "ns_http run" > - ?-body_size size? > - ?-body_chan chan? > - ?-outputfile fn? > - ?-outputchan chan? > * Make clear that "ns_http run" is the preferred interface > (rather than "queue" + "wait") > * Deprecated output variables of "ns_http wait" (since everything > is included in the resulting dict) > > > - New commands: > > * ns_asynclogfile: > Facility for thread-safe writing async > log files of various kinds using the AsyncWriterThread. > Usage example: > # ... at startup > set fd [ns_asynclogfile open /tmp/test.log] > # ... during run > ns_asynclogfile write $fd hello\n > # ... at shutdown > ns_asynclogfile close $fd > > * ns_parsefieldvalue: > > This function parses the provided field value (from an HTTP > request or reply header field) into its parts and returns these > in the form of a list of Tcl dicts or a single Tcl dict depending > on parameters. The syntax of the contents of these header fields > is specified in RFC 7230 section 3.2.6. > > - New features for existing commands/subcommands > > * Binary reform: Some NaviServer commands accepted previously > implicitly binary or non-binary input, some commands used > the flag "-binary" to denote the differences (e.g. ns_return). > For a more uniform and expectable behavior binary input the > explicit option "-binary" flag to the following commands: > > ns_base64decode > ns_base64encode > ns_base64urldecode > ns_base64urlencode > ns_md5 > ns_sha1 > ns_uudecode > ns_uuencode > ns_crypto::aead::decrypt string (for "-key", "-aad", "-iv", "input") > ns_crypto::aead::encrypt string (for "-key", "-aad", "-iv", "input") > ns_crypto::eckey import ("-string") > ns_crypto::eckey sharedsecret ("pubkey") > ns_crypto::hmac add (for "message") > ns_crypto::hmac new (for "key") > ns_crypto::hmac string (for "key", "message") > ns_crypto::md add (for "message") > ns_crypto::md hkdf (for "-salt" "-secret" "-info") > ns_crypto::md string (for "message") > ns_crypto::md vapidsign (for "message") > ns_crypto::scrypt (for "-salt" "-secret") > > Note that this option is not fully backward compatible. Early > versions of "ns_sha1" were expecting always non-binary input, some > new version assumed binary input. The new version is now more > compatible with AOLserver and older NaviServer versions. > > * "ns_server threads" reports now how often a thread of this pool > was started. This statistic can help to identify > installations, where threads are started and stopped in a too > eager fashion. > * "ns_server.... stats": added "sendbodysize" and "replybodysize" > to the dict of the per-pool statistics. > * "ns_server ... connectionratelimit ?value?": Query or set > the default per-connection rate limit. > * "ns_server ... poolratelimit ?value?": Query or set > the pool connection rate limit. > * "ns_writer list" returns now the current transferrates per > writer job. > > * "ns_writer submitfile -offset X -size Y" and "ns_writer size X" > accept now memory units, the option ?-driver driver? was added > to the "ns_writer" subcommands "size", and "streaming" > (defaulting to the current driver). > > One can now write > ns_writer size 150KB > instead of > ns_writer size nssock [expr 150*1024] > > > > > Performance Improvements: > ------------------------- > > > > > > Bug Fixes: > ---------- > > - Blueprint serializer: moved serialized objects in the blueprint > after the namespace imports and ensemble recreators since > constructors can call functions depending on it. > > - Include *xml* and *json* in non-binary MIME types. Note that > there is no exact definition, what MIME types are exactly. > > - nslog: > * Abort in error situation with an error message instead of crashing > * Added server name in log entries to make it easier to distinguish > messages per-server during bootup > > - Range requests: > * Fixed potential problem with too many non-contiguous byte ranges > * Don't silently ignore invalid syntax of range requests > > - Improved handling of already closed connections (data delivery was > already delegated to writer threads, but driver still wants to > return data directly; this could happen in error situations) > > - connchan: > * Fixed potential invalid reads in (error) cases, where > LogConnchanDebug is enabled and the callback is deleted during a > Tcl_Eval(). > * Fixed potential problem, where call to Tcl_Eval() might clean > structures used in the callback handler > > - Make sure, memory allocated by Tcl is returned by Tcl (important, > when NaviServer is compiled with -DSYSTEM_MALLOC) > > - Fixed Tcl argument parsing of "ns_roll", "ns_fmttime", > "nsv_bucket" and "ns_critsec eval". > > - Fixed bug in "nsv_set -default" overwriting pre-existing values > > - Fixed potential crashes in (desperate) error situations, where the > connection was already closed. > > - Fixed leaking tmp file, when writerstraming is activated and > streaming HTML output is used. This bug could appear, when > "writerstreaming" is turned on in the config file (default off). > See also:https://sourceforge.net/p/naviserver/mailman/naviserver- > devel/thread/dd508519-1262-bd91-288a-fa30a7634224%40digital- > concepts.com/#msg36794472 > > - Made table used for enumeration values for Ns_ObjvIndex static. > Background: This is a fix for a tricky Tcl_Obj sharing bug > triggered by the usage of C-level IndexObjs based on volatile > tables. Consider the following example: > > proc foo {x} { > return [bar -value x ...] > } > > The Tcl_Obj "x" is shared as name of argument and as a value of > the non-positional parameter "-value". When bar > (e.g. C-implemented) uses Tcl_GetIndexFromObj*() to lookup "x" in > a table of options, which is volatile, the involved Tcl_Obj will > be converted to an indexObj. > > On a call with wrong number of arguments( e.g."foo 1 2 3"), Tcl > will try to give a nice error message, saying that "foo x" can be > called only with one argument. When printing argument "x", it sees > that "x" is an indexObj, and for these kinds of objects, "x" might > be an abbreviated version of a full name. Since the table behind > the indexObj is in the case above volatile, a crash might happen. > > - Added API call Ns_SockInErrorState() since SSL_shutdown() must not > be called if a previous fatal error has occurred on a connection > i.e. if SSL_get_error() has returned SSL_ERROR_SYSCALL or > SSL_ERROR_SSL. > > - Provided local buffers for OpenSSL ERR_error_string handling to > avoid potential race conditions (OpenSSL keeps otherwise error > messages in static memory). > > - Crypto support: > * New feature: scrypt Password-Based Key Derivation Function (RFC 7914) > > The scrypt function is a modern replacement for crypt and bcrypt > and derives secret keys from a secret string. It is based on > memory- hard functions, which offer added protection against > attacks using custom hardware and GPU arrays. > > The function requires the compilation of NaviServer against > OpenSSL 3.0 or newer (not yet released) > > Example from RFC 7914: > % ::ns_crypto::scrypt -secret "password" -salt NaCl -n 1024 -r 8 -p 16 > fdbabe1c9d3472007856e7190d01e9fe7c6ad7cbc8237830e77376634b373162 > 2eaf30d92e22a3886ff109279d9830dac727afb94a83ee6d8360cbdfa2cc0640 > > * "ns_md" and "ns_hmac": added flag "-encoding" > > * Added options "-passphrase" to all "ns_crypto" commands > accepting a PEM file. In principle, every PEM file can be > protected by a pass phrase. > > * Signing and verifying signatures: "::ns_crypto::md string" can > be used for signing and verifying of cryptgraphic signatures: > > % set sig [::ns_crypto::md string \ > -digest sha1 \ > -encoding binary \ > -sign /usr/local/src/naviserver/myprivate.pem \ > "abcdefghijklmnopqrstuxvwxyz\n"] > > % set vfy [::ns_crypto::md string \ > -digest sha1 \ > -verify /usr/local/src/naviserver/myprivate.pem \ > -signature $sig \ > "abcdefghijklmnopqrstuxvwxyz\n"] > > - nscgi: > * ... > > > Documentation improvements: > --------------------------- > > - Improved cross references between man pages > - Added documentation to undocumented flags > - Added more example > - Improved spelling > > - The following 48 man pages were updated since the last release: > > admin-db.man, admin-install.man, admin-maintenance.man, > admin-tuning.man, commandlist.man, ns_adp.man, ns_adp_ctl.man, > ns_adp_info.man, ns_adp_parse.man, ns_asynclogfile.man, > ns_base64.man, ns_conn.man, ns_connchan.man, ns_crypt.man, > ns_crypto.man, ns_driver.man, ns_filestat.man, ns_hmac.man, > ns_hotp.man, ns_http.man, ns_job.man, ns_locationproc.man, > ns_log.man, ns_md.man, ns_md5.man, ns_parsefieldvalue.man, > ns_parseheader.man, ns_parseurl.man, ns_rand.man, > ns_register.man, ns_return.man, ns_server.man, ns_set.man, > ns_sha1.man, ns_sls.man, ns_sockcallback.man, > ns_sockselect.man, ns_tmpnam.man, ns_totp.man, > ns_urlspace.man, ns_uudecode.man, ns_uuencode.man, > ns_writer.man, nsv.man, returnstatus-cmds.man, > tcl-lib-file.man, tcl-libraries.man, tcl-overview.man > > > Configuration Changes: > ---------------------- > > - Output warning to system log, when NaviServer was compiled without > zlib support, but the config file request compression. Before, the > compress enable request was silently ignored, giving no hint, why > compression attempts of a developer were ignored. > > - Sample configurations: > > * Documented usage of connection pools in sample config files. > > * Added sample entries for new features (e.g. rate limits, log > file sanitizer) > > * openacs-config.tcl: > > . Changed driver installation to "global" in sample configuration > file. This change makes it easier for site admins to add > further servers (e.g. for virtual hosting). > > . Simplified sample config file by using multiple IP addresses > per driver > > * nsd-config.tcl: > > . Changed driver installation to "global" in sample configuration > file. This change makes it easier for site admins to add > > > Code Changes: > ------------- > > - Improved scalability: provide different locks per urlspace IDs. > > - Improved range checking for Ns_ObjvInt, Ns_ObjvLong, > Ns_ObjvWideInt and MemUnits: > > * built-in support for objv parser This eases specification of the > C API, makes accepted ranges clear and provides consistent error > messages. These change fixes several potential crashes, when e.g. > passed-in values are used as array-indices (crash on negative > numbers) or where the values were casted to unsigned types (very > high unsigned values, potential memory alloc problems). > > * Fixed all cases, where Tcl_GetIntFromObj(), Tcl_GetLongFromObj, > and Tcl_GetWideintFromObj was used without a range check. > > > - Argument parser: skip processing of optional arguments when the > number of provided arguments is equal to the number of required > parameters. > > Background: This change allows a fully backward compatible version > of e.g. "ns_md5 $foo", no matter what the value of "$foo" is, > although the new version of "ns_md5" has now a "-binary" option. > > > - Refactored low-level socket I/O: > * All driver operations are now strictly non-blocking > * Use same infrastructure for all network I/O operations > (driver, connchan, ns_http, ...) > * Rewritten socketfile/sendfile handling (including *BSD support) > * Improved handling of closed and detached channels, e.g.delegated > to writers (new macros NS_CONN_REQUIRE_CONNECTED, > NS_CONN_REQUIRE_OPEN, NS_CONN_REQUIRE_CONFIGURED, > NS_CONN_REQUIRE_ALL) > * Improved protection for SIGPIPE signals. > > - New API infrastructure: Ns_DList > > The Ns_DList is similar to Tcl_DString, but operates on pointers > instead of characters. Like Tcl_DString, Ns_DList pre-allocates > some data and extends it when necessary. Ns_DList structures are > especially useful alternative to linked lists, when usually data > is added to the end. Since the data is kept as a dense > (potentially growing) array, the memory locality is much better > than with linked lists, resulting in better CPU cache hit rates. > > - Distinguish between Ns_NormalizePath() and Ns_NormalizeUrl(), > where the first is for the file system and the latter for URLs. > Previously, both cases were handled identically. > > - Function pointer handling: > * Created a Tcl hash type for function pointers, since ANSI/ISO C > forbids casting/comparison of data and function pointers. The > new type is used whenever a hash lookup is performed for a > function pointer. > * Fixed all other occurrences of such comparisons. > > - Regressions testing: > * Removed false positive > * Switch to testing via ns_http in regression tests > (this makes it as well possible for testing https connections) > * Extended tests: > + Added regression tests for "ns_rand" > + Added nsssl tests > + Added automatic certificate generation for test-server > > * The following 35 tests were added or extended since the last release > > adp.test, encoding.test, http.test, http_byteranges.test, > http_chunked.test, http_keep.test, https.test, misc.test, > ns_adp_compress.test, ns_base64.test, ns_cache.test, > ns_conn.test, ns_conn_host.test, ns_crypto.test, > ns_driver.test, ns_hashpath.test, ns_hostbyaddr.test, > ns_info.test, ns_limits.test, ns_md5.test, ns_nsv.test, > ns_pagepath.test, ns_parsefieldvalue.test, ns_proxy.test, > ns_reflow_text.test, ns_schedule.test, ns_server.test, > ns_serverpath.test, ns_set.test, ns_sha1.test, > ns_urlencode.test, ns_urlspace.test, ns_uuencode.test, > ns_writer.test, tclconnio.test, tclresp.test > > - Improved portability: > * improved compatibility with LibreSSL (2.7 and 2.9) > * Windows changes (Many thanks to Andrew Piskorski): > + Fixed Ns_LogRoll() to work on Windows. > + WSASend receives as 5th argument flags, but not a pointer to flags. > + Improved makefiles > + Updated _MSC_VER version numbers to include Visual Studio 2019 > + Added macro NS_INLINE to achieve higher portability with (older?) > Microsoft compilers. > > - Added warnings about potential misconfiguration of NaviServer "tcllib" > setting (Tcl written NaviServer modules) > > - Marked compatibility wrappers Ns_SetThreadServer() and > Ns_SetThreadServer() explicitly as deprecated. Use > Ns_ThreadSetName() and Ns_ThreadGetName() instead. > > - Build-system > * Added -DSYSTEM_MALLOC to default compile flags. > * Added nsssl to the default test target. > * Require RSA keys of size 2048: newer versions of OpenSSL refuse to work > with RSA keys of size 1024. > * Improved alignment with Tcl's current .m4 file to get rid of > "-prebind" deprecated message under macOS. > * Improved handling of posix thread library for FreeBSD and OpenBSD > > - Use native thread_local storage for log handling when available > (experimental). > > - Improved type cleanness for function pointers > - Improved code locality > - Improved error messages > - Improved structure packing > - Reduced variable scopes > - Added missing "extern" declarations > - Aligned function prototypes > - Dropped potentially dangerous call to alloca() > - Added typedefs for commonly used functions > - Reduce potential dangling pointer dereferences > - Aligned names of arguments in prototype with function definition > - Introduced use attribute-based approach for denoting > fall through in case statements > - Adding more declarations for PURE and CONST functions > - Improved configurability for clang-tidy > > - Fixed macro name-clash with PostgreSQL > - Improved spelling > > > > Modules: > -------- > > .... > > > _______________________________________________ > naviserver-devel mailing list > nav...@li... > https://lists.sourceforge.net/lists/listinfo/naviserver-devel -- *Wolfgang Winkler* Geschäftsführung wol...@di... mobil +43.699.19971172 dc:*büro* digital concepts Novak Winkler OG Software & Design Landstraße 68, 5. Stock, 4020 Linz www.digital-concepts.com <http://www.digital-concepts.com> tel +43.732.997117.72 tel +43.699.1997117.2 Firmenbuchnummer: 192003h Firmenbuchgericht: Landesgericht Linz |
From: Gustaf N. <ne...@wu...> - 2021-01-04 18:19:03
|
Dear all, on sourceforge is a release candidate for NaviServer 4.99.20 [1]. This release was originally planned to be a small release (essentially the movement from mercurial to git, as forced by bitbucket), but turned out to be the release with the highest number of changes so far. The short summary of changes is (see below for more details): - Support for time units - Subsecond granularity for all NaviServer commands - HTTP client request logging - Added support for "100 CONTINUE" - Improved server liveliness (management of high loads and request queue overruns) - Support for running behind a reverse proxy server - Driver support for listening on multiple ports - Improved HTTPS support (OCSP Stapling, server-side SNI) - Improved support for client-side HTTP requests via ns_http (timeout handling, binary hindling, log file for outgoing requests) - Improved logging for detecting long running background jobs - Dict support for nsvs - More complete set of subcommands for "ns_set" - Improved form-processig (especially for REST processing) - C-level WebSocket support - Improved server statistics - Improved scalablility (reduced lock contention, higher usage of rwlocks, significanly faster nsv_*appends, ...) - Improved portability: many improvements for windows, compiled with macOS 10.14.6, Ubuntu 20.04, Fedora Core 32, OpenBSD 6.8 (LibreSSL 3.1.1), FreeBSD 12.2 The following people have contributed to this release: Andrew Piskorski David Osborne Gustaf Neumann Maksym Zinchenko Oleg Oleinick Zoran Vasiljevic Below is a preliminary summary of changes. Please test if possible. The release should be in about one week. Future goals: - move towards release 5.0 - upgrade license from Mozilla 1.1/GPL -> Mozilla 2.0 (MPL allows these upgrades, change is necessary for debian packaging with with OpenSSL - change to multiple branches (make current master to the 4.99 branch, provide in future for every release an own branch to ease providing fixes for older releases) best regards -gustaf [1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.19/ ======================================= NaviServer 4.99.20, released 2021-01-XX ======================================= 379 files changed, 25411 insertions(+), 11455 deletions(-) New Features: ------------- - Ns_Time reform * Support for time units: . arguments of functions accepting time durations can be provided now with a time unit suffix (such as ms, s, m, d). These values can be written e.g. as 1.5ms, 3h and the like. . units can be used as configuration values wherever appropriate . time values wherever appropriate in documentation . commands using Ns_ObjvTime can be specified now with time units . This change is fully backward compatible for all commands and configuration options accepting seconds, which was normally the case. . All sample configuration files have been updated with time units . nsproxy: The only exception to the precious sentence is nsproxy, where durations we specified in earlier releases as milliseconds. This is a POTENTIAL INCOMPATIBILITY with previous versions. These parameters are * config parameters of the nsproxy module "evaltimeout", "gettimeout", "idletimeout", "recvtimeout", "sendtimeout", "waittimeout" * ns_proxy get: changed "-timeout" parameter to time unit value * ns_proxy wait: changed optional "timeout" parameter to time unit value * Change granularity from seconds to Ns_Time (internally and in Tcl API) . Added proper handling of negative values in Ns_Time (was broken/not implemented) . Upgraded scheduling machinery from seconds to Ns_Time values . Extended Ns_Time arithmetic (Ns_DiffTime()) to handle positive and negative Ns_Time values . Extended regression test for ns_time significantly . Ns_After(), Ns_ScheduleProc(), Ns_ScheduleProcEx(): change argument from int secs to Ns_Time . Marked Ns_ScheduleProc() as deprecated, Ns_ScheduleProcEx() should be used instead. . ns_schedule_proc, ns_after: support time specs with time units (instead of only full seconds) . Ns_TimeToMilliseconds(): new API call for converting positive and negative Ns_Time values to ms . Ns_DStringAppendTime(): format times in a more compact form (no trailing zeros) . Use consistently Ns_TimeToMilliseconds() to convert Ns_Time to ms . Ns_ObjvTimeRange, Ns_CheckTimeRange(): support to specify time ranges in argument handling . Trim trailing zeros after comma point when printing Ns_Time, represent as int when possible . "ns_info scheduled" returns now timestamps in sub-second resolution (with a second faction). By this, the runtime of the scheduled procedure can be determined on the sub-second level. (potential incompatibility) . Added handling of "schedmaxelapsed" on the sub-second level. - HTTP client request logging NaviServer can now optionally write a logfile for outgoing HTTP requests initiated via "ns_http". This is important for webservers that are dependent on other web services for the heathiness and performance. The logfile is similar to the access.log and suitable for automated analysis. Previously, NaviServer was "blind" on that data. Now it is easier to provide data about the performance of external services (e.g. cloud services such as authentication, Google Firebase Cloud Messaging, Office 365, ...) To activate HTTP client request logging, a new config-file section "ns/server/$server/httpclient" was created where logroll, logrollfmt and the other usual logging parameter can be specified. - New command line option for NaviServer: "-T" When this option is used, NaviServer just checks the configuration file (specified as usual by "-t") for syntax errors. This option should be used on production sites to reduce the downtime of a server in case of typos. - Added support for "100 CONTINUE" This feature is in HTTP since a very long time, but not widely supported although useful (best support is currently in curl). When the client sends "Expect: 100-continue", the server can check the header fields and provide an early feedback, whether it is useful to transmit the full body. This way, e.g. content which is too large can be rejected quickly. - Improved server liveliness under high load (improved and configurable handling of connection queue overrun) When NaviServer processes an HTTP request, it is received (head and body) and associated with a connection thread pool. If the thread pool has a thread free, the request is directly assigned to this thread. Otherwise, the request is added to a queue of the connection pool, waiting for a connection thread. Since the queue has a limited size (limited by "maxconnections"), this queue might overrun. In this situation, the received request is added to a waiting list. Here is, where the behavior is now improved. 1) previously, as soon as there were entries in the waiting list, no more fresh requests were accepted by the driver (just working on waiting requests). While this behavior is sensible if only one connection pool is available, it causes a blocking behavior in cases, when a single connection pool overruns but others could continue. Now, more fresh requests are still accepted in this situation. 2) In addition, it is now possible to configure the overrun behavior via two pool parameters: a) pool parameter "rejectoverrun": when set to true, NaviServer sends on queue overruns an error "503 service unavailable". The overflowing request is not added to the waiting list. b) when (a) is set, it is possible to specify parameter "retryafter" with a seconds interval to add reply header field "Retry-After" to the 503 response. (see https://tools.ietf.org/html/rfc7231#section-7.1.3) When "rejectoverrun" is false (default) then behavior is similar to before, accept the server does not fully block. The request is kept in the waiting list, which is checked, whenever a request finishes. For the client, there is no need to resubmit the request, which will be proceed eventually. Technically, this means an arbirary large queue. When "rejectoverrun" is true, then the request causing the overrun is rejected, and will have to be resubmitted by the client (maybe automatically, when "Retry-After" is specified. However, the client support for "Retry-After" is rather bad. For busy sites, "rejectoverrun" is probably the better behavior, since no background queuing in the waiting list is performed. For now, we keep the old behavior as default for better backwards compatibility. The default might change in the future. - General support for running NaviServer behind a reverse proxy server This change provides general support for running NaviServer behind a reverse proxy. Previously, the support was rather punctual (e.g. for nslog). Reverse proxy mode is activated by setting the global parameter (in section "ns/parameters") named "reverseproxymode" to true (default false). Current limitation: In the current implementation, NaviServer supports only the "X-Forwarded-For" header field, which is supported by most proxy servers, treating the first provided IP-address as the client IP address (see https://en.wikipedia.org/wiki/X-Forwarded-For). The newer "Forwarded" header field is currently not supported. In the reverse proxy mode, the client address is taken as forwarded by the reverse proxy- This effects: * Reported IP address by the following commands. ns_conn peeraddr ns_server all|running ns_writer list ns_connchan list * Reported IP address in the access.log * Context filters (used e.g. for mapping requests to pools) In reverse proxy mode, all IP-address specific context filters refer to the x-forwarded-for addresses. This makes it possible to use context filters (of URLspace commands) based on IP-addresses (IPv4, IPv6, with masks) also behind reverse proxy servers. Previously the IP-based mapping was just possible when no reverse proxy was in use. - Allow a single driver to listen on multiple ports: This new feature simplifies setups, where a single server is listening on multiple ports using the same driver. Previously, it was necessary to define separate drivers for such cases, which need different names but which are often configured identically. Together with the change introduced in 4.99.19 (support for multiple listen addresses) this means that when multiple addresses and multiple ports are specified for a single driver, it will listen on the cross product of all combinations. The maximum number of listening ports of a driver is limited by a compile time constant to MAX_LISTEN_ADDR_PER_DRIVER (per default 16). To use this feature, specify in the configuration file for "port" in the network driver section a space separated list of multiple ports. This change is fully backwards compatible, old configuration files will continue to work. - SSL/TLS improvements: * Support for OCSP Stapling . NaviServer obtains and provides information about the validity of the used certificates. For proving this information the server performs two level of caching: in-memory caching and disk caching. When the server receives the first TLS request with OCSP stapling turned on, it checks for an already retrieved OCSP response. The disk cache file is saved in the "log" directory of the server and uses the serial number of the certificate to be checked as filename with ".der" as extension. When the disk cache file does not exist, an http/https request is made to the server issuing the servers certificate as defined by the Authority Information Access (AIA) Extension. The names of the file and the http/https request for the OCSP response can be obtained from the system log of the server . OCSP Stapling can be turned on in the configuration files (see sample configuration files) . OCSP Stapling requires OpenSSL 1.1.0 or newer * Support for server-side SNI (Server Name Indication (SNI) Server-side SNI can be used to use different certificates for virtual hosts. In addition to the top-level certificate, different certificates can be defined per virtual server (below: srv2). The certificate will be registered for all alternate names specified for the server. In addition to the certificate, also all other OpenSSL specific parameters (ciphers, protocols, ...) will be loaded form the same place as the certificate. The following example shows the setup for two virtual servers "srv1" and "srv2", where the first one is the default server. ns_section ns/modules { # ... ns_param nsssl ${bindir}/nsssl.so } ns_section ns/module/nsssl { # ... ns_param defaultserver srv1 ns_param certificate /usr/local/ns/modules/nsssl/srv1.pem } ns_section "ns/module/nsssl/servers" { ns_param srv1 test.org ns_param srv1 www.test.org ns_param srv2 someotherhost.org } ns_section "ns/server/srv2/module/nsssl" { # ... ns_param certificate /usr/local/ns/modules/nsssl/srv2.pem ns_param ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 } SNI support includes OCSP, DH ephemeral key exchange, reuse of SSLcontexts. * Support for OpenSSL "ciphersuites" (TLSv1.3 feature) * Support for client-side SNI: new parameter "hostname" for "ns_connchan open" * Support for removing TLSv1.0, TLSv1.1, TLSv1.2, and TLSv1.3 via configuration file - ns_http: * Clear separation of "-timeout" and "-expire": . "-timeout" is used for single communication acts (e.g. connect, read write) . "-expire" is used for limiting the full duration of the request (also useful for slow read/write attacks). Scripts already using the -timeout will most probably not see any change unless they are handling huge up/downloads. In such cases we will now continue with the request, instead of terminating it in the middle. * added "-binary" flag to transmit data binary and perform no automatic encoding transformations. - moved code repository on BitBucket from mercurial to git. * adjusted documentation * adjusted build system (Makefiles) - Logging improvements: * New log severity "Debug(timeout)" to force NaviServer to write an entry to the systemlog, whenever a timeout occurs. Therefore, the command ns_logctl severity Debug(timeout) on can be used to debug cases, where NaviServer runs into HTTP status code 503 (Service Unavailable). * Logging of commands with high latency: Provide a consistent interface and naming convention for logging slow commands for nsproxy, ns_jobs, scheduled procedures and SQL (nsdb drivers). . nsproxy and nsjobs have a new configuration option "logminduration" This configuration option specifies a time-limit for logging (long) eval operations to the system log (similar to "logminduration" in the database drivers). Set it to a high value to avoid logging (e.g. 1d). The default is 1s. . scheduled procedures: Changed "schedmaxelapsed" to "schedlogminduration" for consistency. This is a potential incompatibility for old configurations. - New commands: * "nsv_dict": The new command "nsv_dict" provides thread-safe support for setting, getting, updating, and unsetting dict values in nsv array. Motivation: Some larger OpenACS installations have currently more than 1 mio (!) nsv arrays, which comes mostly from creating nsv arrays per packages. Most of these arrays have only a few keys with very short values. The number of arrays can be greatly reduced by supporting complex values (dictionaries), similar to REDIS hash values. . "nsv_dict" supports nested dicts like in Tcl . The following dict commands: nsv_dict append array key dictkey dictvalue:0..n nsv_dict exists array key dictkey:1..n nsv_dict get array key dictkey:0..n nsv_dict getdef array key dictkey:0..n default nsv_dict incr array key dictkey ?increment? nsv_dict keys array key ?pattern? nsv_dict lappend array key dictkey dictvalue:0..n nsv_dict set array key dictkey:1..n dictvalue nsv_dict size array key nsv_dict unset array key dictkey:1..n . added 41 test cases in regression test for nsv_dict * "ns_dbquotelist": similar to "ns_dbquotevalue", but provides quoting for Tcl lists. * "ns_trim": multiline trim command: ns_trim ?-subst? ?-delimiter delimiter? text - New features for existing commands/subcommands * New subcommands "keys" and "values" for "ns_set": "ns_set keys /setId/ ?pattern?" "ns_set values /setId/ ?pattern?" The commands are similar to the Tcl command "dict keys" and "dict values" and can reduce the necessity for loops. * Form processing: . "ns_getformfile": return a list of filenames when "multiple" attribute is provided in the input field and multiple files are submitted . Decouple content-type processing from HTTP methods: Previously, parsing of form data in these two formats happened only on POST requests. However, several REST services use often also PUT with form data and distinguish POST and PUT methods mostly by "add item to a collection" (POST) and process single item (PUT) but not by their content-type processing. This change eases implementation of REST services. . Perform more strict parsing based on "ns_parsefieldvalue" and "ns_parseheader" in file-based form parser (for spooled content). Previously, there was a bug with the regexp-based parsing, when e.g. a filename of and upload file contained e.g. a semicolon. * "ns_conn": . new subcommand "headerlength" . "ns_conn auth" returns now Information about "Bearer" authentication (RFC 6750) when applicable, used e.g. in OAuth2 . "ns_conn peeraddr": New flag "-source" for specifying detailed information about the peer, especially useful in combination with reverse proxy mode; allowed values are "configured", "direct", "forwarded". * "ns_cookie": Support explicit setting of "same-site=none" In previous version of browsers, no explicit setting of the samesite flag was exactly the same as explicit setting. Since some browsers switched to a default of "lax", explicit setting became necessary. * "ns_dbquotevalue": provided a C implementation; was scripted before. * "ns_connchan": . Added option "-server" to "ns_connchan close" to be able to cleaning stale handles of every server. . Added C-level WebSocket support and buffered output handling. The new code is about 200x faster than the Tcl version (especially when bytewise XORs are needed), simpler to use (no Tcl-leval handling of incomplete frames, split frames, remaining leftover from partial writes) and more complete (handling of segmented WebSocket messages). . "ns_connchan read -websocket ....": returns a dict containing the websocket data (including payload, websocket status bits, buffering state, etc.; see documentation and NaviServer websocket module for details) . "ns_connchan write -buffered ..." handles partial writes via an internal buffer (Tcl developer has just to take care about writable callbacks when necessary) . "ns_connchan wsencode ...": WebSocket frame generation, returns binary data based on input parameters suitable for sending on a WebSocket channel. . "ns_connchan status ..." returns a dict containing the buffering status of a channel. * "ns_urelencode": added one more option "oauth1" for the "-part" argument. The "oauth1" encoding is defined in RFC 5849 Section 3.6 for the construction of the signature base string and the "Authorization" header field. * "ns_ictl": . "ns_ictl trace idle ...": new tracetype "idle", which fires, when a connection thread is idle for the threadtimeout period, but minthreads is already reached, and the thread won't terminate. . "ns_ictl maxconcurrentupdates ?max?": Configuration option for specifying the maximum number of concurrent updates for interpreter releads (when epoch incrases, e.g. reloads on OpenACS). Background: For large (say 50-100 connection threads) and busy sites (sustained 800 requests per second) with large blueprints (e.g. OpenACS) blueprint updates can cause delays when all blueprint are currently updates. The blueprint updates happen when the epoch is incremented (e.g. in OpenACS: reloads of packages). A too high number of concurrent blueprint updates cases (a) delays in request processing (b) increased update times per thread (e.g. up to 10 seconds). As a consequnce, queueing will happen. This new option limits the number of concurrent updates (e.g. to 10) leading to smooth operations in such situations. When the maximum number of concurrent blueprint is reached, the pending updates are delayed until the limit is sufficiently low. By default, the value is unlimited. * Improved statistics: . "ns_server stats" returns now as well dropped requests from queue overruns. . "ns_info locks": report number of read and write operations for rwlocks. . "ns_proxy stats": report also the number of running worker processes. * "ns_getcsv": . provide compatibility with RFC 4180 . accept multi-line input . handle quotes in quotes and empty elements . optionally stripping unquoted elements (not covered by RFC) . optional support for a different quote character (option "-quotechar"). Although RFC 4180 specifies double quotes as quoting chars, sometimes also different quoting characters, which can be now successfully decoded. * "ns_parseurl": Support for fragment parsing. When provided URL contains a tail element, this is now parsed info a "query" and/or "fragment" component in the result. Previously, the query and fragmet part was included in the "tail" element. * Improved thread naming for log-files: previously, it was not clear, from which server some output of the "main" thread originated. The new version adds always a server name specific suffix. * "ns_driver info": return also "port" and "defaultport" Bug Fixes: ---------- - Improved support for newer versions of OpenSSL (handling of SSL_R_UNEXPECTED_EOF_WHILE_READING) - ns_cache: Base base cache size calculation on the full size of a cache entry. In previous versions, the cache size calculation was based only on the value size, leaving the size of the key and the size of the overhead (size of cache entry, about 126 bytes on a 64-bit machine) aside. This lead to misleading interpretations, when the cache value size is 0 bytes (just using the cache key as a flag), leaving admins wondering about the memory consumption. - Report to log file, when ns_ictl traces are not executed. Previously, these were silently ignored (e.g. in some error cases). - Fixed a bug with PUT methods, when query-parameters were passed in addition to the put content. Previously, these parameters were only available via [ns_queryget ...] when the content file was not spooled. - Fixed bug for custom error pages for HTTP methods other than GET (Custom error handles are defined as before via redirect sections): * Always use GET in the redirection not the original HTTP method. When e.g. "PUT /somepage" is requested, where no PUT handler but a custom error page /405.tcl, the previoud code tried a PUT /405.tcl, leading the same error. Now, the error page us delivered like with a GET request. * Provided a hint as a warning, where NaviServer searches for non-existing pages. * Extended regression test: cover HEAD requests for existing and non-existing pages, and various forms of redirects - Fixed issue noted by Maksym Zinchenko on the naviserver developer list A recent change of tDOM 0.9.2 introduced a namespace local command "namespace" which was picked up by the blueprint generator incorrectly instead of the global ::namespace command. As a fix, all "namespace" invocations in the blueprint generator were replaced by "::namespace" to avoid considering invocation context for every occurrence. - Fixed potential abortion of regression testruns under macOS Under macOS, tests runs were sometimes abruptly ended with a broken pipe signal. The call "NsBlockSignal(NS_SIGPIPE);" does not avoid these signals. So we ignore bluntly all such signals by using "NsBlockSignal(NS_SIGPIPE);" - Fixed charset handling for spooled requests. - Fixed several cases of invalid chunked data (potential buffer overflows). - Don't swallow errors in braced config sections. - Blueprint generation: * Fixed bug with interp aliases defined without fully qualified namespace path * Improve robustness when "rename" is used (when source exists and target does not exist) - Support calling "ns_conn status" (with the argument setting the HTTP return status) in traces (without open connection). - Make code more robust in case Tcl and NaviServer were compiled with different memory allocators. - Race Conditions: * Fixed several race conditions in "ns_http" * Fixed race conditions in "ns_connchan list" and "ns_connchan create" - Use time_t for secs in Ns_Time, since long is just 32-bit on 64-bit windows, but time_t is 64 bits. - "ns_writefp": fixed to act as advertised (read to end of file when no length is provided) - "ns_sendmail": Convert leading "." in message bodies to ".." as required by RFC 5321 section 4.5.2 - "ns_cache" transactions: in certain situations, where during concurrent entry creates, and one of the create attempts failed and the other succeeded, and the one of the cache transitions is rolled back, updates could lead to an error. - "ns_connchan": Fixed registry of connchans from all servers to the default server (not the best scalable solution, but useful as long we have only one "socks" thread (for more details see: https://sourceforge.net/p/naviserver/mailman/message/37179476/) Performance improvements: ------------------------- - nsv_lappend: The old implementation showed a linear slowdown for nsv_lappend operations depending on the number of elements in the list. The new implementation avoids frequent string to list and vice versa operations by using Tcl_DStrings. It is essentially independent on the number of elements in the nsv. For a list with 10K elements, the performance improvement is about a factor of 400. Test: nsv_set d 1 1 time {nsv_lappend d 1 1} 10000 OLD: 403.7255 microseconds per iteration NEW: 0.860547 microseconds per iteration This changes makes it possible to use the OpenACS developer support on busy sites (before, this lead to a quick standstill). - nsv_append: The performance gain is much less than with nsv_lappend (see previous commit), but still noticeable (about 35% faster, 10K ns_append operatiopns went from 0.94 microseconds to 0.62 - Locking: * Reduced mutex-locked regions in driver. * Use ptrhreads interface for Ns_RWLock* when available So far, Ns_RWLock* used an implementation based on mutexes and cond vars. This implementation has the advantage of being portable. However, since on all *nix platforms, NaviServer uses the pthread library, using the POSIX rwlock implementation poses no additional dependency. Performance comparison fur using mutex/old rwlocks/pther rwlocks on ns_urlspace; macOS (8 cores) rwlock (POSIX) 0.3105 2,27 % rwlock (ns) 11.5263 84,23 % mutex (ns) 13.6839 100,00 % Linux (6 cores) rwlock (POSIX) 0.6081 43,38 % rwlock (ns) 3.6259 258,66 % mutex (ns) 1.4018 100,00 % This data shows that the POSIX rwlocks implementation (pthread_rwlock_rdlock etc.) is always better than the classical NavisSrver rwlock implementation, and that significant improvements can be achieved by using rwlocks. A more detailed performance comparison was provided on the developer mailing list. This change introduces: a) Ns_RWLock* based on pthreads when available b) switch to recommended configure setup for pthreads (AX_PTHREAD) c) New API call: Ns_RWLockSetName2() modeled after Ns_MutexSetName2() d) New API call: Ns_RWLockList() modeld after Ns_MutexList() (removed dependency of rwlocks to mutex statistics) * Support rwlocks for nsvs: Per default, rwlocks are used for all nsv* commands. The default behavior can be altered by setting the configuration variable "nsvrwlocks" to false in the section ns/server/${server}/tcl of the configuration file. This change can lead to improved scalability especially on multi-core machines since in read cases, the variables can be accessd unblocked for parallel executions. In most applications nsv are typically more often read than written rwlocks offer better scalability for large sites. The following table compares the busy locks with mutex operations vs the number of busy locks with rwlocks for openacs.org: With mutex (24h) locks busy nsv:3:openacs.org 27 4.71M 1.3K nsv:6:openacs.org 24 4.88M 1.03K nsv:2:openacs.org 28 3.37M 784 nsv:7:openacs.org 23 9.11M 755 nsv:5:openacs.org 25 2.88M 460 With rwlocks (24h) nsv:7:openacs.org 1 7.22M 0 nsv:6:openacs.org 2 3.92M 143 nsv:3:openacs.org 5 3.31M 1 nsv:2:openacs.org 6 2.23M 16 nsv:5:openacs.org 3 2.16M 0 On more busy sites, the improved scalability si much higher. * rwlocks are better in cases, where the number of readers is significantly higher than the number of writers. So, it has to be carefully considered in which cases rwlocks should be preferred. The new version uses rwlocks for ns_connchan, filters and tcl.cache. - Avoid multiple library initializations for OpenSSL 1.1.0 or newer. Documentation improvements: --------------------------- - Update references to Tcl website (replace http://tcl.tk by https://core.tcl-lang.org) - Added hints about required OpenSSL version to man pages - Made man pages more consistent - Consistent indentation of program examples in documentation - Made source code documentation more consistent - Improved source code documentation (provide more details, remove obsolete comments) - Removed deprecated man pages - Removed obsolete sections in man pages - Follow the recommended spelling convention of the Linux documentation project - Improved spelling, typesetting, and cross references - Reduced technical jargon - Made code listings in documentation more consistent (e.g. indenting) - Added missing commands in command list (overview page) - Updated listing of deprecated commands and suggested alternatives - Replaced several "slave process" by "child process" - 161 man pages were updated since the last release Configuration Changes: ---------------------- - New configuration options: * Section "ns/parameters" . "reverseproxymode": Activate reverse proxy mode. The former configuration parameter of "nslog" named "checkforproxy" is obsolete now. . "joblogminduration": report jobs taking longer in systemlog . "schedlogminduration": report scheduled jobs taking longer in systemlog * Section "ns/server/${server}" . "rejectoverrun": when true send 503 when queue overruns; default "false" . "retryafter": time for Retry-After in 503 cases * Section "ns/server/${server}" . "nsvrwlocks": use RWloccks for nsv (default: true) * Section "ns/server/$server/httpclient" . "logging": (default: off) . "logfile": name of http client logfile . "logrollfmt": format appended to log filename . "logmaxbackup": max number of backup log files (default: 10) . "logroll": perform automatic rolling (default: true) . "logrollonsignal": (default: false) . "logrollhour": specify at which hour to roll (default: 0) - Sample configurations: * Added values with time units where appropriate * Added new values mentioned above with short comments * For multi-server configuration files: use always the first (and not the last) server as default server. * openacs-config.tcl: * sample-config.tcl: . fixed config parameter: use "dnscachetimeout" instead of "keepwaittimeout" (name was changed in Naviserver ages ago, but change was not reflected in config files * nsd-config.tcl: * win32/test/nsd.tcl modernized test config file * nsproxy: Changed term "maxslaves" to "maxworkers" (the old term is deprecated) Code Changes: ------------- - Fixed potential access to a out-of-scope stack variable. - Added more declarations for PURE and CONST functions - Replaced usage of reserved identifier [cert-dcl37-c,cert-dcl51-cpp] - Replaced sscanf() by strtol*() to increase code-safety - Improved structure packing - Marked "ns_adp_mime" explicitly as deprecated, and provide deprecated messages for it. The command was deprecated more than 10 years ago, ns_adp_mimetype should be used - Added mime types for "heif" and "heic" which have become a preferred image type for iOS and macOS. - Regressions testing: * Reduced usages of old-style nstest::http-0.9 (socket and not ns_http based tests) * Make test more robust concerning crlf results (Windows) * Extended tests: 367 tests were added since the last release (changes in 38 test files) - Improved logging: * Added metering for long wait times of Ns_CsEnter locks (e.g. Ns_MasterLock) * Added concurrency level to log message of interp updates * Added warnings, when an NsTclRequestProc or an ADP request runs into a timeout. * Reduced time sensitivity on time when rolling log files_ Rolling log files happens often at midnight, using often a day precision. When e.g. a scheduled procedure the time when this function is called might be slightly after the scheduled time, which might lead to a day jump. The problem aggrevates, when multiple log files are rotated. The new code identifies day wraps and uses in such cases the earlier day. - Improved portability: * Improved compatibility with the forthcoming OpenSSL 3.0 * Improved compatibility with various versions of LibreSSL * Improved Windows (MSVC) compatibility . simplified Makefile.win32 for nmake . fixed thread exit result handling for 64-bit Windows . fixed thread handle access . turned on OpenSSL by default . minimal support for running regression tests under windows . Fixed windows size incompatibility for range requests (64-bit windows uses a 32-bit sized offset type off_t) * Use consequently NS_EAGAIN (can be used in *nix and Windows) * Fixed potential problem with GNUC_PURE for older version of gcc * Fixed regression test warnings with older versions of curl for IPv6 addresses * Fixed m4 configuration for "crypt_r" (broken since many years) * Compiled with macOS 10.14.6, Ubuntu 20.04, Fedora Core 32, OpenBSD 6.8 (LibreSSL 3.1.1), FreeBSD 12.2 - More OpenSSL changes: * Reactivate CRYPTO_set_mem_functions() for OpenSSL 1.1.0 or newer (interface om OpenSSL has changed) * Switched to SSL_CTX_set_dh_auto() from low-level functions: In newer versions of OpenSSL, all low-level DH* functions were deprecated. - Build-system * compute version tag from git slimilarly like to the mercurial version - improved support for MS windows * Windows math.h does not include round(), add simple implementation. - Added cross-platform function ns_localtime_r (localtime with user-provided storage): The previously defined function ns_localtime() uses a single per-thread local storage, so, it was not possible to have in one thread two time buffers with different values. - Improved error code handling on driver receive operations Previously, higher level of API calls were not able to access realiably the error information detected on the lower-level (async I/O-based or OpenSSL based) driver operations. This made it impossible to provide good error messages e.g. in "ns_connchan" and we could not provide error processing based on Tcl error codes. Furthermore, the new interface works for POSIX errnos and OpenSSL errors and improves the situation (currently just for driver read operations). * Ns_Sock* based setting of Tcl error codes: NsSockSetRecvErrorCode(const Sock *sockPtr, Tcl_Interp *interp) Set the Tcl error code in the interpreter based on the information provided by Sock* (actually by recvErrno); the function works for OpenSSL and plain POSIX style error codes. * Setting Tcl error codes for POSIX errno: Ns_PosixSetErrorCode(Tcl_Interp *interp, int errorNum) Set the Tcl error code in the interpreter based on the POSIX errorNum and return the error message. * Setting Tcl error codes for OpenSSL error values Ns_SSLSetErrorCode(Tcl_Interp *interp, unsigned long sslERRcode) Set the Tcl error code in the interpreter based on the OpenSSL error value and return the error message. * Return the last error from a socket. Ns_SockErrorCode(Tcl_Interp *interp, NS_SOCKET sock) The function Ns_SockErrorCode() returns the last error from a socket. When the provided interp is not NULL it sets the Tcl error code based on the error code. Modules: -------- ### AAA files changed, BBBB insertions(+), CCCC deletions(-) nsdb: added support for time units for nsdb LogMinDuration nsdbmysql: nssmtpd: nsdns: nsudp: nszlib: nsimap: nsphp: nsstats: nsdbi: nsdbipg: nsoracle: nswebsocket: revproxy: letsencrypt: nswebpush: nsladp: |
From: oleg <oo...@ua...> - 2021-01-10 16:05:56
Attachments:
naviserver.mingw.diff
|
Hello. We have tested Naviserver RC. No problem found on Ubuntu 20.04 and Windows10/MSVC2019/32bit. Have an issue for Windows10/MSVC2019/64bit build. Deprecated ns_httpget/post does not work with the error "select failed: no such file or directory". Code that causes an error: lassign [ns_sockopen -nonblock httpbin.org 80] rfd wfd ns_sockselect -timeout 1000 {} $wfd {} It looks like the problem is using DuplicateHandle on the socket handle and passing duplicated handle to the select call. >From DuplicateHandle docs: "Sockets. No error is returned, but the duplicate handle may not be recognized by Winsock at the target process." >From googling: should work in most cases. I was just unlucky. Crosscompiled (gcc/mingw) for Windows 64bit Naviserver works perfectly but there are difficulties in configure/make/install: We had to make changes to the code (patch attached): nsthread.h - some variables are not defined, connchan.c - definition of beXXtoh/htobeXX, as for MSVC, dstring.c - realloc dstrings as for older MSVC, exec.c - lowercased header. Configure magic does not detect some settings for MINGW, they need to be specified explicitly ./configure --host=x86_64-w64-mingw32 --enable-64bit \ --prefix=<path> --with-zlib=<path> --with-openssl=<path> --with-tcl=<path>/lib CFLAGS="-DHAVE_INET_PTON -DHAVE_INET_NTON -DHAVE_GETADDRINFO -D_WIN32_WINNT=0x600" \ LDFLAGS="-static-libgcc" \ make LIBLIBS="-Wl,-Bstatic -lpthread -Wl,-Bdynamic" Installation script does not expect .exe extension, workaround cp nsthread/nsthreadtest.exe nsthread/nsthreadtest cp nsd/nsd.exe nsd/nsd cp nsproxy/nsproxy.exe nsproxy/nsproxy make install Regards, Oleg. |
From: Gustaf N. <ne...@wu...> - 2021-01-17 18:18:10
|
Dear all, The release of NaviServer 4.99.20 is now available on sourceforge [1]. This release was originally planned to be a small release (essentially the movement from mercurial to git, as forced by bitbucket), but turned out to be the release with the highest number of changes so far. The short summary of changes is (see below for more details): - Support for time units (specify time units in minutes, seconds, milliseconds...) - Subsecond granularity for all NaviServer commands - HTTP client request logging - Added support for "100 CONTINUE" - Improved server liveliness (management of high loads and request queue overruns) - Support for running behind a reverse proxy server - Driver support for listening on multiple ports - Improved HTTPS support (OCSP Stapling, server-side SNI) - Improved support for client-side HTTP requests via ns_http (timeout handling, binary hindling, log file for outgoing requests) - Improved logging for detecting long running background jobs - Dict support for nsvs - More complete set of subcommands for "ns_set" - Improved form-processig (especially for REST processing) - C-level WebSocket support - Improved server statistics - Improved scalablility (reduced lock contention, higher usage of rwlocks, significanly faster nsv_*appends, ...) - Improved portability: many improvements for windows, compiled with macOS 10.14.6, Ubuntu 20.04, Fedora Core 32, OpenBSD 6.8 (LibreSSL 3.1.1), FreeBSD 12.2 The following people have contributed to this release: Andrew Piskorski David Osborne Gustaf Neumann Hector Romojaro Maksym Zinchenko Oleg Oleinick Zoran Vasiljevic Below is a summary of changes. The major changes since RC1 are improved robustness in case sections are missing in the configuration file and improved windows support (also addressing the problem with ns_sockselect under windows 64-bit that Oleg reported on Jan 10). Under windows, the regression tests reports now 1611 passes and 65 fails.... we are improving. Future goals: - Move towards release 5.0 - Upgrade license from Mozilla 1.1/GPL -> Mozilla 2.0 (MPL allows these upgrades, change is necessary for debian packaging with OpenSSL) - Change to multiple branches (make current master to the 4.99 branch, provide in future for every release an own branch to ease providing fixes for older releases; the last digit of the version number should be just the patch level; tags are unchanged on the patch level) best regards -gustaf [1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.20/ ======================================= NaviServer 4.99.20, released 2021-01-17 ======================================= 380 files changed, 27639 insertions(+), 12103 deletions(-) New Features: ------------- - Ns_Time reform * Support for time units: . arguments of functions accepting time durations can be provided now with a time unit suffix (such as ms, s, m, d). These values can be written e.g. as 1.5ms, 3h and the like. . units can be used as configuration values wherever appropriate . time values wherever appropriate in documentation . commands using Ns_ObjvTime can be specified now with time units . This change is fully backward compatible for all commands and configuration options accepting seconds, which was normally the case. . All sample configuration files have been updated with time units . nsproxy: The only exception to the precious sentence is nsproxy, where durations we specified in earlier releases as milliseconds. This is a POTENTIAL INCOMPATIBILITY with previous versions. These parameters are * config parameters of the nsproxy module "evaltimeout", "gettimeout", "idletimeout", "recvtimeout", "sendtimeout", "waittimeout" * ns_proxy get: changed "-timeout" parameter to time unit value * ns_proxy wait: changed optional "timeout" parameter to time unit value * Change granularity from seconds to Ns_Time (internally and in Tcl API) . Added proper handling of negative values in Ns_Time (was broken/not implemented) . Upgraded scheduling machinery from seconds to Ns_Time values . Extended Ns_Time arithmetic (Ns_DiffTime()) to handle positive and negative Ns_Time values . Extended regression test for ns_time significantly . Ns_After(), Ns_ScheduleProc(), Ns_ScheduleProcEx(): change argument from int secs to Ns_Time . Marked Ns_ScheduleProc() as deprecated, Ns_ScheduleProcEx() should be used instead. . ns_schedule_proc, ns_after: support time specs with time units (instead of only full seconds) . Ns_TimeToMilliseconds(): new API call for converting positive and negative Ns_Time values to ms . Ns_DStringAppendTime(): format times in a more compact form (no trailing zeros) . Use consistently Ns_TimeToMilliseconds() to convert Ns_Time to ms . Ns_ObjvTimeRange, Ns_CheckTimeRange(): support to specify time ranges in argument handling . Trim trailing zeros after comma point when printing Ns_Time, represent as int when possible . "ns_info scheduled" returns now timestamps in sub-second resolution (with a second faction). By this, the runtime of the scheduled procedure can be determined on the sub-second level. (potential incompatibility) . Added handling of "schedmaxelapsed" on the sub-second level. - HTTP client request logging NaviServer can now optionally write a logfile for outgoing HTTP requests initiated via "ns_http". This is important for webservers that are dependent on other web services for the healthiness and performance. The logfile is similar to the access.log and suitable for automated analysis. Previously, NaviServer was "blind" on that data. Now it is easier to provide data about the performance of external services (e.g. cloud services such as authentication, Google Firebase Cloud Messaging, Office 365, ...) To activate HTTP client request logging, a new config-file section "ns/server/$server/httpclient" was created where logroll, logrollfmt and the other usual logging parameter can be specified. - New command line option for NaviServer: "-T" When this option is used, NaviServer just checks the configuration file (specified as usual by "-t") for syntax errors. This option should be used on production sites to reduce the downtime of a server in case of typos. - Added support for "100 CONTINUE" This feature is in HTTP since a very long time, but not widely supported although useful (best support is currently in curl). When the client sends "Expect: 100-continue", the server can check the header fields and provide an early feedback, whether it is useful to transmit the full body. This way, e.g. content which is too large can be rejected quickly. - Improved server liveliness under high load (improved and configurable handling of connection queue overrun) When NaviServer processes an HTTP request, it is received (head and body) and associated with a connection thread pool. If the thread pool has a thread free, the request is directly assigned to this thread. Otherwise, the request is added to a queue of the connection pool, waiting for a connection thread. Since the queue has a limited size (limited by "maxconnections"), this queue might overrun. In this situation, the received request is added to a waiting list. Here is, where the behavior is now improved. 1) previously, as soon as there were entries in the waiting list, no more fresh requests were accepted by the driver (just working on waiting requests). While this behavior is sensible if only one connection pool is available, it causes a blocking behavior in cases, when a single connection pool overruns but others could continue. Now, more fresh requests are still accepted in this situation. 2) In addition, it is now possible to configure the overrun behavior via two pool parameters: a) pool parameter "rejectoverrun": when set to true, NaviServer sends on queue overruns an error "503 service unavailable". The overflowing request is not added to the waiting list. b) when (a) is set, it is possible to specify parameter "retryafter" with a time interval to add reply header field "Retry-After" to the 503 response. (see https://tools.ietf.org/html/rfc7231#section-7.1.3) When "rejectoverrun" is false (default) then behavior is similar to before, accept the server does not fully block. The request is kept in the waiting list, which is checked, whenever a request finishes. For the client, there is no need to resubmit the request, which will be proceed eventually. Technically, this means an arbitrary large queue. When "rejectoverrun" is true, then the request causing the overrun is rejected, and will have to be resubmitted by the client (maybe automatically, when "Retry-After" is specified. However, the client support for "Retry-After" is rather bad. For busy sites, "rejectoverrun" is probably the better behavior, since no background queuing in the waiting list is performed. For now, we keep the old behavior as default for better backwards compatibility. The default might change in the future. - General support for running NaviServer behind a reverse proxy server This change provides general support for running NaviServer behind a reverse proxy. Previously, the support was rather punctual (e.g. for nslog). Reverse proxy mode is activated by setting the global parameter (in section "ns/parameters") named "reverseproxymode" to true (default false). Current limitation: In the current implementation, NaviServer supports only the "X-Forwarded-For" header field, which is supported by most proxy servers, treating the first provided IP-address as the client IP address (see https://en.wikipedia.org/wiki/X-Forwarded-For). The newer "Forwarded" header field is currently not supported. In the reverse proxy mode, the client address is taken as forwarded by the reverse proxy. This has the following consequences: * Reported IP address by the following commands. ns_conn peeraddr ns_server all|running ns_writer list ns_connchan list * Reported IP address in the access.log * Context filters (used e.g. for mapping requests to pools) In reverse proxy mode, all IP-address specific context filters refer to the x-forwarded-for addresses. This makes it possible to use context filters (of URLspace commands) based on IP-addresses (IPv4, IPv6, with masks) also behind reverse proxy servers. Previously the IP-based mapping was just possible when no reverse proxy was in use. - Allow a single driver to listen on multiple ports: This new feature simplifies setups, where a single server is listening on multiple ports using the same driver. Previously, it was necessary to define separate drivers for such cases, which need different names but which are often configured identically. Together with the change introduced in 4.99.19 (support for multiple listen addresses) this means that when multiple addresses and multiple ports are specified for a single driver, it will listen on the cross product of all combinations. The maximum number of listening ports of a driver is limited by a compile time constant to MAX_LISTEN_ADDR_PER_DRIVER (per default 16). To use this feature, specify in the configuration file for "port" in the network driver section a space separated list of multiple ports. This change is fully backwards compatible, old configuration files will continue to work. - SSL/TLS improvements: * Support for OCSP Stapling . NaviServer obtains and provides information about the validity of the used certificates. For proving this information, the server performs two level of caching: in-memory caching and disk caching. When the server receives the first TLS request with OCSP stapling turned on, it checks for an already retrieved OCSP response. The disk cache file is saved in the "log" directory of the server and uses the serial number of the certificate to be checked as filename with ".der" as extension. When the disk cache file does not exist, an http/https request is made to the server issuing the servers certificate as defined by the Authority Information Access (AIA) Extension. The names of the file and the http/https request for the OCSP response can be obtained from the system log of the server . OCSP Stapling can be turned on in the configuration files (see sample configuration files) . OCSP Stapling requires OpenSSL 1.1.0 or newer * Support for server-side SNI (Server Name Indication (SNI) Server-side SNI can be used to use different certificates for virtual hosts. In addition to the top-level certificate, different certificates can be defined per virtual server (below: srv2). The certificate will be registered for all alternate names specified for the server. In addition to the certificate, also all other OpenSSL specific parameters (ciphers, protocols, ...) will be loaded form the same place as the certificate. The following example shows the setup for two virtual servers "srv1" and "srv2", where the first one is the default server. ns_section ns/modules { # ... ns_param nsssl ${bindir}/nsssl.so } ns_section ns/module/nsssl { # ... ns_param defaultserver srv1 ns_param certificate /usr/local/ns/modules/nsssl/srv1.pem } ns_section "ns/module/nsssl/servers" { ns_param srv1 test.org ns_param srv1 www.test.org ns_param srv2 someotherhost.org } ns_section "ns/server/srv2/module/nsssl" { # ... ns_param certificate /usr/local/ns/modules/nsssl/srv2.pem ns_param ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 } SNI support includes OCSP, DH ephemeral key exchange, reuse of SSLcontexts. * Support for OpenSSL "ciphersuites" (TLSv1.3 feature) * Support for client-side SNI: new parameter "hostname" for "ns_connchan open" * Support for removing TLSv1.0, TLSv1.1, TLSv1.2, and TLSv1.3 via configuration file - ns_http: * Clear separation of "-timeout" and "-expire": . "-timeout" is used for single communication acts (e.g. connect, read write) . "-expire" is used for limiting the full duration of the request (also useful for slow read/write attacks). Scripts already using the -timeout will most probably not see any change unless they are handling huge up/downloads. In such cases we will now continue with the request, instead of terminating it in the middle. * added "-binary" flag to transmit data binary and perform no automatic encoding transformations. - moved code repository on BitBucket from mercurial to git. * adjusted documentation * adjusted build system (Makefiles) - Logging improvements: * New log severity "Debug(timeout)" to force NaviServer to write an entry to the systemlog, whenever a timeout occurs. Therefore, the command ns_logctl severity Debug(timeout) on can be used to debug cases, where NaviServer runs into HTTP status code 503 (Service Unavailable). * Logging of commands with high latency: Provide a consistent interface and naming convention for logging slow commands for nsproxy, ns_jobs, scheduled procedures and SQL (nsdb drivers). . nsproxy and nsjobs have a new configuration option "logminduration" This configuration option specifies a time-limit for logging (long) eval operations to the system log (similar to "logminduration" in the database drivers). Set it to a high value to avoid logging (e.g. 1d). The default is 1s. . scheduled procedures: Changed "schedmaxelapsed" to "schedlogminduration" for consistency. This is a potential incompatibility for old configurations. - New commands: * "nsv_dict": The new command "nsv_dict" provides thread-safe support for setting, getting, updating, and unsetting dict values in nsv array. Motivation: Some larger OpenACS installations have currently more than 1 mio (!) nsv arrays, which comes mostly from creating nsv arrays per packages. Most of these arrays have only a few keys with very short values. The number of arrays can be greatly reduced by supporting complex values (dictionaries), similar to REDIS hash values. . "nsv_dict" supports nested dicts like in Tcl . The following dict commands: nsv_dict append array key dictkey dictvalue:0..n nsv_dict exists array key dictkey:1..n nsv_dict get array key dictkey:0..n nsv_dict getdef array key dictkey:0..n default nsv_dict incr array key dictkey ?increment? nsv_dict keys array key ?pattern? nsv_dict lappend array key dictkey dictvalue:0..n nsv_dict set array key dictkey:1..n dictvalue nsv_dict size array key nsv_dict unset array key dictkey:1..n . added 41 test cases in regression test for nsv_dict * "ns_dbquotelist": similar to "ns_dbquotevalue", but provides quoting for Tcl lists. * "ns_trim": multiline trim command: ns_trim ?-subst? ?-delimiter delimiter? text - New features for existing commands/subcommands * New subcommands "keys" and "values" for "ns_set": "ns_set keys /setId/ ?pattern?" "ns_set values /setId/ ?pattern?" The commands are similar to the Tcl command "dict keys" and "dict values" and can reduce the necessity for loops. * Form processing: . "ns_getformfile": return a list of filenames when "multiple" attribute is provided in the input field and multiple files are submitted . Decouple content-type processing from HTTP methods: Previously, parsing of form data in these two formats happened only on POST requests. However, several REST services use often also PUT with form data and distinguish POST and PUT methods mostly by "add item to a collection" (POST) and process single item (PUT) but not by their content-type processing. This change eases implementation of REST services. . Perform more strict parsing based on "ns_parsefieldvalue" and "ns_parseheader" in file-based form parser (for spooled content). Previously, there was a bug with the regexp-based parsing, when e.g. a filename of and upload file contained e.g. a semicolon. * "ns_conn": . new subcommand "headerlength" . "ns_conn auth" returns now Information about "Bearer" authentication (RFC 6750) when applicable, used e.g. in OAuth2 . "ns_conn peeraddr": New flag "-source" for specifying detailed information about the peer, especially useful in combination with reverse proxy mode; allowed values are "configured", "direct", "forwarded". * "ns_cookie": Support explicit setting of "same-site=none" In previous version of browsers, no explicit setting of the samesite flag was exactly the same as explicit setting. Since some browsers switched to a default of "lax", explicit setting became necessary. * "ns_dbquotevalue": provided a C implementation; was scripted before. * "ns_connchan": . Added option "-server" to "ns_connchan close" to be able to cleaning stale handles of every server. . Added C-level WebSocket support and buffered output handling. The new code is about 200x faster than the Tcl version (especially when bytewise XORs are needed), simpler to use (no Tcl-leval handling of incomplete frames, split frames, remaining leftover from partial writes) and more complete (handling of segmented WebSocket messages). . "ns_connchan read -websocket ....": returns a dict containing the WebSocket data (including payload, WebSocket status bits, buffering state, etc.; see documentation and NaviServer websocket module for details) . "ns_connchan write -buffered ..." handles partial writes via an internal buffer (Tcl developer has just to take care about writable callbacks when necessary) . "ns_connchan wsencode ...": WebSocket frame generation, returns binary data based on input parameters suitable for sending on a WebSocket channel. . "ns_connchan status ..." returns a dict containing the buffering status of a channel. * "ns_urelencode": added one more option "oauth1" for the "-part" argument. The "oauth1" encoding is defined in RFC 5849 Section 3.6 for the construction of the signature base string and the "Authorization" header field. * "ns_ictl": . "ns_ictl trace idle ...": new tracetype "idle", which fires, when a connection thread is idle for the threadtimeout period, but minthreads is already reached, and the thread won't terminate. . "ns_ictl maxconcurrentupdates ?max?": Configuration option for specifying the maximum number of concurrent updates for interpreter reloads (when epoch increases, e.g. reloads on OpenACS). Background: For large (say 50-100 connection threads) and busy sites (sustained 800 requests per second) with large blueprints (e.g. OpenACS) blueprint updates can cause delays when all these operations are concurrently performed. The blueprint updates happen when the epoch is incremented (e.g. in OpenACS: reloads of packages). A too high number of concurrent blueprint updates cases (a) delays in request processing (b) increased update times per thread (e.g. up to 10 seconds). As a consequence, queueing will happen. This new option limits the number of concurrent updates (e.g. to 10) leading to smooth operations in such situations. When the maximum number of concurrent blueprints is reached, the pending updates are delayed until the limit is sufficiently low. By default, the value is unlimited. * Improved statistics: . "ns_server stats" returns now as well dropped requests from queue overruns. . "ns_info locks": report number of read and write operations for rwlocks. . "ns_proxy stats": report also the number of running worker processes. * "ns_getcsv": . provide compatibility with RFC 4180 . accept multi-line input . handle quotes in quotes and empty elements . optionally stripping unquoted elements (not covered by RFC) . optional support for a different quote character (option "-quotechar"). Although RFC 4180 specifies double quotes as quoting chars, sometimes also different quoting characters, which can be now successfully decoded. * "ns_parseurl": Support for fragment parsing. When provided URL contains a tail element, this is now parsed info a "query" and/or "fragment" component in the result. Previously, the query and fragment part was included in the "tail" element. * Improved thread naming for log-files: previously, it was not clear, from which server some output of the "main" thread originated. The new version adds always a server name specific suffix. * "ns_driver info": return also "port" and "defaultport" Bug Fixes: ---------- - Improved support for newer versions of OpenSSL (handling of SSL_R_UNEXPECTED_EOF_WHILE_READING) - ns_cache: Base cache size calculation on the full size of a cache entry. In previous versions, the cache size calculation was based only on the value size, leaving the size of the key and the size of the overhead (size of cache entry, about 126 bytes on a 64-bit machine) aside. This is misleading, when the cache value size is 0 bytes (just using the cache key as a flag), leaving admins wondering about the memory consumption. - Report to log file, when ns_ictl traces are not executed. Previously, these were silently ignored (e.g. in some error cases). - Fixed a bug with PUT methods, when query-parameters were passed in addition to the put content. Previously, these parameters were only available via [ns_queryget ...] when the content file was not spooled. - Fixed bug for custom error pages for HTTP methods other than GET (Custom error handles are defined as before via redirect sections): * Always use GET in the redirection not the original HTTP method. When e.g. "PUT /somepage" is requested, where no PUT handler but a custom error page /405.tcl, the previous code tried a PUT /405.tcl, leading the same error. Now, the error page us delivered like with a GET request. * Provided a hint as a warning, where NaviServer searches for non-existing pages. * Extended regression test: cover HEAD requests for existing and non-existing pages, and various forms of redirects. - Fixed issue with tDOM as noted by Maksym Zinchenko on the naviserver developer list: A recent change of tDOM 0.9.2 introduced a namespace local command "namespace" which was picked up by the blueprint generator incorrectly instead of the global ::namespace command. As a fix, all "namespace" invocations in the blueprint generator were replaced by "::namespace" to avoid considering invocation context for every occurrence. - Fixed potential abortion of regression test runs under macOS: Under macOS, tests runs were sometimes abruptly ended with a broken pipe signal. The call "NsBlockSignal(NS_SIGPIPE);" does not avoid these signals. So, we ignore bluntly all such signals by using "NsBlockSignal(NS_SIGPIPE);" - Fixed charset handling for spooled requests. - Fixed several cases of invalid chunked data (potential buffer overflows). - Don't swallow errors in braced config sections in the configuration file. - Fixed a potential crash, when certain configuration section were missing, but code tried to obtain configuration values from such sections. - Blueprint generation: * Fixed bug with interp aliases defined without fully qualified namespace path * Improve robustness when "rename" is used (when source exists and target does not exist) - Support calling "ns_conn status" (with the argument setting the HTTP return status) in traces (without open connection). - Make code more robust in case Tcl and NaviServer were compiled with different memory allocators. The mix if memory allocators could crash NaviServer. - Race Conditions: * Fixed several race conditions in "ns_http" * Fixed race conditions in "ns_connchan list" and "ns_connchan create" - Use time_t for secs in Ns_Time, since long is just 32-bit on 64-bit windows, but time_t is 64 bits. - "ns_writefp": fixed to act as advertised (read to end of file when no length is provided) - "ns_sendmail": Convert leading "." in message bodies to ".." as required by RFC 5321 section 4.5.2 - "ns_cache" transactions: in certain situations, where during concurrent entry creates, and one of the create attempts failed and the other succeeded, and the one of the cache transitions is rolled back, updates could lead to an error. - "ns_connchan": Fixed registry of connchans from all servers to the default server (not the best scalable solution, but useful as long we have only one "socks" thread (for more details see: https://sourceforge.net/p/naviserver/mailman/message/37179476/) - Fixed a 20-year-old time bomb in HTTP time parsing. Server was rejecting to parse HTTP time strings for 2021. Performance improvements: ------------------------- - nsv_lappend: The old implementation showed a linear slowdown for nsv_lappend operations depending on the number of elements in the list. The new implementation avoids frequent string to list and vice versa operations by using Tcl_DStrings. It is essentially independent on the number of elements in the nsv. For a list with 10K elements, the performance improvement is about a factor of 400. Test: nsv_set d 1 1 time {nsv_lappend d 1 1} 10000 OLD: 403.7255 microseconds per iteration NEW: 0.860547 microseconds per iteration This change makes it possible to use the OpenACS developer support on busy sites (before, this led to a quick standstill). - nsv_append: The performance gain is much less than with nsv_lappend (see previous commit), but still noticeable (about 35% faster, 10K ns_append operations went from 0.94 microseconds to 0.62 - Locking: * Reduced mutex-locked regions in driver. * Use ptrhreads interface for Ns_RWLock* when available So far, Ns_RWLock* used an implementation based on mutexes and cond vars. This implementation has the advantage of being portable. However, since on all *nix platforms, NaviServer uses the pthread library, using the POSIX rwlock implementation poses no additional dependency. Performance comparison fur using mutex/old rwlocks/pthread rwlocks on ns_urlspace; macOS (8 cores) rwlock (POSIX) 0.3105 2,27 % rwlock (ns) 11.5263 84,23 % mutex (ns) 13.6839 100,00 % Linux (6 cores) rwlock (POSIX) 0.6081 43,38 % rwlock (ns) 3.6259 258,66 % mutex (ns) 1.4018 100,00 % This data shows that the POSIX rwlocks implementation (pthread_rwlock_rdlock etc.) is always better than the classical NaviServer rwlock implementation, and that significant improvements can be achieved by using rwlocks. A more detailed performance comparison was provided on the developer mailing list. This change introduces: a) Ns_RWLock* based on pthreads when available b) switch to recommended configure setup for pthreads (AX_PTHREAD) c) New API call: Ns_RWLockSetName2() modeled after Ns_MutexSetName2() d) New API call: Ns_RWLockList() modeld after Ns_MutexList() (removed dependency of rwlocks to mutex statistics) * Support rwlocks for nsvs: Per default, rwlocks are used for all nsv* commands. The default behavior can be altered by setting the configuration variable "nsvrwlocks" to false in the section ns/server/${server}/tcl of the configuration file. This change can lead to improved scalability especially on multi-core machines since in read cases, the variables can be accessd unblocked for parallel executions. In most applications nsv are typically more often read than written rwlocks offer better scalability for large sites. The following table compares the busy locks with mutex operations vs the number of busy locks with rwlocks for openacs.org: With mutex (24h) locks busy nsv:3:openacs.org 27 4.71M 1.3K nsv:6:openacs.org 24 4.88M 1.03K nsv:2:openacs.org 28 3.37M 784 nsv:7:openacs.org 23 9.11M 755 nsv:5:openacs.org 25 2.88M 460 With rwlocks (24h) nsv:7:openacs.org 1 7.22M 0 nsv:6:openacs.org 2 3.92M 143 nsv:3:openacs.org 5 3.31M 1 nsv:2:openacs.org 6 2.23M 16 nsv:5:openacs.org 3 2.16M 0 On more busy sites, the improved scalability si much higher. * rwlocks are better in cases, where the number of readers is significantly higher than the number of writers. So, it has to be carefully considered in which cases rwlocks should be preferred. The new version uses rwlocks for nsv, ns_connchan, filters and tcl.cache (rwlocks for nsv and filters can be deactivated via configuration file). Earlier version used rwlocks already for adp tags, epoch handling and permissions. - Avoid multiple library initializations for OpenSSL 1.1.0 or newer. Documentation improvements: --------------------------- - Update references to Tcl website (replace http://tcl.tk by https://core.tcl-lang.org) - Added hints about required OpenSSL version to man pages - Made manual pages more consistent - Consistent indentation of program examples in documentation - Made source code documentation more consistent - Improved source code documentation (provide more details, remove obsolete comments) - Removed deprecated man pages - Removed obsolete sections in manual pages - Follow the recommended spelling convention of the Linux documentation project - Improved spelling, typesetting, and cross references - Reduced technical jargon - Made code listings in documentation more consistent (e.g. indenting) - Added missing commands in command list (overview page) - Updated listing of deprecated commands and suggested alternatives - Replaced several "slave process" by "child process" - 161 manual pages were updated since the last release Configuration Changes: ---------------------- - New configuration options: * Section "ns/parameters" . "reverseproxymode": Activate reverse proxy mode. The former configuration parameter of "nslog" named "checkforproxy" is obsolete now. . "joblogminduration": report jobs taking longer in systemlog . "schedlogminduration": report scheduled jobs taking longer in systemlog * Section "ns/server/${server}" . "rejectoverrun": when true send 503 when queue overruns; default "false" . "retryafter": time for Retry-After in 503 cases * Section "ns/server/${server}" . "filterrwlocks": use RWlocks for filters (default: true) * Section "ns/server/${server}/tcl" . "nsvrwlocks": use RWlocks for nsv (default: true) * Section "ns/server/$server/httpclient" . "logging": (default: off) . "logfile": name of http client logfile . "logrollfmt": format appended to log filename . "logmaxbackup": max number of backup log files (default: 10) . "logroll": perform automatic rolling (default: true) . "logrollonsignal": (default: false) . "logrollhour": specify at which hour to roll (default: 0) - Sample configurations: * Added values with time units where appropriate * Added new values mentioned above with short comments * For multi-server configuration files: use always the first (and not the last) server as default server. * openacs-config.tcl: * sample-config.tcl: . fixed config parameter: use "dnscachetimeout" instead of "keepwaittimeout" (name was changed in NaviServer ages ago, but change was not reflected in config files * nsd-config.tcl: * win32/test/nsd.tcl modernized test config file * nsproxy: Changed term "maxslaves" to "maxworkers" (the old term is deprecated) Code Changes: ------------- - Fixed potential access to an out-of-scope stack variable. - Added more declarations for PURE and CONST functions - Replaced usage of reserved identifier [cert-dcl37-c,cert-dcl51-cpp] - Replaced sscanf() by strtol*() to increase code-safety - Improved structure packing - Marked "ns_adp_mime" explicitly as deprecated, and provide deprecated messages for it. The command was deprecated more than 10 years ago, ns_adp_mimetype should be used - Added mime types for "heif" and "heic" which have become a preferred image type for iOS and macOS. - Regressions testing: * Reduced usages of old-style nstest::http-0.9 (socket and not ns_http-based tests) * Make test more robust concerning CRLF results (Windows) * Extended tests: 367 tests were added since the last release (changes in 38 test files) - Improved logging: * Added metering for long wait times of Ns_CsEnter locks (e.g. Ns_MasterLock) * Added concurrency level to log message of interp updates * Added warnings, when an NsTclRequestProc or an ADP request runs into a timeout. * Reduced time sensitivity on time when rolling log files_ Rolling log files happens often at midnight, using often a day precision. When e.g. a scheduled procedure the time when this function is called might be slightly after the scheduled time, which might lead to a day jump. The problem aggravates, when multiple log files are rotated. The new code identifies day wraps and uses in such cases the earlier day. - Improved portability: * Improved compatibility with the forthcoming OpenSSL 3.0 * Improved compatibility with various versions of LibreSSL * Improved Windows (MSVC) compatibility . Simplified Makefile.win32 for nmake . Fixed thread exit result handling for 64-bit Windows . Fixed thread handle access . Fixed default temp directory for windows . Turned on OpenSSL by default . Minimal support for running regression tests under windows . Fixed windows size incompatibility for range requests (64-bit Windows uses a 32-bit sized offset type off_t) . Fixed ns_sockselect for 64-bit windows installations . Fixes for cross-compiling for Windows 64-bit (using gcc/mingw) * Use consequently NS_EAGAIN (can be used in *nix and Windows) * Fixed potential problem with GNUC_PURE for older version of gcc * Removed dependency of tests from an explicit "/tmp" directory * Fixed regression test warnings with older versions of curl for IPv6 addresses * Fixed m4 configuration for "crypt_r" (broken since many years) * Compiled with macOS 10.14.6, Ubuntu 20.04, Fedora Core 32, OpenBSD 6.8 (LibreSSL 3.1.1), FreeBSD 12.2 - More OpenSSL changes: * Reactivate CRYPTO_set_mem_functions() for OpenSSL 1.1.0 or newer (interface om OpenSSL has changed) * Switched to SSL_CTX_set_dh_auto() from low-level functions: In newer versions of OpenSSL, all low-level DH* functions were deprecated. - Build-system * Compute version tag from git similarly like to the mercurial version. - improved support for MS windows * Windows math.h does not include round(), add simple implementation. - Added cross-platform function ns_localtime_r (localtime with user-provided storage): The previously defined function ns_localtime() uses a single per-thread local storage, so, it was not possible to have in one thread two time buffers with different values. - Improved error code handling on driver receive operations Previously, higher level of API calls were not able to access realiably the error information detected on the lower-level (async I/O-based or OpenSSL based) driver operations. This made it impossible to provide good error messages e.g. in "ns_connchan" and we could not provide error processing based on Tcl error codes. Furthermore, the new interface works for POSIX errnos and OpenSSL errors and improves the situation (currently just for driver read operations). * Ns_Sock* based setting of Tcl error codes: NsSockSetRecvErrorCode(const Sock *sockPtr, Tcl_Interp *interp) Set the Tcl error code in the interpreter based on the information provided by Sock* (actually by recvErrno); the function works for OpenSSL and plain POSIX style error codes. * Setting Tcl error codes for POSIX errno: Ns_PosixSetErrorCode(Tcl_Interp *interp, int errorNum) Set the Tcl error code in the interpreter based on the POSIX errorNum and return the error message. * Setting Tcl error codes for OpenSSL error values Ns_SSLSetErrorCode(Tcl_Interp *interp, unsigned long sslERRcode) Set the Tcl error code in the interpreter based on the OpenSSL error value and return the error message. * Return the last error from a socket. Ns_SockErrorCode(Tcl_Interp *interp, NS_SOCKET sock) The function Ns_SockErrorCode() returns the last error from a socket. When the provided interp is not NULL it sets the Tcl error code based on the error code. Modules: -------- 18 files changed, 1858 insertions(+), 1195 deletions(-) nsdb: * Added support for time units for nsdb LogMinDuration nsdbpg: * Force bindvars to start with alnum or underscore (otherwise, surprises with e.g. lines ending with a colon might arise). * Align spelling with LDP recommendations. * Avoid shadowing of symbol "bind". nssmtpd: * Escape all leading dots according to RFC 5321 section 4.5.2 in mail bodies * Fixed check, if relay host announced STARTTLS * Provide compatibility for versions before 4.99.19, when NS_FALL_THROUGH was not defined yet * Subsecond time resolution (Handled interface changes in post 4.99.19) * Support for 'ciphersuites' and 'protocols' parameters of 'Ns_TLS_CtxServerCreate' * I/O overhaul - Replaced select() call by poll() to ensure scalability - Pushed handling of EAGAIN etc. to the lowest level - Removed one layer of partial write handling - Replaced huge send buffer attempts, since these seem to be rejected sometimes - Improved robustness (never go into an infinite retry loop) - Handle potential "resource is not available" while sending SMTP data * Improved cleanness of compilation * Improved documentation * Align spelling with LDP * Code cleanup (C and Tcl code) * Extended regression test nsdns: * Code Cleanup - Added NS_EXPORT to Ns_ModuleInit - Improved cleanness of compilation nsudp: * Minor code cleanup (type cleanness, documentation) nsphp: * Adjust to changed parameter types in PHP 7 * Improved cleanness of compilation * Improved documentation * Brought test environment in sync with naviserver nsstats: * Improved XSS protection for potentially user supplied strings * Improved statistics: - Added calculated maximum number of reqs/pool - Added "maxconnections" to the per-pool statistics, since it determines the queue length - Improved nsproxy output in "process" page (include seconds, don't truncate everything to 0 digits after the dot) - Added write-percentage for RWLock in lock statistics - Add statistics for dropped requests (queue overruns) - Added nsv usage statistics * Added web-interface for toggling between raw and human centric values * Aligned with fractional seconds interface in NaviServer 4.99.20 * Improve help messages nsoracle: * Changed the include order to avoid bad interactions between oci.h and zlib. with the previous include order, we run with recent oracle versions (19.5) into /usr/include/zlib.h:1677:4: error: unknown type name ‘off64_t’ on 64-bit Linux systems (centos 7) * Added ORA_CFLAGS to allow developers to pass arbitrary include and library flags. Background: for some (newer?) version of the oracle client library, ORACLE_HOME (see README) seems to be gone. * Changes to build nsoracle on Windows with new Makefile.win32 file. * Improved cleanness of compilation websocket: * Major rewrite, using C-level implementing in ns_connchan This version make uses of the new C-level support for WebSockets. The new version is several orders of magnitudes faster than the previous version, it is more robust (C-level handling of partial read and write operations) and it is more complete than previous versions (handling segmented messages). * Added deprecated function ::ws::build_msg for providing better backward compatibility * Provide name for mutex lock to ease tracing of long locks * Added client-side SNI (requires NaviServer 4.99.20) * Added simple WebSocket client interface revproxy: * Improved exception handling (use try interface) * Preserve existing X-Forwarded-For IPs in cases where nsproxy is itself behind a proxy or load balancer * Switch to "ns_connchan write -buffered" to simplify leftover management for partial writes (requires NaviServer 4.99.20) * Fixed bug fix for potential stale handles * Code cleanup - Fine-tuning logging messages - Breaking overlong lines, reduce verbosity letsencrypt: * Minor changes in documentation nsldap: * Fixed result setting of "connected" subcommand * Updated modules interface to usual conventions * Added config parameter "schema" to specify other schemata * Improved documentation in README file * Reduced number of deprecated calls nsshell: * Added HTML-escaping on results of Tcl commands before these are displayed in the shell. |
From: oleg <oo...@ua...> - 2021-01-18 10:00:46
Attachments:
Makefile.win32
|
On Sun, 17 Jan 2021 19:17:51 +0100 Gustaf Neumann <ne...@wu...> wrote: > Under windows, the regression tests reports now 1611 passes > and 65 fails.... we are improving. I have 23 failed tests for mingw build and 24 - for msvc32/64 (with ~1660 successed). It looks like the mentioned test were executed by Naviserver w/o zlib. Oleg. PS. To fix I use attached Makefile.win32 in the nsd/ folder. This fix may use zlib sources from then Tcl8.6 distribution. |
From: Gustaf N. <ne...@wu...> - 2021-04-02 12:47:51
|
Dear all, on sourceforge is a release candidate for NaviServer 4.99.21 [1]. This is essentially a bug-fix release and cleans everything up, what was reported after the larger release of 4.99.20 as an issue. The in-depth look of the changes of oleg (as reported before the release of 4.99.20) are still on our agenda. I have tested the code with Ubuntu 20.04, macOS 10.14.6, FreeBSD-12.2, OpenBSD 6.8. The code is already running on openacs.org. Below is a preliminary summary of changes. Please test if possible. The release should be in about one week. -gustaf [1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.21/ ======================================= NaviServer 4.99.21, released 2021-01-XX ======================================= 59 files changed, 808 insertions(+), 499 deletions(-) New Features: ------------- - Logging improvements: Added parameter "logsec" and "logthread" for section "ns/parameter" to make logging of full timestamps and thread info optional. This give more control to a developer to shorten the log entries e.g. on development instances by excluding certain values. - Made "ns_http ... -decompress ... "obsolete in favour of new option "-raw". This fixes as well a crash. Bug Fixes: ---------- - Fixed computation of "location" of a request for cases, where the network driver is not installed globally. The value of [ns_conn location] (and the redirect target) is often, but not always determined by the host header field as used for virtual servers. - Fixed handling of sendBodyLength in ns_http - Fixed handling of user-ids in nscp. - Fixed potential segmentation violation in debug output, when (a) the network driver is installed locally, and (b) the content of host header field cannot be located in virtual server map (fallback to validation via driver data). - Fixed potential segmentation violation when nsd is called with '-c' and '-t' and the config file contains no "home" parameter in the config file. The fix simplifies the logic, and performs same configuration of nsconf.home whenever "-c" is used in the command line. - Increase sysadmin-friendliness of OCSP handling in error cases: In case OCSP (Online Certificate Status Protocol) is activated and the connection to the AIA-server is blocked, do not reject all traffic but behave like in a configuration without having OCSP activated, and complain the system log. Background: Sometimes requests to the let's encrypt AIA server seems to be blocked by firewalls. Previously, in such stituations, all traffic was blocked. - Relax bailing out on malloc(0), since at least the OpenSSL 3a* versions use this frequently on certain configurations. Calls of malloc(0) are essentially useless and dangerous, but we cannot avoid these. Performance improvements: ------------------------- - Added support for TCP_NODELAY for https sockets via configuration option "nodelay" in the nsssl driver section. For whatever reasons, NaviServer handled so far this option. The option can reduce the time of connection calls via https (clearly measurable in benchmark setups). Documentation improvements: --------------------------- - improved the following man pages doc/src/manual/admin-config.man doc/src/manual/admin-tuning.man doc/src/manual/main-history.man doc/src/naviserver/ns_http.man doc/src/naviserver/ns_http.man doc/src/naviserver/ns_write.man nsdb/doc/mann/ns_db.man Configuration Changes: ---------------------- - New options "logsec" and "logthread" for section "ns/parameter" (see above) Code Changes: ------------- - Extended regression test to cover e.g. the error cases from above. - Code Cleanup . Removed obsolete and deprecated functions ctime_r() and asctime_r() in favor of strftime(). . Removed calls to deprecated Tcl functions - Don't pass (implementation-defined) NULL as last argument of a variadic function. - Usual cleanup, such as reduced implicit type conversions, removed uselass casts, removed dead assignments - Improve comments, fixed typos |
From: Gustaf N. <ne...@wu...> - 2021-04-09 09:42:33
Attachments:
Attached Message Part
Attached Message Part
|
Dear all, The release of NaviServer 4.99.21 is now available on sourceforge [1]. This is essentially a bug-fix release and cleans everything up, what was reported after the larger release of 4.99.20 as an issue. I will make as well as joint release annouce of 4.99.20 and .21 at the Tcl forum. The code is tested at least with Ubuntu 20.04, macOS 10.14.6, 11.2.3, FreeBSD-12.2, OpenBSD 6.8. Below is a summary of changes since the last release. All the best -gustaf [1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.21/ ======================================= NaviServer 4.99.21, released 2021-04-09 ======================================= 64 files changed, 939 insertions(+), 536 deletions(-) New Features: ------------- - Logging improvements: Added parameter "logsec" and "logthread" for section "ns/parameter" to make logging of full timestamps and thread info optional. This give more control to a developer to shorten the log entries e.g. on development instances by excluding certain values. - Made "ns_http ... -decompress ... "obsolete in favor of new option "-raw". This fixes as well a crash. Bug Fixes: ---------- - Fixed computation of "location" of a request for cases where the network driver is not installed globally. The value of [ns_conn location] (and the redirect target) is often, but not always determined by the host header field as used for virtual servers. - Fixed handling of sendBodyLength in ns_http - Fixed handling of user-ids in nscp. - Fixed potential segmentation violation in debug output, when (a) the network driver is installed locally, and (b) the content of host header field cannot be located in virtual server map (fallback to validation via driver data). - Fixed potential segmentation violation when nsd is called with '-c' and '-t' and the config file contains no "home" parameter in the config file. The fix simplifies the logic, and performs same configuration of nsconf.home whenever "-c" is used in the command line. - Increase sysadmin-friendliness of OCSP handling in error cases: In case OCSP (Online Certificate Status Protocol) is activated and the connection to the AIA-server is blocked, do not reject all traffic but behave like in a configuration without having OCSP activated, and complain the system log. Background: Sometimes requests to the let's encrypt AIA server seems to be blocked by firewalls. Previously, in such stituations, all traffic was blocked. - Relax bailing out on malloc(0), since at least the OpenSSL 3a* versions use this frequently on certain configurations. Calls of malloc(0) are essentially useless and dangerous, but we cannot avoid these. Performance improvements: ------------------------- - Added support for TCP_NODELAY for https sockets via configuration option "nodelay" in the nsssl driver section. For whatever reasons, NaviServer handled so far this option. The option can reduce the time of connection calls via https (clearly measurable in benchmark setups). Documentation improvements: --------------------------- - improved the following man pages doc/src/manual/admin-config.man doc/src/manual/admin-tuning.man doc/src/manual/main-history.man doc/src/naviserver/ns_http.man doc/src/naviserver/ns_http.man doc/src/naviserver/ns_write.man nsdb/doc/mann/ns_db.man Configuration Changes: ---------------------- - New options "logsec" and "logthread" for section "ns/parameter" (see above) Code Changes: ------------- - Extended regression test to cover e.g. the error cases from above. - Code Cleanup . Removed obsolete and deprecated functions ctime_r() and asctime_r() in favor of strftime(). . Removed calls to deprecated Tcl functions - Don't pass (implementation-defined) NULL as last argument of a variadic function. - Usual cleanup, such as reduced implicit type conversions, removed uselass casts, removed dead assignments - Improve comments, fixed typos |
From: Gustaf N. <ne...@wu...> - 2021-01-10 18:35:24
|
On 10.01.21 15:58, oleg wrote: > > We have tested Naviserver RC. > No problem found on Ubuntu 20.04 and Windows10/MSVC2019/32bit. > Great! > > Have an issue for Windows10/MSVC2019/64bit build. > > lassign [ns_sockopen -nonblock httpbin.org 80] rfd wfd > > ns_sockselect -timeout 1000 {} $wfd {} > This is an old and known issue. "ns_sockopen" is a somewhat deprecated call, which was used mostly for the regression test (most NaviServer applications will work happily without it), so there was seemingly never a high urgency to fix this. I am surprised that this works with MSVC2019/32bit - but maybe this was the "lucky" part. I would not see this as showstopper for the upcoming release. > Crosscompiled (gcc/mingw) for Windows 64bit Naviserver works perfectly > but there are difficulties in configure/make/install: Cool! I have added the mingw changes to the code base and added your instructions to the README file ... and placed rc3 to sourceforge https://sourceforge.net/projects/naviserver/files/naviserver/4.99.20/ If nothing else shows up, i'll finish the release in the next few days. Oleg, many thanks for testing and the good suggestions. -gn |