naviserver-devel Mailing List for NaviServer

NaviServer, a high performance web server written in C and Tcl

Brought to you by: eide, gneumann, itannir, sdeasey, and 2 others

naviserver-devel — Developer discussion

You can subscribe to this list here.

2005	Jan	Feb (53)	Mar (62)	Apr (88)	May (55)	Jun (204)	Jul (52)	Aug	Sep (1)	Oct (94)	Nov (15)	Dec (68)
2006	Jan (130)	Feb (105)	Mar (34)	Apr (61)	May (41)	Jun (92)	Jul (176)	Aug (102)	Sep (247)	Oct (69)	Nov (32)	Dec (140)
2007	Jan (58)	Feb (51)	Mar (11)	Apr (20)	May (34)	Jun (37)	Jul (18)	Aug (60)	Sep (41)	Oct (105)	Nov (19)	Dec (14)
2008	Jan (3)	Feb	Mar (7)	Apr (5)	May (123)	Jun (5)	Jul (1)	Aug (29)	Sep (15)	Oct (21)	Nov (51)	Dec (3)
2009	Jan	Feb (36)	Mar (29)	Apr	May	Jun (7)	Jul (4)	Aug	Sep (4)	Oct	Nov (13)	Dec
2010	Jan	Feb	Mar (9)	Apr (11)	May (16)	Jun	Jul	Aug	Sep (1)	Oct	Nov	Dec
2011	Jan	Feb	Mar	Apr	May (1)	Jun	Jul	Aug	Sep	Oct	Nov	Dec
2012	Jan (7)	Feb (3)	Mar	Apr	May	Jun (3)	Jul	Aug	Sep	Oct (92)	Nov (28)	Dec (16)
2013	Jan (9)	Feb (2)	Mar	Apr (4)	May (4)	Jun (6)	Jul (14)	Aug (12)	Sep (4)	Oct (13)	Nov (1)	Dec (6)
2014	Jan (23)	Feb (19)	Mar (10)	Apr (14)	May (11)	Jun (6)	Jul (11)	Aug (15)	Sep (41)	Oct (95)	Nov (23)	Dec (11)
2015	Jan (3)	Feb (9)	Mar (19)	Apr (3)	May (1)	Jun (3)	Jul (11)	Aug (1)	Sep (15)	Oct (5)	Nov (2)	Dec
2016	Jan (7)	Feb (11)	Mar (8)	Apr (1)	May (3)	Jun (17)	Jul (12)	Aug (3)	Sep (5)	Oct (19)	Nov (12)	Dec (6)
2017	Jan (30)	Feb (23)	Mar (12)	Apr (32)	May (27)	Jun (7)	Jul (13)	Aug (16)	Sep (6)	Oct (11)	Nov	Dec (12)
2018	Jan (1)	Feb (5)	Mar (6)	Apr (7)	May (23)	Jun (3)	Jul (2)	Aug (1)	Sep (6)	Oct (6)	Nov (10)	Dec (3)
2019	Jan (26)	Feb (15)	Mar (9)	Apr	May (8)	Jun (14)	Jul (10)	Aug (10)	Sep (4)	Oct (2)	Nov (20)	Dec (10)
2020	Jan (10)	Feb (14)	Mar (29)	Apr (11)	May (25)	Jun (21)	Jul (23)	Aug (12)	Sep (19)	Oct (6)	Nov (8)	Dec (12)
2021	Jan (29)	Feb (9)	Mar (8)	Apr (8)	May (2)	Jun (2)	Jul (9)	Aug (9)	Sep (3)	Oct (4)	Nov (12)	Dec (13)
2022	Jan (4)	Feb	Mar (4)	Apr (12)	May (15)	Jun (7)	Jul (10)	Aug (2)	Sep	Oct (1)	Nov (8)	Dec
2023	Jan (15)	Feb	Mar (23)	Apr (1)	May (2)	Jun (10)	Jul	Aug (22)	Sep (19)	Oct (2)	Nov (20)	Dec
2024	Jan (1)	Feb	Mar (16)	Apr (15)	May (6)	Jun (4)	Jul (1)	Aug (1)	Sep	Oct (13)	Nov (18)	Dec (6)
2025	Jan (12)	Feb	Mar (2)	Apr (1)	May (11)	Jun (5)	Jul (4)	Aug	Sep	Oct	Nov	Dec

Flat | Threaded

1 2 3 .. 178 > >> (Page 1 of 178)

[naviserver-devel] New NaviServer Users Mailing List

From: Gustaf N. (sslmail) <ne...@wu...> - 2025-07-15 17:45:58

Dear all,

As discussed and requested at last week’s EuroTcl/OpenACS conference, I’ve created a new mailing list: naviserver-users, now available on SourceForge:

🔗 https://sourceforge.net/p/naviserver/mailman/

While the interface isn’t particularly polished, this setup consolidates everything in one place and gets us started. That said, we can certainly consider alternative platforms - such as openacs.org - or any others you might suggest.

Looking forward to your feedback and participation.

All the best,
Gustaf



PS: The videos from the conference are already online (many thanks to Antonio!), they need still linkage on the conference site.

 https://learn.wu.ac.at/eurotcl2025/

Re: [naviserver-devel] Announcement: NaviServer 5.0.0 Release

From: Stefan S. <ste...@wu...> - 2025-07-09 15:37:27

Congratulations, what a heavy lift! Enjoy Bologna and have NS 5 celebrated!

Ciao, Stefan

> Dear all,
> 
> We are glad to announce the final release of NaviServer 5.0.0, marking 
> the most significant update in the project’s history with over 70,000 
> lines of changes. This release delivers robust security enhancements, 
> modernized infrastructure, and improved scalability. Key highlights include:
> 
>   *
> 
>     *Security by Default*: HTTP client requests now validate server
>     certificates automatically, with configurable trust exceptions and a
>     bundled CA root store. A new pluggable authorization framework
>     (|ns_auth|) enables scriptable request- and user-level access control.
> 
>   *
> 
>     *Modernized Core*: Unix domain socket support, case-insensitive
>     |ns_set|, Argon2 password hashing, and streamlined configuration via
>     environment variables (ideal for containers). The reverse proxy is
>     now built into the core, with new forward proxy capabilities.
> 
>   *
> 
>     *Enhanced Observability and Scalability*: Improved logging (dynamic
>     paths, rotation, and new "Security" severity), detailed connection
>     diagnostics (|ns_conn urldict|, |ns_connchan debug|), runtime
>     introspection (|ns_http keepalives|, |ns_info buildinfo|),
>     persistent client connections, and streaming HTTP client processing.
> 
> *Backward Compatibility & Future-Readiness*
> While introducing cutting-edge features, NaviServer 5.0.0 maintains 
> strong backward compatibility, with deprecated APIs clearly documented 
> and modern replacements provided (e.g., |ns_urlencode| → 
> |ns_percentencode|). The release introduces/extends over 50 Tcl commands 
> and subcommands. The upgrade to MPL 1.2 and Tcl 9 support ensures 
> long-term sustainability. Over 20 community-contributed modules (like 
> |nsdbpg|, |nssmtpd|) have been updated in lockstep.
> 
> The following people contributed to this release:
> 
>       Alexander Danilov, Andrew Piskorski, Antonio Pisano, Brendan 
> Graves, Brian Fenton, Danilo Raynor, David F, David Osborne,
> 
>       Filip Minic, Georg Lehner, Gustaf Neumann, Hector Romojaro, Joe 
> Oldak, Oleg Oleinick, Priyank Jalu, Thomas Renner, Zoran Vasiljevic
> 
> *Resources*:
> 
>   *
> 
>     Download
>     <https://sourceforge.net/projects/naviserver/files/naviserver/5.0.0/> | GitHub <https://github.com/naviserver-project/naviserver> | Docs <https://naviserver.sourceforge.io/5.0/toc.html>
> 
> This release is a major leap forward in security, flexibility, and 
> maintainability. See below for the detailed changes: Upgrade today!
> 
> /The NaviServer Team/
> 
> 
>     Tcl API Changes
> 
> 
>       |ns_cache|
> 
>   * New option |cachingmode|: Accepts |full| or |none| to toggle caching
>     behavior.
> 
> 
>       |ns_configsection|
> 
>   * New option |-filter|: Filters variables by |unread|, |defaulted|, or
>     |defaults|.
> 
> 
>       |ns_configure_variables|
> 
>   * New command to pull in configuration variables from environment
>     variables (for use in configuration files)
> 
> 
>       |ns_conn|
> 
>   *
> 
>     *General enhancements*:
> 
>       o |ns_conn host|, |ns_conn port|, |ns_conn protocol|: Now return
>         appropriate values for all request types (not limited to forward
>         proxy requests).
>   *
> 
>     *New subcommands*:
> 
>       o |ns_conn target|: Returns the complete request target, including
>         query parameters, from the HTTP start line.
>       o |ns_conn fragment|: Returns the fragment identifier (after |#|)
>         if present.
>       o |ns_conn urldict|: Parses the request URL into a Tcl dictionary
>         for structured access.
>       o |ns_conn host|: Supports an optional default value if the host
>         is not determined.
>       o |ns_conn details|: Returns a dictionary with driver-specific
>         connection metadata.
> 
> 
>       |ns_connchan|
> 
>   *
> 
>     *Security and diagnostics improvements*:
> 
>       o |ns_connchan debug|: Set or query the debug level.
>       o |ns_connchan connect| / |ns_connchan open|: Now accept
>         additional security-related options: |-cafile|, |-capath|,
>         |-cert|, |-hostname|, |-insecure|, |-driver|, and |-unixsocket|.
>       o |ns_connchan status|: Added |-server| option for multi-server
>         introspection.
>       o |ns_connchan write|: Now performs buffered writes.
>   *
> 
>     More detailed timeout handling
> 
> 
>       |ns_driver|
> 
>   * |info|, |names|, |stats|, and |threads| support the |-server| option
>     to allow multi-server diagnostics.
> 
> 
>       |ns_http|
> 
>   *
> 
>     *Connection handling and introspection*:
> 
>       o
> 
>         Persistent HTTP connections supported via the |keepalive| option
>         in |httpclient| configuration.
> 
>       o
> 
>         Introduced support for multiple task threads.
> 
>       o
> 
>         Streaming response support for incremental data handling (useful
>         for large file transfers and LLM interaction).
> 
>       o
> 
>         Default timeout configuration for requests.
> 
>       o
> 
>         New callbacks: |response_header_callback| and
>         |response_data_callback|.
> 
>       o
> 
>         Renamed |-donecallback| to |-done_callback| (old name deprecated).
> 
>       o
> 
>         Added support for informational HTTP status codes (e.g., 100
>         Continue).
> 
>       o
> 
>         New subcommands:
> 
>           + |ns_http keepalives|: Displays active persistent connections.
>           + |ns_http taskthreads|: Provides insight into HTTP client
>             threads.
>       o
> 
>         Major internal refactoring of |tclhttp.c| for modularity and
>         maintainability.
> 
> 
>       |ns_ictl|
> 
>   * |getmodules|: Now supports the |-server| option for per-server
>     introspection.
> 
> 
>       |ns_info|
> 
>   *
> 
>     Added subcommands:
> 
>       o |argv|: Returns the original argument vector.
>       o |bindir|: Returns the binary directory path.
>       o |buildinfo|: Displays build-time config and version info.
>       o |logdir|: Path to the log directory.
>       o |meminfo|: Displays memory statistics (when using |tcmalloc|).
> 
> 
>       |ns_ip|
> 
>   *
> 
>     Added subcommands:
> 
>       o |inany|: Checks if an IP matches any configured address.
>       o |properties|: Returns metadata about configured IPs.
>       o |public|: Determines if an IP is publicly routable.
>       o |trusted|: Checks if an IP is trusted.
>       o |valid|: Validates IP syntax.
>   *
> 
>     These enhancements supersede the need for |ns_subnetmatch|.
> 
> 
>       |ns_issmallint|
> 
>   * New command to check if a value is a valid small integer (replaces
>     the old, now deprecated |issmallint|).
> 
> 
>       |ns_logctl|
> 
>   * New subcommand |grep|: Searches log files with pattern matching,
>     stripping color codes and handling line continuations.
> 
> 
>       |ns_register*| Enhancements
> 
>   * All |ns_register_*| commands now support the |-constraints| option,
>     enabling context-sensitive behavior.
>   * New command: |ns_register_auth| allows registration of user- and
>     request-level authorization handlers.
> 
> 
>       |ns_server|
> 
>   * New subcommands:
>       o |authprocs|: Lists registered request/user auth procedures.
>       o |hosts|: Lists registered hostnames for the server.
>       o |logdir|: Returns the server log directory.
>       o |realm|: returns or sets the realm of the server
>       o |serverdir|: Returns the base directory for the virtual server
>         (|-effective| returns the resolved runtime path).
>       o |vhostenabled|: Returns a boolean indicating virtual hosting status.
>   * |ns_server mapped|: New option |-all| returns a dict with handler
>     and pool info.
> 
> 
>       |ns_set|
> 
>   *
> 
>     *Case-insensitive support*:
> 
>       o New |-nocase| flag for |ns_set create| and related operations.
>       o Deprecated the |i*| subcommands (e.g., |iget|, |ifind|) in favor
>         of unified interface.
>   *
> 
>     *Multi-valued key support*:
> 
>       o |-all| option retrieves all values for a given key in
>         |ns_set| and |ns_config|.
>   *
> 
>     *New subcommands*:
> 
>       o |format|: Pretty-prints the set contents.
>       o |stats|: Returns memory usage statistics.
>   *
> 
>     |delkey|: Now returns a boolean success flag.
> 
>   *
> 
>     *Internal improvements*:
> 
>       o Refactored |ns_set| internals using dense storage for lower
>         memory use and better cache locality.
>       o Replaced deeply nested switch logic with modular, maintainable code.
> 
> 
>       |ns_thread|
> 
>   *
> 
>     Unified thread creation:
> 
>       o Use |ns_thread create| for all thread types.
>       o Deprecated |begin| and |begindetached| to align with |ns_cond|,
>         |ns_mutex|, and |ns_sema| usage.
> 
> 
>           |ns_urlspace|
> 
>   *
> 
>     Wildcard matching now supports path-segment-level matches
>     (previously limited to leaf nodes).
> 
>   *
> 
>     Applied improved matching in |nscgi| to support directory-wide CGI
>     mapping.
> 
>   *
> 
>     Renamed option |-contextfilter| to |-constraints| in |ns_urlspace
>     set/unset| to match the new registration API.
> 
> 
>       Database Enhancements
> 
>   * |ns_db info|: Returns a dictionary of metadata for a given DB handle.
>   * |ns_dbpooldescription|: Renamed from the previous
>     |ns_pooldescription| for consistent naming.
>   * |ns_db rowcount|: Fixed in this release to return actual row counts.
> 
> 
>       New Utilities
> 
>   * |ns_fseekchars|: Efficiently scans a stream for a string (e.g., for
>     multipart/form-data parsing).
>   * |ns_joinurl|: Constructs well-formed URLs from path components.
>   * |ns_mkdtemp|: Creates a unique temporary directory (POSIX-style
>     |mkdtemp()|).
>   * |ns_parsehtml|: Parses HTML fragments into a structured Tcl dict.
>   * |ns_parsemessage|: Parses MIME-style messages (e.g., emails or HTTP
>     headers).
>   * |ns_percentencode| / |ns_percentdecode|: Replace deprecated
>     |ns_urlencode| and |ns_urldecode| for robust URL-safe encoding/decoding.
> 
> 
>     Changes in Core Modules
> 
> 
>       |nscgi|
> 
>   * New command: |ns_register_cgi| for dynamic CGI handler registration
>     (supports |-noinherit|, |-path|, etc.).
>   * Supports unregistration via |ns_unregister_op|.
>   * New environment variables: |SCRIPT_FILENAME|, |REQUEST_URI|.
>   * Refactored to support major web apps (e.g., WordPress, Joomla).
>   * See commit |36027b70215| for implementation details.
> 
> 
>       |nscp|
> 
>   * Integrated with new authorization system (e.g., via |nsperm users|).
>   * New command |nscp users|: Lists registered users for nscp
>     authentication.
> 
> 
>       |nsperm|
> 
>   * Acts as a pluggable authorization provider for request/user scopes.
>   * New config parameter: |allowLoopbackEmptyUser|—permits
>     unauthenticated loopback access when enabled.
>   * Added support for setting default server start pages and |nsstats|.
> 
> 
>       |nsproxy|
> 
>   * New subcommand: |ns_proxy workers|—provides detailed runtime info on
>     proxy worker processes.
> 
> ------------------------------------------------------------------------
> 
> 
>     Bug Fixes
> 
> 
>       Stability and Crash Resolutions
> 
>   *
> 
>     *Addressed multiple potential crash scenarios:*
> 
>       o Robust handling of HTTP |CONNECT| requests.
>       o Fixed crashes caused by missing or empty argument lists in
>         commands such as |ns_filestat|, |ns_sockcallback|, and |ns_ictl
>         oncleanup|.
>       o Prevented crash in |ns_log| when invoked with an empty message.
>       o Resolved a 24-year-old bug in |Ns_AdjTime()| that could lead to
>         fatal errors due to microsecond overflow in multithreaded
>         environments.
>       o Fixed crash in |ns_conn copy| when operating on empty content.
>       o Prevented crash in |ns_sema release| with invalid semaphore counts.
>       o Corrected off-by-one error in |ns_adp_bind_args| that could
>         access uninitialized |Tcl_Obj| values.
>       o Fixed crash in |ns_inet_ntop| due to unsafe memory operations
>         with overlapping regions (notably on aarch64 with musl).
>       o Resolved crash in |Ns_SetIUpdateSz()| caused by case mismatch in
>         header keys with the legacy C API.
>       o Fixed crash during |nscp| startup when the |users| section was
>         not configured.
>       o Prevented crash in debug mode when the |Host:| header could not
>         be mapped to a virtual server and the driver was installed locally.
>       o Avoided crash when launching |nsd| with |-c| and |-t| options
>         and no |home| parameter defined.
>       o Fixed crash during computation of |ns_conn location| when the
>         network driver was not globally installed (global installation
>         now recommended).
>       o Fixed crash due to self-destructive header replacement when
>         |ns_conn outputheaders| are passed via |ns_respond ... -headers ...|
> 
> 
>       Functional Correctness and Logic Fixes:
> 
>   * |ns_conn status|: Fixed issue where updated status codes were
>     silently ignored.
>   * |ns_conn doneCallback|: Ensured this callback is always invoked in
>     |ns_http|.
>   * |nsv_dict get|: Fixed a memory leak in value retrieval.
>   * |ns_conn peeraddr|: Resolved race condition in pipelined requests
>     that could yield incorrect peer addresses.
>   * |ns_cache_eval -force|: Fixed race condition that could produce
>     obsolete results.
>   * |ns_sema create|: Corrected handling of initial count values (e.g.,
>     |1000|).
>   * |ns_trim|: Fixed spacing logic to trim only leading and trailing
>     whitespace, preserving internal spacing.
>   * |ns_config -int|: Corrected fallback behavior when invalid values
>     are supplied, now correctly using documented defaults.
>   * |ns_sockcallback|: Now gracefully handles missing arguments.
>   * |ns_socknread|: Fixed inaccurate results for buffered connections.
>   * |ns_crypto::aead::encrypt/decrypt|: Restored compatibility with
>     OpenSSL 1.1.1.
>   * Fixed incorrect parsing of encoded backslashes in URLs.
>   * |Ns_StrTrimRight()|: Corrected UTF-8 handling in right-side string
>     trimming.
>   * Fixed |ns_conn location| when running behind a reverse proxy to
>     always returns a value, even in broken configurations.
> 
> 
>       OpenSSL fixes
> 
>   * Added support for detecting and validating OCSP Must-Staple and AIA
>     presence in certificates. Without that, NaviServer might crash, when
>     OCSP is turned on, and NULL values are passed for AIA URLs (letsencrypt)
>   * Introduced stable output buffers for |send| operations to support
>     retries after |SSL_ERROR_WANT_WRITE|, preventing connection failures
>     under high load.
>   * Improved error handling in OpenSSL integration by draining the error
>     stack via DrainErrorStack().
> 
> 
>       HTTP Client Fixes
> 
>   * |ns_http|:
>       o Reordered initialization in |NsInitServer()| to ensure
>         submodules can access a fully configured server state.
> 
> 
>       HTML and ADP Parsing Fixes
> 
>   * ADP Parser:
>       o Enhanced support for quoted |>| characters inside attribute
>         values, aligning with modern HTML parsing rules.
>   * |return-notice| handling:
>       o Suppressed spurious error messages when fallback ADP templates
>         are processed outside of a full ADP context.
>   * |ns_striphtml|:
>       o Fixed long-standing bug where adjacent HTML entities were
>         incorrectly decoded—only the first entity was processed.
> 
> 
>       Logging and Diagnostics Fixes
> 
>   * Fixed misleading log output when dynamically changing extended
>     headers via |ns_accesslog extendedheaders ...|.
>   * Improved error messages when port binding fails due to conflicts
>     with driver assignment.
>   * Fixed off-by-one error in the virtual server port configuration logic.
>   * Clarified or corrected multiple logging messages across modules for
>     better diagnostics.
> 
> 
>       nscgi Module Fixes
> 
>   * Fixed file upload failures when uploads were internally spooled by
>     NaviServer — previously returned 500 errors.
>   * Corrected processing of CGI script exit codes not 0.
>   * Fixed hostname and port reporting for |SERVER_NAME|.
> 
> 
>       Database Fixes
> 
>   * |ns_db rowcount|: Fixed regression where the row count logic was not
>     invoked, rendering the command a no-op.
> 
> 
>       Build and Compatibility Fixes
> 
>   * Suppressed obsolete |--enable-threads| warning for Tcl versions
>     where thread support is now enabled by default.
>   * Fixed compilation failure with glibc 2.38+ due to
>     |PTHREAD_STACK_MIN| becoming dynamic via |sysconf()|.
> 
> 
>       General Cleanups
> 
>   * Fixed small memory leak triggered by |serverrootproc| reset.
>   * Numerous minor typo corrections, comment clarifications, and small
>     logic cleanups across modules.
> 
> ------------------------------------------------------------------------
> 
> 
>     C-Level Infrastructure, C API Enhancements, and Build System
> 
> 
>       Core Infrastructure Improvements
> 
>   *
> 
>     *Socket Layer Enhancements*:
> 
>       o Added |sendErrno| field to the |Sock| structure to improve
>         diagnostics and tracking of write errors.
>   *
> 
>     *Modernized Initialization*:
> 
>       o Introduced |NS_INIT_ONCE()| macro for thread-safe one-time
>         initialization, replacing legacy double-lock patterns.
>   *
> 
>     *Data Structure Utilities*:
> 
>       o Added utility functions |Ns_DListSaveString()| and
>         |Ns_DListFreeElements()| to simplify dynamic list management.
>       o Replaced use of legacy |Ns_DString| functions with standard
>         |Tcl_DString|, modernizing internal data handling.
>   *
> 
>     *Debugging Support*:
> 
>       o Added |NsHexPrint()| for hex-dumping of byte sequences, aiding
>         in low-level debugging and analysis.
>   *
> 
>     *Introspection Enhancements*:
> 
>       o Introduced |Ns_TclReturnCodeString()| and
>         |Ns_ReturnCodeString()| to convert internal return codes into
>         readable strings, improving log clarity and diagnostics.
>   *
> 
>     *Code Quality and Performance*:
> 
>       o Applied extensive internal refactorings to improve performance,
>         cache locality, and maintainability across multiple subsystems.
> 
> 
>       C API Enhancements
> 
>   *
> 
>     *General*:
> 
>       o Added typedefs for |Ns_AuthorizeRequestProc|,
>         |Ns_AuthorizeUserProc|,|Ns_UrlSpaceMatchInfo|,
>         |Ns_DriverConnInfoProc|
>       o Added enum for |Ns_RequestType|, |Ns_UrlSpaceOp|,
>         |Ns_DriverClientInitArg|
>       o Added API calls in ns.h: |Ns_ConfigFilename|, |Ns_ConnTarget|,
>         |Ns_UrlSpaceMatchInfo|, |Ns_ConnServPtr|,
>         |Ns_DStringAppendSockState|, |Ns_RegisterFilter2|,
>         |Ns_TaskQueueLength|, |Ns_TaskQueueName|,
>         |Ns_TaskQueueRequests|, |Ns_ObjvTablePrint|, |Ns_InfoLogPath|,
>         |Ns_LogPath|, |Ns_ServerLogDir|, |Ns_ServerRootProcEnabled|,
>         |Ns_ServerLogGetFd|, |Ns_ServerLogCloseAll|,
>         |Ns_ServerLogRollAll|, |Ns_GetServer|, |Ns_ServerName|,
>         |Ns_SockSetSendErrno|, |Ns_SockGetSendErrno|,
>         |Ns_SockGetSendRejected|, |Ns_SockGetSendCount|,
>         |Ns_SockFlagAdd|, |Ns_SockFlagClear|, |Ns_SockSendBufsEx|,
>         |Ns_SockConnectUnix|, |Ns_SockGetClientSockAddr|,
>         |Ns_SockGetConfiguredSockAddr|, |Ns_SockaddrPublicIpAddress|,
>         |Ns_SockaddrTrustedReverseProxy|, |Ns_SockaddrInAny|,
>         |Ns_SockaddrAddToDictIpProperties|, |Ns_TclReturnCodeString|,
>         |Ns_ReturnCodeString|, |Ns_TclInterpServPtr|,
>         |Ns_LogDeprecatedParameter|, |Ns_RegisterFastUrl2File|
>       o Extended |Ns_Request|, |Ns_TclCallback|
>       o Refactored |Ns_ConnReturnMoved()| and
>         |Ns_ConnReturnRedirect()| to unify redirection logic.
>   *
> 
>     *Secure Communication and Validation*:
> 
>       o Added fine-grained certificate validation APIs
>       o Improved error stack draining for OpenSSL operations.
>   *
> 
>     *Extended C API for |ns_connchan|*:
> 
>       o Introduced a minimal C-level API for |connchan| connections,
>         enabling finer control over lower-level network operations.
>   *
> 
>     *Build Platform Compatibility*:
> 
>       o Updated Windows build system to use |NS_IMPORT| in place of
>         deprecated |NS_EXTERN|.
>       o Avoided use of deprecated C functions such as |mktemp()| to
>         improve portability and security.
> 
> 
>       Build System and Tooling
> 
>   *
> 
>     *Configuration and Sample Support*:
> 
>       o Replaced hard-coded OS commands (ls, mv, ...) consistently with
>         build variables
>       o Replaced hard-coded OpenSSL binary references with the
>         |$(OPENSSL)| build variable.
>       o Enhanced log path and configuration variable handling to support
>         more dynamic setups.
>   *
> 
>     *Test Infrastructure*:
> 
>       o Extended regression and compatibility test coverage to validate
>         edge cases and cross-platform behavior.
>   *
> 
>     *Build Metadata Introspection*:
> 
>       o Added support for build-time introspection of environment
>         settings, including allocator type (e.g., |malloc|), compiler
>         version, and Tcl build info.
>   *
> 
>     *Optional Deprecation-Free Builds*:
> 
>       o Introduced build flag |NS_NO_DEPRECATED| to exclude deprecated
>         functions and APIs, mirroring Tcl’s |TCL_NO_DEPRECATED| mechanism.
>   *
> 
>     *CI/CD Modernization*:
> 
>       o Upgraded GitHub Actions workflows from version 3 to version 4
>         for improved performance and compatibility.
>   *
> 
>     *Sample Configuration Files*:
> 
>       o Improved sample configurations (|nsd-config| and
>         |openacs-config|) to use environment-specific settings via
>         |ns_configure_variables|
>       o prefer names |http| and |https| instead of |nsock| and
>         |nsssl| inside sample configuration files to ease configuration
>         for new users.
> 
> ------------------------------------------------------------------------
> 
> 
>     Documentation Updates
> 
> 
>       Comprehensive Overhaul
> 
>   *
> 
>     Conducted a thorough review and restructuring of all documentation:
> 
>       o
> 
>         *Tcl Command Documentation*:
> 
>           + Ensured all implemented Tcl commands and their options are
>             fully documented.
>           + Removed documentation for obsolete or unimplemented commands.
>           + Deprecated commands are no longer advertised in manuals or
>             used in examples.
>           + Introduced a dedicated section listing deprecated commands.
>           + Enabled automated generation of a complete command reference.
>       o
> 
>         *Test Alignment*:
> 
>           + Verified that every implemented Tcl command is covered by
>             regression tests.
>           + Ensured that all documented commands are also implemented
>             and tested.
> 
> 
>       Syntax and Formatting Consistency
> 
>   * Standardized placeholder syntax across the documentation and syntax
>     error messages (see commit |ffbd32774db|for details).
> 
> 
>       Manual Page Improvements
> 
>   *
> 
>     *nscgi*: Significantly revised documentation and usage examples for
>     clarity and completeness.
> 
>   *
> 
>     *admin-config.man*: Added a new section on “Customizing File Locations”.
> 
>   *
> 
>     Added and updated sections covering:
> 
>       o Basic templating features.
>       o Error handling mechanisms.
>       o Recently introduced Tcl commands and options.
> 
> 
>       Visual and Structural Enhancements
> 
>   *
> 
>     Updated diagrams and usage examples related to:
> 
>       o Request processing flow.
>       o Reverse proxy (revproxy) behavior.
> 
> 
>       Additional Improvements
> 
>   * Numerous corrections to spelling, grammar, formatting, and internal
>     linking throughout the documentation.
> 
> 
>       Deprecation Management Infrastructure
> 
>   * Introduced compiler warnings for usage of deprecated
>     |Ns_DString*| functions
>   * Added log severity level |Deprecated| to better surface deprecated
>     usage at runtime
>   * Documented all deprecated commands in a dedicated section of the
>     command reference
>   * Deprecated the use of manual double-checked locking for one-time
>     initialization replaced by the |NS_INIT_ONCE()| macro for safer and
>     cleaner initialization semantics
> 
> ------------------------------------------------------------------------
> 
> 
>     Deprecations
> 
> 
>       Tcl-Level Command Deprecations
> 
>   *
> 
>     Deprecated the following Tcl commands:
> 
>       o
> 
>         |ns_set print| → replaced by |ns_set format|
> 
>       o
> 
>         |ns_checkurl| and |ns_requestauthorize| → replaced by |ns_auth
>         request|
> 
>       o
> 
>         |ns_thread begin| → replaced by |ns_thread create|
> 
>       o
> 
>         |ns_thread begindetached| → replaced by |ns_thread create -detached|
> 
>       o
> 
>         |ns_event| → replaced by |ns_cond|
> 
>       o
> 
>         |ns_pooldescription| → replaced by |ns_dbpooldescription|
> 
>       o
> 
>         |keyldel|, |keylget|, |keylkeys|, |keylset| (from TclX) →
>         replaced by native |dict| functionality in Tcl
> 
>       o
> 
>         Legacy experimental functions (marked TBD for over 20 years) are
>         now deprecated:
> 
>           + |ns_browsermatch|, |ns_choosecharset|, |ns_cookiecharset|,
>             |ns_formfieldcharset|, |ns_formvalueput|, |ns_paren|,
>             |ns_tagelement|, |ns_tagelementset|
>       o
> 
>         Deprecated non-namespaced functions:
> 
>           + |getformdata|, |issmallint|
>       o
> 
>         |ns_parsetime|: now officially deprecated (was internally marked
>         "To be removed" for ~15 years)
> 
>       o
> 
>         |ns_set_precision|: deprecated in favor of standard Tcl idioms
> 
> 
>       Tcl-Level Option Deprecations
> 
>   *
> 
>     Deprecated options:
> 
>       o |-buffered| in |ns_connchan write|
>       o |-donecallback| in |ns_http| → replaced by
>         |-done_callback| (naming consistency)
>       o |-binary| (previously used to indicate Tcl objects with binary
>         data) → replaced by |-data|, aligning with conventions where
>         |-binary| is a boolean flag
> 
> 
>       C-Level API Deprecations
> 
>   *
> 
>     Deprecated or removed C functions:
> 
>       o |Ns_ObjvFlags()| → replaced by |Ns_ObjvIndex()| for option
>         parsing (removed redundancy)
>       o |Ns_SockSendBufs2()| → replaced by |Ns_SockSendBufsEx()| which
>         returns an additional |errorCode|
>       o |Ns_TclInitInterps()| → removed (marked as deprecated since 2005)
>       o Deprecated internal usage of OpenSSL and Tcl functions that have
>         been marked as deprecated upstream
>       o All C functions previously marked as deprecated in source
>         comments are now officially deprecated
> 
> 
>       Configuration Parameter Deprecations
> 
>   *
> 
>     Global config parameters:
> 
>       o |logroll|: → replaced by |logrollonsignal| (standardized log
>         rotation behavior on |SIGHUP|)
>       o |serverlog|: → replaced by |systemlog| to reduce ambiguity
>         between system and per-server logs
>   *
> 
>     Section parameter changes:
> 
>       o |serverdir| in the |fastpath| section is now deprecated → use
>         |serverdir| in the main per-server section instead (reflecting
>         its broader usage scope)
> 
> 
>     Changes in extra Modules
> 
> 
>       letsencrypt
> 
>   * Tcl9 compatibility changes
> 
> 
>       nsauthpam
> 
>   * Tcl9 compatibility changes
>   * License upgrade
> 
> 
>       nscoap
> 
>   * Tcl9 compatibility changes
>   * NaviServer 5 compatibility
>   * License upgrade
> 
> 
>       nsdbbdb
> 
>   * Replaced deprecated Ns_DString operations
>   * Support for LMDB
> 
> 
>       nsdbi
> 
>   * Removed calls to deprecated functions
>   * Tcl9 compatibility changes
> 
> 
>       nsdbilite
> 
>   * Removed calls to deprecated functions
> 
> 
>       nsdbimy
> 
>   * Removed calls to deprecated functions
> 
> 
>       nsdbipg
> 
>   * Removed calls to deprecated functions
> 
> 
>       nsdbmysql
> 
>   * Removed calls to deprecated functions
> 
> 
>       nsdbpg
> 
>   * Tcl9 compatibility changes
>   * NaviServer 5 compatibility
>   * Removed calls to deprecated functions
>   * License upgrade
> 
> 
>       nsdbsqlite
> 
>   * Fixed execution of DML queries.
>   * Removed calls to deprecated functions
>   * Pulled in new upstream version (3.47.2) of qlite3.c
> 
> 
>       nsdns
> 
>   * Removed calls to deprecated functions
>   * Code cleanup
> 
> 
>       nsldap
> 
>   * Added support for LDAP URIs in pool configuration
>   * Removed calls to deprecated functions
>   * Removed unneeded compile macro
>   * Fixed result setting of "connected" subcommand
>   * Refactor nsldap Tcl command into modular subcommands
>   * Tcl9 compatibility changes
>   * Updated documentation
> 
> 
>       nsloopctl
> 
>   * Tcl9 compatibility changes
>   * Minor code cleanup
> 
> 
>       nsmemcache
> 
>   * Removed calls to deprecated functions
> 
> 
>       nsoracle
> 
>   * Added experimental boolean configuration parameter
>     |convertEncoding|: When this Boolean parameter is true, the
>     connection is created with OCIEnvNlsCreate() instead of
>     OCIEnvCreate(). OCIEnvNlsCreate was introduced with Oracle9 and is
>     an enhanced version of OCIEnvCreate() which is used by default
>     (legacy setting) or when ConvertEncoding is not true. This parameter
>     performs on-the-fly conversions when connecting to a Latin1 encoded
>     database.
>   * Removed calls to deprecated functions
>   * Modernized configuration (range checks) and use Ns_ReturnCode when
>     appropriate
>   * Tcl9 compatibility changes
> 
> 
>       nsphp
> 
>   * Removed calls to deprecated functions
> 
> 
>       nsshell
> 
>   * Updated command list for command completion for NaviServer 5
>   * Updated most JavaScript libraries
> 
> 
>       nssmtpd
> 
>   * Simplified configuration (provided defaults were possible)
>   * Added support for providing mail server password via userinfo in
>     |relayhost|
>   * Fixed multi-line response parsing
>   * Tcl9 compatibility changes
>   * NaviServer 5 compatibility
>   * Removed calls to deprecated functions
>   * Updated documentation
>   * License upgrade
> 
> 
>       nssnmp
> 
>   * Removed calls to deprecated functions
> 
> 
>       nsstats
> 
>   * More detailed reports:
>       o added list of loaded modules per server
>       o added list of all registered domain names for a driver
>       o added server log dir to per-server information
>   * Refactored web UI (adjusted layout for NaviServer 4.99 and 5)
>   * Added support for dark mode
>   * Added support for |nsperm| module, when installed
>   * Made parsing of http client log file more robust
>   * License upgrade
> 
> 
>       nssyslogd
> 
>   * Removed calls to deprecated functions
>   * NaviServer 5 compatibility
> 
> 
>       nstk
> 
>   * Tcl9 compatibility changes
> 
> 
>       nsudp
> 
>   * Tcl9 compatibility changes
>   * NaviServer 5 compatibility
>   * License upgrade
> 
> 
>       nswebpush
> 
>   * Tcl9 compatibility changes
>   * Updated code for newer and more picky web servers
>   * Fixed code for aes128gcm
> 
> 
>       revproxy
> 
>   * moved code to main repository
> 
> 
>       websocket
> 
>   * NaviServer 5 compatibility
>   * Avoid deprecated messages for |ns_conn write -buffered|
>   * Improved documentation
>   * More detailed timeout handling
> 
> 
> 
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

[naviserver-devel] Announcement: NaviServer 5.0.0 Release

From: Gustaf N. (sslmail) <ne...@wu...> - 2025-07-08 19:13:04

Dear all,
We are glad to announce the final release of NaviServer 5.0.0, marking the most significant update in the project’s history with over 70,000 lines of changes. This release delivers robust security enhancements, modernized infrastructure, and improved scalability. Key highlights include:

Security by Default: HTTP client requests now validate server certificates automatically, with configurable trust exceptions and a bundled CA root store. A new pluggable authorization framework (ns_auth) enables scriptable request- and user-level access control.

Modernized Core: Unix domain socket support, case-insensitive ns_set, Argon2 password hashing, and streamlined configuration via environment variables (ideal for containers). The reverse proxy is now built into the core, with new forward proxy capabilities.

Enhanced Observability and Scalability: Improved logging (dynamic paths, rotation, and new "Security" severity), detailed connection diagnostics (ns_conn urldict, ns_connchan debug), runtime introspection (ns_http keepalives, ns_info buildinfo), persistent client connections, and streaming HTTP client processing.

Backward Compatibility & Future-Readiness
While introducing cutting-edge features, NaviServer 5.0.0 maintains strong backward compatibility, with deprecated APIs clearly documented and modern replacements provided (e.g., ns_urlencode → ns_percentencode). The release introduces/extends over 50 Tcl commands and subcommands. The upgrade to MPL 1.2 and Tcl 9 support ensures long-term sustainability. Over 20 community-contributed modules (like nsdbpg, nssmtpd) have been updated in lockstep.

The following people contributed to this release:

Alexander Danilov, Andrew Piskorski, Antonio Pisano, Brendan Graves, Brian Fenton, Danilo Raynor, David F, David Osborne,

Filip Minic, Georg Lehner, Gustaf Neumann, Hector Romojaro, Joe Oldak, Oleg Oleinick, Priyank Jalu, Thomas Renner, Zoran Vasiljevic

Resources:

Download <https://sourceforge.net/projects/naviserver/files/naviserver/5.0.0/> | GitHub <https://github.com/naviserver-project/naviserver> | Docs <https://naviserver.sourceforge.io/5.0/toc.html>
This release is a major leap forward in security, flexibility, and maintainability. See below for the detailed changes: Upgrade today!

The NaviServer Team

Tcl API Changes

ns_cache

New option cachingmode: Accepts full or none to toggle caching behavior.
ns_configsection

New option -filter: Filters variables by unread, defaulted, or defaults.
ns_configure_variables

New command to pull in configuration variables from environment variables (for use in configuration files)
ns_conn

General enhancements:

ns_conn host, ns_conn port, ns_conn protocol: Now return appropriate values for all request types (not limited to forward proxy requests).
New subcommands:

ns_conn target: Returns the complete request target, including query parameters, from the HTTP start line.
ns_conn fragment: Returns the fragment identifier (after #) if present.
ns_conn urldict: Parses the request URL into a Tcl dictionary for structured access.
ns_conn host: Supports an optional default value if the host is not determined.
ns_conn details: Returns a dictionary with driver-specific connection metadata.
ns_connchan

Security and diagnostics improvements:

ns_connchan debug: Set or query the debug level.
ns_connchan connect / ns_connchan open: Now accept additional security-related options: -cafile, -capath, -cert, -hostname, -insecure, -driver, and -unixsocket.
ns_connchan status: Added -server option for multi-server introspection.
ns_connchan write: Now performs buffered writes.
More detailed timeout handling

ns_driver

info, names, stats, and threads support the -server option to allow multi-server diagnostics.
ns_http

Connection handling and introspection:

Persistent HTTP connections supported via the keepalive option in httpclient configuration.

Introduced support for multiple task threads.

Streaming response support for incremental data handling (useful for large file transfers and LLM interaction).

Default timeout configuration for requests.

New callbacks: response_header_callback and response_data_callback.

Renamed -donecallback to -done_callback (old name deprecated).

Added support for informational HTTP status codes (e.g., 100 Continue).

New subcommands:

ns_http keepalives: Displays active persistent connections.
ns_http taskthreads: Provides insight into HTTP client threads.
Major internal refactoring of tclhttp.c for modularity and maintainability.

ns_ictl

getmodules: Now supports the -server option for per-server introspection.
ns_info

Added subcommands:

argv: Returns the original argument vector.
bindir: Returns the binary directory path.
buildinfo: Displays build-time config and version info.
logdir: Path to the log directory.
meminfo: Displays memory statistics (when using tcmalloc).
ns_ip

Added subcommands:

inany: Checks if an IP matches any configured address.
properties: Returns metadata about configured IPs.
public: Determines if an IP is publicly routable.
trusted: Checks if an IP is trusted.
valid: Validates IP syntax.
These enhancements supersede the need for ns_subnetmatch.

ns_issmallint

New command to check if a value is a valid small integer (replaces the old, now deprecated issmallint).
ns_logctl

New subcommand grep: Searches log files with pattern matching, stripping color codes and handling line continuations.
ns_register* Enhancements

All ns_register_* commands now support the -constraints option, enabling context-sensitive behavior.
New command: ns_register_auth allows registration of user- and request-level authorization handlers.
ns_server

New subcommands:
authprocs: Lists registered request/user auth procedures.
hosts: Lists registered hostnames for the server.
logdir: Returns the server log directory.
realm: returns or sets the realm of the server
serverdir: Returns the base directory for the virtual server (-effective returns the resolved runtime path).
vhostenabled: Returns a boolean indicating virtual hosting status.
ns_server mapped: New option -all returns a dict with handler and pool info.
ns_set

Case-insensitive support:

New -nocase flag for ns_set create and related operations.
Deprecated the i* subcommands (e.g., iget, ifind) in favor of unified interface.
Multi-valued key support:

-all option retrieves all values for a given key in ns_set and ns_config.
New subcommands:

format: Pretty-prints the set contents.
stats: Returns memory usage statistics.
delkey: Now returns a boolean success flag.

Internal improvements:

Refactored ns_set internals using dense storage for lower memory use and better cache locality.
Replaced deeply nested switch logic with modular, maintainable code.
ns_thread

Unified thread creation:

Use ns_thread create for all thread types.
Deprecated begin and begindetached to align with ns_cond, ns_mutex, and ns_sema usage.
ns_urlspace

Wildcard matching now supports path-segment-level matches (previously limited to leaf nodes).

Applied improved matching in nscgi to support directory-wide CGI mapping.

Renamed option -contextfilter to -constraints in ns_urlspace set/unset to match the new registration API.

Database Enhancements

ns_db info: Returns a dictionary of metadata for a given DB handle.
ns_dbpooldescription: Renamed from the previous ns_pooldescription for consistent naming.
ns_db rowcount: Fixed in this release to return actual row counts.
New Utilities

ns_fseekchars: Efficiently scans a stream for a string (e.g., for multipart/form-data parsing).
ns_joinurl: Constructs well-formed URLs from path components.
ns_mkdtemp: Creates a unique temporary directory (POSIX-style mkdtemp()).
ns_parsehtml: Parses HTML fragments into a structured Tcl dict.
ns_parsemessage: Parses MIME-style messages (e.g., emails or HTTP headers).
ns_percentencode / ns_percentdecode: Replace deprecated ns_urlencode and ns_urldecode for robust URL-safe encoding/decoding.
Changes in Core Modules

nscgi

New command: ns_register_cgi for dynamic CGI handler registration (supports -noinherit, -path, etc.).
Supports unregistration via ns_unregister_op.
New environment variables: SCRIPT_FILENAME, REQUEST_URI.
Refactored to support major web apps (e.g., WordPress, Joomla).
See commit 36027b70215 for implementation details.
nscp

Integrated with new authorization system (e.g., via nsperm users).
New command nscp users: Lists registered users for nscp authentication.
nsperm

Acts as a pluggable authorization provider for request/user scopes.
New config parameter: allowLoopbackEmptyUser—permits unauthenticated loopback access when enabled.
Added support for setting default server start pages and nsstats.
nsproxy

New subcommand: ns_proxy workers—provides detailed runtime info on proxy worker processes.
Bug Fixes

Stability and Crash Resolutions

Addressed multiple potential crash scenarios:

Robust handling of HTTP CONNECT requests.
Fixed crashes caused by missing or empty argument lists in commands such as ns_filestat, ns_sockcallback, and ns_ictl oncleanup.
Prevented crash in ns_log when invoked with an empty message.
Resolved a 24-year-old bug in Ns_AdjTime() that could lead to fatal errors due to microsecond overflow in multithreaded environments.
Fixed crash in ns_conn copy when operating on empty content.
Prevented crash in ns_sema release with invalid semaphore counts.
Corrected off-by-one error in ns_adp_bind_args that could access uninitialized Tcl_Obj values.
Fixed crash in ns_inet_ntop due to unsafe memory operations with overlapping regions (notably on aarch64 with musl).
Resolved crash in Ns_SetIUpdateSz() caused by case mismatch in header keys with the legacy C API.
Fixed crash during nscp startup when the users section was not configured.
Prevented crash in debug mode when the Host: header could not be mapped to a virtual server and the driver was installed locally.
Avoided crash when launching nsd with -c and -t options and no home parameter defined.
Fixed crash during computation of ns_conn location when the network driver was not globally installed (global installation now recommended).
Fixed crash due to self-destructive header replacement when ns_conn outputheaders are passed via ns_respond ... -headers ...
Functional Correctness and Logic Fixes:

ns_conn status: Fixed issue where updated status codes were silently ignored.
ns_conn doneCallback: Ensured this callback is always invoked in ns_http.
nsv_dict get: Fixed a memory leak in value retrieval.
ns_conn peeraddr: Resolved race condition in pipelined requests that could yield incorrect peer addresses.
ns_cache_eval -force: Fixed race condition that could produce obsolete results.
ns_sema create: Corrected handling of initial count values (e.g., 1000).
ns_trim: Fixed spacing logic to trim only leading and trailing whitespace, preserving internal spacing.
ns_config -int: Corrected fallback behavior when invalid values are supplied, now correctly using documented defaults.
ns_sockcallback: Now gracefully handles missing arguments.
ns_socknread: Fixed inaccurate results for buffered connections.
ns_crypto::aead::encrypt/decrypt: Restored compatibility with OpenSSL 1.1.1.
Fixed incorrect parsing of encoded backslashes in URLs.
Ns_StrTrimRight(): Corrected UTF-8 handling in right-side string trimming.
Fixed ns_conn location when running behind a reverse proxy to always returns a value, even in broken configurations.
OpenSSL fixes

Added support for detecting and validating OCSP Must-Staple and AIA presence in certificates. Without that, NaviServer might crash, when OCSP is turned on, and NULL values are passed for AIA URLs (letsencrypt)
Introduced stable output buffers for send operations to support retries after SSL_ERROR_WANT_WRITE, preventing connection failures under high load.
Improved error handling in OpenSSL integration by draining the error stack via DrainErrorStack().
HTTP Client Fixes

ns_http:
Reordered initialization in NsInitServer() to ensure submodules can access a fully configured server state.
HTML and ADP Parsing Fixes

ADP Parser:
Enhanced support for quoted > characters inside attribute values, aligning with modern HTML parsing rules.
return-notice handling:
Suppressed spurious error messages when fallback ADP templates are processed outside of a full ADP context.
ns_striphtml:
Fixed long-standing bug where adjacent HTML entities were incorrectly decoded—only the first entity was processed.
Logging and Diagnostics Fixes

Fixed misleading log output when dynamically changing extended headers via ns_accesslog extendedheaders ....
Improved error messages when port binding fails due to conflicts with driver assignment.
Fixed off-by-one error in the virtual server port configuration logic.
Clarified or corrected multiple logging messages across modules for better diagnostics.
nscgi Module Fixes

Fixed file upload failures when uploads were internally spooled by NaviServer — previously returned 500 errors.
Corrected processing of CGI script exit codes not 0.
Fixed hostname and port reporting for SERVER_NAME.
Database Fixes

ns_db rowcount: Fixed regression where the row count logic was not invoked, rendering the command a no-op.
Build and Compatibility Fixes

Suppressed obsolete --enable-threads warning for Tcl versions where thread support is now enabled by default.
Fixed compilation failure with glibc 2.38+ due to PTHREAD_STACK_MIN becoming dynamic via sysconf().
General Cleanups

Fixed small memory leak triggered by serverrootproc reset.
Numerous minor typo corrections, comment clarifications, and small logic cleanups across modules.
C-Level Infrastructure, C API Enhancements, and Build System

Core Infrastructure Improvements

Socket Layer Enhancements:

Added sendErrno field to the Sock structure to improve diagnostics and tracking of write errors.
Modernized Initialization:

Introduced NS_INIT_ONCE() macro for thread-safe one-time initialization, replacing legacy double-lock patterns.
Data Structure Utilities:

Added utility functions Ns_DListSaveString() and Ns_DListFreeElements() to simplify dynamic list management.
Replaced use of legacy Ns_DString functions with standard Tcl_DString, modernizing internal data handling.
Debugging Support:

Added NsHexPrint() for hex-dumping of byte sequences, aiding in low-level debugging and analysis.
Introspection Enhancements:

Introduced Ns_TclReturnCodeString() and Ns_ReturnCodeString() to convert internal return codes into readable strings, improving log clarity and diagnostics.
Code Quality and Performance:

Applied extensive internal refactorings to improve performance, cache locality, and maintainability across multiple subsystems.
C API Enhancements

General:

Added typedefs for Ns_AuthorizeRequestProc, Ns_AuthorizeUserProc,Ns_UrlSpaceMatchInfo, Ns_DriverConnInfoProc
Added enum for Ns_RequestType, Ns_UrlSpaceOp, Ns_DriverClientInitArg
Added API calls in ns.h: Ns_ConfigFilename, Ns_ConnTarget, Ns_UrlSpaceMatchInfo, Ns_ConnServPtr, Ns_DStringAppendSockState, Ns_RegisterFilter2, Ns_TaskQueueLength, Ns_TaskQueueName, Ns_TaskQueueRequests, Ns_ObjvTablePrint, Ns_InfoLogPath, Ns_LogPath, Ns_ServerLogDir, Ns_ServerRootProcEnabled, Ns_ServerLogGetFd, Ns_ServerLogCloseAll, Ns_ServerLogRollAll, Ns_GetServer, Ns_ServerName, Ns_SockSetSendErrno, Ns_SockGetSendErrno, Ns_SockGetSendRejected, Ns_SockGetSendCount, Ns_SockFlagAdd, Ns_SockFlagClear, Ns_SockSendBufsEx, Ns_SockConnectUnix, Ns_SockGetClientSockAddr, Ns_SockGetConfiguredSockAddr, Ns_SockaddrPublicIpAddress, Ns_SockaddrTrustedReverseProxy, Ns_SockaddrInAny, Ns_SockaddrAddToDictIpProperties, Ns_TclReturnCodeString, Ns_ReturnCodeString, Ns_TclInterpServPtr, Ns_LogDeprecatedParameter, Ns_RegisterFastUrl2File
Extended Ns_Request, Ns_TclCallback
Refactored Ns_ConnReturnMoved() and Ns_ConnReturnRedirect() to unify redirection logic.
Secure Communication and Validation:

Added fine-grained certificate validation APIs
Improved error stack draining for OpenSSL operations.
Extended C API for ns_connchan:

Introduced a minimal C-level API for connchan connections, enabling finer control over lower-level network operations.
Build Platform Compatibility:

Updated Windows build system to use NS_IMPORT in place of deprecated NS_EXTERN.
Avoided use of deprecated C functions such as mktemp() to improve portability and security.
Build System and Tooling

Configuration and Sample Support:

Replaced hard-coded OS commands (ls, mv, ...) consistently with build variables
Replaced hard-coded OpenSSL binary references with the $(OPENSSL) build variable.
Enhanced log path and configuration variable handling to support more dynamic setups.
Test Infrastructure:

Extended regression and compatibility test coverage to validate edge cases and cross-platform behavior.
Build Metadata Introspection:

Added support for build-time introspection of environment settings, including allocator type (e.g., malloc), compiler version, and Tcl build info.
Optional Deprecation-Free Builds:

Introduced build flag NS_NO_DEPRECATED to exclude deprecated functions and APIs, mirroring Tcl’s TCL_NO_DEPRECATED mechanism.
CI/CD Modernization:

Upgraded GitHub Actions workflows from version 3 to version 4 for improved performance and compatibility.
Sample Configuration Files:

Improved sample configurations (nsd-config and openacs-config) to use environment-specific settings via ns_configure_variables
prefer names http and https instead of nsock and nsssl inside sample configuration files to ease configuration for new users.
Documentation Updates

Comprehensive Overhaul

Conducted a thorough review and restructuring of all documentation:

Tcl Command Documentation:

Ensured all implemented Tcl commands and their options are fully documented.
Removed documentation for obsolete or unimplemented commands.
Deprecated commands are no longer advertised in manuals or used in examples.
Introduced a dedicated section listing deprecated commands.
Enabled automated generation of a complete command reference.
Test Alignment:

Verified that every implemented Tcl command is covered by regression tests.
Ensured that all documented commands are also implemented and tested.
Syntax and Formatting Consistency

Standardized placeholder syntax across the documentation and syntax error messages (see commit ffbd32774dbfor details).
Manual Page Improvements

nscgi: Significantly revised documentation and usage examples for clarity and completeness.

admin-config.man: Added a new section on “Customizing File Locations”.

Added and updated sections covering:

Basic templating features.
Error handling mechanisms.
Recently introduced Tcl commands and options.
Visual and Structural Enhancements

Updated diagrams and usage examples related to:

Request processing flow.
Reverse proxy (revproxy) behavior.
Additional Improvements

Numerous corrections to spelling, grammar, formatting, and internal linking throughout the documentation.
Deprecation Management Infrastructure

Introduced compiler warnings for usage of deprecated Ns_DString* functions
Added log severity level Deprecated to better surface deprecated usage at runtime
Documented all deprecated commands in a dedicated section of the command reference
Deprecated the use of manual double-checked locking for one-time initialization replaced by the NS_INIT_ONCE() macro for safer and cleaner initialization semantics
Deprecations

Tcl-Level Command Deprecations

Deprecated the following Tcl commands:

ns_set print → replaced by ns_set format

ns_checkurl and ns_requestauthorize → replaced by ns_auth request

ns_thread begin → replaced by ns_thread create

ns_thread begindetached → replaced by ns_thread create -detached

ns_event → replaced by ns_cond

ns_pooldescription → replaced by ns_dbpooldescription

keyldel, keylget, keylkeys, keylset (from TclX) → replaced by native dict functionality in Tcl

Legacy experimental functions (marked TBD for over 20 years) are now deprecated:

ns_browsermatch, ns_choosecharset, ns_cookiecharset, ns_formfieldcharset, ns_formvalueput, ns_paren, ns_tagelement, ns_tagelementset
Deprecated non-namespaced functions:

getformdata, issmallint
ns_parsetime: now officially deprecated (was internally marked "To be removed" for ~15 years)

ns_set_precision: deprecated in favor of standard Tcl idioms

Tcl-Level Option Deprecations

Deprecated options:

-buffered in ns_connchan write
-donecallback in ns_http → replaced by -done_callback (naming consistency)
-binary (previously used to indicate Tcl objects with binary data) → replaced by -data, aligning with conventions where -binary is a boolean flag
C-Level API Deprecations

Deprecated or removed C functions:

Ns_ObjvFlags() → replaced by Ns_ObjvIndex() for option parsing (removed redundancy)
Ns_SockSendBufs2() → replaced by Ns_SockSendBufsEx() which returns an additional errorCode
Ns_TclInitInterps() → removed (marked as deprecated since 2005)
Deprecated internal usage of OpenSSL and Tcl functions that have been marked as deprecated upstream
All C functions previously marked as deprecated in source comments are now officially deprecated
Configuration Parameter Deprecations

Global config parameters:

logroll: → replaced by logrollonsignal (standardized log rotation behavior on SIGHUP)
serverlog: → replaced by systemlog to reduce ambiguity between system and per-server logs
Section parameter changes:

serverdir in the fastpath section is now deprecated → use serverdir in the main per-server section instead (reflecting its broader usage scope)
Changes in extra Modules

letsencrypt

Tcl9 compatibility changes
nsauthpam

Tcl9 compatibility changes
License upgrade
nscoap

Tcl9 compatibility changes
NaviServer 5 compatibility
License upgrade
nsdbbdb

Replaced deprecated Ns_DString operations
Support for LMDB
nsdbi

Removed calls to deprecated functions
Tcl9 compatibility changes
nsdbilite

Removed calls to deprecated functions
nsdbimy

Removed calls to deprecated functions
nsdbipg

Removed calls to deprecated functions
nsdbmysql

Removed calls to deprecated functions
nsdbpg

Tcl9 compatibility changes
NaviServer 5 compatibility
Removed calls to deprecated functions
License upgrade
nsdbsqlite

Fixed execution of DML queries.
Removed calls to deprecated functions
Pulled in new upstream version (3.47.2) of qlite3.c
nsdns

Removed calls to deprecated functions
Code cleanup
nsldap

Added support for LDAP URIs in pool configuration
Removed calls to deprecated functions
Removed unneeded compile macro
Fixed result setting of "connected" subcommand
Refactor nsldap Tcl command into modular subcommands
Tcl9 compatibility changes
Updated documentation
nsloopctl

Tcl9 compatibility changes
Minor code cleanup
nsmemcache

Removed calls to deprecated functions
nsoracle

Added experimental boolean configuration parameter convertEncoding: When this Boolean parameter is true, the connection is created with OCIEnvNlsCreate() instead of OCIEnvCreate(). OCIEnvNlsCreate was introduced with Oracle9 and is an enhanced version of OCIEnvCreate() which is used by default (legacy setting) or when ConvertEncoding is not true. This parameter performs on-the-fly conversions when connecting to a Latin1 encoded database.
Removed calls to deprecated functions
Modernized configuration (range checks) and use Ns_ReturnCode when appropriate
Tcl9 compatibility changes
nsphp

Removed calls to deprecated functions
nsshell

Updated command list for command completion for NaviServer 5
Updated most JavaScript libraries
nssmtpd

Simplified configuration (provided defaults were possible)
Added support for providing mail server password via userinfo in relayhost
Fixed multi-line response parsing
Tcl9 compatibility changes
NaviServer 5 compatibility
Removed calls to deprecated functions
Updated documentation
License upgrade
nssnmp

Removed calls to deprecated functions
nsstats

More detailed reports:
added list of loaded modules per server
added list of all registered domain names for a driver
added server log dir to per-server information
Refactored web UI (adjusted layout for NaviServer 4.99 and 5)
Added support for dark mode
Added support for nsperm module, when installed
Made parsing of http client log file more robust
License upgrade
nssyslogd

Removed calls to deprecated functions
NaviServer 5 compatibility
nstk

Tcl9 compatibility changes
nsudp

Tcl9 compatibility changes
NaviServer 5 compatibility
License upgrade
nswebpush

Tcl9 compatibility changes
Updated code for newer and more picky web servers
Fixed code for aes128gcm
revproxy

moved code to main repository
websocket

NaviServer 5 compatibility
Avoid deprecated messages for ns_conn write -buffered
Improved documentation
More detailed timeout handling

[naviserver-devel] Forthcoming OpenACS/EuroTcl conference in Bologna

From: Gustaf N. (sslmail) <ne...@wu...> - 2025-07-02 16:04:49

Dear all,

The preliminary program for this year’s conference (next week in Bologna)  is now available:
👉 https://openacs.km.at/evaluate/org/129998253/companyhelp/schedule

Please note that we will not have live-streams this year. However, we plan to make recordings of the presentations available afterward.

As in previous years, the program includes an OpenACS and NaviServer panel to discuss future directions for both projects. If you have suggestions for improvements or areas that need attention - whether technical, community-related, or otherwise, or features you need in your projects - please feel free to email me your thoughts. I’ll make sure your input is brought into the discussion.

Looking forward to hearing from you!

All the best,
-g

[naviserver-devel] NaviServer 5 RC1 available

From: Gustaf N. (sslmail) <ne...@wu...> - 2025-06-26 17:46:42

Hi everyone,

I’m glad to report that NaviServer 5.0 RC 1 is now available on SourceForge:
https://sourceforge.net/projects/naviserver/files/NaviServer5.0/

The updated online documentation can be found here:
https://naviserver.org/docs/5.0/

This marks our feature freeze - over the coming weeks we’ll focus strictly on bug fixes and stability improvements. Our goal is to deliver the final 5.0 release before the OpenACS/EuroTcl/Conference in Bologna (July 10–11, 2025): https://openacs.km.at/

Below is the preliminary changelog for the core package. We continue work on a summary of the main changes in the NaviServer modules repositories.
Since raw changelog since the 4.99 release is quite large (>22K lines), something may have slipped through. If you spot any omissions that should be on our radar, please let me know.

The changes are quite extensive:

522 files changed, 78016 insertions(+), 32993 deletions(-)

According to our documentation, this is the largest number of changes between releases.
Please, if possible, download RC 1, give it a spin, and let us know of any regressions or show-stopper bugs by opening an issue on our tracker.
https://github.com/naviserver-project/naviserver

Thank you for your testing and feedback - let’s make the 5.0 release rock solid for Bologna!

Best regards,
The NaviServer Team

Changelog

New Features

Security Enhancements

Secure by Default when NaviServer is used as an HTTP client (ns_http and ns_connchan)
All HTTP client requests now validate the peer server certificate by default.

Introduced the -insecure flag to bypass validation in trusted scenarios.

Bundled a ca-bundle.crt with well-known root certificates for out-of-the-box verification support.

Fine-Grained Validation Management:

Configure validation exceptions in the httpclient section to allow specific certificate issues (e.g., self-signed certs) for defined IP ranges.
Define a directory for storing rejected certificates, enabling later review, manual trust decisions, or audit trails.
For details, consult the admin-config manual.
Certificate Management Added new commands:

ns_certctl list – List loaded certificates.
ns_certctl reload – Reload certificates at runtime.
Pluggable Authorization Framework

Introduced flexible, scriptable request- and user-level authorization:
Supports multiple authorization handlers ("authorities") with ordered evaluation.
Handlers can return continuation codes (TCL_OK, TCL_BREAK, TCL_RETURN) to control the authorization chain.
New Tcl commands:
ns_auth request – Evaluate configured request-level auth handlers.
ns_auth user – Evaluate user-level authorization logic.
Improved Reverse Proxy Trust Management

Define trusted upstream proxies via CIDR ranges.
Configurable handling of X-Forwarded-For chains, including:
Skipping non-routable addresses.
Alternative resolution strategies.
Improves accuracy and security of client IP logging (e.g., in access.log).
For implementation details, see commit ab23158ece.
Argon2 Password Hashing New command: ns_crypto::argon2 – Provides support for modern, secure password hashing.

User Interface and HTML Improvements

Modernized Default Start Page

Redesigned layout with responsive and accessible design.
Added modal dialog for password updates.
Improved security guidance for publicly exposed installations.
Installer now fetches resources from GitHub (previously Bitbucket).
Dark Mode Support

Automatically respects system preferences via the prefers-color-scheme media query.
Supported across modern versions of Windows, macOS, iOS, and Android.
Consistent Visual Styling

Unified look and feel across:
Default start page
NaviServer documentation
nsstats module interface
Configuration and File Management

Configuration via environment variables

Added ability to pull in environment variables with a certain prefix as configuration variables for configuration files (new command ns_configure_variables). This helps configuration significantly for container-based deployments, where ports etc. have to be adjusted on standardized configuration files.
New File Location Options

Unified log path resolution via logdir or serverdir.
Moved serverdir definition from fastpath to the top-level server section; it now acts as the root for both logdirand page directories.
Added bindir parameter for custom binary module and nsproxy worker locations.
Improved Log Configuration

Relative log file paths are now resolved consistently based on:
serverrootproc (for mass virtual hosting), or
serverdir.
Supports dynamic log file naming (e.g., using Host header) for per-vhost logging.
Log rotation now applies across multiple log targets.
Improved Section Parameter Updates

New -update flag for ns_section, allowing ns_param to overwrite previously defined values rather than appending.
Proxy Features

Reverse Proxy Integration

The reverse proxy module is now built into the NaviServer core.
Leverages improvements in ns_http, including:
Persistent connections
Streaming request handling
New features:
-use_target_host_header flag for ::revproxy::upstream to control Host header forwarding.
Configurable transport mechanism using either ns_http or ns_connchan.
Support for backend connections via Unix domain sockets.
Forward Proxy Implementation

Base implementation included, built on the same scalable architecture as the reverse proxy.
Full support for the HTTP CONNECT method:
Enables tunneling HTTPS connections through the proxy.
Eliminates the need for insecure plaintext fallbacks.
Additional Functional Improvements

Unix Domain Socket Support

Both ns_http and ns_connchan now support connections via Unix domain sockets.
Multibyte HTML Entity Support

ns_striphtml and ns_unquotehtml now correctly decode multibyte numeric HTML entities.
New Log Severities

Security – For security-related events.
Deprecated – For usage of deprecated features (e.g., API calls, parameters).
TclPro Debugger Support

Significantly improved integration with the TclPro debugger.
Resolved missing file references and corrected documentation.
See the ns_adp_debug manual for usage.
Tcl 9 Compatibility

Internal updates to support Tcl 9 features and constraints.
Support for Tcl command argument counts >2^31.
License Upgrade

Changed from MPL 1.1 + GPL to MPL 1.2.
Announced previously on the mailing list and Tcl forums.
Tcl API Changes

ns_cache

New option cachingmode: Accepts full or none to toggle caching behavior.
ns_configsection

New option -filter: Filters variables by unread, defaulted, or defaults.
ns_configure_variables

New command to pull in configuration variables from environment variables (to be used in configuration files)
ns_conn

General enhancements:

ns_conn host, ns_conn port, ns_conn protocol: Now return appropriate values for all request types (not limited to forward proxy requests).
ns_conn target: Returns the complete request target, including query parameters, from the HTTP start line.
ns_conn fragment: Returns the fragment identifier (after #) if present.
ns_conn urldict: Parses the request URL into a Tcl dictionary for structured access.
ns_conn host: Supports an optional default value if the host is not determined.
ns_conn details: Returns a dictionary with driver-specific connection metadata.
ns_connchan

Security and diagnostics improvements:

info, names, stats, and threads support the -server option to allow multi-server diagnostics.
ns_http

Connection handling and introspection:

Persistent HTTP connections supported via the keepalive option in httpclient configuration.

Introduced support for multiple task threads.

Streaming response support for incremental data handling (useful for large file transfers and LLM interaction).

Default timeout configuration for requests.

New callbacks: response_header_callback and response_data_callback.

Renamed -donecallback to -done_callback (old name deprecated).

Added support for informational HTTP status codes (e.g., 100 Continue).

New subcommands:

ns_http keepalives: Displays active persistent connections.
ns_http taskthreads: Provides insight into HTTP client threads.
Major internal refactoring of tclhttp.c for modularity and maintainability.

ns_ictl

getmodules: Now supports the -server option for per-server introspection.
ns_info

Added subcommands:

ns_issmallint

New command to check if a value is a valid small integer (replaces the old, now deprecated issmallint).
ns_logctl

New subcommand grep: Searches log files with pattern matching, stripping color codes and handling line continuations.
ns_register* Enhancements

Case-insensitive support:

New -nocase flag for ns_set create and related operations.
Deprecated the i* subcommands (e.g., iget, ifind) in favor of unified interface.
Multi-valued key support:

-all option retrieves all values for a given key in ns_set and ns_config.
New subcommands:

format: Pretty-prints the set contents.
stats: Returns memory usage statistics.
delkey: Now returns a boolean success flag.

Internal improvements:

Refactored ns_set internals using dense storage for lower memory use and better cache locality.
Replaced deeply nested switch logic with modular, maintainable code.
ns_thread

Unified thread creation:

Use ns_thread create for all thread types.
Deprecated begin and begindetached to align with ns_cond, ns_mutex, and ns_sema usage.
ns_urlspace

Wildcard matching now supports path-segment-level matches (previously limited to leaf nodes).

Applied improved matching in nscgi to support directory-wide CGI mapping.

Renamed option -contextfilter to -constraints in ns_urlspace set/unset to match the new registration API.

Database Enhancements

nscgi

Integrated with new authorization system (e.g., via nsperm users).
New command nscp users: Lists registered users for nscp authentication.
nsperm

New subcommand: ns_proxy workers—provides detailed runtime info on proxy worker processes.
Bug Fixes

Stability and Crash Resolutions

Addressed multiple potential crash scenarios:

ns_http:
Reordered initialization in NsInitServer() to ensure submodules can access a fully configured server state.
HTML and ADP Parsing Fixes

ns_db rowcount: Fixed regression where the row count logic was not invoked, rendering the command a no-op.
Build and Compatibility Fixes

Core Infrastructure Improvements

Socket Layer Enhancements:

Added sendErrno field to the Sock structure to improve diagnostics and tracking of write errors.
Modernized Initialization:

Introduced NS_INIT_ONCE() macro for thread-safe one-time initialization, replacing legacy double-lock patterns.
Data Structure Utilities:

Added NsHexPrint() for hex-dumping of byte sequences, aiding in low-level debugging and analysis.
Introspection Enhancements:

Introduced Ns_TclReturnCodeString() and Ns_ReturnCodeString() to convert internal return codes into readable strings, improving log clarity and diagnostics.
Code Quality and Performance:

Applied extensive internal refactorings to improve performance, cache locality, and maintainability across multiple subsystems.
C API Enhancements

General:

Added fine-grained certificate validation APIs
Improved error stack draining for OpenSSL operations.
Extended C API for ns_connchan:

Introduced a minimal C-level API for connchan connections, enabling finer control over lower-level network operations.
Build Platform Compatibility:

Updated Windows build system to use NS_IMPORT in place of deprecated NS_EXTERN.
Avoided use of deprecated C functions such as mktemp() to improve portability and security.
Build System and Tooling

Configuration and Sample Support:

Extended regression and compatibility test coverage to validate edge cases and cross-platform behavior.
Build Metadata Introspection:

Added support for build-time introspection of environment settings, including allocator type (e.g., malloc), compiler version, and Tcl build info.
Optional Deprecation-Free Builds:

Introduced build flag NS_NO_DEPRECATED to exclude deprecated functions and APIs, mirroring Tcl’s TCL_NO_DEPRECATED mechanism.
CI/CD Modernization:

Upgraded GitHub Actions workflows from version 3 to version 4 for improved performance and compatibility.
Sample Configuration Files:

Comprehensive Overhaul

Conducted a thorough review and restructuring of all documentation:

Tcl Command Documentation:

Verified that every implemented Tcl command is covered by regression tests.
Ensured that all documented commands are also implemented and tested.
Syntax and Formatting Consistency

Standardized placeholder syntax across the documentation and syntax error messages (see commit ffbd32774db for details).
Manual Page Improvements

nscgi: Significantly revised documentation and usage examples for clarity and completeness.

admin-config.man: Added a new section on “Customizing File Locations”.

Added and updated sections covering:

Basic templating features.
Error handling mechanisms.
Recently introduced Tcl commands and options.
Visual and Structural Enhancements

Updated diagrams and usage examples related to:

Request processing flow.
Reverse proxy (revproxy) behavior.
Additional Improvements

Numerous corrections to spelling, grammar, formatting, and internal linking throughout the documentation.

Improved WebSocket documentation with expanded details on:

Timeout handling.
Use of connchan for bidirectional communication.
Deprecations

Tcl-Level Command Deprecations

Deprecated the following Tcl commands:

ns_set print → replaced by ns_set format

ns_checkurl and ns_requestauthorize → replaced by ns_auth request

ns_thread begin → replaced by ns_thread create

ns_thread begindetached → replaced by ns_thread create -detached

ns_event → replaced by ns_cond

ns_pooldescription → replaced by ns_dbpooldescription

keyldel, keylget, keylkeys, keylset (from TclX) → replaced by native dict functionality in Tcl

Legacy experimental functions (marked TBD for over 20 years) are now deprecated:

ns_browsermatch, ns_choosecharset, ns_cookiecharset, ns_formfieldcharset, ns_formvalueput, ns_paren, ns_tagelement, ns_tagelementset
Deprecated non-namespaced functions:

getformdata, issmallint
ns_parsetime: now officially deprecated (was internally marked "To be removed" for ~15 years)

ns_set_precision: deprecated in favor of standard Tcl idioms

Tcl-Level Option Deprecations

Deprecated options:

Deprecated or removed C functions:

Global config parameters:

serverdir in the fastpath section is now deprecated → use serverdir in the main per-server section instead (reflecting its broader usage scope)
Deprecation Management Infrastructure

Re: [naviserver-devel] nsauthn: prototypish Tcl module using ns_register_auth

From: Gustaf N. (sslmail) <ne...@wu...> - 2025-06-25 10:22:00


> On 24.06.2025, at 22:21, Georg Lehner <jor...@ma...> wrote:
> What is needed is a configuration option for the LDAP version to use, so that one can adapt to the capabilities of the given server.
> 
I do not see this as urgent. LDAPv3 was defined in RFC2251 (Dec 97) and is supported by OpenLDAP since 25 years (full compliance in 2000).
the nsldap module uses ldap_sasl_bind_s(), which was not supported by v2, so, without v3 there is not much the module can do.

>>> Global Initialzation of a Tcl Module
>>> 
> As I understand it, with the provided nsd-config.tcl the global libraries are served from /usr/local/ns/tcl, as well as the private libraries for the server default.
> 
> I created a module as a subdirectory of /usr/local/ns/tcl and put an initialization proc into my module into the file init.tcl and also execute it within this file.  When I load the module globally, the initialization proc is not run and the module is not loaded. When I load the module in the default server it is loaded/executed as expected.
> 
> Code within /usr/local/ns/tcl/init.tcl is executed, but I do not want to touch this file, since it is part of the installation of Naviserver.
> 

I think the code works as designed. If you load the module globally, then the global library files are used. If you load the module per server, then the per-server library files are used. 
Do you say, this is not the case? If so, this is a bug.
Do you expect that a globally loaded module uses the per-server library files?

All the best
-g

Re: [naviserver-devel] nsauthn: prototypish Tcl module using ns_register_auth

From: Georg L. <jor...@ma...> - 2025-06-24 20:22:05

On 18.06.25 10:43, Gustaf Neumann (sslmail) wrote:
>>
>>   * .... The pre-processor macro LDAPV3 has to be defined manually
>>     when building in order to work with current LDAP server versions.
>>
> As i see it, the macro is not necessary (i’ve replaced it with a 
> proper version compare, with macros provided by the standard includes.
> Also without the macro being defined, nsdap compiles fine and works 
> (but without setting the LDAP to LDAPV3). Is this maybe Alpine specific?

Sorry for not being specific, what I meant to say is: Since the LDAP 
version is hardcoded and by default is lower then LDAPV3 the module will 
not work with an LDAP server expecting LDAPV3. To remedy one has to 
recompile.

What is needed is a configuration option for the LDAP version to use, so 
that one can adapt to the capabilities of the given server.

>
>>   * Ns_ModuleInit() returns extraneously when scheduling a background
>>     procedure: I worked around by setting maxidle to 0 in the pool
>>     configuration.
>>
> what means  "returns extraneously”? crash? Tcl exception? I would 
> expect, Ns_ModuleInit() to be executed during startup (when all 
> modules are loaded), not in a background procedure.
At the end of Ns_ModuleInit() background procedures are scheduled in a 
loop with return Ns_ScheduleProcEx(...). Therefore Ns_ModuleInit() exits 
at this point with the integer id of the first scheduled proc. I can see 
though, that you already fixed this in the source code.
>>
>>   * Thorough documentation of all configuration options is missing.
>>
> ditto
I already started to complement the documentation (plus clarifying and 
reformatting) in a fork of the repo. I'll be happy to include 
documentation of your improvements and create a pull request.
>>
>>  *
>>
>>
>>   * Current LDAP client library implementations encourage the use of
>>     LDAP URI's instead of host, port and schema. This would be a
>>     useful enhancement.
>>
> i have added this to the repository. Please test if possible, i have 
> no usable LDAP server installation.
I'll do! thanks for the improvements.
>>
>>
>>       Controlled Creation of the auth Chain
>>
>> Initially I had the idea of separating authentication and 
>> authorization and have the admin of the server decide how and in 
>> which order to run respective "filters".  The current implementation 
>> does not give exact control over the order of registration of auth 
>> callbacks.  I'll have to learn more on the details of Naviserver 
>> configuration.
>>
> what do you mean by “no control”? This is exactly like for filters, 
> you can append at the end or at the begin,ut

It was a problem in my approach to structure the configuration sections 
for the module. I already figured out a better way to do it.

>
>>       Global Initialzation of a Tcl Module
>>
>> The nsauthn module implements caching of credentials from passwd 
>> files.  For this I wanted to use a global initialization procedure, 
>> run before any server is initialized. Alas I did not find if this is 
>> even possible. Any info on this is appreciated.
>>
> In general, the global (shared) Tcl libraries are loaded before the 
> per-server (private) libraries.
> You get the path of the global libraries via [ns_library shared] and 
> the per-server ones via ns_library private]
>
> https://naviserver.sourceforge.io/5.0/manual/files/tcl-libraries.html

As I understand it, with the provided nsd-config.tcl the global 
libraries are served from /usr/local/ns/tcl, as well as the private 
libraries for the server default.

I created a module as a subdirectory of /usr/local/ns/tcl and put an 
initialization proc into my module into the file init.tcl and also 
execute it within this file.  When I load the module globally, the 
initialization proc is not run and the module is not loaded. When I load 
the module in the default server it is loaded/executed as expected.

Code within /usr/local/ns/tcl/init.tcl is executed, but I do not want to 
touch this file, since it is part of the installation of Naviserver.

...

Best Regards,

   Georg

Re: [naviserver-devel] nsauthn: prototypish Tcl module using ns_register_auth

From: Gustaf N. (sslmail) <ne...@wu...> - 2025-06-18 08:44:07


> nsldap Issues
> I'm inclined to do some maintenance of the nsldap module, just for the record:
> 
> Deprecation warnings during build.
i have replaced the ones concerning Ns_DString* operations in the repository.

When compiling under macOS of the form:

nsldap.c:652:15: warning: 'ldap_initialize' is deprecated: first deprecated in macOS 10.11 - use OpenDirectory Framework [-Wdeprecated-declarations]

These will go away when using openLDAP under macOS. I assume, you are just referring to the prior ones.

> The pre-processor macro LDAPV3 has to be defined manually when building in order to work with current LDAP server versions.
As i see it, the macro is not necessary (i’ve replaced it with a proper version compare, with macros provided by the standard includes.
Also without the macro being defined, nsdap compiles fine and works (but without setting the LDAP to LDAPV3). Is this maybe Alpine specific?

> Ns_ModuleInit() returns extraneously when scheduling a background procedure: I worked around by setting maxidle to 0 in the pool configuration.
what means  "returns extraneously”? crash? Tcl exception? I would expect, Ns_ModuleInit() to be executed during startup (when all modules are loaded), not in a background procedure.

> The ns_ldap search documentation mentions that attributes to be extracted can be specified after the filter.
> This is not consistently documented.
> If using an empty filter *and* an attribute to extract, an assertion is triggered and nsd exits.
this requires certainly work
> Thorough documentation of all configuration options is missing.
ditto
> 
> Current LDAP client library implementations encourage the use of LDAP URI's instead of host, port and schema. This would be a useful enhancement.
i have added this to the repository. Please test if possible, i have no usable LDAP server installation.
> Controlled Creation of the auth Chain
> 
> Initially I had the idea of separating authentication and authorization and have the admin of the server decide how and in which order to run respective "filters".  The current implementation does not give exact control over the order of registration of auth callbacks.  I'll have to learn more on the details of Naviserver configuration.
> 
what do you mean by “no control”? This is exactly like for filters, you can append at the end or at the begin,

> Global Initialzation of a Tcl Module
> 
> The nsauthn module implements caching of credentials from passwd files.  For this I wanted to use a global initialization procedure, run before any server is initialized. Alas I did not find if this is even possible. Any info on this is appreciated.
> 
In general, the global (shared) Tcl libraries are loaded before the per-server (private) libraries.
You get the path of the global libraries via [ns_library shared] and the per-server ones via ns_library private] 

https://naviserver.sourceforge.io/5.0/manual/files/tcl-libraries.html

> - - -
> 
> Best Regards,
> 
>   Georg
> 
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

[naviserver-devel] nsauthn: prototypish Tcl module using ns_register_auth

From: Georg L. <jor...@ma...> - 2025-06-17 15:15:30

Hello,

My first excursion into Naviserver module development is nsauthn:

https://github.com/jorge-leon/nsauthn

The module builds on the new `ns_register_auth` function and implements 
a minimum of the ideas discussed about a month ago, namely Basic HTTP 
Authentication against passwd files and against LDAP/AD.

The repository contains a README.md file with hopefully sufficient 
detail to get started. Most likely I will set up a documented demo on 
https://naviserver.magma-soft.at in the near future.

I ran into some issues I could not overcome yet.


      Passing on Information to the request handler:

I tried to use the [ns_conn auth] ns_set from within the registered 
callback to pass on information to the rest of the request processing 
chain. This did not work out, I received: Warning: authorize script 
error: no connection

Is there some other way to get to ns_conn at that place, or to pass on 
information for later consumption?


      nsldap Issues

I'm inclined to do some maintenance of the nsldap module, just for the 
record:

  * Deprecation warnings during build.
  * The pre-processor macro LDAPV3 has to be defined manually when
    building in order to work with current LDAP server versions.
  * Ns_ModuleInit() returns extraneously when scheduling a background
    procedure: I worked around by setting maxidle to 0 in the pool
    configuration.
  * The ns_ldap search documentation mentions that attributes to be
    extracted can be specified after the filter.
      o This is not consistently documented.
      o If using an empty filter *and* an attribute to extract, an
        assertion is triggered and nsd exits.
  * Thorough documentation of all configuration options is missing.
  * Current LDAP client library implementations encourage the use of
    LDAP URI's instead of host, port and schema. This would be a useful
    enhancement.


      Controlled Creation of the auth Chain

Initially I had the idea of separating authentication and authorization 
and have the admin of the server decide how and in which order to run 
respective "filters".  The current implementation does not give exact 
control over the order of registration of auth callbacks.  I'll have to 
learn more on the details of Naviserver configuration.


      Global Initialzation of a Tcl Module

The nsauthn module implements caching of credentials from passwd files.  
For this I wanted to use a global initialization procedure, run before 
any server is initialized. Alas I did not find if this is even possible. 
Any info on this is appreciated.

- - -

Best Regards,

   Georg