naviserver-devel — Developer discussion

Re: [naviserver-devel] nswebpush & "invalid JWT provided"

[Georg L. <jor...@ma...>]

Hello,

Google changed from JWT to OAuth for the FCM HTTP v1 API

https://firebase.google.com/docs/cloud-messaging/auth-server?hl=en

Have you upgraded from legacy to v1?

Best Regards,

   Georg

On 8/9/23 11:37, Gustaf Neumann wrote:
>
> Hi David,
>
> We do not have nswebpush somewhere in production. Can you tell more 
> precisely, what "suddenly" means?
> Does this mean, that you have not changed anything in your 
> environment, but google started to refuse it?
>
> The implementation in nswebpush uses for JWT the algorithm ES256 
> (based on elliptic curves), which seems not supported by google cloud 
> endpoints, whereas [2] uses ES256, there is as well support in 
> firebase/php-jwt [3]. Not sure, where to start to look for helping you.
>
> -g
>
> [1] 
> https://cloud.google.com/endpoints/docs/frameworks/python/troubleshoot-jwt?hl=en
> [2] 
> https://cloud.google.com/iap/docs/signed-headers-howto?hl=en#securing_iap_headers
> [3] https://github.com/firebase/php-jwt/blob/main/src/JWT.php
>
> On 08.08.23 17:32, David Osborne wrote:
>> Hi there,
>>
>> We have a chat implementation based on the Naviserver nswebpush 
>> module which recently stopped working with Google endpoints (eg. 
>> https://fcm.googleapis.com/fcm/send...).
>> Suddenly it's complaining about invalid JWTs.
>>
>> We went back to reference the nswebpush code.
>> https://bitbucket.org/naviserver/nswebpush/src/main/
>> We installed it on a clean Debian Bullseye server with 
>> latest Naviserver from bitbucket.
>>
>> When we ran the "make test" we also get a 403 from Google... more 
>> specifically, the reply was:
>> Webpush failed with reply status 403 time 0:88018 headers d8 body 
>> {invalid JWT provided } https {sslversion TLSv1.3 cipher 
>> TLS_AES_256_GCM_SHA384}
>>
>> Is anyone else experiencing this or can make any suggestions as to 
>> what has changed?
>>
>> -- 
>>
>> *David
>> *
>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

[naviserver-devel] NaviServer 4.99.27 available

[Gustaf N. <ne...@wu...>]

Dear all,

I am glad to announce that the release of NaviServer 4.99.27 is 
available at SourceForge [1].  This release is a pure bug-fix and 
maintenance release, which fixes a bug annoying for some OpenACS users.
See below for a summary of the changes.

Furthermore, the online documentation at sourceforge

https://naviserver.sourceforge.io/n/toc.html

is now more mobile-friendly and contains a simple version switcher for the
stable release branch (4.99) and the main branch, which will be released 
as 5.0.

All the best!

-gustaf neumann

[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.27/

=======================================
NaviServer 4.99.27, released 2023-08-09
=======================================

  46 files changed, 739 insertions(+), 172 deletions(-)

New Features:
-------------
  - None

Bug Fixes:
----------

  - Bug fix: fixed potential crash when fallbackCharset is not defined
    in the configuration file (using outdated configuration)

  - Bug fix for fastpath in connection with "ns_serverrootproc"

    Earlier versions of NaviServer initialized for fastpath the
    interpreter with the connection conditionally, and rather late.
    When "ns_serverrootproc" is configured, the interpreter with its
    linkage to the connection is needed very early in the request.  Now
    it is ensured, that the interpreted is registered for the
    connection when the callback is issued.

    NaviServer 5 will have more changes in this respect.

    Many thanks to Georg Lehner for reporting this bug and testing.

  - Fixed various spelling errors in source code and documentation

  - Documentation:
     * fixed documentation bugs
     * modernized examples

  - Ease debugging, how values for "ns_conn location" are determined

Configuration Files:
--------------------

- Added sample section for charset mapping to sample-config.tcl

Modules:
--------
The following list contains just bug fixes, new features will
be documented with NaviServer 5.0.

  - nsdbpg:
     * Fixed potential crash in Tcl, when Database contains UTF-8
       characters invalid to Tcl 8.6. Crash was observed in
       "string tolower $var".
     * Fixed potential crash, when bind variables are passed via
       explicit "-bind" option, but variable binding was missing

  - nsoracle:
     * Reduced verbosity: For debug messages of the driver
       implementation, use again "Debug" severity to avoid too much
       verbosity, when looking for slow queries ("Debug(sql)" severity
       of nsdb).

Re: [naviserver-devel] nswebpush & "invalid JWT provided"

[Gustaf N. <ne...@wu...>]

Hi David,

We do not have nswebpush somewhere in production. Can you tell more 
precisely, what "suddenly" means?
Does this mean, that you have not changed anything in your environment, 
but google started to refuse it?

The implementation in nswebpush uses for JWT the algorithm ES256 (based 
on elliptic curves), which seems not supported by google cloud 
endpoints, whereas [2] uses ES256, there is as well support in 
firebase/php-jwt [3]. Not sure, where to start to look for helping you.

-g

[1] 
https://cloud.google.com/endpoints/docs/frameworks/python/troubleshoot-jwt?hl=en
[2] 
https://cloud.google.com/iap/docs/signed-headers-howto?hl=en#securing_iap_headers
[3] https://github.com/firebase/php-jwt/blob/main/src/JWT.php

On 08.08.23 17:32, David Osborne wrote:
> Hi there,
>
> We have a chat implementation based on the Naviserver nswebpush module 
> which recently stopped working with Google endpoints (eg. 
> https://fcm.googleapis.com/fcm/send...).
> Suddenly it's complaining about invalid JWTs.
>
> We went back to reference the nswebpush code.
> https://bitbucket.org/naviserver/nswebpush/src/main/
> We installed it on a clean Debian Bullseye server with 
> latest Naviserver from bitbucket.
>
> When we ran the "make test" we also get a 403 from Google... more 
> specifically, the reply was:
> Webpush failed with reply status 403 time 0:88018 headers d8 body 
> {invalid JWT provided } https {sslversion TLSv1.3 cipher 
> TLS_AES_256_GCM_SHA384}
>
> Is anyone else experiencing this or can make any suggestions as to 
> what has changed?
>
> -- 
>
> *David
> *

[naviserver-devel] nswebpush & "invalid JWT provided"

[David O. <da...@qc...>]

Hi there,

We have a chat implementation based on the Naviserver nswebpush module
which recently stopped working with Google endpoints (eg.
https://fcm.googleapis.com/fcm/send...).
Suddenly it's complaining about invalid JWTs.

We went back to reference the nswebpush code.
https://bitbucket.org/naviserver/nswebpush/src/main/
We installed it on a clean Debian Bullseye server with latest Naviserver
from bitbucket.

When we ran the "make test" we also get a 403 from Google... more
specifically, the reply was:
Webpush failed with reply status 403 time 0:88018 headers d8 body {invalid
JWT provided } https {sslversion TLSv1.3 cipher TLS_AES_256_GCM_SHA384}

Is anyone else experiencing this or can make any suggestions as to what has
changed?

-- 


*David*

[naviserver-devel] Forthcoming OpenACS + EuroTcl/Tk conference

[Gustaf N. <ne...@wu...>]

Dear all

At the forthcoming OpenACS + EuroTcl/Tk conference (details below),  we 
are considering a "Wishlist" session, where users can place their 
needs/demands, such we can consider this with higher priority in the 
development. This also covers NaviServer.

If you have something on your mind, please send a short email with such 
topic to me. We might discuss these with the community at the conference.

All the best

-gn


    June 30th, 2023: Registration ends
    July 19th, 2023: Meet & greet
    July 20th - July 21st, 2023:  Conference

For details, see:https://openacs.org/conf2023/

-- 
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"

Re: [naviserver-devel] NaviServer/4.99.16 vs NaviServer/4.99.25

[Sassy N. <sa...@gm...>]

You are a magician !

On Sun, Jun 18, 2023 at 1:44 PM Gustaf Neumann <ne...@wu...> wrote:

> On 17.06.23 19:08, Sassy Natan wrote:
>
> Is there any way to keep it as it was?
> So encoding convertfrom utf-8 will work like in the past?
>
> The "encoding convertfrom" has not changed, but its output is now properly
> encoded.
>
> The following will probably work for you: add the following section to your
> configuration file (and for other extensions, when you need this).
>
>    ns_section ns/encodings {
>        ns_param .adp iso8859-1
>    }
>
> I am not sure, what this will mean for Tcl9 though.
>
> all the best
>
> -g
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>


-- 
Regards,

Sassy Natan
972-(0)54-2203702

Re: [naviserver-devel] NaviServer/4.99.16 vs NaviServer/4.99.25

[Gustaf N. <ne...@wu...>]

On 17.06.23 19:08, Sassy Natan wrote:
> Is there any way to keep it as it was?
> So encoding convertfrom utf-8 will work like in the past?

The "encoding convertfrom" has not changed, but its output is now 
properly encoded.

The following will probably work for you: add the following section to your
configuration file (and for other extensions, when you need this).

    ns_section ns/encodings {
        ns_param .adp iso8859-1
    }

I am not sure, what this will mean for Tcl9 though.

all the best

-g

Re: [naviserver-devel] NaviServer/4.99.16 vs NaviServer/4.99.25

[Sassy N. <sa...@gm...>]

Thank you!

The problem is that in my code I was using the encoding convertfrom utf-8
in many places
and since I'm in the process of upgrading my system to the new version of
NaviSever I will need to maintain two versions of the code.

One for the old server and one for the new server.
It is not trivial as it sounds as the development cycle of my product is
running in parallel in both versions.


Is there any way to keep it as it was?
So encoding convertfrom utf-8 will work like in the past? even if there is
a security bridge?

Thank You



On Sat, Jun 17, 2023 at 7:27 PM Gustaf Neumann <ne...@wu...> wrote:

> Dear Sassy,
>
> Why do you say that the output right next to text utf-8 is broken?
>
> You simply do not need the extra [encoding convertfrom utf-8 ... ] in
> versions after 4.99.23 in text pages.The UTF-8 handling in earlier
> versions was broken.
>
> The changes in 4.99.23 were triggered by requests of full emoji support
> (in UTF-8 multi-byte characters, up to 4 bytes). The changes effect
> also the DB-Interface, such that now such multi-byte characters are
> fully supported in display and storing. Look e.g.:
> https://openacs.org/xowiki/text-with-hebrew
>
>  > One more thing to note is that the log file which also has UTF-8
> characters is not translated. I see the escape characters instead of the
> data.
>
> This is exactly the same answer as in you earlier question. There are
> several known attacks based on Unicode, which can as well be used to
> trick the person reading the log-file.
>
> If you do not care about the log file sanitizing and you prefer the old
> behavior, turn sanitizing off.
>
> all the best
>
> -gn
>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>


-- 
Regards,

Sassy Natan
972-(0)54-2203702

Re: [naviserver-devel] NaviServer/4.99.16 vs NaviServer/4.99.25

[Gustaf N. <ne...@wu...>]

Dear Sassy,

Why do you say that the output right next to text utf-8 is broken?

You simply do not need the extra [encoding convertfrom utf-8 ... ] in 
versions after 4.99.23 in text pages.The UTF-8 handling in earlier 
versions was broken.

The changes in 4.99.23 were triggered by requests of full emoji support 
(in UTF-8 multi-byte characters, up to 4 bytes). The changes effect  
also the DB-Interface, such that now such multi-byte characters are 
fully supported in display and storing. Look e.g.: 
https://openacs.org/xowiki/text-with-hebrew

 > One more thing to note is that the log file which also has UTF-8 
characters is not translated. I see the escape characters instead of the 
data.

This is exactly the same answer as in you earlier question. There are 
several known attacks based on Unicode, which can as well be used to 
trick the person reading the log-file.

If you do not care about the log file sanitizing and you prefer the old 
behavior, turn sanitizing off.

all the best

-gn

Re: [naviserver-devel] NaviServer 4.99.25 available

[Sassy N. <sa...@gm...>]

thank you

On Wed, Jun 14, 2023 at 1:19 PM Gustaf Neumann <ne...@wu...> wrote:

> Dear Sassy,
>
> You are referring to a change that was released more than 3 years ago
> (4.99.19).
>
> Log file sanitizing works as expected, namely it prints invisible
> characters in a hex representation (in your case tab characters).
> Sanitizing was required by security  audits, since otherwise, it would be
> possible to execute code by looking into the log file, one could obfuscate
> the log file, and confuse log file analyzer that alarm when suspect
> activities are noticed, etc.
>
> Certainly, the harm caused by a tab character alone is limited, but when
> debugging and analyze problems, it is often important to distinguish
> between a tab and some spaces.
>
> You can easily replace the hex notation in the log file via "sed", such as
> e.g.
>
>    cat log/error.log | sed -En 's/\\x09/\t/p' | more
>
> or the like.
>
> all the best
>
> -gn
> On 13.06.23 20:51, Sassy Natan wrote:
>
> Hi All,
>
> After upgrading my server to the latest version, my log file is broken.
>
> For example I see UTF-8 as special characters.
>
> I check the readme but the sanitizelogfiles 2 doesn't seems to work as
> expected.
> I have in my configuration:
>
> set        debug                                true
> ns_logctl  severity "Debug(sql)"                on
>
> Any ideas?
>
> Here is example:
> :    ],is_break_pcols)) as hrs_wo_break
> :    FROM wt_payroll_analysis_pp
> :     join (select sid,agg_array(case when is_break_pcol=true then 99999
> else 0 end) as is_break_pcols
> :    \x09\x09\x09from ( select * from wt_et_cols_defs where sid=410000
>  order by  pcol_number )pc_defs group by sid) pdefs using (sid)
> :    \x09\x09\x09\x09WHERE sid=410000 and employee_id =40599
> :    \x09\x09\x09\x09and  to_date(pp_year||'-'||pp||'-01','yyyy-mm-dd')
>  between to_date('2023-6-01','yyyy-mm-dd')- interval '6 month'
> :    \x09\x09\x09\x09\x09\x09\x09and ( to_date('2023-6-01','yyyy-mm-dd')
>  - interval '1 day' )::date
> :    \x09\x09\x09\x09UNION
> :    \x09\x09\x09\x09 select
> sid,employee_id,pp,pp_year,ppid,to_date(pp_year||'-'||pp||'-01','yyyy-mm-dd')
> as day,null2zero(hrs_wo_break)*3600
> :    \x09\x09\x09\x09                                from pp_pa_table
> :    \x09\x09\x09\x09) ggg
> :    \x09\x09\x09\x09group by  sid,employee_id,pp,pp_year
>
>
> Thanks
> Sassy
>
> On Mon, May 1, 2023 at 6:55 PM Gustaf Neumann <ne...@wu...> wrote:
>
>> Dear all,
>>
>> I am glad to announce that the release of NaviServer 4.99.25 is
>> available at SourceForge [1].  This release is mostly a bug-fix
>> release. The forthcoming version 5.0 of NaviServer will contain
>> several new features omitted in this bug-fix branch. In case, you are
>> building NaviServer from the Bitbucket repository, please note that
>> the release 4.99.25 is in the branch release/4.99 (bug fix branch for
>> the NaviServer 4.99 family). New development happens in the "main"
>> branch of the repository, leading to NaviServer 5.*.
>>
>> See below for a summary of the changes.
>>
>> Many thanks to the contributors of this release:
>>
>>      Andrew Piskorski
>>      Antonio Pisano
>>      Brian Fenton
>>      Gustaf Neumann
>>      Hector Romojaro
>>      Joe Oldak
>>      Khy Huang
>>      Oleg Oleinick
>>      Zoran Vasiljevic
>>
>> All the best!
>>
>> -gustaf neumann
>>
>> [1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.25/
>>
>> =======================================
>> NaviServer 4.99.25, released 2023-05-01
>> =======================================
>>
>>  132 files changed, 3957 insertions(+), 2068 deletions(-)
>>
>> New Features:
>> -------------
>>
>>  - Added meta-information to configuration values
>>
>>    NaviServer can now report, what configuration values provided in
>>    the configuration file were actually used, what their default
>>    values are, and whether these values were specified or not (using
>>    the default values). With this information, administration
>>    (e.g. migration) becomes easier. The NaviServer module "nsstats"
>>    shows this meta information via the web interface.
>>
>>    This functionality is provided via the new option "-filter" for the
>>    command "ns_configsection ... section". When the "-filter" option
>>    is used, different kinds of information about the parameters is
>>    returned from the specified section.
>>
>>     "-filter unread":
>>
>>         Returns the parameters, which were set during configuration (i.e.,
>>         in the configuration file) but which were not read in the startup
>>         phase of the server. This option is useful to determine
>>         e.g. typographical errors of specified parameter names.
>>
>>     "-filter defaulted":
>>
>>         Returns the parameters, from which the default values were read
>>         (i.e., which were not explicitly set)
>>
>>     "-filter defaults":
>>
>>         Returns the defaults of the parameter. This is useful for
>>         contrasting the actual values with the default values for
>>         Parameters, e.g. in a web based interface.
>>
>>  - ns_set reform (per default deactivated in 4.99, but activated in 5.*)
>>
>>    The classical implementation for ns_sets uses separately malloced
>>    storage for every attribute name and attribute value. So, e.g., for
>>    1000 ns_sets with 20 members each, this means 1,000*20*2 = 40,000
>>    malloc/free operations, e.g., for a single db query! Although the
>>    malloc implementations have improved over the years, these will
>>    require many lock operations, especially under load, where many
>>    threads might perform many concurrent malloc operations. One other
>>    consequence is that the allocated memory will be scattered over
>>    address space, which has bad implications for CPU caching.
>>
>>    The new implementation uses for one "ns_set" a single Tcl_DString
>>    keeping all attribute names and attribute values. This reduces the
>>    malloc operations and improves memory locality, such that cache
>>    hits will improve.
>>
>>    One caveat of this change is that modules using "ns_set" have to be
>>    recompiled, since the full C-level data structure of the "ns_set"
>>    is exposed. Therefore, adding a member causes a binary
>>    incompatibility. One other potential problem is that C-level
>>    modules using the Ns_Set* API have to make sure that long-living
>>    string values are copied (this was necessary before as well, but
>>    was in many cases no problem, when the "ns_sets" were seldom
>>    updated).
>>
>>    For high compatibility, this feature is deactivated per default in
>>    the 4.99.* series and can be activated by setting the compile-time
>>    C macro "NS_SET_DSTRING".
>>
>>
>>
>> API changes:
>> ------------
>>
>>    API extensions:
>>     - Provide a new interface ending with *Sz to provide string sizes.
>>       This reduces the need of strlen() operations.
>>       * Ns_SetCreateSz()
>>       * Ns_SetIUpdateSz()
>>       * Ns_SetPutSz()
>>       * Ns_SetPutValueSz()
>>       * Ns_SetUpdateSz()
>>
>>     - New API calls for "ns_set" reform
>>       * Ns_SetClearValues(): clear the values for all keys
>>       * Ns_SetDataPrealloc(): creating ns_sets with preallocated values
>>         to avoid resize operations
>>       * NsSetResize()
>>       * NsHeaderSetGet()
>>
>>     - Ns_ConfigSet(const char *section, const char *key, const char *name)
>>       The last argument is new and allows one to create named sets
>>       (previously, all such sets were unnamed)
>>
>>     - NsHexPrint(): Print the potentially binary content of a buffer
>>       in human-readable form.
>>
>>     - Ns_RelativeTime(Ns_Time *relTimePtr, Ns_Time *timePtr)
>>       This call implements the inverse operation of Ns_AbsoluteTime(),
>>       and is used mostly to make debug messages eye-friendly.
>>
>>
>> Performance Improvements:
>> -------------------------
>>
>>  - Replaced malloc operation per log entry by thread local variable in
>>    system log implementation.
>>
>>  - When NaviServer 4.99.25 is compiled with NS_SET_DSTRING supportm
>>    the following preliminary performance results were measured from
>>    the "ns_set" reform (see above).  The tests were performed on
>>    openacs.org (Xeon Gold 6226R CPU @ 2.90GHz, 32 cores,
>>    hyper-threading enabled). The test executes the SQL query
>>
>>         select * from acs_objects limit 1000
>>
>>    100 times in sequence. This test is run in 1 to 30 concurrent
>>    threads. With 30 threads, 3mio tuples are retrieved, and 72 mio
>>    malloc/free operations are needed alone for the retrieved values.
>>
>>    Before (classical ns_set with many mallocs):
>>
>>         threads 1 total 4606.787 ms avg 3285.25 ms
>>         threads 5 total 4595.358 ms avg 3493.07 ms
>>         threads 10 total 4804.193 ms avg 3755.93 ms
>>         threads 20 total 6279.524 ms avg 4569.16 ms
>>         threads 30 total 8966.427 ms avg 6618.58 ms
>>
>>    After reform (using common Tcl_DString per tuple):
>>
>>         threads 1 total 4524.645 ms avg 3242.54 ms
>>         threads 5 total 4251.266 ms avg 3450.09 ms
>>         threads 10 total 4656.795 ms avg 3665.31 ms
>>         threads 20 total 5934.105 ms avg 4671.38 ms
>>         threads 30 total 7384.591 ms avg 5642.76 ms
>>
>>    To summarize, the improvement increases under higher load (with
>>    more parallel threads). E.g., with 30 threads, the total time
>>    improved by 17%.... leading also to a smaller RSS. These tests were
>>    not performed under "clinical" conditions.
>>
>>    The new Tcl API call "ns_set size" can be used to pre-allocate
>>    larger ns_sets, such that the usual Tcl_DString growing policy does
>>    not kick in, reducing further realloc() operations.
>>
>>
>> Bug Fixes:
>> ----------
>>
>>  - Fixed potential crash in "ns_accesslog extendedheaders XXXX".
>>    Setting extended headers via configuration file was correct, but
>>    changing it dynamically via "ns_accesslog extendedheaders .." was
>>    broken. (Issue https://sourceforge.net/p/naviserver/bugs/91/)
>>
>>  - "ns_conn location": Fixed potential race condition
>>
>>     It was possible that "ns_conn location" could return inconsistent
>>     results in a single request, when the underlying sockPtr was
>>     aggressively reused. Now, the value of the location member is
>>     copied to the connection structure instead of being shared with
>>     the socket structure (as before).
>>
>>  - "ns_cache_eval -force": Fixed potential race condition
>>
>>     There was a problem with "ns_cache_eval -force", where the system
>>     relied on the existence of a pre-existing entry, but in case the
>>     entry was flushed in the meantime problems a crash might have
>>     happened. Now the value during the "-force" call is cleared exactly
>>     like in the case of an unset operation. The null-value operations
>>     are already protected until these are finished in various places
>>     in the code.
>>
>>  - "ns_socknread": Fix potentially wrong result for buffered channels
>>
>>  - Bug fixes for problems showing up under MS Windows:
>>
>>     * Make sure that the output variable of Ns_ObjvIndex() is always an
>>       integer. Previously, the output variable was in two places a
>>       character variable, causing crashes under MS Windows.
>>
>>     * Handle incompatibility in setlocale() under MS Windows.  Under
>>       MS Windows, later calls to setlocale() overwrite the string
>>       returned by former calls. So, it is necessary to copy of the
>>       returned string of a setlocale() call under MS Windows.
>>
>>     * Handling linking problems: MS Windows requires explicit handling
>>       when importing symbols from .dll files (Ns_LogSqlDebug,
>>       NS_intTypePtr)
>>
>>     * Pass error codes from low-level function SockRecv() and
>>       SockSend() via variables.  This change makes sure the real error
>>       code (immediately after the I/O operation) is passed to the
>>       caller to avoid missed error cases and weird error message
>>       (e.g. under windows).
>>
>>
>>  - Bug fixes for ADP parser:
>>
>>     * Support for greater than sign ">" inside attribute values.
>>
>>       Previously, NaviServer determined the terminating end-of-tag
>>       character as literally the first greater than sign, no matter if
>>       this was used as attribute values between single or double
>>       quotes.
>>
>>       The new version supports such values, since the "Living Standard
>>       of HTML" [1] allows the use of less than "<" and greater than
>>       ">" signs inside attribute values as long these are between
>>       single or double quotes.  The guide [2] just recommends using
>>       character escapes for "<", ">" and "&".
>>
>>       [1] https://html.spec.whatwg.org/multipage/syntax.html#syntax-attribute-value
>>       [2] https://www.w3.org/International/questions/qa-escapes
>>
>>       This problem was reported by Wolfgang Winkler
>>
>>     * Clear ADP flags in case of errors (this error was present for
>>       many years)
>>
>>       Previously, The following ADP page could lead to a full
>>       breakdown of the server, since the error states of the server
>>       were never cleared, and subsequent requests served be the same
>>       interpreter could lead to the old error states.
>>
>>       Many thanks to Oleg Oleinick for reporting and the great test
>>       cases.
>>
>>  - Provide better HTTP status code for early errors: When the driver
>>    terminates already a request, it might be the case that the error
>>    flags were not read out. In such cases, we provide now more
>>    specific status codes rather than 400.
>>
>>  - ns_http:
>>
>>    * Improved robustness with domain names resolving against many IP
>>      addresses: When a domain name is resolved against many IP
>>      addresses, and all these IP addresses block (connect returns "in
>>      progress") then the old code might have looped infinitely. Now
>>      the code respects the provided timeout (default set to 5s) per
>>      resolved IP address.
>>
>>      Many thanks to Joe Oldak for pointing out the problem and
>>      leading us to the solution.
>>
>>    * Fixed timeout handling during TLS handshake: This problem could
>>      appear, when the TCP connection to a server succeeded quickly,
>>      but the TLS handshake was taking a long time, without a raising
>>      a timeout exception, although the timeout time has expired.
>>
>>  - Set the default server before the init-scripts is called to make it
>>    accessible from there
>>
>>  - Provide compatibility between in-memory and file-based form-data
>>    handling for invalid characters: Since the file based
>>    implementation uses a fallback-charset of iso8859-1 when parsing
>>    form data in POST requests (in order to be able to extract
>>    "_charset_" data), the in-memory based variant does now the
>>    same. The decision of which parser is used is taken based on the
>>    size of the form data.... and should therefore be consistent.
>>
>>  - nscp: Fixed problem, when the nscp module is activated but no
>>    "users" section is specified. Many thanks to Andrew Piskorski for
>>    reporting the problem.
>>
>>  - Security Improvement: Added protection against sneaking in fake
>>    NAME.tmpfile entries into form fields, when performing file upload
>>    operations.
>>
>>
>> Documentation improvements:
>> ---------------------------
>>
>>  - Improved the following man pages:
>>
>>        doc/src/manual/adp-overview.man
>>        doc/src/manual/tcl-lib-nstrace.man
>>        doc/src/naviserver/commandlist.man
>>        doc/src/naviserver/ns_adp.man
>>        doc/src/naviserver/ns_adp_ctl.man
>>        doc/src/naviserver/ns_config.man
>>        doc/src/naviserver/ns_conn.man
>>        doc/src/naviserver/ns_connchan.man
>>        doc/src/naviserver/ns_cookie.man
>>        doc/src/naviserver/ns_crypto.man
>>        doc/src/naviserver/ns_http.man
>>        doc/src/naviserver/ns_ictl.man
>>        doc/src/naviserver/ns_locationproc.man
>>        doc/src/naviserver/ns_parsehostport.man
>>        doc/src/naviserver/ns_return.man
>>        doc/src/naviserver/ns_set.man
>>        doc/src/naviserver/ns_setprivileges.man
>>
>>
>> Configuration Files:
>> --------------------
>>
>>  - New feature for the sample configuration file openacs-config.tcl
>>
>>    The following variables (with prefix oacs_) can be taken from the
>>    shell variables. This makes it easier to run similar variants of
>>    NaviServer instances from a single configuration file, while
>>    providing variables from the command line. This is e.g. useful for
>>    Docker or cluster setups.
>>
>>        CookieNamespace
>>        bindir
>>        cachingmode
>>        db_host
>>        db_name
>>        db_port
>>        db_user
>>        homedir
>>        hostname
>>        httpport
>>        httpsport
>>        ipaddress
>>        logroot
>>        nscpport
>>        server
>>        serverroot
>>        smtpdport
>>
>>    One can use e.g.the following command to change some ports and the
>>    log file during startup
>>
>>     oacs_httpport=8000 oacs_httpsport=8443 oacs_smtpdport=2526 \
>>         oacs_logroot=/var/www/XXXX/log-node1/ \
>>         /usr/local/ns/bin/nsd -t /var/www/XXXX/etc/openacs-config.tcl ...
>>
>>    - further updates for openacs-config.tcl:
>>       * Added sample nssmtpd configuration
>>       * Added placeholder for ClusterSecret
>>       * Reflect recent Oracle requirements (tested with Oracle 19c)
>>       * Added documentation for "StaticCSP", "CookieNamespace",
>>         "NsShutdownWithNonZeroExitCode", "LogIncludeUserId"
>>
>>    - Updated all sample configuration files
>>
>>
>> Code Changes:
>> -------------
>>
>>  - Added and updated predefined MIME types based on
>>     https://www.iana.org/assignments/media-types/media-types.xhtml
>>
>>  - Added compile time macro NS_VERBOSE_MALLOC to identify frequent
>>    *alloc operations
>>
>>  - Print version of zlib during startup
>>
>>  - OpenSSL maintenance:
>>
>>     * Improved robustness for OCSP with OpenSSL 3.*
>>
>>       The previous version crashed with OpenSSL 3.*, when OCSP was
>>       tried on self-signed certificates. Aside of the fact that OCSP
>>       does not make sense for self-signed certificates, the server
>>       should not crash in such situations.
>>
>>     * Fixed ns_crypto::aead::encrypt/decrypt test under OpenSSL 1.1.1
>>       (as shipped per default, e.g. on Ubuntu 18.04.4)
>>
>>       In short, the problem was that with this version of OpenSSL,
>>       setting empty additional authenticated data (AAD) behaved
>>       differently from other versions, namely it was clearing
>>       incorrectly (forgetting) the information that the initialization
>>       vector (IV) was already set. An upgrade of OpenSSL fixed the
>>       problem. However, with these changes, also the stock version of
>>       OpenSSL can be used. As a byproduct, better error messages are
>>       now produced, the code received more cleanup (e.g. explicit
>>       initialization, etc.)
>>
>>
>>  - Automated testing:
>>
>>     * Setup if Bitbucket + GitHub pipelines for automated regression
>>       testing with multiple versions of components
>>
>>       For NaviServer 4.99 the current setup performs tests with gcc-10
>>       + gcc-11, Tcl 8.6.13 + 8.7a5, NSF 2.4,0 + 2.4.0, tDOM 0.9.1 +
>>       0.9.3, extra modules: nsdbpg nsdbi nsdbipg nsudp nscoap nssmtpd.
>>
>>       https://github.com/nm-wu/naviserver-mirror/actions
>>
>>     * Improved robustness of regression test when running with the
>>       docker networking setup
>>
>>     * Force nonzero exit code when regression test fails
>>
>>     * Added parameter "-timeout" to call of regression test cases
>>       Previously, the timeout was hard-wired to 3 seconds. One can now
>>       call a test with e.g. "nstest::http -timeout 1s ... GET ..."
>>
>>     * adp_compress.test: removed trailing newline to ease
>>       cross-platform regression tests
>>
>>     * Prefer standard Tcl test constraint "macOrUnix" over own solution
>>
>>     * Extended regression tests with more test cases
>>
>>  - Code management:
>>     * Changed name of branch from "master" to "main"
>>
>>  - Code Cleanup
>>      * Improved type cleanness
>>      * Removed deprecated calls to "sprintf"
>>      * Improved portability for Tcl 8.7* (handling of binary data)
>>
>>  - Improved comments, fixed typos
>>
>>
>> Changes in NaviServer Modules:
>> ==============================
>>
>>  39 files changed, 9658 insertions(+), 1781 deletions(-)
>>
>> General:
>>   Adjust to necessary API changes in NaviServer (contains as well
>>   support for the forthcoming release of NaviServer 5)
>>
>>
>> nsdbpg:
>> -------
>>
>>   - new pg-driver specific command: ns_pg_prepare /sql/
>>
>>     Return a dict building a prepared statement for the passed-in SQL
>>     statement. The dict contains the keys "sql" and "args". The
>>     function is used by e.g. OpenACS to generate prepared statements
>>     from SQL commands with bind variables.
>>
>>   - Raise exception when a value for a bind variable contains a NUL character.
>>     This value is explicitly forbidden in text strings passed to PostgreSQL.
>>
>>   - Let "ns_pg" report available subcommands even when handle is not
>>     specified.  This makes the command compatible with the "icanuse"
>>     feature in OpenACS.
>>
>>
>> nsstats:
>> --------
>>
>>   - HTTP client log analysis:
>>      * Provide charts for performance (using highcharts via CDN)
>>      * Provide charts on request frequency (using highcharts via CDN)
>>      * Provide a summary table for HTTP client requests
>>      * Improved robustness against invalid URLs (containing unescaped
>>        spaces)
>>      * Added support for selection of different HTTP client log files
>>        via web interface
>>
>>   - "Process" page:
>>       * Added percentage of request distribution over connection pools
>>       * Added information about the connected client
>>       * Added more detailed version information
>>
>>   - Added cache configuration to output when looking at a single cache
>>
>>   - Improved "log file" analysis
>>     * Automated stripping of color codes
>>
>>     * Added filter option. The filter can be used to grep for (ID)
>>       strings in both the system and access logs, providing a summary
>>       for the traces of a request in the access log and system log on
>>       a single place.
>>
>>   - Added default and usage information to "Config Parameters" page
>>
>>
>> nsoracle:
>> ---------
>>
>>   - Fixed bug when streaming LOB content to connection. The old code
>>     did not distinguish between binary and non-binary content.  This
>>     bug was discussed in
>>     https://openacs.org/forums/message-view?message_id=5693661
>>
>>     Bumped version number to 2.9
>>
>> - switched to plain Debug handling for debugging the driver
>>
>>     The handling of Ns_LogSqlDebug is performed inside nsdb, including
>>     also the printout of (long) SQL statements. Previously, the driver
>>     was too chatty when Debug(sql) was turned on.
>>
>>   - Added support for output columns of type SQLT_TIMESTAMP or SQLT_TIMESTAMP_TZ
>>
>>     This change fixes a bug, where SQL queries of the form
>>
>>          SELECT TO_TIMESTAMP(sysdate) FROM dual
>>
>>     lead to errors for the form:
>>
>>          Database operation "getrow" failed (exception 1406, "nsoracle.c:3659:Ns_OracleGetRow:
>>          error in `OCIStmtFetch ()': ORA-01406: fetched column value was truncated
>>
>>     The driver needs for several output types special rules, where the
>>     timestamp cases were not supported so far. It is also recommended
>>     to set the according environment variables specifying the output
>>     format in the configuration server of NaviServer, such as e.g.
>>
>>         set ::env(NLS_TIMESTAMP_FORMAT) "YYYY-MM-DD HH24:MI:SS.FF6"
>>         set ::env(NLS_TIMESTAMP_TZ_FORMAT) "YYYY-MM-DD HH24:MI:SS.FF6 TZH:TZM"
>>
>>     For testing in you local Oracle installation, you might test the
>>     output formats (and the required sizes with the following snippet
>>     for sqlplus:
>>
>>        COLUMN localtimestamp format a40
>>        COLUMN systimestamp format a40
>>        COLUMN ts_bytes format a80
>>
>>        alter session set nls_timestamp_format='YYYY-MM-DD HH24:MI:SS.FF6';
>>        select localtimestamp, length(localtimestamp), dump(localtimestamp) ts_bytes from dual;
>>
>>        alter session set nls_timestamp_tz_format='YYYY-MM-DD HH24:MI:SS.FF6 TZH:TZM';
>>        select systimestamp, length(systimestamp), dump(systimestamp) ts_bytes from dual;
>>
>>        alter session set nls_timestamp_tz_format='YYYY-MM-DD HH24:MI:SS.FF6 TZR';
>>        select systimestamp, length(systimestamp), dump(systimestamp) ts_bytes from dual;
>>
>>
>> letsencrypt:
>> ------------
>>
>>   - Added option to produce certificates with ECDSA:
>>
>>     Prior to this change, all certificates were using RSA keys.
>>     Since a while, keys based on elliptic curves became the preference
>>     of letsencrypt.
>>
>>
>> nsshell:
>> --------
>>
>>    - Fixed a bug in "ns_conn" emulation, when the "kernel" was not correctly identified
>>
>>
>> _______________________________________________
>> naviserver-devel mailing list
>> nav...@li...
>> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>>
>
>
> --
> Regards,
>
> Sassy Natan
> 972-(0)54-2203702
>
>
> _______________________________________________
> naviserver-devel mailing lis...@li...https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>
> --
> Univ.Prof. Dr. Gustaf Neumann
> Head of the Institute of Information Systems and New Media
> of Vienna University of Economics and Business
> Program Director of MSc "Information Systems"
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>


-- 
Regards,

Sassy Natan
972-(0)54-2203702

[naviserver-devel] NaviServer/4.99.16 vs NaviServer/4.99.25

[Sassy N. <sa...@gm...>]

Hi Group,

I'm facing an issue on my new Ubuntu 22.04 installation with NaviServer/
4.99.25.

First here is the code of my adp page:

<html>
<meta charset="utf-8">
<body>
encoding system:<%=[encoding system]%><br>
text utf-8 שליחת מייל <br>
function encoding convertfrom utf-8 : <%=[encoding convertfrom utf-8 "\שלח
למייל"]%> <br>
</body>
</html>

In NaviServer/4.99.16 everything is working as expected.
However in NaviServer/4.99.25 the encoding convertfrom utf-8 is somehow
broken.


[image: image.png]

I have check the locate setting in both servers and they are the same:

LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

The naviserver in both server print this on the load:




*[-main:conf-] Notice: initialized locale en_US.UTF-8 from environment
variable LANG[16/Jun/2023:18:51:58][854016.7f8dc402af80][-main:conf-]
Warning: config mimtypes: redefine predefined mime type for .adp value
'text/html' with different value: text/html;
charset=UTF-8[16/Jun/2023:18:51:58][854016.7f8dc402af80][-main:conf-]
Debug: encoding: loaded: utf-8*


I also run curl  to check the headers:

HTTP/1.1 200 OK
*Server: NaviServer/4.99.25*
Date: Fri, 16 Jun 2023 18:53:45 GMT
Set-Cookie: ad_browser_id=414419283; Path=/; Expires=Fri, 01-Jan-2010
01:00:00 GMT
Set-Cookie:
ad_session_id=414419284,0,c4rYRb4RyIH2tMMORokgBz44T0g9U6XF,1686941625;
Path=/; Max-Age=14400
Content-Type: text/html; charset=UTF-8
Expires: now
Content-Length: 0
Connection: keep-alive

HTTP/1.1 200 OK
*Server: NaviServer/4.99.16*
Date: Fri, 16 Jun 2023 18:47:05 GMT
Set-Cookie: ad_browser_id=622810778; Path=/; Expires=Fri, 01-Jan-2010
01:00:00 GMT
Set-Cookie:
ad_session_id=622810779,0,P9pbnqGCncOosO.v5xKMTLosjbGdyLQs,1686941225;
Path=/; Max-Age=14400
Content-Type: text/html; charset=UTF-8
Expires: now
Content-Length: 0
Connection: keep-alive


Running the command in a local tclsh gives the same output so I don't think
it is a tcl8.6 issue.


tclsh
% encoding convertfrom utf-8 "שלח למייל"
éÜ× ÜÞÙÙÜ
%


The *Server: NaviServer/4.99.25 is running * tcl 8.6.12+dfsg vs *Server:
NaviServer/4.99.16 tcl * 8.6.8+dfsg-3

One more thing to note is that the log file which also has UTF-8 characters
is not translated. I see the escape characters instead of the data.

Any ideas? What am I missing?
Both servers run the same naviserver configuration.

I have build the naviserver as a debing pkgs - I can share the code if
needed:

[image: image.png]

-- 
Regards,

Sassy Natan
972-(0)54-2203702

[naviserver-devel] NaviServer 4.99.26 available

[Gustaf N. <ne...@wu...>]

Dear all,

I am glad to announce that the release of NaviServer 4.99.26 is 
available at SourceForge [1].  This release is a pure bug-fix and 
maintenance release, which fixes a bug annoying for some OpenACS users 
(wrong value when hostname is retrieved from the configuration values 
via "ns_config ...").
See below for a summary of the changes.

All the best!

-gustaf neumann

[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.26/

=======================================
NaviServer 4.99.26, released 2023-06-16
=======================================

  18 files changed, 109 insertions(+), 64 deletions(-)

New Features:
-------------
  - None

Bug Fixes:
----------

  - Bug fix: don't update the value of an ns_set entry using the
    identical string.

    Since the update of an ns_set entry requires a free operation of
    the old memory and a copy operation of the new value, in case the
    pointers are identical, the new value might be freed in the first
    step, leading to an invalid value. This condition can happen only
    from the C level API.  Unfortunately, this happened with the sample
    configuration file for OpenACS.

  - Increase portability for older compilers and windows.
    NaviServer 4.99.* uses now same idiom for NS_BITELEMENTS
    as in NaviServer 5

  - improved error message on socket errors (reported unknown error
    before)

Configuration Files:
--------------------
  - Modernize recommended cipher lists in OpenSSL configuration
    files

Re: [naviserver-devel] NaviServer 4.99.25 available

[Gustaf N. <ne...@wu...>]

Dear Sassy,

You are referring to a change that was released more than 3 years ago 
(4.99.19).

Log file sanitizing works as expected, namely it prints invisible 
characters in a hex representation (in your case tab characters). 
Sanitizing was required by security  audits, since otherwise, it would 
be possible to execute code by looking into the log file, one could 
obfuscate the log file, and confuse log file analyzer that alarm when 
suspect activities are noticed, etc.

Certainly, the harm caused by a tab character alone is limited, but when 
debugging and analyze problems, it is often important to distinguish 
between a tab and some spaces.

You can easily replace the hex notation in the log file via "sed", such 
as e.g.

    cat log/error.log | sed -En 's/\\x09/\t/p' | more

or the like.

all the best

-gn

On 13.06.23 20:51, Sassy Natan wrote:
> Hi All,
>
> After upgrading my server to the latest version, my log file is broken.
>
> For example I see UTF-8 as special characters.
>
> I check the readme but the sanitizelogfiles 2 doesn't seems to work as 
> expected.
> I have in my configuration:
>
> set        debug                                true
> ns_logctl  severity "Debug(sql)"                on
>
> Any ideas?
>
> Here is example:
> :    ],is_break_pcols)) as hrs_wo_break
> :    FROM wt_payroll_analysis_pp
> :     join (select sid,agg_array(case when is_break_pcol=true then 
> 99999 else 0 end) as is_break_pcols
> :    \x09\x09\x09from ( select * from wt_et_cols_defs where sid=410000 
>  order by  pcol_number )pc_defs group by sid) pdefs using (sid)
> :    \x09\x09\x09\x09WHERE sid=410000 and employee_id =40599
> :    \x09\x09\x09\x09and 
>  to_date(pp_year||'-'||pp||'-01','yyyy-mm-dd')  between 
> to_date('2023-6-01','yyyy-mm-dd')- interval '6 month'
> :    \x09\x09\x09\x09\x09\x09\x09and ( 
> to_date('2023-6-01','yyyy-mm-dd')  - interval '1 day' )::date
> :    \x09\x09\x09\x09UNION
> :    \x09\x09\x09\x09 select 
> sid,employee_id,pp,pp_year,ppid,to_date(pp_year||'-'||pp||'-01','yyyy-mm-dd') 
> as day,null2zero(hrs_wo_break)*3600
> :    \x09\x09\x09\x09                                from pp_pa_table
> :    \x09\x09\x09\x09) ggg
> :    \x09\x09\x09\x09group by  sid,employee_id,pp,pp_year
>
>
> Thanks
> Sassy
>
> On Mon, May 1, 2023 at 6:55 PM Gustaf Neumann <ne...@wu...> wrote:
>
>     Dear all,
>
>     I am glad to announce that the release of NaviServer 4.99.25 is
>     available at SourceForge [1].  This release is mostly a bug-fix
>     release. The forthcoming version 5.0 of NaviServer will contain
>     several new features omitted in this bug-fix branch. In case, you are
>     building NaviServer from the Bitbucket repository, please note that
>     the release 4.99.25 is in the branch release/4.99 (bug fix branch for
>     the NaviServer 4.99 family). New development happens in the "main"
>     branch of the repository, leading to NaviServer 5.*.
>
>     See below for a summary of the changes.
>
>     Many thanks to the contributors of this release:
>
>          Andrew Piskorski
>          Antonio Pisano
>          Brian Fenton
>          Gustaf Neumann
>          Hector Romojaro
>          Joe Oldak
>          Khy Huang
>          Oleg Oleinick
>          Zoran Vasiljevic
>
>     All the best!
>
>     -gustaf neumann
>
>     [1]
>     https://sourceforge.net/projects/naviserver/files/naviserver/4.99.25/
>
>     =======================================
>     NaviServer 4.99.25, released 2023-05-01
>     =======================================
>
>       132 files changed, 3957 insertions(+), 2068 deletions(-)
>
>     New Features:
>     -------------
>
>       - Added meta-information to configuration values
>
>         NaviServer can now report, what configuration values provided in
>         the configuration file were actually used, what their default
>         values are, and whether these values were specified or not (using
>         the default values). With this information, administration
>         (e.g. migration) becomes easier. The NaviServer module "nsstats"
>         shows this meta information via the web interface.
>
>         This functionality is provided via the new option "-filter" for the
>         command "ns_configsection ... section". When the "-filter" option
>         is used, different kinds of information about the parameters is
>         returned from the specified section.
>
>          "-filter unread":
>
>              Returns the parameters, which were set during configuration (i.e.,
>              in the configuration file) but which were not read in the startup
>              phase of the server. This option is useful to determine
>              e.g. typographical errors of specified parameter names.
>
>          "-filter defaulted":
>
>              Returns the parameters, from which the default values were read
>              (i.e., which were not explicitly set)
>
>          "-filter defaults":
>
>              Returns the defaults of the parameter. This is useful for
>              contrasting the actual values with the default values for
>              Parameters, e.g. in a web based interface.
>
>       - ns_set reform (per default deactivated in 4.99, but activated in 5.*)
>
>         The classical implementation for ns_sets uses separately malloced
>         storage for every attribute name and attribute value. So, e.g., for
>         1000 ns_sets with 20 members each, this means 1,000*20*2 = 40,000
>         malloc/free operations, e.g., for a single db query! Although the
>         malloc implementations have improved over the years, these will
>         require many lock operations, especially under load, where many
>         threads might perform many concurrent malloc operations. One other
>         consequence is that the allocated memory will be scattered over
>         address space, which has bad implications for CPU caching.
>
>         The new implementation uses for one "ns_set" a single Tcl_DString
>         keeping all attribute names and attribute values. This reduces the
>         malloc operations and improves memory locality, such that cache
>         hits will improve.
>
>         One caveat of this change is that modules using "ns_set" have to be
>         recompiled, since the full C-level data structure of the "ns_set"
>         is exposed. Therefore, adding a member causes a binary
>         incompatibility. One other potential problem is that C-level
>         modules using the Ns_Set* API have to make sure that long-living
>         string values are copied (this was necessary before as well, but
>         was in many cases no problem, when the "ns_sets" were seldom
>         updated).
>
>         For high compatibility, this feature is deactivated per default in
>         the 4.99.* series and can be activated by setting the compile-time
>         C macro "NS_SET_DSTRING".
>
>
>
>     API changes:
>     ------------
>
>         API extensions:
>          - Provide a new interface ending with *Sz to provide string sizes.
>            This reduces the need of strlen() operations.
>            * Ns_SetCreateSz()
>            * Ns_SetIUpdateSz()
>            * Ns_SetPutSz()
>            * Ns_SetPutValueSz()
>            * Ns_SetUpdateSz()
>
>          - New API calls for "ns_set" reform
>            * Ns_SetClearValues(): clear the values for all keys
>            * Ns_SetDataPrealloc(): creating ns_sets with preallocated values
>              to avoid resize operations
>            * NsSetResize()
>            * NsHeaderSetGet()
>
>          - Ns_ConfigSet(const char *section, const char *key, const char *name)
>            The last argument is new and allows one to create named sets
>            (previously, all such sets were unnamed)
>
>          - NsHexPrint(): Print the potentially binary content of a buffer
>            in human-readable form.
>
>          - Ns_RelativeTime(Ns_Time *relTimePtr, Ns_Time *timePtr)
>            This call implements the inverse operation of Ns_AbsoluteTime(),
>            and is used mostly to make debug messages eye-friendly.
>
>
>     Performance Improvements:
>     -------------------------
>
>       - Replaced malloc operation per log entry by thread local variable in
>         system log implementation.
>
>       - When NaviServer 4.99.25 is compiled with NS_SET_DSTRING supportm
>         the following preliminary performance results were measured from
>         the "ns_set" reform (see above).  The tests were performed on
>         openacs.org  <http://openacs.org>  (Xeon Gold 6226R CPU @ 2.90GHz, 32 cores,
>         hyper-threading enabled). The test executes the SQL query
>
>              select * from acs_objects limit 1000
>
>         100 times in sequence. This test is run in 1 to 30 concurrent
>         threads. With 30 threads, 3mio tuples are retrieved, and 72 mio
>         malloc/free operations are needed alone for the retrieved values.
>
>         Before (classical ns_set with many mallocs):
>
>              threads 1 total 4606.787 ms avg 3285.25 ms
>              threads 5 total 4595.358 ms avg 3493.07 ms
>              threads 10 total 4804.193 ms avg 3755.93 ms
>              threads 20 total 6279.524 ms avg 4569.16 ms
>              threads 30 total 8966.427 ms avg 6618.58 ms
>
>         After reform (using common Tcl_DString per tuple):
>
>              threads 1 total 4524.645 ms avg 3242.54 ms
>              threads 5 total 4251.266 ms avg 3450.09 ms
>              threads 10 total 4656.795 ms avg 3665.31 ms
>              threads 20 total 5934.105 ms avg 4671.38 ms
>              threads 30 total 7384.591 ms avg 5642.76 ms
>
>         To summarize, the improvement increases under higher load (with
>         more parallel threads). E.g., with 30 threads, the total time
>         improved by 17%.... leading also to a smaller RSS. These tests were
>         not performed under "clinical" conditions.
>
>         The new Tcl API call "ns_set size" can be used to pre-allocate
>         larger ns_sets, such that the usual Tcl_DString growing policy does
>         not kick in, reducing further realloc() operations.
>
>
>     Bug Fixes:
>     ----------
>
>       - Fixed potential crash in "ns_accesslog extendedheaders XXXX".
>         Setting extended headers via configuration file was correct, but
>         changing it dynamically via "ns_accesslog extendedheaders .." was
>         broken. (Issuehttps://sourceforge.net/p/naviserver/bugs/91/)
>
>       - "ns_conn location": Fixed potential race condition
>
>          It was possible that "ns_conn location" could return inconsistent
>          results in a single request, when the underlying sockPtr was
>          aggressively reused. Now, the value of the location member is
>          copied to the connection structure instead of being shared with
>          the socket structure (as before).
>
>       - "ns_cache_eval -force": Fixed potential race condition
>
>          There was a problem with "ns_cache_eval -force", where the system
>          relied on the existence of a pre-existing entry, but in case the
>          entry was flushed in the meantime problems a crash might have
>          happened. Now the value during the "-force" call is cleared exactly
>          like in the case of an unset operation. The null-value operations
>          are already protected until these are finished in various places
>          in the code.
>
>       - "ns_socknread": Fix potentially wrong result for buffered channels
>
>       - Bug fixes for problems showing up under MS Windows:
>
>          * Make sure that the output variable of Ns_ObjvIndex() is always an
>            integer. Previously, the output variable was in two places a
>            character variable, causing crashes under MS Windows.
>
>          * Handle incompatibility in setlocale() under MS Windows.  Under
>            MS Windows, later calls to setlocale() overwrite the string
>            returned by former calls. So, it is necessary to copy of the
>            returned string of a setlocale() call under MS Windows.
>
>          * Handling linking problems: MS Windows requires explicit handling
>            when importing symbols from .dll files (Ns_LogSqlDebug,
>            NS_intTypePtr)
>
>          * Pass error codes from low-level function SockRecv() and
>            SockSend() via variables.  This change makes sure the real error
>            code (immediately after the I/O operation) is passed to the
>            caller to avoid missed error cases and weird error message
>            (e.g. under windows).
>
>
>       - Bug fixes for ADP parser:
>
>          * Support for greater than sign ">" inside attribute values.
>
>            Previously, NaviServer determined the terminating end-of-tag
>            character as literally the first greater than sign, no matter if
>            this was used as attribute values between single or double
>            quotes.
>
>            The new version supports such values, since the "Living Standard
>            of HTML" [1] allows the use of less than "<" and greater than
>            ">" signs inside attribute values as long these are between
>            single or double quotes.  The guide [2] just recommends using
>            character escapes for "<", ">" and "&".
>
>            [1]https://html.spec.whatwg.org/multipage/syntax.html#syntax-attribute-value
>            [2]https://www.w3.org/International/questions/qa-escapes
>
>            This problem was reported by Wolfgang Winkler
>
>          * Clear ADP flags in case of errors (this error was present for
>            many years)
>
>            Previously, The following ADP page could lead to a full
>            breakdown of the server, since the error states of the server
>            were never cleared, and subsequent requests served be the same
>            interpreter could lead to the old error states.
>
>            Many thanks to Oleg Oleinick for reporting and the great test
>            cases.
>
>       - Provide better HTTP status code for early errors: When the driver
>         terminates already a request, it might be the case that the error
>         flags were not read out. In such cases, we provide now more
>         specific status codes rather than 400.
>
>       - ns_http:
>
>         * Improved robustness with domain names resolving against many IP
>           addresses: When a domain name is resolved against many IP
>           addresses, and all these IP addresses block (connect returns "in
>           progress") then the old code might have looped infinitely. Now
>           the code respects the provided timeout (default set to 5s) per
>           resolved IP address.
>
>           Many thanks to Joe Oldak for pointing out the problem and
>           leading us to the solution.
>
>         * Fixed timeout handling during TLS handshake: This problem could
>           appear, when the TCP connection to a server succeeded quickly,
>           but the TLS handshake was taking a long time, without a raising
>           a timeout exception, although the timeout time has expired.
>
>       - Set the default server before the init-scripts is called to make it
>         accessible from there
>
>       - Provide compatibility between in-memory and file-based form-data
>         handling for invalid characters: Since the file based
>         implementation uses a fallback-charset of iso8859-1 when parsing
>         form data in POST requests (in order to be able to extract
>         "_charset_" data), the in-memory based variant does now the
>         same. The decision of which parser is used is taken based on the
>         size of the form data.... and should therefore be consistent.
>
>       - nscp: Fixed problem, when the nscp module is activated but no
>         "users" section is specified. Many thanks to Andrew Piskorski for
>         reporting the problem.
>
>       - Security Improvement: Added protection against sneaking in fake
>         NAME.tmpfile entries into form fields, when performing file upload
>         operations.
>
>
>     Documentation improvements:
>     ---------------------------
>
>       - Improved the following man pages:
>
>             doc/src/manual/adp-overview.man
>             doc/src/manual/tcl-lib-nstrace.man
>             doc/src/naviserver/commandlist.man
>             doc/src/naviserver/ns_adp.man
>             doc/src/naviserver/ns_adp_ctl.man
>             doc/src/naviserver/ns_config.man
>             doc/src/naviserver/ns_conn.man
>             doc/src/naviserver/ns_connchan.man
>             doc/src/naviserver/ns_cookie.man
>             doc/src/naviserver/ns_crypto.man
>             doc/src/naviserver/ns_http.man
>             doc/src/naviserver/ns_ictl.man
>             doc/src/naviserver/ns_locationproc.man
>             doc/src/naviserver/ns_parsehostport.man
>             doc/src/naviserver/ns_return.man
>             doc/src/naviserver/ns_set.man
>             doc/src/naviserver/ns_setprivileges.man
>
>
>     Configuration Files:
>     --------------------
>
>       - New feature for the sample configuration file openacs-config.tcl
>
>         The following variables (with prefix oacs_) can be taken from the
>         shell variables. This makes it easier to run similar variants of
>         NaviServer instances from a single configuration file, while
>         providing variables from the command line. This is e.g. useful for
>         Docker or cluster setups.
>
>             CookieNamespace
>             bindir
>             cachingmode
>             db_host
>             db_name
>             db_port
>             db_user
>             homedir
>             hostname
>             httpport
>             httpsport
>             ipaddress
>             logroot
>             nscpport
>             server
>             serverroot
>             smtpdport
>
>         One can use e.g.the following command to change some ports and the
>         log file during startup
>
>          oacs_httpport=8000 oacs_httpsport=8443 oacs_smtpdport=2526 \
>              oacs_logroot=/var/www/XXXX/log-node1/ \
>              /usr/local/ns/bin/nsd -t /var/www/XXXX/etc/openacs-config.tcl ...
>
>         - further updates for openacs-config.tcl:
>            * Added sample nssmtpd configuration
>            * Added placeholder for ClusterSecret
>            * Reflect recent Oracle requirements (tested with Oracle 19c)
>            * Added documentation for "StaticCSP", "CookieNamespace",
>              "NsShutdownWithNonZeroExitCode", "LogIncludeUserId"
>
>         - Updated all sample configuration files
>
>
>     Code Changes:
>     -------------
>
>       - Added and updated predefined MIME types based on
>          https://www.iana.org/assignments/media-types/media-types.xhtml
>
>       - Added compile time macro NS_VERBOSE_MALLOC to identify frequent
>         *alloc operations
>
>       - Print version of zlib during startup
>
>       - OpenSSL maintenance:
>
>          * Improved robustness for OCSP with OpenSSL 3.*
>
>            The previous version crashed with OpenSSL 3.*, when OCSP was
>            tried on self-signed certificates. Aside of the fact that OCSP
>            does not make sense for self-signed certificates, the server
>            should not crash in such situations.
>
>          * Fixed ns_crypto::aead::encrypt/decrypt test under OpenSSL 1.1.1
>            (as shipped per default, e.g. on Ubuntu 18.04.4)
>
>            In short, the problem was that with this version of OpenSSL,
>            setting empty additional authenticated data (AAD) behaved
>            differently from other versions, namely it was clearing
>            incorrectly (forgetting) the information that the initialization
>            vector (IV) was already set. An upgrade of OpenSSL fixed the
>            problem. However, with these changes, also the stock version of
>            OpenSSL can be used. As a byproduct, better error messages are
>            now produced, the code received more cleanup (e.g. explicit
>            initialization, etc.)
>
>
>       - Automated testing:
>
>          * Setup if Bitbucket + GitHub pipelines for automated regression
>            testing with multiple versions of components
>
>            For NaviServer 4.99 the current setup performs tests with gcc-10
>            + gcc-11, Tcl 8.6.13 + 8.7a5, NSF 2.4,0 + 2.4.0, tDOM 0.9.1 +
>            0.9.3, extra modules: nsdbpg nsdbi nsdbipg nsudp nscoap nssmtpd.
>
>            https://github.com/nm-wu/naviserver-mirror/actions
>
>          * Improved robustness of regression test when running with the
>            docker networking setup
>
>          * Force nonzero exit code when regression test fails
>
>          * Added parameter "-timeout" to call of regression test cases
>            Previously, the timeout was hard-wired to 3 seconds. One can now
>            call a test with e.g. "nstest::http -timeout 1s ... GET ..."
>
>          * adp_compress.test: removed trailing newline to ease
>            cross-platform regression tests
>
>          * Prefer standard Tcl test constraint "macOrUnix" over own solution
>
>          * Extended regression tests with more test cases
>
>       - Code management:
>          * Changed name of branch from "master" to "main"
>
>       - Code Cleanup
>           * Improved type cleanness
>           * Removed deprecated calls to "sprintf"
>           * Improved portability for Tcl 8.7* (handling of binary data)
>
>       - Improved comments, fixed typos
>
>
>     Changes in NaviServer Modules:
>     ==============================
>
>       39 files changed, 9658 insertions(+), 1781 deletions(-)
>
>     General:
>        Adjust to necessary API changes in NaviServer (contains as well
>        support for the forthcoming release of NaviServer 5)
>
>
>     nsdbpg:
>     -------
>
>        - new pg-driver specific command: ns_pg_prepare /sql/
>
>          Return a dict building a prepared statement for the passed-in SQL
>          statement. The dict contains the keys "sql" and "args". The
>          function is used by e.g. OpenACS to generate prepared statements
>          from SQL commands with bind variables.
>
>        - Raise exception when a value for a bind variable contains a NUL character.
>          This value is explicitly forbidden in text strings passed to PostgreSQL.
>
>        - Let "ns_pg" report available subcommands even when handle is not
>          specified.  This makes the command compatible with the "icanuse"
>          feature in OpenACS.
>
>
>     nsstats:
>     --------
>
>        - HTTP client log analysis:
>           * Provide charts for performance (using highcharts via CDN)
>           * Provide charts on request frequency (using highcharts via CDN)
>           * Provide a summary table for HTTP client requests
>           * Improved robustness against invalid URLs (containing unescaped
>             spaces)
>           * Added support for selection of different HTTP client log files
>             via web interface
>
>        - "Process" page:
>            * Added percentage of request distribution over connection pools
>            * Added information about the connected client
>            * Added more detailed version information
>
>        - Added cache configuration to output when looking at a single cache
>
>        - Improved "log file" analysis
>          * Automated stripping of color codes
>
>          * Added filter option. The filter can be used to grep for (ID)
>            strings in both the system and access logs, providing a summary
>            for the traces of a request in the access log and system log on
>            a single place.
>
>        - Added default and usage information to "Config Parameters" page
>
>
>     nsoracle:
>     ---------
>
>        - Fixed bug when streaming LOB content to connection. The old code
>          did not distinguish between binary and non-binary content.  This
>          bug was discussed in
>          https://openacs.org/forums/message-view?message_id=5693661
>
>          Bumped version number to 2.9
>
>     - switched to plain Debug handling for debugging the driver
>
>          The handling of Ns_LogSqlDebug is performed inside nsdb, including
>          also the printout of (long) SQL statements. Previously, the driver
>          was too chatty when Debug(sql) was turned on.
>
>        - Added support for output columns of type SQLT_TIMESTAMP or SQLT_TIMESTAMP_TZ
>
>          This change fixes a bug, where SQL queries of the form
>
>               SELECT TO_TIMESTAMP(sysdate) FROM dual
>
>          lead to errors for the form:
>
>               Database operation "getrow" failed (exception 1406, "nsoracle.c:3659:Ns_OracleGetRow:
>               error in `OCIStmtFetch ()': ORA-01406: fetched column value was truncated
>
>          The driver needs for several output types special rules, where the
>          timestamp cases were not supported so far. It is also recommended
>          to set the according environment variables specifying the output
>          format in the configuration server of NaviServer, such as e.g.
>
>              set ::env(NLS_TIMESTAMP_FORMAT) "YYYY-MM-DD HH24:MI:SS.FF6"
>              set ::env(NLS_TIMESTAMP_TZ_FORMAT) "YYYY-MM-DD HH24:MI:SS.FF6 TZH:TZM"
>
>          For testing in you local Oracle installation, you might test the
>          output formats (and the required sizes with the following snippet
>          for sqlplus:
>
>             COLUMN localtimestamp format a40
>             COLUMN systimestamp format a40
>             COLUMN ts_bytes format a80
>
>             alter session set nls_timestamp_format='YYYY-MM-DD HH24:MI:SS.FF6';
>             select localtimestamp, length(localtimestamp), dump(localtimestamp) ts_bytes from dual;
>
>             alter session set nls_timestamp_tz_format='YYYY-MM-DD HH24:MI:SS.FF6 TZH:TZM';
>             select systimestamp, length(systimestamp), dump(systimestamp) ts_bytes from dual;
>
>             alter session set nls_timestamp_tz_format='YYYY-MM-DD HH24:MI:SS.FF6 TZR';
>             select systimestamp, length(systimestamp), dump(systimestamp) ts_bytes from dual;
>
>
>     letsencrypt:
>     ------------
>
>        - Added option to produce certificates with ECDSA:
>
>          Prior to this change, all certificates were using RSA keys.
>          Since a while, keys based on elliptic curves became the preference
>          of letsencrypt.
>
>
>     nsshell:
>     --------
>
>         - Fixed a bug in "ns_conn" emulation, when the "kernel" was not correctly identified
>
>     _______________________________________________
>     naviserver-devel mailing list
>     nav...@li...
>     https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>
>
>
> -- 
> Regards,
>
> Sassy Natan
> 972-(0)54-2203702
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

-- 
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"

Re: [naviserver-devel] NaviServer 4.99.25 available

[Sassy N. <sa...@gm...>]

Hi All,

After upgrading my server to the latest version, my log file is broken.

For example I see UTF-8 as special characters.

I check the readme but the sanitizelogfiles 2 doesn't seems to work as
expected.
I have in my configuration:

set        debug                                true
ns_logctl  severity "Debug(sql)"                on

Any ideas?

Here is example:
:    ],is_break_pcols)) as hrs_wo_break
:    FROM wt_payroll_analysis_pp
:     join (select sid,agg_array(case when is_break_pcol=true then 99999
else 0 end) as is_break_pcols
:    \x09\x09\x09from ( select * from wt_et_cols_defs where sid=410000
 order by  pcol_number )pc_defs group by sid) pdefs using (sid)
:    \x09\x09\x09\x09WHERE sid=410000 and employee_id =40599
:    \x09\x09\x09\x09and  to_date(pp_year||'-'||pp||'-01','yyyy-mm-dd')
 between to_date('2023-6-01','yyyy-mm-dd')- interval '6 month'
:    \x09\x09\x09\x09\x09\x09\x09and ( to_date('2023-6-01','yyyy-mm-dd')  -
interval '1 day' )::date
:    \x09\x09\x09\x09UNION
:    \x09\x09\x09\x09 select
sid,employee_id,pp,pp_year,ppid,to_date(pp_year||'-'||pp||'-01','yyyy-mm-dd')
as day,null2zero(hrs_wo_break)*3600
:    \x09\x09\x09\x09                                from pp_pa_table
:    \x09\x09\x09\x09) ggg
:    \x09\x09\x09\x09group by  sid,employee_id,pp,pp_year


Thanks
Sassy

On Mon, May 1, 2023 at 6:55 PM Gustaf Neumann <ne...@wu...> wrote:

> Dear all,
>
> I am glad to announce that the release of NaviServer 4.99.25 is
> available at SourceForge [1].  This release is mostly a bug-fix
> release. The forthcoming version 5.0 of NaviServer will contain
> several new features omitted in this bug-fix branch. In case, you are
> building NaviServer from the Bitbucket repository, please note that
> the release 4.99.25 is in the branch release/4.99 (bug fix branch for
> the NaviServer 4.99 family). New development happens in the "main"
> branch of the repository, leading to NaviServer 5.*.
>
> See below for a summary of the changes.
>
> Many thanks to the contributors of this release:
>
>      Andrew Piskorski
>      Antonio Pisano
>      Brian Fenton
>      Gustaf Neumann
>      Hector Romojaro
>      Joe Oldak
>      Khy Huang
>      Oleg Oleinick
>      Zoran Vasiljevic
>
> All the best!
>
> -gustaf neumann
>
> [1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.25/
>
> =======================================
> NaviServer 4.99.25, released 2023-05-01
> =======================================
>
>  132 files changed, 3957 insertions(+), 2068 deletions(-)
>
> New Features:
> -------------
>
>  - Added meta-information to configuration values
>
>    NaviServer can now report, what configuration values provided in
>    the configuration file were actually used, what their default
>    values are, and whether these values were specified or not (using
>    the default values). With this information, administration
>    (e.g. migration) becomes easier. The NaviServer module "nsstats"
>    shows this meta information via the web interface.
>
>    This functionality is provided via the new option "-filter" for the
>    command "ns_configsection ... section". When the "-filter" option
>    is used, different kinds of information about the parameters is
>    returned from the specified section.
>
>     "-filter unread":
>
>         Returns the parameters, which were set during configuration (i.e.,
>         in the configuration file) but which were not read in the startup
>         phase of the server. This option is useful to determine
>         e.g. typographical errors of specified parameter names.
>
>     "-filter defaulted":
>
>         Returns the parameters, from which the default values were read
>         (i.e., which were not explicitly set)
>
>     "-filter defaults":
>
>         Returns the defaults of the parameter. This is useful for
>         contrasting the actual values with the default values for
>         Parameters, e.g. in a web based interface.
>
>  - ns_set reform (per default deactivated in 4.99, but activated in 5.*)
>
>    The classical implementation for ns_sets uses separately malloced
>    storage for every attribute name and attribute value. So, e.g., for
>    1000 ns_sets with 20 members each, this means 1,000*20*2 = 40,000
>    malloc/free operations, e.g., for a single db query! Although the
>    malloc implementations have improved over the years, these will
>    require many lock operations, especially under load, where many
>    threads might perform many concurrent malloc operations. One other
>    consequence is that the allocated memory will be scattered over
>    address space, which has bad implications for CPU caching.
>
>    The new implementation uses for one "ns_set" a single Tcl_DString
>    keeping all attribute names and attribute values. This reduces the
>    malloc operations and improves memory locality, such that cache
>    hits will improve.
>
>    One caveat of this change is that modules using "ns_set" have to be
>    recompiled, since the full C-level data structure of the "ns_set"
>    is exposed. Therefore, adding a member causes a binary
>    incompatibility. One other potential problem is that C-level
>    modules using the Ns_Set* API have to make sure that long-living
>    string values are copied (this was necessary before as well, but
>    was in many cases no problem, when the "ns_sets" were seldom
>    updated).
>
>    For high compatibility, this feature is deactivated per default in
>    the 4.99.* series and can be activated by setting the compile-time
>    C macro "NS_SET_DSTRING".
>
>
>
> API changes:
> ------------
>
>    API extensions:
>     - Provide a new interface ending with *Sz to provide string sizes.
>       This reduces the need of strlen() operations.
>       * Ns_SetCreateSz()
>       * Ns_SetIUpdateSz()
>       * Ns_SetPutSz()
>       * Ns_SetPutValueSz()
>       * Ns_SetUpdateSz()
>
>     - New API calls for "ns_set" reform
>       * Ns_SetClearValues(): clear the values for all keys
>       * Ns_SetDataPrealloc(): creating ns_sets with preallocated values
>         to avoid resize operations
>       * NsSetResize()
>       * NsHeaderSetGet()
>
>     - Ns_ConfigSet(const char *section, const char *key, const char *name)
>       The last argument is new and allows one to create named sets
>       (previously, all such sets were unnamed)
>
>     - NsHexPrint(): Print the potentially binary content of a buffer
>       in human-readable form.
>
>     - Ns_RelativeTime(Ns_Time *relTimePtr, Ns_Time *timePtr)
>       This call implements the inverse operation of Ns_AbsoluteTime(),
>       and is used mostly to make debug messages eye-friendly.
>
>
> Performance Improvements:
> -------------------------
>
>  - Replaced malloc operation per log entry by thread local variable in
>    system log implementation.
>
>  - When NaviServer 4.99.25 is compiled with NS_SET_DSTRING supportm
>    the following preliminary performance results were measured from
>    the "ns_set" reform (see above).  The tests were performed on
>    openacs.org (Xeon Gold 6226R CPU @ 2.90GHz, 32 cores,
>    hyper-threading enabled). The test executes the SQL query
>
>         select * from acs_objects limit 1000
>
>    100 times in sequence. This test is run in 1 to 30 concurrent
>    threads. With 30 threads, 3mio tuples are retrieved, and 72 mio
>    malloc/free operations are needed alone for the retrieved values.
>
>    Before (classical ns_set with many mallocs):
>
>         threads 1 total 4606.787 ms avg 3285.25 ms
>         threads 5 total 4595.358 ms avg 3493.07 ms
>         threads 10 total 4804.193 ms avg 3755.93 ms
>         threads 20 total 6279.524 ms avg 4569.16 ms
>         threads 30 total 8966.427 ms avg 6618.58 ms
>
>    After reform (using common Tcl_DString per tuple):
>
>         threads 1 total 4524.645 ms avg 3242.54 ms
>         threads 5 total 4251.266 ms avg 3450.09 ms
>         threads 10 total 4656.795 ms avg 3665.31 ms
>         threads 20 total 5934.105 ms avg 4671.38 ms
>         threads 30 total 7384.591 ms avg 5642.76 ms
>
>    To summarize, the improvement increases under higher load (with
>    more parallel threads). E.g., with 30 threads, the total time
>    improved by 17%.... leading also to a smaller RSS. These tests were
>    not performed under "clinical" conditions.
>
>    The new Tcl API call "ns_set size" can be used to pre-allocate
>    larger ns_sets, such that the usual Tcl_DString growing policy does
>    not kick in, reducing further realloc() operations.
>
>
> Bug Fixes:
> ----------
>
>  - Fixed potential crash in "ns_accesslog extendedheaders XXXX".
>    Setting extended headers via configuration file was correct, but
>    changing it dynamically via "ns_accesslog extendedheaders .." was
>    broken. (Issue https://sourceforge.net/p/naviserver/bugs/91/)
>
>  - "ns_conn location": Fixed potential race condition
>
>     It was possible that "ns_conn location" could return inconsistent
>     results in a single request, when the underlying sockPtr was
>     aggressively reused. Now, the value of the location member is
>     copied to the connection structure instead of being shared with
>     the socket structure (as before).
>
>  - "ns_cache_eval -force": Fixed potential race condition
>
>     There was a problem with "ns_cache_eval -force", where the system
>     relied on the existence of a pre-existing entry, but in case the
>     entry was flushed in the meantime problems a crash might have
>     happened. Now the value during the "-force" call is cleared exactly
>     like in the case of an unset operation. The null-value operations
>     are already protected until these are finished in various places
>     in the code.
>
>  - "ns_socknread": Fix potentially wrong result for buffered channels
>
>  - Bug fixes for problems showing up under MS Windows:
>
>     * Make sure that the output variable of Ns_ObjvIndex() is always an
>       integer. Previously, the output variable was in two places a
>       character variable, causing crashes under MS Windows.
>
>     * Handle incompatibility in setlocale() under MS Windows.  Under
>       MS Windows, later calls to setlocale() overwrite the string
>       returned by former calls. So, it is necessary to copy of the
>       returned string of a setlocale() call under MS Windows.
>
>     * Handling linking problems: MS Windows requires explicit handling
>       when importing symbols from .dll files (Ns_LogSqlDebug,
>       NS_intTypePtr)
>
>     * Pass error codes from low-level function SockRecv() and
>       SockSend() via variables.  This change makes sure the real error
>       code (immediately after the I/O operation) is passed to the
>       caller to avoid missed error cases and weird error message
>       (e.g. under windows).
>
>
>  - Bug fixes for ADP parser:
>
>     * Support for greater than sign ">" inside attribute values.
>
>       Previously, NaviServer determined the terminating end-of-tag
>       character as literally the first greater than sign, no matter if
>       this was used as attribute values between single or double
>       quotes.
>
>       The new version supports such values, since the "Living Standard
>       of HTML" [1] allows the use of less than "<" and greater than
>       ">" signs inside attribute values as long these are between
>       single or double quotes.  The guide [2] just recommends using
>       character escapes for "<", ">" and "&".
>
>       [1] https://html.spec.whatwg.org/multipage/syntax.html#syntax-attribute-value
>       [2] https://www.w3.org/International/questions/qa-escapes
>
>       This problem was reported by Wolfgang Winkler
>
>     * Clear ADP flags in case of errors (this error was present for
>       many years)
>
>       Previously, The following ADP page could lead to a full
>       breakdown of the server, since the error states of the server
>       were never cleared, and subsequent requests served be the same
>       interpreter could lead to the old error states.
>
>       Many thanks to Oleg Oleinick for reporting and the great test
>       cases.
>
>  - Provide better HTTP status code for early errors: When the driver
>    terminates already a request, it might be the case that the error
>    flags were not read out. In such cases, we provide now more
>    specific status codes rather than 400.
>
>  - ns_http:
>
>    * Improved robustness with domain names resolving against many IP
>      addresses: When a domain name is resolved against many IP
>      addresses, and all these IP addresses block (connect returns "in
>      progress") then the old code might have looped infinitely. Now
>      the code respects the provided timeout (default set to 5s) per
>      resolved IP address.
>
>      Many thanks to Joe Oldak for pointing out the problem and
>      leading us to the solution.
>
>    * Fixed timeout handling during TLS handshake: This problem could
>      appear, when the TCP connection to a server succeeded quickly,
>      but the TLS handshake was taking a long time, without a raising
>      a timeout exception, although the timeout time has expired.
>
>  - Set the default server before the init-scripts is called to make it
>    accessible from there
>
>  - Provide compatibility between in-memory and file-based form-data
>    handling for invalid characters: Since the file based
>    implementation uses a fallback-charset of iso8859-1 when parsing
>    form data in POST requests (in order to be able to extract
>    "_charset_" data), the in-memory based variant does now the
>    same. The decision of which parser is used is taken based on the
>    size of the form data.... and should therefore be consistent.
>
>  - nscp: Fixed problem, when the nscp module is activated but no
>    "users" section is specified. Many thanks to Andrew Piskorski for
>    reporting the problem.
>
>  - Security Improvement: Added protection against sneaking in fake
>    NAME.tmpfile entries into form fields, when performing file upload
>    operations.
>
>
> Documentation improvements:
> ---------------------------
>
>  - Improved the following man pages:
>
>        doc/src/manual/adp-overview.man
>        doc/src/manual/tcl-lib-nstrace.man
>        doc/src/naviserver/commandlist.man
>        doc/src/naviserver/ns_adp.man
>        doc/src/naviserver/ns_adp_ctl.man
>        doc/src/naviserver/ns_config.man
>        doc/src/naviserver/ns_conn.man
>        doc/src/naviserver/ns_connchan.man
>        doc/src/naviserver/ns_cookie.man
>        doc/src/naviserver/ns_crypto.man
>        doc/src/naviserver/ns_http.man
>        doc/src/naviserver/ns_ictl.man
>        doc/src/naviserver/ns_locationproc.man
>        doc/src/naviserver/ns_parsehostport.man
>        doc/src/naviserver/ns_return.man
>        doc/src/naviserver/ns_set.man
>        doc/src/naviserver/ns_setprivileges.man
>
>
> Configuration Files:
> --------------------
>
>  - New feature for the sample configuration file openacs-config.tcl
>
>    The following variables (with prefix oacs_) can be taken from the
>    shell variables. This makes it easier to run similar variants of
>    NaviServer instances from a single configuration file, while
>    providing variables from the command line. This is e.g. useful for
>    Docker or cluster setups.
>
>        CookieNamespace
>        bindir
>        cachingmode
>        db_host
>        db_name
>        db_port
>        db_user
>        homedir
>        hostname
>        httpport
>        httpsport
>        ipaddress
>        logroot
>        nscpport
>        server
>        serverroot
>        smtpdport
>
>    One can use e.g.the following command to change some ports and the
>    log file during startup
>
>     oacs_httpport=8000 oacs_httpsport=8443 oacs_smtpdport=2526 \
>         oacs_logroot=/var/www/XXXX/log-node1/ \
>         /usr/local/ns/bin/nsd -t /var/www/XXXX/etc/openacs-config.tcl ...
>
>    - further updates for openacs-config.tcl:
>       * Added sample nssmtpd configuration
>       * Added placeholder for ClusterSecret
>       * Reflect recent Oracle requirements (tested with Oracle 19c)
>       * Added documentation for "StaticCSP", "CookieNamespace",
>         "NsShutdownWithNonZeroExitCode", "LogIncludeUserId"
>
>    - Updated all sample configuration files
>
>
> Code Changes:
> -------------
>
>  - Added and updated predefined MIME types based on
>     https://www.iana.org/assignments/media-types/media-types.xhtml
>
>  - Added compile time macro NS_VERBOSE_MALLOC to identify frequent
>    *alloc operations
>
>  - Print version of zlib during startup
>
>  - OpenSSL maintenance:
>
>     * Improved robustness for OCSP with OpenSSL 3.*
>
>       The previous version crashed with OpenSSL 3.*, when OCSP was
>       tried on self-signed certificates. Aside of the fact that OCSP
>       does not make sense for self-signed certificates, the server
>       should not crash in such situations.
>
>     * Fixed ns_crypto::aead::encrypt/decrypt test under OpenSSL 1.1.1
>       (as shipped per default, e.g. on Ubuntu 18.04.4)
>
>       In short, the problem was that with this version of OpenSSL,
>       setting empty additional authenticated data (AAD) behaved
>       differently from other versions, namely it was clearing
>       incorrectly (forgetting) the information that the initialization
>       vector (IV) was already set. An upgrade of OpenSSL fixed the
>       problem. However, with these changes, also the stock version of
>       OpenSSL can be used. As a byproduct, better error messages are
>       now produced, the code received more cleanup (e.g. explicit
>       initialization, etc.)
>
>
>  - Automated testing:
>
>     * Setup if Bitbucket + GitHub pipelines for automated regression
>       testing with multiple versions of components
>
>       For NaviServer 4.99 the current setup performs tests with gcc-10
>       + gcc-11, Tcl 8.6.13 + 8.7a5, NSF 2.4,0 + 2.4.0, tDOM 0.9.1 +
>       0.9.3, extra modules: nsdbpg nsdbi nsdbipg nsudp nscoap nssmtpd.
>
>       https://github.com/nm-wu/naviserver-mirror/actions
>
>     * Improved robustness of regression test when running with the
>       docker networking setup
>
>     * Force nonzero exit code when regression test fails
>
>     * Added parameter "-timeout" to call of regression test cases
>       Previously, the timeout was hard-wired to 3 seconds. One can now
>       call a test with e.g. "nstest::http -timeout 1s ... GET ..."
>
>     * adp_compress.test: removed trailing newline to ease
>       cross-platform regression tests
>
>     * Prefer standard Tcl test constraint "macOrUnix" over own solution
>
>     * Extended regression tests with more test cases
>
>  - Code management:
>     * Changed name of branch from "master" to "main"
>
>  - Code Cleanup
>      * Improved type cleanness
>      * Removed deprecated calls to "sprintf"
>      * Improved portability for Tcl 8.7* (handling of binary data)
>
>  - Improved comments, fixed typos
>
>
> Changes in NaviServer Modules:
> ==============================
>
>  39 files changed, 9658 insertions(+), 1781 deletions(-)
>
> General:
>   Adjust to necessary API changes in NaviServer (contains as well
>   support for the forthcoming release of NaviServer 5)
>
>
> nsdbpg:
> -------
>
>   - new pg-driver specific command: ns_pg_prepare /sql/
>
>     Return a dict building a prepared statement for the passed-in SQL
>     statement. The dict contains the keys "sql" and "args". The
>     function is used by e.g. OpenACS to generate prepared statements
>     from SQL commands with bind variables.
>
>   - Raise exception when a value for a bind variable contains a NUL character.
>     This value is explicitly forbidden in text strings passed to PostgreSQL.
>
>   - Let "ns_pg" report available subcommands even when handle is not
>     specified.  This makes the command compatible with the "icanuse"
>     feature in OpenACS.
>
>
> nsstats:
> --------
>
>   - HTTP client log analysis:
>      * Provide charts for performance (using highcharts via CDN)
>      * Provide charts on request frequency (using highcharts via CDN)
>      * Provide a summary table for HTTP client requests
>      * Improved robustness against invalid URLs (containing unescaped
>        spaces)
>      * Added support for selection of different HTTP client log files
>        via web interface
>
>   - "Process" page:
>       * Added percentage of request distribution over connection pools
>       * Added information about the connected client
>       * Added more detailed version information
>
>   - Added cache configuration to output when looking at a single cache
>
>   - Improved "log file" analysis
>     * Automated stripping of color codes
>
>     * Added filter option. The filter can be used to grep for (ID)
>       strings in both the system and access logs, providing a summary
>       for the traces of a request in the access log and system log on
>       a single place.
>
>   - Added default and usage information to "Config Parameters" page
>
>
> nsoracle:
> ---------
>
>   - Fixed bug when streaming LOB content to connection. The old code
>     did not distinguish between binary and non-binary content.  This
>     bug was discussed in
>     https://openacs.org/forums/message-view?message_id=5693661
>
>     Bumped version number to 2.9
>
> - switched to plain Debug handling for debugging the driver
>
>     The handling of Ns_LogSqlDebug is performed inside nsdb, including
>     also the printout of (long) SQL statements. Previously, the driver
>     was too chatty when Debug(sql) was turned on.
>
>   - Added support for output columns of type SQLT_TIMESTAMP or SQLT_TIMESTAMP_TZ
>
>     This change fixes a bug, where SQL queries of the form
>
>          SELECT TO_TIMESTAMP(sysdate) FROM dual
>
>     lead to errors for the form:
>
>          Database operation "getrow" failed (exception 1406, "nsoracle.c:3659:Ns_OracleGetRow:
>          error in `OCIStmtFetch ()': ORA-01406: fetched column value was truncated
>
>     The driver needs for several output types special rules, where the
>     timestamp cases were not supported so far. It is also recommended
>     to set the according environment variables specifying the output
>     format in the configuration server of NaviServer, such as e.g.
>
>         set ::env(NLS_TIMESTAMP_FORMAT) "YYYY-MM-DD HH24:MI:SS.FF6"
>         set ::env(NLS_TIMESTAMP_TZ_FORMAT) "YYYY-MM-DD HH24:MI:SS.FF6 TZH:TZM"
>
>     For testing in you local Oracle installation, you might test the
>     output formats (and the required sizes with the following snippet
>     for sqlplus:
>
>        COLUMN localtimestamp format a40
>        COLUMN systimestamp format a40
>        COLUMN ts_bytes format a80
>
>        alter session set nls_timestamp_format='YYYY-MM-DD HH24:MI:SS.FF6';
>        select localtimestamp, length(localtimestamp), dump(localtimestamp) ts_bytes from dual;
>
>        alter session set nls_timestamp_tz_format='YYYY-MM-DD HH24:MI:SS.FF6 TZH:TZM';
>        select systimestamp, length(systimestamp), dump(systimestamp) ts_bytes from dual;
>
>        alter session set nls_timestamp_tz_format='YYYY-MM-DD HH24:MI:SS.FF6 TZR';
>        select systimestamp, length(systimestamp), dump(systimestamp) ts_bytes from dual;
>
>
> letsencrypt:
> ------------
>
>   - Added option to produce certificates with ECDSA:
>
>     Prior to this change, all certificates were using RSA keys.
>     Since a while, keys based on elliptic curves became the preference
>     of letsencrypt.
>
>
> nsshell:
> --------
>
>    - Fixed a bug in "ns_conn" emulation, when the "kernel" was not correctly identified
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>


-- 
Regards,

Sassy Natan
972-(0)54-2203702

[naviserver-devel] Joint OpenACS and EuroTcl Conference 2023

[Gustaf N. <ne...@wu...>]

Dear all,

The deadline for submissions of abstracts for presentation at the 
forthcoming joint OpenACS / EuroTcl conference is in about one month. 
Presentations about NaviServer applications are very welcome.

Important dates:

     June 15th, 2023: Deadline for submissions of abstracts (max. 2 
pages, min. 250 words);
     June 19th, 2023: Notification of acceptance
     June 30th, 2023: Registration ends
     July 19th, 2023: Meet & greet
     July 20th - July 21st, 2023:  Conference

For details, see: https://openacs.org/conf2023/

Gustaf Neumann and Harald Oehlmann

-- 
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"

[naviserver-devel] NaviServer 4.99.25 available

[Gustaf N. <ne...@wu...>]

Dear all,

I am glad to announce that the release of NaviServer 4.99.25 is
available at SourceForge [1].  This release is mostly a bug-fix
release. The forthcoming version 5.0 of NaviServer will contain
several new features omitted in this bug-fix branch. In case, you are
building NaviServer from the Bitbucket repository, please note that
the release 4.99.25 is in the branch release/4.99 (bug fix branch for
the NaviServer 4.99 family). New development happens in the "main"
branch of the repository, leading to NaviServer 5.*.

See below for a summary of the changes.

Many thanks to the contributors of this release:

      Andrew Piskorski
      Antonio Pisano
      Brian Fenton
      Gustaf Neumann
      Hector Romojaro
      Joe Oldak
      Khy Huang
      Oleg Oleinick
      Zoran Vasiljevic

All the best!

-gustaf neumann

[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.25/


=======================================
NaviServer 4.99.25, released 2023-05-01
=======================================

  132 files changed, 3957 insertions(+), 2068 deletions(-)

New Features:
-------------

  - Added meta-information to configuration values

    NaviServer can now report, what configuration values provided in
    the configuration file were actually used, what their default
    values are, and whether these values were specified or not (using
    the default values). With this information, administration
    (e.g. migration) becomes easier. The NaviServer module "nsstats"
    shows this meta information via the web interface.

    This functionality is provided via the new option "-filter" for the
    command "ns_configsection ... section". When the "-filter" option
    is used, different kinds of information about the parameters is
    returned from the specified section.

     "-filter unread":

         Returns the parameters, which were set during configuration (i.e.,
         in the configuration file) but which were not read in the startup
         phase of the server. This option is useful to determine
         e.g. typographical errors of specified parameter names.

     "-filter defaulted":

         Returns the parameters, from which the default values were read
         (i.e., which were not explicitly set)

     "-filter defaults":

         Returns the defaults of the parameter. This is useful for
         contrasting the actual values with the default values for
         Parameters, e.g. in a web based interface.

  - ns_set reform (per default deactivated in 4.99, but activated in 5.*)

    The classical implementation for ns_sets uses separately malloced
    storage for every attribute name and attribute value. So, e.g., for
    1000 ns_sets with 20 members each, this means 1,000*20*2 = 40,000
    malloc/free operations, e.g., for a single db query! Although the
    malloc implementations have improved over the years, these will
    require many lock operations, especially under load, where many
    threads might perform many concurrent malloc operations. One other
    consequence is that the allocated memory will be scattered over
    address space, which has bad implications for CPU caching.

    The new implementation uses for one "ns_set" a single Tcl_DString
    keeping all attribute names and attribute values. This reduces the
    malloc operations and improves memory locality, such that cache
    hits will improve.

    One caveat of this change is that modules using "ns_set" have to be
    recompiled, since the full C-level data structure of the "ns_set"
    is exposed. Therefore, adding a member causes a binary
    incompatibility. One other potential problem is that C-level
    modules using the Ns_Set* API have to make sure that long-living
    string values are copied (this was necessary before as well, but
    was in many cases no problem, when the "ns_sets" were seldom
    updated).

    For high compatibility, this feature is deactivated per default in
    the 4.99.* series and can be activated by setting the compile-time
    C macro "NS_SET_DSTRING".



API changes:
------------

    API extensions:
     - Provide a new interface ending with *Sz to provide string sizes.
       This reduces the need of strlen() operations.
       * Ns_SetCreateSz()
       * Ns_SetIUpdateSz()
       * Ns_SetPutSz()
       * Ns_SetPutValueSz()
       * Ns_SetUpdateSz()

     - New API calls for "ns_set" reform
       * Ns_SetClearValues(): clear the values for all keys
       * Ns_SetDataPrealloc(): creating ns_sets with preallocated values
         to avoid resize operations
       * NsSetResize()
       * NsHeaderSetGet()

     - Ns_ConfigSet(const char *section, const char *key, const char *name)
       The last argument is new and allows one to create named sets
       (previously, all such sets were unnamed)

     - NsHexPrint(): Print the potentially binary content of a buffer
       in human-readable form.

     - Ns_RelativeTime(Ns_Time *relTimePtr, Ns_Time *timePtr)
       This call implements the inverse operation of Ns_AbsoluteTime(),
       and is used mostly to make debug messages eye-friendly.


Performance Improvements:
-------------------------

  - Replaced malloc operation per log entry by thread local variable in
    system log implementation.

  - When NaviServer 4.99.25 is compiled with NS_SET_DSTRING supportm
    the following preliminary performance results were measured from
    the "ns_set" reform (see above).  The tests were performed on
    openacs.org (Xeon Gold 6226R CPU @ 2.90GHz, 32 cores,
    hyper-threading enabled). The test executes the SQL query

         select * from acs_objects limit 1000

    100 times in sequence. This test is run in 1 to 30 concurrent
    threads. With 30 threads, 3mio tuples are retrieved, and 72 mio
    malloc/free operations are needed alone for the retrieved values.

    Before (classical ns_set with many mallocs):

         threads 1 total 4606.787 ms avg 3285.25 ms
         threads 5 total 4595.358 ms avg 3493.07 ms
         threads 10 total 4804.193 ms avg 3755.93 ms
         threads 20 total 6279.524 ms avg 4569.16 ms
         threads 30 total 8966.427 ms avg 6618.58 ms

    After reform (using common Tcl_DString per tuple):

         threads 1 total 4524.645 ms avg 3242.54 ms
         threads 5 total 4251.266 ms avg 3450.09 ms
         threads 10 total 4656.795 ms avg 3665.31 ms
         threads 20 total 5934.105 ms avg 4671.38 ms
         threads 30 total 7384.591 ms avg 5642.76 ms

    To summarize, the improvement increases under higher load (with
    more parallel threads). E.g., with 30 threads, the total time
    improved by 17%.... leading also to a smaller RSS. These tests were
    not performed under "clinical" conditions.

    The new Tcl API call "ns_set size" can be used to pre-allocate
    larger ns_sets, such that the usual Tcl_DString growing policy does
    not kick in, reducing further realloc() operations.


Bug Fixes:
----------

  - Fixed potential crash in "ns_accesslog extendedheaders XXXX".
    Setting extended headers via configuration file was correct, but
    changing it dynamically via "ns_accesslog extendedheaders .." was
    broken. (Issuehttps://sourceforge.net/p/naviserver/bugs/91/)

  - "ns_conn location": Fixed potential race condition

     It was possible that "ns_conn location" could return inconsistent
     results in a single request, when the underlying sockPtr was
     aggressively reused. Now, the value of the location member is
     copied to the connection structure instead of being shared with
     the socket structure (as before).

  - "ns_cache_eval -force": Fixed potential race condition

     There was a problem with "ns_cache_eval -force", where the system
     relied on the existence of a pre-existing entry, but in case the
     entry was flushed in the meantime problems a crash might have
     happened. Now the value during the "-force" call is cleared exactly
     like in the case of an unset operation. The null-value operations
     are already protected until these are finished in various places
     in the code.

  - "ns_socknread": Fix potentially wrong result for buffered channels

  - Bug fixes for problems showing up under MS Windows:

     * Make sure that the output variable of Ns_ObjvIndex() is always an
       integer. Previously, the output variable was in two places a
       character variable, causing crashes under MS Windows.

     * Handle incompatibility in setlocale() under MS Windows.  Under
       MS Windows, later calls to setlocale() overwrite the string
       returned by former calls. So, it is necessary to copy of the
       returned string of a setlocale() call under MS Windows.

     * Handling linking problems: MS Windows requires explicit handling
       when importing symbols from .dll files (Ns_LogSqlDebug,
       NS_intTypePtr)

     * Pass error codes from low-level function SockRecv() and
       SockSend() via variables.  This change makes sure the real error
       code (immediately after the I/O operation) is passed to the
       caller to avoid missed error cases and weird error message
       (e.g. under windows).


  - Bug fixes for ADP parser:

     * Support for greater than sign ">" inside attribute values.

       Previously, NaviServer determined the terminating end-of-tag
       character as literally the first greater than sign, no matter if
       this was used as attribute values between single or double
       quotes.

       The new version supports such values, since the "Living Standard
       of HTML" [1] allows the use of less than "<" and greater than
       ">" signs inside attribute values as long these are between
       single or double quotes.  The guide [2] just recommends using
       character escapes for "<", ">" and "&".

       [1]https://html.spec.whatwg.org/multipage/syntax.html#syntax-attribute-value
       [2]https://www.w3.org/International/questions/qa-escapes

       This problem was reported by Wolfgang Winkler

     * Clear ADP flags in case of errors (this error was present for
       many years)

       Previously, The following ADP page could lead to a full
       breakdown of the server, since the error states of the server
       were never cleared, and subsequent requests served be the same
       interpreter could lead to the old error states.

       Many thanks to Oleg Oleinick for reporting and the great test
       cases.

  - Provide better HTTP status code for early errors: When the driver
    terminates already a request, it might be the case that the error
    flags were not read out. In such cases, we provide now more
    specific status codes rather than 400.

  - ns_http:

    * Improved robustness with domain names resolving against many IP
      addresses: When a domain name is resolved against many IP
      addresses, and all these IP addresses block (connect returns "in
      progress") then the old code might have looped infinitely. Now
      the code respects the provided timeout (default set to 5s) per
      resolved IP address.

      Many thanks to Joe Oldak for pointing out the problem and
      leading us to the solution.

    * Fixed timeout handling during TLS handshake: This problem could
      appear, when the TCP connection to a server succeeded quickly,
      but the TLS handshake was taking a long time, without a raising
      a timeout exception, although the timeout time has expired.

  - Set the default server before the init-scripts is called to make it
    accessible from there

  - Provide compatibility between in-memory and file-based form-data
    handling for invalid characters: Since the file based
    implementation uses a fallback-charset of iso8859-1 when parsing
    form data in POST requests (in order to be able to extract
    "_charset_" data), the in-memory based variant does now the
    same. The decision of which parser is used is taken based on the
    size of the form data.... and should therefore be consistent.

  - nscp: Fixed problem, when the nscp module is activated but no
    "users" section is specified. Many thanks to Andrew Piskorski for
    reporting the problem.

  - Security Improvement: Added protection against sneaking in fake
    NAME.tmpfile entries into form fields, when performing file upload
    operations.


Documentation improvements:
---------------------------

  - Improved the following man pages:

        doc/src/manual/adp-overview.man
        doc/src/manual/tcl-lib-nstrace.man
        doc/src/naviserver/commandlist.man
        doc/src/naviserver/ns_adp.man
        doc/src/naviserver/ns_adp_ctl.man
        doc/src/naviserver/ns_config.man
        doc/src/naviserver/ns_conn.man
        doc/src/naviserver/ns_connchan.man
        doc/src/naviserver/ns_cookie.man
        doc/src/naviserver/ns_crypto.man
        doc/src/naviserver/ns_http.man
        doc/src/naviserver/ns_ictl.man
        doc/src/naviserver/ns_locationproc.man
        doc/src/naviserver/ns_parsehostport.man
        doc/src/naviserver/ns_return.man
        doc/src/naviserver/ns_set.man
        doc/src/naviserver/ns_setprivileges.man


Configuration Files:
--------------------

  - New feature for the sample configuration file openacs-config.tcl

    The following variables (with prefix oacs_) can be taken from the
    shell variables. This makes it easier to run similar variants of
    NaviServer instances from a single configuration file, while
    providing variables from the command line. This is e.g. useful for
    Docker or cluster setups.

        CookieNamespace
        bindir
        cachingmode
        db_host
        db_name
        db_port
        db_user
        homedir
        hostname
        httpport
        httpsport
        ipaddress
        logroot
        nscpport
        server
        serverroot
        smtpdport

    One can use e.g.the following command to change some ports and the
    log file during startup

     oacs_httpport=8000 oacs_httpsport=8443 oacs_smtpdport=2526 \
         oacs_logroot=/var/www/XXXX/log-node1/ \
         /usr/local/ns/bin/nsd -t /var/www/XXXX/etc/openacs-config.tcl ...

    - further updates for openacs-config.tcl:
       * Added sample nssmtpd configuration
       * Added placeholder for ClusterSecret
       * Reflect recent Oracle requirements (tested with Oracle 19c)
       * Added documentation for "StaticCSP", "CookieNamespace",
         "NsShutdownWithNonZeroExitCode", "LogIncludeUserId"

    - Updated all sample configuration files


Code Changes:
-------------

  - Added and updated predefined MIME types based on
     https://www.iana.org/assignments/media-types/media-types.xhtml

  - Added compile time macro NS_VERBOSE_MALLOC to identify frequent
    *alloc operations

  - Print version of zlib during startup

  - OpenSSL maintenance:

     * Improved robustness for OCSP with OpenSSL 3.*

       The previous version crashed with OpenSSL 3.*, when OCSP was
       tried on self-signed certificates. Aside of the fact that OCSP
       does not make sense for self-signed certificates, the server
       should not crash in such situations.

     * Fixed ns_crypto::aead::encrypt/decrypt test under OpenSSL 1.1.1
       (as shipped per default, e.g. on Ubuntu 18.04.4)

       In short, the problem was that with this version of OpenSSL,
       setting empty additional authenticated data (AAD) behaved
       differently from other versions, namely it was clearing
       incorrectly (forgetting) the information that the initialization
       vector (IV) was already set. An upgrade of OpenSSL fixed the
       problem. However, with these changes, also the stock version of
       OpenSSL can be used. As a byproduct, better error messages are
       now produced, the code received more cleanup (e.g. explicit
       initialization, etc.)


  - Automated testing:

     * Setup if Bitbucket + GitHub pipelines for automated regression
       testing with multiple versions of components

       For NaviServer 4.99 the current setup performs tests with gcc-10
       + gcc-11, Tcl 8.6.13 + 8.7a5, NSF 2.4,0 + 2.4.0, tDOM 0.9.1 +
       0.9.3, extra modules: nsdbpg nsdbi nsdbipg nsudp nscoap nssmtpd.

       https://github.com/nm-wu/naviserver-mirror/actions

     * Improved robustness of regression test when running with the
       docker networking setup

     * Force nonzero exit code when regression test fails

     * Added parameter "-timeout" to call of regression test cases
       Previously, the timeout was hard-wired to 3 seconds. One can now
       call a test with e.g. "nstest::http -timeout 1s ... GET ..."

     * adp_compress.test: removed trailing newline to ease
       cross-platform regression tests

     * Prefer standard Tcl test constraint "macOrUnix" over own solution

     * Extended regression tests with more test cases

  - Code management:
     * Changed name of branch from "master" to "main"

  - Code Cleanup
      * Improved type cleanness
      * Removed deprecated calls to "sprintf"
      * Improved portability for Tcl 8.7* (handling of binary data)

  - Improved comments, fixed typos


Changes in NaviServer Modules:
==============================

  39 files changed, 9658 insertions(+), 1781 deletions(-)

General:
   Adjust to necessary API changes in NaviServer (contains as well
   support for the forthcoming release of NaviServer 5)


nsdbpg:
-------

   - new pg-driver specific command: ns_pg_prepare /sql/

     Return a dict building a prepared statement for the passed-in SQL
     statement. The dict contains the keys "sql" and "args". The
     function is used by e.g. OpenACS to generate prepared statements
     from SQL commands with bind variables.

   - Raise exception when a value for a bind variable contains a NUL character.
     This value is explicitly forbidden in text strings passed to PostgreSQL.

   - Let "ns_pg" report available subcommands even when handle is not
     specified.  This makes the command compatible with the "icanuse"
     feature in OpenACS.


nsstats:
--------

   - HTTP client log analysis:
      * Provide charts for performance (using highcharts via CDN)
      * Provide charts on request frequency (using highcharts via CDN)
      * Provide a summary table for HTTP client requests
      * Improved robustness against invalid URLs (containing unescaped
        spaces)
      * Added support for selection of different HTTP client log files
        via web interface

   - "Process" page:
       * Added percentage of request distribution over connection pools
       * Added information about the connected client
       * Added more detailed version information

   - Added cache configuration to output when looking at a single cache

   - Improved "log file" analysis
     * Automated stripping of color codes

     * Added filter option. The filter can be used to grep for (ID)
       strings in both the system and access logs, providing a summary
       for the traces of a request in the access log and system log on
       a single place.

   - Added default and usage information to "Config Parameters" page


nsoracle:
---------

   - Fixed bug when streaming LOB content to connection. The old code
     did not distinguish between binary and non-binary content.  This
     bug was discussed in
     https://openacs.org/forums/message-view?message_id=5693661

     Bumped version number to 2.9

- switched to plain Debug handling for debugging the driver

     The handling of Ns_LogSqlDebug is performed inside nsdb, including
     also the printout of (long) SQL statements. Previously, the driver
     was too chatty when Debug(sql) was turned on.

   - Added support for output columns of type SQLT_TIMESTAMP or SQLT_TIMESTAMP_TZ

     This change fixes a bug, where SQL queries of the form

          SELECT TO_TIMESTAMP(sysdate) FROM dual

     lead to errors for the form:

          Database operation "getrow" failed (exception 1406, "nsoracle.c:3659:Ns_OracleGetRow:
          error in `OCIStmtFetch ()': ORA-01406: fetched column value was truncated

     The driver needs for several output types special rules, where the
     timestamp cases were not supported so far. It is also recommended
     to set the according environment variables specifying the output
     format in the configuration server of NaviServer, such as e.g.

         set ::env(NLS_TIMESTAMP_FORMAT) "YYYY-MM-DD HH24:MI:SS.FF6"
         set ::env(NLS_TIMESTAMP_TZ_FORMAT) "YYYY-MM-DD HH24:MI:SS.FF6 TZH:TZM"

     For testing in you local Oracle installation, you might test the
     output formats (and the required sizes with the following snippet
     for sqlplus:

        COLUMN localtimestamp format a40
        COLUMN systimestamp format a40
        COLUMN ts_bytes format a80

        alter session set nls_timestamp_format='YYYY-MM-DD HH24:MI:SS.FF6';
        select localtimestamp, length(localtimestamp), dump(localtimestamp) ts_bytes from dual;

        alter session set nls_timestamp_tz_format='YYYY-MM-DD HH24:MI:SS.FF6 TZH:TZM';
        select systimestamp, length(systimestamp), dump(systimestamp) ts_bytes from dual;

        alter session set nls_timestamp_tz_format='YYYY-MM-DD HH24:MI:SS.FF6 TZR';
        select systimestamp, length(systimestamp), dump(systimestamp) ts_bytes from dual;


letsencrypt:
------------

   - Added option to produce certificates with ECDSA:

     Prior to this change, all certificates were using RSA keys.
     Since a while, keys based on elliptic curves became the preference
     of letsencrypt.


nsshell:
--------

    - Fixed a bug in "ns_conn" emulation, when the "kernel" was not correctly identified

Re: [naviserver-devel] Some questions about Docker images for Naviserver

[Brian F. <bri...@ai...>]

Hi

thanks for the response, that was really helpful to get a report of real-world usage. I think it's definitely worth our while exploring this further.

I'm currently trying to compile the Oracle driver on Vlad V's Docker image. I will post separately about an issue I'm having there.

thanks!
Brian
________________________________
From: Maksym Zinchenko <siq...@gm...>
Sent: Friday 31 March 2023 1:23 pm
To: Brian Fenton <bri...@ai...>
Cc: nav...@li... <nav...@li...>
Subject: Re: [naviserver-devel] Some questions about Docker images for Naviserver

Hello, I'm using Docker Naviserver in production, but it's not a big one. I can recommend it. In my case I'm using only Naviserver, not OpenACS. I have some images on docker hub under siqsuruq/naviserver, it's not official, but they work for me. They are based on a custom ubuntu image I made with Tcl. They are kinda big, about 300 mb but it works for me. About S6 and non S6, difference is only on type of init system used in the image, but i'm not pro at this question. The S6 version of Docker images uses an init system called s6-overlay, the non-S6 version of Docker images typically use a different init system, such as Systemd or SysVinit. In my case I'm using Postgres only, at host not in a Docker container. it looks like its working faster this way.
So, in general, you can use a Docker container with Naviserver without any problem.

Maksym

On Fri, 31 Mar 2023, 10:44 Brian Fenton, <bri...@ai...<mailto:bri...@ai...>> wrote:
Hello all

We're exploring reviving the Linux version of our Naviserver product. As part of this we'd like to investigate the feasibility of using Docker, initially for ease of development, and potentially later for production systems. I have a couple of questions that would be good to get the community's responses to.

1. Is there anyone using Docker Naviserver in production? If so would you recommend it, and are there any peculiarities that we should know about?

2. There are quite a few Naviserver images on on https://hub.docker.com, but I don't see an official one. It seems from posts on OpenACS.org <https://openacs.org/xowiki/setup-with-docker-s6> that Vlad V's build has had a lot of community input, and seems to be the de facto build. Would it be worth trying to get that version tagged as an official Naviserver Docker<https://docs.docker.com/docker-hub/official_images/> build to give newcomers clarity? I must admit to some confusion regarding the difference between the S6 and the non-S6 versions - should we as a community pick one as the official one?

3. For those of us using Oracle, what would be involved in getting Oracle driver support added as a build/run option? I'd be happy to help with this in any way I can.

thanks in advance
Brian

_______________________________________________
naviserver-devel mailing list
nav...@li...<mailto:nav...@li...>
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] Some questions about Docker images for Naviserver

[Gustaf N. <ne...@wu...>]

Here is some background, what's good about s6-overlay:
https://kreuzwerker.de/post/managing-multi-process-applications-in-containers-using-s6

In short: it is sufficiently simple to use and has a couple of useful 
utilities...

-g

On 31.03.23 14:23, Maksym Zinchenko wrote:
> Hello, I'm using Docker Naviserver in production, but it's not a big 
> one. I can recommend it. In my case I'm using only Naviserver, not 
> OpenACS. I have some images on docker hub under siqsuruq/naviserver, 
> it's not official, but they work for me. They are based on a custom 
> ubuntu image I made with Tcl. They are kinda big, about 300 mb but it 
> works for me. About S6 and non S6, difference is only on type of init 
> system used in the image, but i'm not pro at this question. The S6 
> version of Docker images uses an init system called s6-overlay, the 
> non-S6 version of Docker images typically use a different init system, 
> such as Systemd or SysVinit. In my case I'm using Postgres only, at 
> host not in a Docker container. it looks like its working faster this way.
> So, in general, you can use a Docker container with Naviserver without 
> any problem.
>
> Maksym
>
> On Fri, 31 Mar 2023, 10:44 Brian Fenton, 
> <bri...@ai...> wrote:
>
>     Hello all
>
>     We're exploring reviving the Linux version of our Naviserver
>     product. As part of this we'd like to investigate the feasibility
>     of using Docker, initially for ease of development, and
>     potentially later for production systems. I have a couple of
>     questions that would be good to get the community's responses to.
>
>     1. Is there anyone using Docker Naviserver in production? If so
>     would you recommend it, and are there any peculiarities that we
>     should know about?
>
>     2. There are quite a few Naviserver images on on
>     https://hub.docker.com, but I don't see an official one. It seems
>     from posts on OpenACS.org
>     <https://openacs.org/xowiki/setup-with-docker-s6>that Vlad V's
>     build has had a lot of community input, and seems to be the /de
>     facto /build. Would it be worth trying to get that version tagged
>     as an official Naviserver Docker
>     <https://docs.docker.com/docker-hub/official_images/> build to
>     give newcomers clarity? I must admit to some confusion regarding
>     the difference between the S6 and the non-S6 versions - should we
>     as a community pick one as the official one?
>
>     3. For those of us using Oracle, what would be involved in getting
>     Oracle driver support added as a build/run option? I'd be happy to
>     help with this in any way I can.
>
>     thanks in advance
>     Brian
>
>     _______________________________________________
>     naviserver-devel mailing list
>     nav...@li...
>     https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

-- 
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"

Re: [naviserver-devel] Some questions about Docker images for Naviserver

[Maksym Z. <siq...@gm...>]

Hello, I'm using Docker Naviserver in production, but it's not a big one. I
can recommend it. In my case I'm using only Naviserver, not OpenACS. I have
some images on docker hub under siqsuruq/naviserver, it's not official, but
they work for me. They are based on a custom ubuntu image I made with Tcl.
They are kinda big, about 300 mb but it works for me. About S6 and non S6,
difference is only on type of init system used in the image, but i'm not
pro at this question. The S6 version of Docker images uses an init system
called s6-overlay, the non-S6 version of Docker images typically use a
different init system, such as Systemd or SysVinit. In my case I'm using
Postgres only, at host not in a Docker container. it looks like its working
faster this way.
So, in general, you can use a Docker container with Naviserver without any
problem.

Maksym

On Fri, 31 Mar 2023, 10:44 Brian Fenton, <bri...@ai...>
wrote:

> Hello all
>
> We're exploring reviving the Linux version of our Naviserver product. As
> part of this we'd like to investigate the feasibility of using Docker,
> initially for ease of development, and potentially later for production
> systems. I have a couple of questions that would be good to get the
> community's responses to.
>
> 1. Is there anyone using Docker Naviserver in production? If so would you
> recommend it, and are there any peculiarities that we should know about?
>
> 2. There are quite a few Naviserver images on on https://hub.docker.com,
> but I don't see an official one. It seems from posts on OpenACS.org
> <https://openacs.org/xowiki/setup-with-docker-s6>that Vlad V's build has
> had a lot of community input, and seems to be the *de facto *build. Would
> it be worth trying to get that version tagged as an official Naviserver
> Docker <https://docs.docker.com/docker-hub/official_images/> build to
> give newcomers clarity? I must admit to some confusion regarding the
> difference between the S6 and the non-S6 versions - should we as a
> community pick one as the official one?
>
> 3. For those of us using Oracle, what would be involved in getting Oracle
> driver support added as a build/run option? I'd be happy to help with this
> in any way I can.
>
> thanks in advance
> Brian
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>

[naviserver-devel] Some questions about Docker images for Naviserver

[Brian F. <bri...@ai...>]

Hello all

We're exploring reviving the Linux version of our Naviserver product. As part of this we'd like to investigate the feasibility of using Docker, initially for ease of development, and potentially later for production systems. I have a couple of questions that would be good to get the community's responses to.

1. Is there anyone using Docker Naviserver in production? If so would you recommend it, and are there any peculiarities that we should know about?

2. There are quite a few Naviserver images on on https://hub.docker.com, but I don't see an official one. It seems from posts on OpenACS.org <https://openacs.org/xowiki/setup-with-docker-s6> that Vlad V's build has had a lot of community input, and seems to be the de facto build. Would it be worth trying to get that version tagged as an official Naviserver Docker<https://docs.docker.com/docker-hub/official_images/> build to give newcomers clarity? I must admit to some confusion regarding the difference between the S6 and the non-S6 versions - should we as a community pick one as the official one?

3. For those of us using Oracle, what would be involved in getting Oracle driver support added as a build/run option? I'd be happy to help with this in any way I can.

thanks in advance
Brian

[naviserver-devel] Support for Argon2

[Gustaf N. <ne...@wu...>]

Dear all,

For all security hungry NaviServer users:

NaviServer supports now Argon2, which is currently the best known
password hashing function (well more than this, it is a key derivation
algorithm). For details, see [1]. It may take still some time until
OpenSSL 3.2 is available in the main Linux distributions.

With this change, NaviServer provides direct support for the two most
recommended password hashing algorithms of the OWASP project [2],
namely Argon2 and scrypt, along with SCRAM-sha-256 (actually PBKDF2)
which is the most secure algorithm implemented in PostgreSQL.

All the best

-g

[1] 
https://bitbucket.org/naviserver/naviserver/commits/4d634d54b77d1ce6b61f07944871f3dcf1a330a5
[2] 
https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html#password-hashing-algorithms

Re: [naviserver-devel] nscp segfaults server when missing module/nscp/users

[Gustaf N. <ne...@wu...>]

Thanks for the report. The problem could be repeated and is fixed in the
public repository [1]. The problem was introduced in 2022-07-11, but
not included in any released version.

all the best -g

[1] 
https://bitbucket.org/naviserver/naviserver/commits/168eebbe67d0d631df5948e1647a36e6e0d9baf0

On 18.03.23 20:07, Andrew Piskorski wrote:
> Probably nscp shouldn't segfault if it's missing that section?

[naviserver-devel] nscp segfaults server when missing module/nscp/users

[Andrew P. <at...@pi...>]

Using the Naviserver head on Linux, the installed "conf/nsd-config.tcl"
and "conf/simple-config.tcl" config fieles both work fine for testing
that a newly compliled server starts up correctly.  However, simple-config.tcl
includes a commented-out line loading the nscp module, like so:

    #ns_param   nscp            nscp

If you uncomment that line, then the server crashes every time on startup.
If you ALSO copy this "module/nscp/users" section from nsd-config.tcl,
then the crash goes away and server again starts fine:

ns_section ns/server/default/module/nscp/users {
    ns_param user "::"
}

Probably nscp shouldn't segfault if it's missing that section?  Or
maybe we should either just add that section into simple-config.tcl,
or remove the commented-out nscp line entirely.

When the server crashes because of the missing nscp configuration, the
backtrace looked like this:

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007f4a13c2c859 in __GI_abort () at abort.c:79
#2  0x00007f4a13ea4288 in Panic () from /usr/local/pkg/nsd-20230317-1/lib/libnsd.so
#3  0x00007f4a13af358d in Tcl_PanicVA (format=<optimized out>, argList=argList@entry=0x7ffcecfc0b40) at /home/nobackup/co/tcl/tcl-core-8-6-branch/generic/tclPanic.c:99
#4  0x00007f4a13af36ff in Tcl_Panic (format=<optimized out>) at /home/nobackup/co/tcl/tcl-core-8-6-branch/generic/tclPanic.c:160
#5  0x00007f4a13f096c7 in Abort () from /usr/local/pkg/nsd-20230317-1/lib/libnsd.so
#6  <signal handler called>
#7  0x00007f4a13ebd18d in Ns_SetFindCmp () from /usr/local/pkg/nsd-20230317-1/lib/libnsd.so
#8  0x00007f4a13ebd287 in Ns_SetFind () from /usr/local/pkg/nsd-20230317-1/lib/libnsd.so
#9  0x00007f4a13ebc951 in Ns_SetUpdateSz () from /usr/local/pkg/nsd-20230317-1/lib/libnsd.so
#10 0x00007f4a13e22a6e in LoadUsers () from /home/local-20.04/pkg/nsd-20230317-1/bin/nscp.so
#11 0x00007f4a13e22eb0 in Ns_ModuleInit () from /home/local-20.04/pkg/nsd-20230317-1/bin/nscp.so
#12 0x00007f4a13ea6e61 in Ns_ModuleLoad () from /usr/local/pkg/nsd-20230317-1/lib/libnsd.so
#13 0x00007f4a13ea7141 in NsTclModuleLoadObjCmd () from /usr/local/pkg/nsd-20230317-1/lib/libnsd.so

-- 
Andrew Piskorski <at...@pi...>

Re: [naviserver-devel] test ns_config-7.4.1, Stack around the variable 'filter' was corrupted

[Andrew P. <at...@pi...>]

On Thu, Mar 16, 2023 at 02:36:27PM -0400, Andrew Piskorski wrote:
> > >   Debug Error!
> > >   Program:  C:\web\ns-fork-pub\naviserver\nsd\libnsd.dll
> > >   Run-Time Check Failure #2 - Stack around the variable 'filter' was corrupted.

> I put the same code that the ns_config-7.4.1 test runs into a simple
> *.tcl page like this:
> 
>   set xx [ns_set array [ns_configsection -filter "unread" ns/testconfig]]
>   ns_return 200 {text/plain} "Result:  $xx"

I found this helpful article on trying to find the source of stack
corruption via memory access breakpoints:
 
  https://www.timdbg.com/posts/debugger-lies-part-1/#memory-access-breakpoints

Below is my attempt to do that; I got stuck without clear results.
My memory access breakpoint fired, but the stack trace said we were
still in NsTclConfigSectionObjCmd(), seemingly still at the beginning
of that function, with just this cryptic (to me) output:

  (41d0.f7c): Break instruction exception - code 80000003 (first chance)
  libnsd!failwithmessage+0x234:
  00007ff8`c7101364 cc              int     3

I'm not very skilled at using WinDbg, so it's likely I missed or
misinterpreted something.


Here's what I did to get that far:
------------------------------------------------------------
## I downloaded and installed Windows SDK 10.0.22621:
https://developer.microsoft.com/en-us/windows/downloads/windows-sdk/

## That gave me WinDbg installed here:
C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\windbg.exe

## Build NaviServer on Windows my usual way:
"C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvars64.bat"
z: & cd Z:\src\web\ns-fork-pub\naviserver
nmake -f Makefile.win32 clean-core clean-mod
nmake -f Makefile.win32 all-core all-mod
# Fix permissions if necessary!  e.g.:
#   find . -type f \( -name "*.exe" -o -name "*.dll" \) -print | sudo xargs chmod 775
nmake -f Makefile.win32 _install
nmake -f Makefile.win32 _test

## Start up nsd.exe under WinDbg:
"C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\windbg.exe" -g C:\P\nsd\nsd-fp_2023-03-15-2\bin\nsd.exe -t Z:/src/web/ns-fork-pub/naviserver/tests/test.nscfg

## Tell WinDbg about my Naviserver code:
.srcpath Z:\src\web\ns-fork-pub\naviserver
.sympath C:\P\nsd\nsd-fp_2023-03-15-2\lib;C:\P\Tcl-64-8.6.12\lib;SRV*C:\MyLocalSymbols*http://msdl.microsoft.com/download/symbols
!sym noisy
lml
.reload libnsd.dll
.reload nsd.exe

## Set breakpoint 0 on NsTclConfigSectionObjCmd():
bp NsTclConfigSectionObjCmd
g

## Trigger the bug.  I like to use Cygwin like so:
wget -O - -q http://localhost:8000/atp-crash.tcl

## Now in WinDbg were are in NsTclConfigSectionObjCmd().
## Show stack:
k
## Show current stack pointer:
dx @$csp
## Set memory access breakpoint 1 on the stack pointer:
ba w 8 @$csp
bl
g

## This is the cryptic output I got:
(41d0.f7c): Break instruction exception - code 80000003 (first chance)
libnsd!failwithmessage+0x234:
00007ff8`c7101364 cc              int     3

## Disable the stack pointer breakpoint:
bd 1
------------------------------------------------------------

-- 
Andrew Piskorski <at...@pi...>

Re: [naviserver-devel] test ns_config-7.4.1, Stack around the variable 'filter' was corrupted

[Andrew P. <at...@pi...>]

On Wed, Mar 15, 2023 at 03:57:40PM -0400, Andrew Piskorski wrote:
> On Tue, Mar 14, 2023 at 04:44:21PM -0400, Andrew Piskorski wrote:
> 
> >   Debug Error!
> >   Program:  C:\web\ns-fork-pub\naviserver\nsd\libnsd.dll
> >   Run-Time Check Failure #2 - Stack around the variable 'filter' was corrupted.

> On Windows, running the "ns_config.test" tests triggers that one.  All
> the tests through ns_config-7.4.0, pass, then it stops with no further
> output.  Yep, the next test, ns_config-7.4.1, is sufficient to trigger
> the problem all by itself.

I put the same code that the ns_config-7.4.1 test runs into a simple
*.tcl page like this:

  set xx [ns_set array [ns_configsection -filter "unread" ns/testconfig]]
  ns_return 200 {text/plain} "Result:  $xx"

Hitting that web page, I the ns_return never runs, because nsd.exe has
broken before getting that far.

WinDbg says the stack corruption is happening in NsTclConfigSectionObjCmd()
(in "nsd/tclconf.c"), but it doesn't notice until it gets to the end
of that function.  I think most of the work there is in Ns_ParseObjv(),
so maybe that, or something it calls, is the most likely place for
some sort of array bounds overrun to be hiding.  I don't know though,
it could be anything that NsTclConfigSectionObjCmd() calls.

I don't know how else to further track down this bug, but if anybody
has further suggestions I'm willing to try them.

-- 
Andrew Piskorski <at...@pi...>