naviserver-devel — Developer discussion

[naviserver-devel] Repository work: Renaming default branch from "master" to "main"

[Gustaf N. <ne...@wu...>]

Dear all, The default branch of NaviServer was changed from "master" to 
"main". I have done this for the "naviserver" main repository and for 
the 51 modules below. Fresh checkouts are now on the branch "main", old 
checkouts should switch to the main branch after a "git pull" with: git 
branch main Since git 2.28, one can also change the default branch with 
the built-in git command: git config --global init.defaultBranch main We 
were actually quite late with this change. Details for the main branch 
reaming are here: 
https://dev.to/lukeocodes/change-git-s-default-branch-from-master-19le 
All the best -g Changed modules: letsencrypt nsaccess nsaspell nsauthpam 
nschartdir nsclamav nscoap nsconf nsdbbdb nsdbi nsdbilite nsdbimy 
nsdbipg nsdbmysql nsdbpg nsdbsqlite nsdbtds nsdhcpd nsdns nsexample 
nsexpat nsfortune nsgdchart nsicmp nsimap nsldap nsldapd nsloopctl 
nsmemcache nsocaml nsodbc nsoracle nsphp nsradiusd nsrtsp nssavi nsshell 
nssip nssmtpd nssnmp nsstats nssys nssyslogd nstftpd nstk nsudp nsvfs 
nswebpush nszlib revproxy websocket

Re: [naviserver-devel] ns_returnfile return original file name

[Oscar R. F. <oro...@vr...>]

I didn't know that.

It seems that most browsers are quite benevolent parsing this with
wrong space character encoding.

I have to update my code. 

Thank you very much! 

-----Mensaje original-----
De: Gustaf Neumann <ne...@wu...>
Responder a: nav...@li...
Para: nav...@li...
Asunto: Re: [naviserver-devel] ns_returnfile return original file
name
Fecha: Wed, 20 Jul 2022 22:08:05 +0200

Dear Oscar,
In principle, you are right concerning the encoding. A few comments:

- on most browsers, also utf-8 filenames are accepted,
   but the use of UTF-8 in the header fields is not recommended.

- For the percent encoding, one should use "ns_urlencode -path",
e.g., 

         ns_urlencode -part path {£ and € rates}

   since the "ns_urlencode" without parameters (as in your example)
uses a special rule for the space character.
One should actually provide an API along the lines of:
    ns_header_field_parameter -charset /charset/ -language /language/
name string
to ease its usage. ... maybe something for the next release...
All the best
-g
PS: Minor nitpick: RFC 5987 was obsoleted by RFC 8187
On 19.07.22 23:25, Oscar Rodriguez Fonseca wrote:
 
> Dear all,
> 
> Just for the sake of completeness. If you need an utf-8 encoded
> filename you may need to encode the filename (as per RFC 5987):
> 
> ns_set update [ns_conn outputheaders] Content-Disposition
> "attachment; filename*=UTF-8''[ns_urlencode $filename]"
> 
> ns_set update [ns_conn outputheaders] Content-Disposition
> "filename*=UTF-8''[ns_urlencode $filename]"
> 
> AFAIK it works well in any modern browser.
> 
> Best regards.
> 
> 
> -----Mensaje original-----
> De: Maksym Zinchenko <siq...@gm...>
> Responder a: nav...@li...
> Para: nav...@li...
> Asunto: Re: [naviserver-devel] ns_returnfile return original file
> name
> Fecha: Tue, 19 Jul 2022 16:33:58 -0100
> 
> Thank you all, that's exactly what I needed.
>  
> On Mon, Jul 18, 2022 at 1:00 PM Wolfgang Winkler via naviserver-
> devel <nav...@li...> wrote:
>  
> > Hi!
> > We use
> > ns_set update [ns_conn outputheaders] Content-Disposition
> > "attachment; filename=\"${filename}\""
> > for downloading and
> > ns_set update [ns_conn outputheaders] Content-Disposition
> > "filename=\"${filename}\""
> > for viewing files
> > Am 15.07.22 um 16:50 schrieb Maksym Zinchenko:
> >  
> >  
> > > Hello, I have a question about how to return the original file
> > > name. For example:
> > > 
> > > ns_register_proc GET /dev/rtrn_file ::dev::rtrn_file
> > >  proc rtrn_file {args} {
> > >  set f [file join /tmp test.csv]
> > >  ns_returnfile 200 [ns_guesstype "$f"] $f
> > >  }
> > > 
> > > When I do GET request  I'm getting "rtrn_file.csv" instead of
> > > "test.csv"
>  
>  
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
_______________________________________________
naviserver-devel mailing list
nav...@li...
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] ns_returnfile return original file name

[Gustaf N. <ne...@wu...>]

Dear Oscar,

In principle, you are right concerning the encoding. A few comments:

- on most browsers, also utf-8 filenames are accepted,
    but the use of UTF-8 in the header fields is not recommended.

- For the percent encoding, one should use "ns_urlencode -path", e.g.,

          ns_urlencode -part path {£ and € rates}

    since the "ns_urlencode" without parameters (as in your example) 
uses a special rule for the space character.

One should actually provide an API along the lines of:

     ns_header_field_parameter -charset /charset/ -language /language/ 
name string

to ease its usage. ... maybe something for the next release...

All the best

-g

PS: Minor nitpick: RFC 5987 was obsoleted by RFC 8187

On 19.07.22 23:25, Oscar Rodriguez Fonseca wrote:
> Dear all,
>
> Just for the sake of completeness. If you need an utf-8 encoded 
> filename you may need to encode the filename (as per RFC 5987):
>
> ns_set update [ns_conn outputheaders] Content-Disposition "attachment; 
> filename*=UTF-8''[ns_urlencode $filename]"
>
> ns_set update [ns_conn outputheaders] Content-Disposition 
> "filename*=UTF-8''[ns_urlencode $filename]"
>
> AFAIK it works well in any modern browser.
>
> Best regards.
>
>
> -----Mensaje original-----
> *De*: Maksym Zinchenko <siq...@gm... 
> <mailto:Maksym%20Zinchenko%20%3cs...@gm...%3e>>
> *Responder a*: nav...@li...
> *Para*: nav...@li...
> *Asunto*: Re: [naviserver-devel] ns_returnfile return original file name
> *Fecha*: Tue, 19 Jul 2022 16:33:58 -0100
>
> Thank you all, that's exactly what I needed.
>
> On Mon, Jul 18, 2022 at 1:00 PM Wolfgang Winkler via naviserver-devel 
> <nav...@li...> wrote:
>>
>> Hi!
>>
>> We use
>>
>> ns_set update [ns_conn outputheaders] Content-Disposition 
>> "attachment; filename=\"${filename}\""
>>
>> for downloading and
>>
>> ns_set update [ns_conn outputheaders] Content-Disposition 
>> "filename=\"${filename}\""
>>
>> for viewing files
>>
>> Am 15.07.22 um 16:50 schrieb Maksym Zinchenko:
>>
>>> Hello, I have a question about how to return the original file name. 
>>> For example:
>>>
>>> ns_register_proc GET /dev/rtrn_file ::dev::rtrn_file
>>> proc rtrn_file {args} {
>>> set f [file join /tmp test.csv]
>>> ns_returnfile 200 [ns_guesstype "$f"] $f
>>> }
>>>
>>> When I do GET request  I'm getting "rtrn_file.csv" instead of "test.csv"
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

-- 
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"

[naviserver-devel] ns_set reform

[Gustaf N. <ne...@wu...>]

Dear all,

over the last days, I've worked on improving scalability of ns_set 
operations in NaviServer, which are used on multiple crucial places such as:

  - ndsb interface (returning tuples as ns_sets)
  - configuration values
  - headers

The classical implementation for ns_sets uses separately malloced 
storage for every attribute name and attribute value. So, e.g., for 1000 
ns_sets with 20 members each, this means 1,000*20*2 = 40,000 malloc/free 
operations, e.g. for a single db query! Although the malloc 
implementations have improved, these will require many lock operations, 
especially under load, where many threads might do as much malloc 
operation. One other consequence is that the allocated memory will be 
scattered over address space, which has bad implications for CPU caching.

The new implementation uses a single Tcl_DString per ns_set keeping all 
attribute names and attribute values. This reduces the malloc operations 
and improved memory locality, such that cache hits will improve.

One caveat is that modules using ns_set have to be recompiled, since the 
full data structure of the ns_set is exposed, and adding a member causes 
a binary incompatibility. One other potential problem is that C-level 
modules using the Ns_Set* API have to make sure that long-living string 
values are copied. Copying was necessary in general before as well, but 
might show up now earlier, when the ns_set sees multiple updates. There 
was one place, inside NaviServer, where a change was necessary. All of 
OpenACS is working fine with these changes, openacs.org runs already a 
version having this feature enabled.

However, since this is technically a large change, the initial commit 
will have this feature deactivated by default (flag NS_SET_DSTRING). My 
plan is to have this feature by default
deactivated for the 4.99* releases, but having it enabled for version 5 
of NaviServer. Other goals for NS5 are Tcl9 compatibility (will require 
source code changes; NSF/XOTcl is already Tcl9 compatible) and the move 
from MPL 1.1 to 2.0 (as previously [1] announced).

Here are some preliminary results from the ns_set reform.  The tests 
were performed on openacs.org (Xeon Gold 6226R CPU @ 2.90GHz, 32 cores, 
hyper-threading enabled). The test executes the SQL query

     select * from acs_objects limit 1000

100 times in sequence. This test is run in 1..30 concurrent threads. 
With 30 threads, 3mio tuples are retrieved, and 72 mio malloc/free 
operations are needed alone for the retrieved values.

Before (classical ns_set with many mallocs):

     threads 1 total 4606.787 ms avg 3285.25 ms
     threads 5 total 4595.358 ms avg 3493.07 ms
     threads 10 total 4804.193 ms avg 3755.93 ms
     threads 20 total 6279.524 ms avg 4569.16 ms
     threads 30 total 8966.427 ms avg 6618.58 ms

After reform (using common Tcl_DString per tuple):

     threads 1 total 4524.645 ms avg 3242.54 ms
     threads 5 total 4251.266 ms avg 3450.09 ms
     threads 10 total 4656.795 ms avg 3665.31 ms
     threads 20 total 5934.105 ms avg 4671.38 ms
     threads 30 total 7384.591 ms avg 5642.76 ms

As one can see,  the improvement increases under higher load (with more 
parallel threads). E.g. with 30 threads, the total time improved by 
17%.... with a smaller RSS. These tests were not performed under 
"clinical" conditions.

While working on ns_set and OpenACS db operations, having substantial 
debugging activated, I've optimized the database operations further, 
such that on high load and large queries, the performance can be now 
several times (!) faster by avoiding duplicates in the memory (but this 
is more an OpenACS topic).

More changes:

Better resource reuse:
- keep Ns_Sets for request and response headers instead of
   deleting/recreating it frequently

API extensions:
- Provide new interface ending with *Sz to provide string sizes.
   This reduces the need of strlen() operations.
   * Ns_SetCreateSz()
   * Ns_SetIUpdateSz()
   * Ns_SetPutSz()
   * Ns_SetPutValueSz()
   * Ns_SetUpdateSz()

- New severity Debug(nsset) for ns_set debugging added, maybe temporary

- New API calls
   * Ns_SetClearValues(): clear the values for all keys
   * Ns_SetDataPrealloc(): creating ns_sets with pre-allocated values
     to avoid resize operations
   * NsSetResize()
   * NsHeaderSetGet()
- Ns_ConfigSet(const char *section, const char *key, const char *name)
   The last argument is now and allows one to created named sets 
(previously, all
   such sets were unnamed)

- Tcl API:
   * provide optional values to "ns_set size" for pre-allocation

- extended regression test

All the best
-g

Re: [naviserver-devel] ns_returnfile return original file name

[Oscar R. F. <oro...@vr...>]

Dear all,

Just for the sake of completeness. If you need an utf-8 encoded
filename you may need to encode the filename (as per RFC 5987):

ns_set update [ns_conn outputheaders] Content-Disposition
"attachment; filename*=UTF-8''[ns_urlencode $filename]"

ns_set update [ns_conn outputheaders] Content-Disposition
"filename*=UTF-8''[ns_urlencode $filename]"

AFAIK it works well in any modern browser.

Best regards.


-----Mensaje original-----
De: Maksym Zinchenko <siq...@gm...>
Responder a: nav...@li...
Para: nav...@li...
Asunto: Re: [naviserver-devel] ns_returnfile return original file
name
Fecha: Tue, 19 Jul 2022 16:33:58 -0100

Thank you all, that's exactly what I needed.

On Mon, Jul 18, 2022 at 1:00 PM Wolfgang Winkler via naviserver-devel
<nav...@li...> wrote:
> Hi!
> We use
> ns_set update [ns_conn outputheaders] Content-Disposition
> "attachment; filename=\"${filename}\""
> for downloading and
> ns_set update [ns_conn outputheaders] Content-Disposition
> "filename=\"${filename}\""
> for viewing files
> Am 15.07.22 um 16:50 schrieb Maksym Zinchenko:
>  
> > Hello, I have a question about how to return the original file
> > name. For example:
> > 
> > ns_register_proc GET /dev/rtrn_file ::dev::rtrn_file
> > proc rtrn_file {args} {
> > set f [file join /tmp test.csv]
> > ns_returnfile 200 [ns_guesstype "$f"] $f
> > }
> > 
> > When I do GET request  I'm getting "rtrn_file.csv" instead of
> > "test.csv"

Re: [naviserver-devel] ns_returnfile return original file name

[Maksym Z. <siq...@gm...>]

Thank you all, that's exactly what I needed.

On Mon, Jul 18, 2022 at 1:00 PM Wolfgang Winkler via naviserver-devel <
nav...@li...> wrote:

> Hi!
>
> We use
>
> ns_set update [ns_conn outputheaders] Content-Disposition "attachment;
> filename=\"${filename}\""
>
> for downloading and
>
> ns_set update [ns_conn outputheaders] Content-Disposition
> "filename=\"${filename}\""
>
> for viewing files
>
>
> Am 15.07.22 um 16:50 schrieb Maksym Zinchenko:
>
> Hello, I have a question about how to return the original file name. For
> example:
>
> ns_register_proc GET /dev/rtrn_file ::dev::rtrn_file
> proc rtrn_file {args} {
> set f [file join /tmp test.csv]
> ns_returnfile 200 [ns_guesstype "$f"] $f
> }
>
> When I do GET request  I'm getting "rtrn_file.csv" instead of "test.csv"
>
> Maybe I'm doing it wrong.
>
> Thank you.
>
>
> _______________________________________________
> naviserver-devel mailing lis...@li...://lists.sourceforge.net/lists/listinfo/naviserver-devel
>
> --
>
> *Wolfgang Winkler*
> Geschäftsführung
> wol...@di...
> mobil +43.699.19971172
>
> dc:*büro*
> digital concepts Novak Winkler OG
> Software & Design
> Landstraße 68, 5. Stock, 4020 Linz
> www.digital-concepts.com
> tel +43.732.997117.72
> tel +43.699.1997117.2
>
> Firmenbuchnummer: 192003h
> Firmenbuchgericht: Landesgericht Linz
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>

Re: [naviserver-devel] ns_returnfile return original file name

[Wolfgang W. <wol...@di...>]

Hi!

We use

ns_set update [ns_conn outputheaders] Content-Disposition "attachment; 
filename=\"${filename}\""

for downloading and

ns_set update [ns_conn outputheaders] Content-Disposition 
"filename=\"${filename}\""

for viewing files


Am 15.07.22 um 16:50 schrieb Maksym Zinchenko:
> Hello, I have a question about how to return the original file name. 
> For example:
>
> ns_register_proc GET /dev/rtrn_file ::dev::rtrn_file
> proc rtrn_file {args} {
> set f [file join /tmp test.csv]
> ns_returnfile 200 [ns_guesstype "$f"] $f
> }
>
> When I do GET request  I'm getting "rtrn_file.csv" instead of "test.csv"
>
> Maybe I'm doing it wrong.
>
> Thank you.
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
-- 

*Wolfgang Winkler*
Geschäftsführung
wol...@di...
mobil +43.699.19971172

dc:*büro*
digital concepts Novak Winkler OG
Software & Design
Landstraße 68, 5. Stock, 4020 Linz
www.digital-concepts.com <http://www.digital-concepts.com>
tel +43.732.997117.72
tel +43.699.1997117.2

Firmenbuchnummer: 192003h
Firmenbuchgericht: Landesgericht Linz

Re: [naviserver-devel] ns_returnfile return original file name

[Gustaf N. <ne...@wu...>]

Dear Maksym,

If no name is provided with the download, the browser can provide a 
name; the behavior might be different dependent on the browser.
Therefore, it is recommended to provide the name via the return header 
field "Content-Disposition", ... or via the "download" attribute in the 
link. For details, see [1]

all the best

-gn

[1] 
https://stackoverflow.com/questions/1628260/downloading-a-file-with-a-different-name-to-the-stored-name


ns_register_proc GET /dev/rtrn_file ::dev::rtrn_file

namespace eval ::dev {
   proc rtrn_file {args} {
      ns_set put [ns_conn outputheaders] Content-Disposition {attachment; filename="test.csv"}
      set f [file join /tmp test.csv]
      ns_returnfile 200 [ns_guesstype $f] $f
   }
}

[naviserver-devel] ns_returnfile return original file name

[Maksym Z. <siq...@gm...>]

Hello, I have a question about how to return the original file name. For
example:

ns_register_proc GET /dev/rtrn_file ::dev::rtrn_file
proc rtrn_file {args} {
set f [file join /tmp test.csv]
ns_returnfile 200 [ns_guesstype "$f"] $f
}

When I do GET request  I'm getting "rtrn_file.csv" instead of "test.csv"

Maybe I'm doing it wrong.

Thank you.

[naviserver-devel] Improved introspection of configuration parameters

[Gustaf N. <ne...@wu...>]

Dear all,

One of the feature of the NaviServer configuration file is that these 
are extensible, i.e. that arbitrary values can also be added for 
application purposes. However, this flexibility comes with the price 
that it is sometimes a problem to detect typos in the configuration 
file, or obsolete settings (e.g. some names from AOLserverver), etc.

The new version on Bitbucket addresses this disadvantage by supporting 
now additional information about configuration values, such as
- value read during startup
- value is defaulted (was not set in the configuration file)
- default of a value.

The information is provided via tool-tips and color coding in nsstats:
- black: value is set from the configuration file (showing the default 
between parenthesis when non-empty)
- gray: value is defaulted
- orange: value was set in the configuration file, but is identical to 
the default (not necessary to set it)
- red: value was not read during startup (not necessarily wrong, but 
might hint to a problem)

Please note, that in order to use this new feature, also the nsstats 
module has to be updated

all the best

-g

[naviserver-devel] OpenACS and EuroTcl conference presentations

[Gustaf N. <ne...@wu...>]

Dear all,

The presentations are now on the conference website, including video and 
slides (on the program page). There are also photo impressions from the 
conference (on the starting page).

    https://openacs.org/conf2022/info/

all the best
-g

[naviserver-devel] OpenACS and EuroTcl Conference Program and Live Streams

[Gustaf N. <ne...@wu...>]

Dear all

The program of the EuroTcl and OpenACS conference starting tomorrow is 
here:
https://openacs.org/conf2022/info/schedule

Live-streams:
Talks on June 30: 
https://learn.wu.ac.at/eurotcl2022/calendar/cal-item-view?cal_item_id=613703025
Talks on July 1: 
https://learn.wu.ac.at/eurotcl2022/calendar/cal-item-view?cal_item_id=613704587

Note, that the times are Vienna local time (CEST)

all the best

-g

-- 
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"

[naviserver-devel] NaviServer 4.99.24 available

[Gustaf N. <ne...@wu...>]

Dear all,

The release of NaviServer 4.99.24 [1] is available at sourceforge. See 
below for the
list of changes.

The code was tested with Ubuntu 20.04, Rocky Linux 8.5, FreeBSD 13.1, 
macOS 11.6.6 (Intel).

Many thanks to the contributors of this release:

      Antonio Pisano
      David Osborne
      Gustaf Neumann
      Zoran Vasiljevic

All the best!

-gustaf neumann

[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.24/

=======================================
NaviServer 4.99.24, released 2022-06-14
=======================================

  77 files changed, 3242 insertions(+), 1100 deletions(-)

New Features:

  - Improved security

    * Added protection against certain attacks in ns_dbquotevalue

      Due to the corrected conversion to external UTF-8 in db-output,
      new potential attack vectors appeared that were protected earlier
      via the Tcl-internal 'modified UTF-8'. E.g., the binary null
      character is stored as an overlong (two-byte) encoding of null
      (0xc0 0x80), so that an actual (embedded) null byte (0x00) never
      appears in the string. Due to the conversion, the internal
      representation is translated back to the binary null
      character. Embedded null byte characters can lead to
      non-terminated string literals via ns_dbquotevalue.

      In the updated version of NaviServer, ns_dbquotevalue raises an
      exception when this occurs. Therefore, the function can be used
      as well as an input checker (together with "try").

    * Raise an exception when trying to use "ns_urldecode" to
      produce invalid UTF-8

      Background: several (external) functions expect valid UTF-8 to be
      passed in and crash if this is not the case. One such example is
      tDOM. These nasty byte sequences are used more intensively by
      vulnerability scanners.

      Therefore, ns_urldecode raises now an exception, when it tries to
      convert to invalid UTF-8. It is still possible to use
      ns_urldecode to convert to other charsets.

         ns_urldecode -charset iso8859-1 -part path "/mot%C3or"

      When urldecode() is called internally and would produce invalid
      UTF-8, it truncates the string (and writes a warning to the
      system log).

      Note that the enw fallback charset

  - Fallback charsets

    In case, a conversion to UTF-8 fails due to invalid byte sequences,
    one can now provide a fallback charset for a second attempt of
    decoding this data. This feature is useful for websites that
    have to deal with requests containing invalid (form) data, typically
    from legacy applications.

    The fallback charset can be provided as optional parameter
    "-fallbackcharset" for the command "ns_getform", "ns_parsequery"
    and "ns_urldecode":

         ns_getform ?-fallbackcharset fallbackcharset? ?charset?
         ns_parsequery ?-charset charset? ?-fallbackcharset fallbackcharset? ?--? querystring
         ns_urldecode ?-charset charset? ?-fallbackcharset fallbackcharset? ?-part part? ?--? string

    In case, the parameter is not specified, it can be also be provided
    to the form-processing commands "ns_getform" and "ns_parsequery" via
    configuration variables:

      * per-server configuration parameter "formFallbackCharset"
        (in the section "ns/server/$server"), or as

      * global server configuration parameter "formFallbackCharset"
        (in the section "ns/parameters").

    The highest precedence has the optional parameter, followed by the
    per-server configuration parameter and the global configuration
    parameter.


  - Provide a hint when cache-entry is too large for caching

    Background: the size of the entry is typically determined after the
    execution of a potentially expensive query. During the eval of the
    command, the cache entry is locked and forces a
    serialization. However, this means that in these cases the
    situation is worse than without a cache, where some queries can be
    executed in parallel. We faced the situation of an expected
    slowdown of the server with many "create entry collision", where
    due to application matters, an entry was becoming too large. This
    situation is not easy to debug, especially in stress
    situations. The log message would have helped a log to identify the
    cause.

  - Added support for multibyte numeric entities

    This change supports conversion of numeric entities representing
    multibyte characters into HTML in "ns_striphtml" and
    "ns_unquotehtml". Technically, the numeric entities represent
    Unicode code points, which are transformed into UTF-8
    serialization.  Every entity represents a single code point; The
    values can be provided in decimal or hexadecimal notation. Before
    this change, only single byte numeric entities were
    supported. ASCII control characters (decimal 0-31) are ignored as
    before.

  - New and extended commands:

    * ns_unquotehtml /text/

      This command is the inverse operation of "ns_quotehtml". It
      replaces the named and numeric entities in the provided string
      with the native values. The command is similar to "ns_striphtml",
      but "ns_striphtml" removes as well other HTML markup which might
      not be desired in all cases.

      This change fixes as well a bug with numeric entities (the old
      code assumed, these are starting directly with a number after the
      ampersand) and it adds support for numeric entities with
      hexadecimal values (so far with the same value range as for
      decimal numeric entities).

    * ns_subnetmatch /subnet/ /ipaddr/

      Determine, if a provided IP address (IPv4 or IPv6) is included in
      a subnet specification, which is provided in CIDR notation. The
      command makes internal NaviServer functionality available at the
      Tcl level.  The regression test was extended to cover the
      functionality.

      The command ns_subnetmatch validates the provided subnet
      specification (IPv4 or IPv6 address followed by slash and number
      of significant bits) and the provided IP address and tests whether
      the IP address is in the implied range. The command returns a
      boolean value as the result. When comparing an IPv4 and IPv6
      address/CIDR specification or vice versa, the result is always
      false.

      The function can be use when e.g. restricting access to certain
      functionality to some subnets. The function can be used as well
      to check, whether an IP address is an IPv4 or IPv6 address.

       Examples:
         % ns_subnetmatch 137.208.0.0/16 137.208.116.31
         1

         % ns_subnetmatch 137.208.0.0/16 112.207.16.33
         0

         % ns_subnetmatch 2001:628:404:74::31/64 [ns_conn peeraddr]
         ...

         # Is IP address a valid IPv6 address?
         % ns_subnetmatch ::/0 $ip

         # Is IP address a valid IPv4 address?
         % ns_subnetmatch 0.0.0.0/0 $ip

    * ns_connchan: Added new subcommand "ns_connchan connect"

      "ns_connchan connect" is similar to "ns_connchan open", except
      that it does not send an HTTP request (HTTP method, URL, and
      header fields) but just opens the connection. It can be used for
      some non-HTTP communication over TCP and TLS over the ns_connchan
      infrastructure.

    * ns_parseheader, Ns_ParseHeader(): return the field
      number (index) of the parsed entry

      Previously, there was no explicit feedback, what field of an
      "ns_set" has been parsed by "ns_parseheader". Now, in success
      cases, the function returns the index of the new/modified
      entry. This function made it possible to generalize and simplify
      the Tcl-level parsing of "multipart/form-data"
      significantly. Additionally, a new optional argument "-prefix"
      was added. When specified, it adds the specified prefix to the
      key.

    * ns_setcookie, ns_deletecookie

      Mozilla and Chrome changed the default value for SameSite of
      cookies from "none" to "lax" in February and Aug 2020. Cookies
      that explicitly set SameSite=None must also set the "Secure"
      attribute.

      In order to mirror this change of policy in NaviServer and to
      reduce necessary code changes, the default behavior for setting
      or deleting cookie is now samesite "lax" (when "-samesite" is not
      explicitly specified).  When trying to set a cookie with
      "-samesite none" without the "-secure" flag, a warning is
      generated, and the "-samesite lax" is assumed, since major
      browsers announced that these will reject these cookies soon.


API changes:

  - ns_getform, ns_parsequery, and ns_urldecode

    New optional parameter "-fallbackcharset". See above for details.

- ns_parsequery: added option "-charset" and raise exception on failure

    The new option "-charset" can be used to add a charset for the
    result encoding of the passed-in HTTP query.

    In case the charset is UTF8 (default on most platforms), and the
    content is invalid UTF-8, an exception is raised (similar to
    ns_urldecode). This can be addressed by parameter
    "-fallbackcharset" (See above for details).

  - ns_deletecookie: added support "-samesite" flag for ns_deletecookie.

    Since "ns_deletecookie" sets internally a cookie, some browsers
    might ignore in the future certain cookie requests (e.g. when
    "-samesite" is not used or set to "none" on non-secure connections).

  - ns_trim enhancements:
    The new option "-prefix ..." can be used to strip a string
    (such as ">> ") from every line starting with it.

  - Potential incompatibilities
     * "ns_urldecode" and "ns_getform" will raise an exception
       when invalid UTF-8 data is tried to be interpreted as UTF-8
       and no fallback charset is provided. Invalid UTF-8 data
       causes trouble with external components such as
       Tdom or databases and opens vulnerability vectors.


Performance Improvements:

  - Improve "cachingmode none"

     "ns_cache_eval" works as follows:
      1) create a temporary cache entry for the key
      2) lock the cache-key (to avoid multiple parallel executions)
      3) execute the query
      4) store the result for the entry on success
      5) unlock the cache-key

     Previously, "cachingmode none" was simply avoiding to store the cached
     values (step 4), but was serializing calls for a cache key as in
     default caching modes. This was leading easily to cache entry collisions.

     Now, "cachingmode none" is avoiding all steps 1..5 (therefore
     no serialization and no cache collisions).

     See also:https://openacs.org/forums/message-view?message_id=5665480

Bug Fixes:

   - Improved robustness of "ns_parseurl" for handling query parameters
     and fragments for partial URLs

     * fix over-eager collecting of URL components in tail
     * extended regression test

   - Fixed Ns_ResetFileVec NOT to invalidate residual Ns_FileVec
     buffer.q (caused problems under Windows).

   - ns_striphtml: Fixed probably very old bug for markup immediately
     after an entity

     This bug fix handles cases, where e.g. two entities are in a text
     right next to each other, like e.g. in the string
     "hello<>world". The old code was correctly decoding the
     first entity, but output the second one literally.

   - Fixed compilation for C++, which was introduced in 4.99.23
     to avoid usage of reserved C identifiers

     Many thanks to Brendan Graves for reporting the problem.

   - Added missing named entities "apos" and "quote". These have been
     missing since ages.

   - Provide an error message when the configured locale is not
     installed on the host.

     This change causes NaviServer to abort, when the configured locale
     is not installed on the host. Typically, this locale is e.g. used
     by "ns_strcoll" for determining the default collating order. The
     configuration file for the regression testing sets the environment
     variable LANG to "en_US.UTF-8". This means that for running the
     stock regression test, this locale must be installed on the
     system. Before this change, NaviServer could crash at runtime when
     trying to access the default locale (as e.g. in "ns_strcoll")

   - Added support for "_charset_" field for default charset in
     multipart/form-data (RFC 7578, section 4.6)

     RFC 7578 (July 2015) defines an optional "_charset_" entry in the
     form (typically provided as hidden form field) to specify the
     charset of text entries. This is now supported as well by
     NaviServer. This is apparently a seldom used feature.

Documentation improvements:
---------------------------

  - Improved the following man pages:

        doc/src/manual/admin-install.man
        doc/src/naviserver/ns_conn.man
        doc/src/naviserver/ns_connchan.man
        doc/src/naviserver/ns_cookie.man
        doc/src/naviserver/ns_crypto.man
        doc/src/naviserver/ns_getform.man
        doc/src/naviserver/ns_http.man
        doc/src/naviserver/ns_httptime.man
        doc/src/naviserver/ns_log.man
        doc/src/naviserver/ns_parseheader.man
        doc/src/naviserver/ns_parsequery.man
        doc/src/naviserver/ns_parseurl.man
        doc/src/naviserver/ns_rlimit.man
        doc/src/naviserver/ns_urldecode.man
        doc/src/naviserver/ns_urlencode.man
        doc/src/naviserver/ns_valid_utf8.man
        doc/src/naviserver/textutil-cmds.man
        nsdb/doc/mann/ns_db.man


Configuration Changes:
----------------------

  - Updated OpenACS sample configuration file
     * reflect recent Oracle (tested with Oracle 19c)

     * added documentation for "StaticCSP", "CookieNamespace",
       "NsShutdownWithNonZeroExitCode", "LogIncludeUserId"


Code Changes:
-------------

  - Set Tcl error code "NS_INVALID_UTF8" for errors due to invalid UTF-8

  - Changed Tcl error code "NSCACHE" to "NS_CACHE". Now all
    NaviServer-specific error codes start with the prefix "NS_".

  - Extended regression test

  - Improve Tcl version compatibility

    * Removed -DTCL_NO_DEPRECATED from default CFLAGS to cope with
      recent deprecation in Tcl 8.7a5

  - Code Cleanup
      . Do not declare reserved C identifiers
      . Improved type cleanness
      . Refactored file-based multipart form parser to make logic
        explicit (Tcl code)

  - Improved comments, fixed typos

  - Marked "ns_set_precision" as deprecated, since there is no
    reason why not setting the Tcl variable ::tcl_precision directly.

  - Don't hard-wire port for HTTPS testing to 8443

    The setup code looks now for a free port for HTTPS connections
    starting with 8443, and remembers the free port in the
    configuration value "tls_listenport" and "tls_listenurl". This is
    now fully analogous to the setup of the plain HTTP testing (setting
    "listenport" and "listenurl")

  - Silence warning with recent versions of gcc when certain values of
    _FORTIFY_SOURCE/-Wstringop-overflow are set.


Changes in NaviServer Modules:
==============================

  22 files changed, 8447 insertions(+), 1429 deletions(-)

  - general: Updated obsolete ChangeLog files and
    replaced these by automatically generated ones.

nsdbpg:
-------

  - Raise exception when a value for a bind variable contains a NUL
    character. This value is explicitly forbidden in text strings
    passed to PostgreSQL.

  - Let "ns_pg" report available subcommands even when handle is not
    specified.  This makes the command compatible with the "icanuse"
    feature in OpenACS.


nsoracle:
---------
  - Added support for output columns of type SQLT_TIMESTAMP or SQLT_TIMESTAMP_TZ

    This change fixes a bug, where SQL queries of the form

          SELECT TO_TIMESTAMP(sysdate) FROM dual

    lead to errors for the form:

          Database operation "getrow" failed (exception 1406, "nsoracle.c:3659:Ns_OracleGetRow:
          error in `OCIStmtFetch ()': ORA-01406: fetched column value was truncated

    The driver needs for several output types special rules, where the
    timestamp cases were not supported so far. It is also recommended
    to set the according environment variables specifying the output
    format in the configuration server of NaviServer, such as e.g.

         set ::env(NLS_TIMESTAMP_FORMAT) "YYYY-MM-DD HH24:MI:SS.FF6"
         set ::env(NLS_TIMESTAMP_TZ_FORMAT) "YYYY-MM-DD HH24:MI:SS.FF6 TZH:TZM"

    For testing in you local Oracle installation, you might test the
    output formats (and the required sizes with the following snippet
    for sqlplus:

        COLUMN localtimestamp format a40
        COLUMN systimestamp format a40
        COLUMN ts_bytes format a80

        alter session set nls_timestamp_format='YYYY-MM-DD HH24:MI:SS.FF6';
        select localtimestamp, length(localtimestamp), dump(localtimestamp) ts_bytes from dual;

        alter session set nls_timestamp_tz_format='YYYY-MM-DD HH24:MI:SS.FF6 TZH:TZM';
        select systimestamp, length(systimestamp), dump(systimestamp) ts_bytes from dual;

        alter session set nls_timestamp_tz_format='YYYY-MM-DD HH24:MI:SS.FF6 TZR';
        select systimestamp, length(systimestamp), dump(systimestamp) ts_bytes from dual;

  - Code cleanup, ensure silent compilation with standard compiler settings
  - Improved spelling

ECDSA
letsencrypt:
------------
  - Added option to produce certificates with ECDSA

    Prior to this change, all certificates were using RSA keys. Since a
    while, keys based on elliptic curves became the preference of
    letsencrypt.

  - Improved spelling

Re: [naviserver-devel] Towards NaviServer 4.99.24

[David O. <da...@qc...>]

Thanks again Gustaf.
We've been happily running rc2 for a few days now without problem.

On Tue, 7 Jun 2022 at 11:13, Gustaf Neumann <ne...@wu...> wrote:

> Dear all,
>
> i've uploaded RC2 to sourceforge.
>
> In short, the changes relative to RC1 are primarily the fallback charsets
> (required to ease the situation for sites that have to deal with invalid
> UTF-8)
> and a change in cookie management to improve future-proofness
> (some browsers announced that these will ignore same-site=none
> without the secure flag, requiring secure connections).
>
> Please test if possible.  The release should be in the near future.
>
> all the best
> -gn
>
> https://sourceforge.net/projects/naviserver/files/naviserver/4.99.24/
>
>
>

Re: [naviserver-devel] Joint OpenACS and EuroTcl conference

[Gustaf N. <ne...@wu...>]

The second sentence should read .... We need some time ....

Sorry for the typo, age weakens eyes...

all the best

-gn

On 09.06.22 19:21, Gustaf Neumann wrote:
> Dear NaviServer Community,
>
> The registration for the joint OpenACS and EuroTcl conference ends in 
> one week. No need some time ahead of the event to reserve sufficient 
> capacities for the social events.
>
> all the best
>
> -gn
>
> https://openacs.org/conf2022/info/

-- 
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"

Re: [naviserver-devel] Joint OpenACS and EuroTcl conference

[Gustaf N. <ne...@wu...>]

Dear NaviServer Community,

The registration for the joint OpenACS and EuroTcl conference ends in 
one week. No need some time ahead of the event to reserve sufficient 
capacities for the social events.

all the best

-gn

https://openacs.org/conf2022/info/

Re: [naviserver-devel] Towards NaviServer 4.99.24

[Gustaf N. <ne...@wu...>]

Dear all,

i've uploaded RC2 to sourceforge.

In short, the changes relative to RC1 are primarily the fallback charsets
(required to ease the situation for sites that have to deal with invalid 
UTF-8)
and a change in cookie management to improve future-proofness
(some browsers announced that these will ignore same-site=none
without the secure flag, requiring secure connections).

Please test if possible.  The release should be in the near future.

all the best
-gn

https://sourceforge.net/projects/naviserver/files/naviserver/4.99.24/


=======================================
NaviServer 4.99.24, released 2022-06-XX
=======================================

  75 files changed, 3234 insertions(+), 1096 deletions(-)

New Features:

  - Improved security

    * Added protection against certain attacks in ns_dbquotevalue
     
      Due to the corrected conversion to external UTF-8 in db-output,
      new potential attack vectors appeared that were protected earlier
      via the Tcl-internal 'modified UTF-8'. E.g., the binary null
      character is stored as an overlong (two-byte) encoding of null
      (0xc0 0x80), so that an actual (embedded) null byte (0x00) never
      appears in the string. Due to the conversion, the internal
      representation is translated back to the binary null
      character. Embedded null byte characters can lead to
      non-terminated string literals via ns_dbquotevalue.
     
      In the updated version of NaviServer, ns_dbquotevalue raises an
      exception when this occurs. Therefore, the function can be used
      as well as an input checker (together with "try").

    * Raise an exception when trying to use "ns_urldecode" to
      produce invalid UTF-8
     
      Background: several (external) functions expect valid UTF-8 to be
      passed in and crash if this is not the case. One such example is
      tDOM. These nasty byte sequences are used more intensively by
      vulnerability scanners.
     
      Therefore, ns_urldecode raises now an exception, when it tries to
      convert to invalid UTF-8. It is still possible to use
      ns_urldecode to convert to other charsets.
     
         ns_urldecode -charset iso8859-1 -part path "/mot%C3or"
     
      When urldecode() is called internally and would produce invalid
      UTF-8, it truncates the string (and writes a warning to the
      system log).

      Note that the enw fallback charset

  - Fallback charsets

    In case, a conversion to UTF-8 fails due to invalid byte sequences,
    one can now provide a fallback charset for a second attempt of
    decoding this data. This feature is useful for websites that
    have to deal with requests containing invalid (form) data, typically
    from legacy applications.

    The fallback charset can be provided as optional parameter
    "-fallbackcharset" for the command "ns_getform", "ns_parsequery"
    and "ns_urldecode":

         ns_getform ?-fallbackcharset fallbackcharset? ?charset?
         ns_parsequery ?-charset charset? ?-fallbackcharset fallbackcharset? ?--? querystring
         ns_urldecode ?-charset charset? ?-fallbackcharset fallbackcharset? ?-part part? ?--? string

    In case, the parameter is not specified, it can be also be provided
    to the form-processing commands "ns_getform" and "ns_parsequery" via
    configuration variables:
    
      * per-server configuration parameter "formFallbackCharset"
        (in the section "ns/server/$server"), or as
        
      * global server configuration parameter "formFallbackCharset"
        (in the section "ns/parameters").
               
    The highest precedence has the optional parameter, followed by the
    per-server configuration parameter and the global configuration
    parameter.


  - Provide a hint when cache-entry is too large for caching
     
    Background: the size of the entry is typically determined after the
    execution of a potentially expensive query. During the eval of the
    command, the cache entry is locked and forces a
    serialization. However, this means that in these cases the
    situation is worse than without a cache, where some queries can be
    executed in parallel. We faced the situation of an expected
    slowdown of the server with many "create entry collision", where
    due to application matters, an entry was becoming too large. This
    situation is not easy to debug, especially in stress
    situations. The log message would have helped a log to identify the
    cause.

  - Added support for multibyte numeric entities
     
    This change supports conversion of numeric entities representing
    multibyte characters into HTML in "ns_striphtml" and
    "ns_unquotehtml". Technically, the numeric entities represent
    Unicode code points, which are transformed into UTF-8
    serialization.  Every entity represents a single code point; The
    values can be provided in decimal or hexadecimal notation. Before
    this change, only single byte numeric entities were
    supported. ASCII control characters (decimal 0-31) are ignored as
    before.

  - New and extended commands:

    * ns_unquotehtml /text/
     
      This command is the inverse operation of "ns_quotehtml". It
      replaces the named and numeric entities in the provided string
      with the native values. The command is similar to "ns_striphtml",
      but "ns_striphtml" removes as well other HTML markup which might
      not be desired in all cases.
     
      This change fixes as well a bug with numeric entities (the old
      code assumed, these are starting directly with a number after the
      ampersand) and it adds support for numeric entities with
      hexadecimal values (so far with the same value range as for
      decimal numeric entities).

    * ns_subnetmatch /subnet/ /ipaddr/
     
      Determine, if a provided IP address (IPv4 or IPv6) is included in
      a subnet specification, which is provided in CIDR notation. The
      command makes internal NaviServer functionality available at the
      Tcl level.  The regression test was extended to cover the
      functionality.
     
      The command ns_subnetmatch validates the provided subnet
      specification (IPv4 or IPv6 address followed by slash and number
      of significant bits) and the provided IP address and tests whether
      the IP address is in the implied range. The command returns a
      boolean value as the result. When comparing an IPv4 and IPv6
      address/CIDR specification or vice versa, the result is always
      false.

      The function can be use when e.g. restricting access to certain
      functionality to some subnets. The function can be used as well
      to check, whether an IP address is an IPv4 or IPv6 address.
     
       Examples:
         % ns_subnetmatch 137.208.0.0/16 137.208.116.31
         1
     
         % ns_subnetmatch 137.208.0.0/16 112.207.16.33
         0
     
         % ns_subnetmatch 2001:628:404:74::31/64 [ns_conn peeraddr]
         ...
     
         # Is IP address a valid IPv6 address?
         % ns_subnetmatch ::/0 $ip
     
         # Is IP address a valid IPv4 address?
         % ns_subnetmatch 0.0.0.0/0 $ip

    * ns_connchan: Added new subcommand "ns_connchan connect"
     
      "ns_connchan connect" is similar to "ns_connchan open", except
      that it does not send an HTTP request (HTTP method, URL, and
      header fields) but just opens the connection. It can be used for
      some non-HTTP communication over TCP and TLS over the ns_connchan
      infrastructure.

    * ns_parseheader, Ns_ParseHeader(): return the field
      number (index) of the parsed entry
     
      Previously, there was no explicit feedback, what field of an
      "ns_set" has been parsed by "ns_parseheader". Now, in success
      cases, the function returns the index of the new/modified
      entry. This function made it possible to generalize and simplify
      the Tcl-level parsing of "multipart/form-data"
      significantly. Additionally, a new optional argument "-prefix"
      was added. When specified, it adds the specified prefix to the
      key.

    * ns_setcookie, ns_deletecookie

      Mozilla and Chrome changed the default value for SameSite of
      cookies from "none" to "lax" in February and Aug 2020. Cookies
      that explicitly set SameSite=None must also set the "Secure"
      attribute.
     
      In order to mirror this change of policy in NaviServer and to
      reduce necessary code changes, the default behavior for setting
      or deleting cookie is now samesite "lax" (when "-samesite" is not
      explicitly specified).  When trying to set a cookie with
      "-samesite none" without the "-secure" flag, a warning is
      generated, and the "-samesite lax" is assumed, since major
      browsers announced that these will reject these cookies soon.


API changes:

  - ns_getform, ns_parsequery, and ns_urldecode

    New optional parameter "-fallbackcharset". See above for details.

- ns_parsequery: added option "-charset" and raise exception on failure
     
    The new option "-charset" can be used to add a charset for the
    result encoding of the passed-in HTTP query.

    In case the charset is UTF8 (default on most platforms), and the
    content is invalid UTF-8, an exception is raised (similar to
    ns_urldecode). This can be addressed by parameter
    "-fallbackcharset" (See above for details).

  - ns_deletecookie: added support "-samesite" flag for ns_deletecookie.

    Since "ns_deletecookie" sets internally a cookie, some browsers
    might ignore in the future certain cookie requests (e.g. when
    "-samesite" is not used or set to "none" on non-secure connections).
  
  - ns_trim enhancements:
    The new option "-prefix ..." can be used to strip a string
    (such as ">> ") from every line starting with it.

  - Potential incompatibilties
     * "ns_urldecode" and "ns_getform" will raise an exception
       when invalid UTF-8 data is tried to be interpreted as UTF-8.
       Such data cause trouble with external components such as
       Tdom or databases and opens vulnerability vectors.


Performance Improvements:

  - Improve "cachingmode none"
     
     "ns_cache_eval" works as follows:
      1) create a temporary cache entry for the key
      2) lock the cache-key (to avoid multiple parallel executions)
      3) execute the query
      4) store the result for the entry on success
      5) unlock the cache-key
     
     Previously, "cachingmode none" was simply avoiding to store the cached
     values (step 4), but was serializing calls for a cache key as in
     default caching modes. This was leading easily to cache entry collisions.
     
     Now, "cachingmode none" is avoiding all steps 1..5 (therefore
     no serialization and no cache collisions).
     
     See also:https://openacs.org/forums/message-view?message_id=5665480

Bug Fixes:

   - Improved robustness of "ns_parseurl" for handling query parameters
     and fragments for partial URLs
     
     * fix over-eager collecting of URL components in tail
     * extended regression test

   - Fixed Ns_ResetFileVec NOT to invalidate residual Ns_FileVec
     buffer.q (caused problems under Windows).

   - ns_striphtml: Fixed probably very old bug for markup immediately
     after an entity
     
     This bug fix handles cases, where e.g. two entities are in a text
     right next to each other, like e.g. in the string
     "hello<>world". The old code was correctly decoding the
     first entity, but output the second one literally.

   - Fixed compilation for C++, which was introduced in 4.99.23
     to avoid usage of reserved C identifiers
     
     Many thanks to Brendan Graves for reporting the problem.

   - Added missing named entities "apos" and "quote". These have been
     missing since ages.

   - Provide an error message when the configured locale is not
     installed on the host.
     
     This change causes NaviServer to abort, when the configured locale
     is not installed on the host. Typically, this locale is e.g. used
     by "ns_strcoll" for determining the default collating order. The
     configuration file for the regression testing sets the environment
     variable LANG to "en_US.UTF-8". This means that for running the
     stock regression test, this locale must be installed on the
     system. Before this change, NaviServer could crash at runtime when
     trying to access the default locale (as e.g. in "ns_strcoll")
     
   - Added support for "_charset_" field for default charset in
     multipart/form-data (RFC 7578, section 4.6)
         
     RFC 7578 (July 2015) defines an optional "_charset_" entry in the
     form (provided typically as hidden form field) to specify the
     charset of text entries. This is now supported as well by
     NaviServer. This is apparently a seldom used feature.
     
Documentation improvements:
---------------------------

  - Improved the following man pages:

        doc/src/manual/admin-install.man
        doc/src/naviserver/ns_conn.man
        doc/src/naviserver/ns_connchan.man
        doc/src/naviserver/ns_crypto.man
        doc/src/naviserver/ns_http.man
        doc/src/naviserver/ns_httptime.man
        doc/src/naviserver/ns_log.man
        doc/src/naviserver/ns_parseheader.man
        doc/src/naviserver/ns_parsequery.man
        doc/src/naviserver/ns_parseurl.man
        doc/src/naviserver/ns_rlimit.man
        doc/src/naviserver/ns_subnetmatch.man
        doc/src/naviserver/ns_urldecode.man
        doc/src/naviserver/ns_urlencode.man
        doc/src/naviserver/textutil-cmds.man
        nsdb/doc/mann/ns_db.man


Configuration Changes:
----------------------

  - Updated OpenACS sample configuration file
     * reflect recent Oracle (tested with Oracle 19c)
     
     * added documentation for "StaticCSP", "CookieNamespace",
       "NsShutdownWithNonZeroExitCode", "LogIncludeUserId"


Code Changes:
-------------

  - Set Tcl error code "NS_INVALID_UTF8" for errors due to invalid UTF-8

  - Changed Tcl error code "NSCACHE" to "NS_CACHE". Now all
    NaviServer-specific error codes start with the prefix "NS_".

  - Extended regression test

  - Improve Tcl version compatibility
  
    * Removed -DTCL_NO_DEPRECATED from default CFLAGS to cope with
      recent deprecation in Tcl 8.7a5
      
  - Code Cleanup
      . Do not declare reserved C identifiers
      . Improved type cleanness
      . Refactored file-based multipart form parser to make logic
        explicit (Tcl code)

  - Improved comments, fixed typos

  - Marked "ns_set_precision" as deprecated, since there is no
    reason why not setting the Tcl variable ::tcl_precision directly.

  - Don't hard-wire port for https testing to 8443
     
    The setup code looks now for a free port for HTTPS connections
    starting with 8443, and remembers the free port in the
    configuration value "tls_listenport" and "tls_listenurl". This is
    now fully analogous to the setup of the plain HTTP testing (setting
    "listenport" and "listenurl")

  - Silence warning with recent versions of gcc when certain values of
    _FORTIFY_SOURCE/-Wstringop-overflow are set.
     

Changes in NaviServer Modules:
==============================

...

On 03.06.22 20:03, Gustaf Neumann wrote:
>
> Dear David,
>
> the automated shortening for the invalid strings is now committed.
>
> https://bitbucket.org/naviserver/naviserver/commits/51f101928be6d27efe5ab78d7d9a9693026791c1
>
> I'll try to make rc2 soon.
>
> all the best
>
> -gn
>

Re: [naviserver-devel] Towards NaviServer 4.99.24

[Gustaf N. <ne...@wu...>]

Dear David,

the automated shortening for the invalid strings is now committed.

https://bitbucket.org/naviserver/naviserver/commits/51f101928be6d27efe5ab78d7d9a9693026791c1

I'll try to make rc2 soon.

all the best

-gn

On 30.05.22 21:01, Gustaf Neumann wrote:
>
> i agree, this might be a lot of logging, especially when the decoded 
> strings are long.
>
> Not sure, what the best approach of this is
> - an configuration parameter LogInvalidUTF8warnings to deactivate 
> these messages (default true)
> - an configuration parameter NrInvalidUTF8warnings to limit these 
> messages (default 99999999)
> - truncate the invalid UTF8 sequence in the log entry to shorten the 
> messages (when longer than e.g. 20 bytes)
> - try to identify the first invalid UTF8 sequence in the string and 
> report just this (even shorter log message).
>
> The last option would be probably the best, but requires an C.API change.
>
> Other options?
>
> -gn
>
>
>
> On 30.05.22 10:41, David Osborne wrote:
>> Thanks Gustaf - the errorCode will be very handy to trap these 
>> encoding errors.
>>
>> We're using the per-server "formfallbackcharset" without issue at the 
>> moment. Working well.
>>
>> One problem we encountered is the Warnings logged when invalid UTF-8 
>> is encountered.
>> Some of the POSTs causing encoding issues were very large. This was 
>> filling logs quite quickly (I think it's possible to have "maxpost" 
>> bytes of data written to the log). Didn't want to disable all 
>> Warnings, so we patched NaviServer to log just a Warning of invalid 
>> UTF-8 but not include the data itself. But there may be a better way 
>> of dealing with this.
>>
>> Regards,
>> David
>>
>>
>> On Sat, 28 May 2022 at 18:45, Gustaf Neumann <ne...@wu...> wrote:
>>
>>     Dear all,
>>
>>     The latest commits
>>     - added the "multipart/form-data" handlingfor fallback charsets,
>>     - provides an error code for invalid UTF-8 and
>>     - adds  support for the "_charset_" field for default charsets
>>     (see RFC
>>     7578, section 4.6).
>>
>>     Also, the regression test got several more tests.
>>
>>     I have still a bug report for ns_connchan (which i could not
>>     reproduce
>>     so far),
>>     if i find something to fix the next days, this will go as well
>>     into the next
>>     release, otherwise we are ready for rc2.
>>
>>     all the best
>>
>>
>>
>> _______________________________________________
>> naviserver-devel mailing list
>> nav...@li...
>> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> -- 
> Univ.Prof. Dr. Gustaf Neumann
> Head of the Institute of Information Systems and New Media
> of Vienna University of Economics and Business
> Program Director of MSc "Information Systems"
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

-- 
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"

Re: [naviserver-devel] Towards NaviServer 4.99.24

[Gustaf N. <ne...@wu...>]

i agree, this might be a lot of logging, especially when the decoded 
strings are long.

Not sure, what the best approach of this is
- an configuration parameter LogInvalidUTF8warnings to deactivate these 
messages (default true)
- an configuration parameter NrInvalidUTF8warnings to limit these 
messages (default 99999999)
- truncate the invalid UTF8 sequence in the log entry to shorten the 
messages (when longer than e.g. 20 bytes)
- try to identify the first invalid UTF8 sequence in the string and 
report just this (even shorter log message).

The last option would be probably the best, but requires an C.API change.

Other options?

-gn



On 30.05.22 10:41, David Osborne wrote:
> Thanks Gustaf - the errorCode will be very handy to trap these 
> encoding errors.
>
> We're using the per-server "formfallbackcharset" without issue at the 
> moment. Working well.
>
> One problem we encountered is the Warnings logged when invalid UTF-8 
> is encountered.
> Some of the POSTs causing encoding issues were very large. This was 
> filling logs quite quickly (I think it's possible to have "maxpost" 
> bytes of data written to the log). Didn't want to disable all 
> Warnings, so we patched NaviServer to log just a Warning of invalid 
> UTF-8 but not include the data itself. But there may be a better way 
> of dealing with this.
>
> Regards,
> David
>
>
> On Sat, 28 May 2022 at 18:45, Gustaf Neumann <ne...@wu...> wrote:
>
>     Dear all,
>
>     The latest commits
>     - added the "multipart/form-data" handlingfor fallback charsets,
>     - provides an error code for invalid UTF-8 and
>     - adds  support for the "_charset_" field for default charsets
>     (see RFC
>     7578, section 4.6).
>
>     Also, the regression test got several more tests.
>
>     I have still a bug report for ns_connchan (which i could not
>     reproduce
>     so far),
>     if i find something to fix the next days, this will go as well
>     into the next
>     release, otherwise we are ready for rc2.
>
>     all the best
>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

-- 
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"

Re: [naviserver-devel] Towards NaviServer 4.99.24

[David O. <da...@qc...>]

Thanks Gustaf - the errorCode will be very handy to trap these encoding
errors.

We're using the per-server "formfallbackcharset" without issue at the
moment. Working well.

One problem we encountered is the Warnings logged when invalid UTF-8 is
encountered.
Some of the POSTs causing encoding issues were very large. This was filling
logs quite quickly (I think it's possible to have "maxpost" bytes of data
written to the log). Didn't want to disable all Warnings, so we patched
NaviServer to log just a Warning of invalid UTF-8 but not include the data
itself. But there may be a better way of dealing with this.

Regards,
David


On Sat, 28 May 2022 at 18:45, Gustaf Neumann <ne...@wu...> wrote:

> Dear all,
>
> The latest commits
> - added the "multipart/form-data" handlingfor fallback charsets,
> - provides an error code for invalid UTF-8 and
> - adds  support for the "_charset_" field for default charsets (see RFC
> 7578, section 4.6).
>
> Also, the regression test got several more tests.
>
> I have still a bug report for ns_connchan (which i could not reproduce
> so far),
> if i find something to fix the next days, this will go as well into the
> next
> release, otherwise we are ready for rc2.
>
> all the best
>
>

Re: [naviserver-devel] Towards NaviServer 4.99.24

[Gustaf N. <ne...@wu...>]

Dear all,

The latest commits
- added the "multipart/form-data" handlingfor fallback charsets,
- provides an error code for invalid UTF-8 and
- adds  support for the "_charset_" field for default charsets (see RFC 
7578, section 4.6).

Also, the regression test got several more tests.

I have still a bug report for ns_connchan (which i could not reproduce 
so far),
if i find something to fix the next days, this will go as well into the next
release, otherwise we are ready for rc2.

all the best

-gn

On 19.05.22 21:19, Gustaf Neumann wrote:
>
> Hi David,
>
> we have not a global and per-server parameter called 
> "formfallbackcharset",
> the flag for "ns_getform" and "ns_parsequery" is now called 
> "fallbackcharset".
>
> In many cases, using e.g. the per-server parameter should be 
> sufficient to handle
> incorrect queries...
>
> still missing: "multipart/form-data" handling and documentation 
> updates, error code
>
> all the best
>

[naviserver-devel] Joint OpenACS and EuroTcl conference

[Gustaf N. <ne...@wu...>]

Dear all,

The deadline for submissions of abstracts for presentation at the 
forthcoming joint OpenACS / EuroTcl
conference is approaching. It is less than one week left. Presentation 
about NaviServer applications
are also very welcome.

Important dates:

May 27th, 2022: Deadline for submissions of abstracts (max. 2 pages, 
min. 250 words);
June 3rd, 2022: Notification of acceptance
June 15th, 2022: Registration ends
June 29th, 2022: Meet & greet
June 30th - July 1st, 2022:  Conference

For details, see: https://openacs.org/conf2022/

Gustaf Neumann and Paul Obermeier

-- 
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"

Re: [naviserver-devel] Towards NaviServer 4.99.24

[Gustaf N. <ne...@wu...>]

Thanks as well, change is welcome! ... i've added the documentation for 
the configuration variables.

-g

On 20.05.22 16:28, David Osborne wrote:
> Thanks Gustaf - I've run some quick tests against the per-server and 
> global fallback and it seems to work well in the cases we're looking 
> at - thanks for the fast work!
> I've yet to try the ns_getform option but will do so shortly. I've 
> submitted a pull req for the ns_getform doc which I'm hoping might be 
> useful to you.
> Thanks again.

Re: [naviserver-devel] Towards NaviServer 4.99.24

[David O. <da...@qc...>]

Thanks Gustaf - I've run some quick tests against the per-server and global
fallback and it seems to work well in the cases we're looking at - thanks
for the fast work!
I've yet to try the ns_getform option but will do so shortly. I've
submitted a pull req for the ns_getform doc which I'm hoping might be
useful to you.
Thanks again.

On Thu, 19 May 2022 at 20:20, Gustaf Neumann <ne...@wu...> wrote:

> Hi David,
>
> we have not a global and per-server parameter called "formfallbackcharset",
> the flag for "ns_getform" and "ns_parsequery" is now called
> "fallbackcharset".
>
> In many cases, using e.g. the per-server parameter should be sufficient to
> handle
> incorrect queries...
>
> still missing: "multipart/form-data" handling and documentation updates,
> error code
>
> all the best
>
> -gn
> On 18.
>

Re: [naviserver-devel] Towards NaviServer 4.99.24

[Gustaf N. <ne...@wu...>]

Hi David,

we have not a global and per-server parameter called "formfallbackcharset",
the flag for "ns_getform" and "ns_parsequery" is now called 
"fallbackcharset".

In many cases, using e.g. the per-server parameter should be sufficient 
to handle
incorrect queries...

still missing: "multipart/form-data" handling and documentation updates, 
error code

all the best

-gn

On 18.05.22 22:00, Gustaf Neumann wrote:
>
> Dear David,
>
> i've committed the option "-fallbackencodings" for the commands  
> "ns_getform" and "ns_parsequery". The implementation covers 
> "ns_getform", where the data is provided as 
> "application/x-www-form-urlencoded"  either when parsing from memory 
> or from the spool file. The "multipart/form-data" implementation (also 
> separate for memory and spoolfile) is not yet covered.
>
> We can also consider a global parameter for the configuration file 
> (like e.g. FormFallbackEncodings). Probably, we should use the term 
> "charset" instead of "encoding", since "charset" is the MIME term, 
> also used for e.g. "URLCharset", while "encoding" is the Tcl name.
>
> Although the names might still change, you might test whether this 
> works for your test cases.
>
> -gn
>
> On 16.05.22 16:16, David Osborne wrote:
>> Hi Gustaf,
>>
>> I spotted that *ns_getform *takes a charset argument from looking at 
>> the source code.
>> The options for overriding charsets  at the moment seem to be:
>>
>> *ns_getform iso8859-1
>> *
>> *
>> *
>> *ns_urlcharset iso8859-1*
>> *ns_getform
>> *
>> *
>> *
>> *ns_conn urlencoding iso8859-1
>> *
>> *ns_getform *
>>
>> We experimented with some code which tried to trap errors from 
>> *ns_getform*, and where the error was due to "invalid UTF-8", try a 
>> fallback charset.
>> All 3 of the above techniques worked OK when the Content-Type header 
>> leaves the charset /unspecified/.
>>
>> The main issues we had were:
>>
>> 1. When a *charset=utf-8* is present in the *Content-Type* header, 
>> this overrides ([1]) any encoding we pass with using the 3 techniques 
>> above.
>> In those cases we have to manipulate the headers' ns_set to remove or 
>> change the charset.
>> eg.
>> *Content-Type: application/x-www-form-urlencoded; charset=utf-8*
>> transform to ->
>> *Content-Type: application/x-www-form-urlencoded*
>> or
>> *Content-Type: application/x-www-form-urlencoded; charset=windows-1252*
>>
>> 2. Trapping the specific "invalid UTF-8" error - this method seems 
>> fragile - would be nice if there was an *errorCode *we would trap.
>> *::try {
>> *
>> *    ns_getform*
>> *} on error {msg options} {*
>> *    if { [string match "*contains invalid UTF-8" $msg] } {*
>> *        # change Content_type charset (if present)*
>> *        # try fallback charset*
>> *    } else {*
>> *        # rethrow error*
>> *    }*
>> *}*
>>
>> But I think this presents us with a way forward in cases where client 
>> apps are not getting the encoding correct.
>>
>> [1] 
>> https://bitbucket.org/naviserver/naviserver/annotate/master/nsd/form.c?at=master#form.c-170
>>
>>