naviserver-devel — Developer discussion

[naviserver-devel] Towards NaviServer 4.99.24

[Gustaf N. <ne...@wu...>]

Dear all,

on sourceforge is a release candidate for NaviServer 4.99.24 [1].

Please test if possible.  The release should be in the near future.
Below is a preliminary summary of changes.

All the best, and have a nice easter weekend!
-g

[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.24/


=======================================
NaviServer 4.99.24, released 2022-04-XX
=======================================

  57 files changed, 1840 insertions(+), 824 deletions(-)

New Features:

  - Improved security

    * Added protection against certain attacks in ns_dbquotevalue
     
      Due to the corrected conversion to external UTF-8 in db-output,
      new potential attack vectors appeared that were protected earlier
      via the Tcl-internal 'modified UTF-8'. E.g., the binary null
      character is stored as an overlong (two-byte) encoding of null
      (0xc0 0x80), so that an actual (embedded) null byte (0x00) never
      appears in the string. Due to the conversion, the internal
      representation is translated back to the binary null
      character. Embedded null byte characters can lead to
      non-terminated string literals via ns_dbquotevalue.
     
      In the updated version of NaviServer, ns_dbquotevalue raises an
      exception when this occurs. Therefore, the function can be used
      as well as an input checker (together with "try").

    * Raise an exception when trying to use "ns_urldecode" to
      produce invalid UTF-8
     
      Background: several (external) functions expect valid UTF-8 to be
      passed in and crash if this is not the case. One such example is
      tDOM. These nasty byte sequences are used more intensively by
      vulnerability scanners.
     
      Therefore, ns_urldecode raises now an exception, when it tries to
      convert to invalid UTF-8. It is still possible to use
      ns_urldecode to convert to other charsets.
     
         ns_urldecode -charset iso8859-1 -part path "/mot%C3or"
     
      When urldecode() is called internally and would produce invalid UTF-8,
      it truncates the string (and writes a warning to the system log).
     
  - Provide a hint when cache-entry was too large for caching
     
    Background: the size of the entry is typically determined after the
    execution of a potentially expensive query. During the eval of the
    command, the cache entry is locked and forces a
    serialization. However, this means that in these cases the
    situation is worse than without a cache, where some queries can be
    executed in parallel. We faced the situation of an expected
    slowdown of the server with many "create entry collision", where
    due to application matters, an entry was becoming too large. This
    situation is not easy to debug, especially in stress
    situations. The log message would have helped a log to identify the
    cause.

  - Added support for multibyte numeric entities
     
    This change supports conversion of numeric entities representing
    multibyte characters into HTML in "ns_striphtml" and
    "ns_unquotehtml". Technically, the numeric entities represent
    Unicode code points, which are transformed into UTF-8
    serialization.  Every entity represents a single code point; The
    values can be provided in decimal or hexadecimal notation. Before
    this change, only single byte numeric entities were
    supported. ASCII control characters (decimal 0-31) are ignored as
    before.

  - New and extended commands:

    * ns_unquotehtml /text/
     
      This command is the inverse operation of "ns_quotehtml". It
      replaces the named and numeric entities in the provided string
      with the native values. The command is similar to "ns_striphtml",
      but "ns_striphtml" removes as well other HTML markup which might
      not be desired in all cases.
     
      This change fixes as well a bug with numeric entities (the old
      code assumed, these are starting directly with a number after the
      ampersand) and it adds support for numeric entities with
      hexadecimal values (so far with the same value range as for
      decimal numeric entities).

    * ns_subnetmatch /subnet/ /ipaddr/
     
      Determine, if a provided IP address (IPv4 or IPv6) is included in
      a subnet specification, which is provided in CIDR notation. The
      command makes internal NaviServer functionality available at the
      Tcl level.  The regression test was extended to cover the
      functionality.
     
      The command ns_subnetmatch validates the provided subnet
      specification (IPv4 or IPv6 address followed by slash and number
      of significant bits) and the provided IP address and tests whether
      the IP address is in the implied range. The command returns a
      boolean value as the result. When comparing an IPv4 and IPv6
      address/CIDR specification or vice versa, the result is always
      false.

      The function can be use when e.g. restricting access to certain
      functionality to some subnets. The function can be used as well
      to check, whether an IP address is an IPv4 or IPv6 address.
     
       Examples:
         % ns_subnetmatch 137.208.0.0/16 137.208.116.31
         1
     
         % ns_subnetmatch 137.208.0.0/16 112.207.16.33
         0
     
         % ns_subnetmatch 2001:628:404:74::31/64 [ns_conn peeraddr]
         ...
     
         # Is IP address a valid IPv6 address?
         % ns_subnetmatch ::/0 $ip
     
         # Is IP address a valid IPv4 address?
         % ns_subnetmatch 0.0.0.0/0 $ip

    * ns_connchan: Added new subcommand "ns_connchan connect"
     
      "ns_connchan connect" is similar to "ns_connchan open", except
      that it does not send an HTTP request (HTTP method, URL, and
      header fields) but just opens the connection. It can be used for
      some non-HTTP communication over TCP and TLS over the ns_connchan
      infrastructure.

    * ns_parseheader, Ns_ParseHeader(): return the field
      number (index) of the parsed entry
     
      Previously, there was no explicit feedback, what field of an
      "ns_set" has been parsed by "ns_parseheader". Now, in success
      cases, the function returns the index of the new/modified
      entry. This function made it possible to generalize and simplify
      the Tcl-level parsing of "multipart/form-data"
      significantly. Additionally, a new optional argument "-prefix"
      was added. When specified, it adds the specified prefix to the
      key.


API changes:

  - ns_parsequery: added option "-charset" and raise exception on failure
     
    The new option "-charset" can be used to add a charset for the
    result encoding of the passed-in HTTP query. In case the charset is
    UTF8 (default on most platforms), and the content is invalid UTF-8,
    an exception is raised (similar to ns_urldecode).


  - Better Unicode support, including emojis requiring 4-byte UTF-8 characters.

  - ns_trim enhancements:
    The new option "-prefix ..." can be used to strip a string
    (such as ">> ") from every line starting with it.

     


Performance Improvements:



Bug Fixes:

   - Improved robustness of "ns_parseurl" for handling query parameters
     and fragments for partial URLs
     
     * fix over-eager collecting of URL components in tail
     * extended regression test

   - Fixed Ns_ResetFileVec NOT to invalidate residual Ns_FileVec
     buffer.q (caused problems under Windows).

   - ns_striphtml: Fixed probably very old bug for markup immediately after an entity
     
     This bug fix handles cases, where e.g. two entities are in a text
     right next to each other, like e.g. in the string
     "hello<>world". The old code was correctly decoding the
     first entity, but output the second one literally.

   - Fixed compilation for C++, which was introduced in 4.99.23
     to avoid usage of reserved C identifiers
     
     Many thanks to Brendan Graves for reporting the problem.

   - Added missing named entities "apos" and "quote". These have been
     missing since ages.

   - Provide an error message when the configured locale is not
     installed on the host.
     
     This change causes NaviServer to abort, when the configured locale
     is not installed on the host. Typically, this locale is e.g. used
     by "ns_strcoll" for determining the default collating order. The
     configuration file for the regression testing sets the environment
     variable LANG to "en_US.UTF-8". This means that for running the
     stock regression test, this locale must be installed on the
     system. Before this change, NaviServer could crash at runtime when
     trying to access the default locale (as e.g. in "ns_strcoll")
     
     
Documentation improvements:
---------------------------

  - Improved the following man pages:

        doc/src/manual/admin-install.man
        doc/src/naviserver/ns_conn.man
        doc/src/naviserver/ns_connchan.man
        doc/src/naviserver/ns_crypto.man
        doc/src/naviserver/ns_http.man
        doc/src/naviserver/ns_httptime.man
        doc/src/naviserver/ns_log.man
        doc/src/naviserver/ns_parseheader.man
        doc/src/naviserver/ns_parsequery.man
        doc/src/naviserver/ns_parseurl.man
        doc/src/naviserver/ns_rlimit.man
        doc/src/naviserver/ns_subnetmatch.man
        doc/src/naviserver/ns_urldecode.man
        doc/src/naviserver/ns_urlencode.man
        doc/src/naviserver/textutil-cmds.man
        nsdb/doc/mann/ns_db.man


Configuration Changes:
----------------------

  - Updated OpenACS sample configuration file
     * reflect recent Oracle (tested with Oracle 19c)
     
     * added documentation for "StaticCSP", "CookieNamespace",
       "NsShutdownWithNonZeroExitCode", "LogIncludeUserId"


Code Changes:
-------------
  - Extended regression test

  - Improve Tcl version compatibility
  
    * Removed -DTCL_NO_DEPRECATED from default CFLAGS to cope with
      recent deprecation in Tcl 8.7a5
      
  - Code Cleanup
      . Do not declare reserved C identifiers
      . Improved type cleanness

  - Improved comments, fixed typos

  - Marked "ns_set_precision" as deprecated, since there is no
    reason why not setting the Tcl variable ::tcl_precision directly.

  - Don't hard-wire port for https testing to 8443
     
    The setup code looks now for a free port for HTTPS connections
    starting with 8443, and remembers the free port in the
    configuration value "tls_listenport" and "tls_listenurl". This is
    now fully analogous to the setup of the plain HTTP testing (setting
    "listenport" and "listenurl")

   - Silence warning with recent versions of gcc when certain values of
     _FORTIFY_SOURCE/-Wstringop-overflow are set.
     
     

Changes in NaviServer Modules:
==============================

...

Re: [naviserver-devel] v4.99.23 Tests failing building on Debian Buster Pt1: encoding_ns_http-1.1

[Gustaf N. <ne...@wu...>]

There are now two changes committed to bitbucket:

a) Provide an error message when the configured locale is not installed 
on the host (misconfiguration)

This change causes NaviServer to abort, when the configured locale is 
not installed on the host. Typically, this locale is e.g. used by 
ns_strcoll for determining the default collating order. The 
configuration file for the regression testing sets the environment 
variable LANG to "en_US.UTF-8". This means that for running the stock
regression test, this locale must be installed on the OS level.

b) Silence warning with recent versions of gcc when certain values of 
_FORTIFY_SOURCE/-Wstringop-overflow are set

Newer versions of gcc support warning of dangerous operations (such as 
e.g. strncat) when these depend on not easy traceable sources. In the 
fixed case, the warning was:

    warning: ‘__builtin_strncat’ specified bound depends on
    the length of the source argument

With FORTIFY_SOURCE whenever possible, GCC tries to use buffer-length 
aware replacements for functions, which was not possible in the case in 
question. The documentation says that with _FORTIFY_SOURCE set to
2, some more checking is added, but some conforming programs might fail.

The case for (b) was a false positive, but it is still better to silence 
these rather than ignoring it.

all the best

-g

Re: [naviserver-devel] v4.99.23 Tests failing building on Debian Buster Pt1: encoding_ns_http-1.1

[Gustaf N. <ne...@wu...>]

Dear David,

The problem with manual compilation on minimal Debian installations is 
that the locale "en_US.UTF-8" is there not installed by default. The 
configuration file for the test sets LC_COLLATE to this value such that 
the collate test will work. ....

The problem with Debian is that when LC_COLLATE is set to any of the 
predefined values, that crash will go away, but the test will fail 
(leading to a different comparison result).  So we have either to skip 
the ns_stroll tests if no proper locale is defined, or refuse to start 
naviserver, if the local is not installed. Since other Unixes have the 
locale "en_US.UTF-8" installed by default, it is probably the best to 
require its installation during startup of nsd. This can avoid surprises 
later.

See below for its installation of the missing locale under Debian; with 
this set-up, the problem with strcoll_l() will disappear.

all the best

-gn

$ locale -a
C
C.UTF-8
POSIX

$ sed -i 's/^# *\(en_US.UTF-8\)/\1/' /etc/locale.gen
$ locale-gen

$ locale -a
C
C.UTF-8
POSIX
en_US.utf8


On 07.04.22 13:17, David Osborne wrote:
> Thanks very much,
>
> For versions of Tcl less than 8.6.11 we're failing because it's a new 
> test exposing an old problem, is that correct?
> This would explain why I don't see any test failures after 
> building v4.99.22 with Tcl 8.6.9 - it's only because 4.99.23 has 
> introduced the tests and not that 4.99.22 doesn't have the problem.
>
> To test Tcl8.6.11 the easiest way for me is to jump to bullseye 
> (Debian v11) which provides 8.6.11+dfsg-1
>
> Strangely I seem to get the same ns_strcoll seg fault with that.
> But if I remove misc.test temporarily, all other tests pass happily.
>
> # uname -a
> Linux ip-172-0-1-190 5.10.0-12-cloud-amd64 #1 SMP Debian 5.10.103-1 
> (2022-03-07) x86_64 GNU/Linux
>
> # cat /etc/debian_version
> 11.3
>
> # ls -l /lib/x86_64-linux-gnu/libc.so.6
> lrwxrwxrwx 1 root root 12 Mar 17 21:37 /lib/x86_64-linux-gnu/libc.so.6 
> -> libc-2.31.so <http://libc-2.31.so>
>
> # apt-cache policy tcl8.6
> tcl8.6:
>   Installed: 8.6.11+dfsg-1
>
> # git clone https://bitbucket.org/naviserver/naviserver.git
> Cloning into 'naviserver'...
>
> # cd naviserver
> # git checkout tags/naviserver-4.99.23
> Note: switching to 'tags/naviserver-4.99.23'.
>
> # ./autogen.sh --with-tcl=/usr/lib/tcl8.6 --enable-rpath 
> --enable-threads --enable-symbols
> # make
>
> Compiler warning for reference:
>
> gcc   -Wall -fPIC -g -O2 
> -fdebug-prefix-map=/build/tcl8.6-qxVr7a/tcl8.6-8.6.11+dfsg=. 
> -fstack-protector-strong -Wformat -Werror=format-security 
> -fno-unit-at-a-time -pipe -Wdate-time -D_FORTIFY_SOURCE=2 -DNDEBUG 
> -DSYSTEM_MALLOC -DTCL_NO_DEPRECATED -std=c99 -I../include 
> -I"/usr/include/tcl8.6"   -DHAVE_CONFIG_H   -c -o tclenv.o tclenv.c
> In file included from /usr/include/string.h:495,
>                  from ../include/nsthread.h:378,
>                  from ../include/ns.h:46,
>                  from nsd.h:38,
>                  from tclenv.c:37:
> In function ‘strncat’,
>     inlined from ‘PutEnv’ at tclenv.c:349:13:
> /usr/include/x86_64-linux-gnu/bits/string_fortified.h:136:10: warning: 
> ‘__builtin_strncat’ specified bound depends on the length of the 
> source argument [ 
> ]8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wstringop-overflow=-Wstringop-overflow= 
> ]8;;]
>   136 |   return __builtin___strncat_chk (__dest, __src, __len, __bos 
> (__dest));
>       |  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> tclenv.c: In function ‘PutEnv’:
> tclenv.c:314:23: note: length computed here
>   314 |         valueLength = strlen(value) + 1;
>       |                       ^~~~~~~~~~~~~
>
> # make memcheck TESTFLAGS="-verbose start -file misc.test"
> ---- ns_random-1.1 start
> ---- ns_fmttime-1.0 start
> ---- ns_fmttime-1.1 start
> ---- ns_trim-0.0 start
> ---- ns_trim-0.1 start
> ---- ns_trim-0.2 start
> ---- ns_trim-1.1 start
> ---- ns_trim-1.2 start
> ---- ns_trim-1.3 start
> ---- ns_trim-1.4 start
> ---- ns_trim-1.5 start
> ---- ns_trim-2.1 start
> ---- ns_trim-2.2 start
> ---- ns_quotehtml start
> ---- ns_strcoll-1.0.0 start
> ==37899== Thread 2:
> ==37899== Invalid read of size 8
> ==37899==    at 0x49E1361: strcoll_l (strcoll_l.c:260)
> ==37899==    by 0x48DA9FF: NsTclStrcollObjCmd (tclmisc.c:2802)
> ==37899==    by 0x4BBC4A1: TclNRRunCallbacks (in 
> /usr/lib/x86_64-linux-gnu/libtcl8.6.so <http://libtcl8.6.so>)
> ==37899==    by 0x4BBD71F: ??? (in 
> /usr/lib/x86_64-linux-gnu/libtcl8.6.so <http://libtcl8.6.so>)
> ==37899==    by 0x4C794D8: Tcl_FSEvalFileEx (in 
> /usr/lib/x86_64-linux-gnu/libtcl8.6.so <http://libtcl8.6.so>)
> ==37899==    by 0x4C818AD: Tcl_MainEx (in 
> /usr/lib/x86_64-linux-gnu/libtcl8.6.so <http://libtcl8.6.so>)
> ==37899==    by 0x4B745AF: NsThreadMain (thread.c:232)
> ==37899==    by 0x4B75A48: ThreadMain (pthread.c:870)
> ==37899==    by 0x521BEA6: start_thread (pthread_create.c:477)
> ==37899==    by 0x4A4DDEE: clone (clone.S:95)
> ==37899==  Address 0x18 is not stack'd, malloc'd or (recently) free'd
> ==37899==
> ==37899==
> ==37899== Process terminating with default action of signal 11 (SIGSEGV)
> ==37899==  Access not within mapped region at address 0x18
> ==37899==    at 0x49E1361: strcoll_l (strcoll_l.c:260)
> ==37899==    by 0x48DA9FF: NsTclStrcollObjCmd (tclmisc.c:2802)
> ==37899==    by 0x4BBC4A1: TclNRRunCallbacks (in 
> /usr/lib/x86_64-linux-gnu/libtcl8.6.so <http://libtcl8.6.so>)
> ==37899==    by 0x4BBD71F: ??? (in 
> /usr/lib/x86_64-linux-gnu/libtcl8.6.so <http://libtcl8.6.so>)
> ==37899==    by 0x4C794D8: Tcl_FSEvalFileEx (in 
> /usr/lib/x86_64-linux-gnu/libtcl8.6.so <http://libtcl8.6.so>)
> ==37899==    by 0x4C818AD: Tcl_MainEx (in 
> /usr/lib/x86_64-linux-gnu/libtcl8.6.so <http://libtcl8.6.so>)
> ==37899==    by 0x4B745AF: NsThreadMain (thread.c:232)
> ==37899==    by 0x4B75A48: ThreadMain (pthread.c:870)
> ==37899==    by 0x521BEA6: start_thread (pthread_create.c:477)
> ==37899==    by 0x4A4DDEE: clone (clone.S:95)
> ==37899==  If you believe this happened as a result of a stack
> ==37899==  overflow in your program's main thread (unlikely but
> ==37899==  possible), you can try to increase the size of the
> ==37899==  main thread stack using the --main-stacksize= flag.
> ==37899==  The main thread stack size used in this run was 8388608.
> ==37899==
> ==37899== HEAP SUMMARY:
> ==37899==     in use at exit: 12,499,085 bytes in 8,840 blocks
> ==37899==   total heap usage: 12,059 allocs, 3,219 frees, 29,314,210 
> bytes allocated
> ==37899==
> ==37899== LEAK SUMMARY:
> ==37899==    definitely lost: 131 bytes in 1 blocks
> ==37899==    indirectly lost: 0 bytes in 0 blocks
> ==37899==      possibly lost: 10,466,879 bytes in 2,978 blocks
> ==37899==    still reachable: 2,032,075 bytes in 5,861 blocks
> ==37899==         suppressed: 0 bytes in 0 blocks
> ==37899== Rerun with --leak-check=full to see details of leaked memory
> ==37899==
> ==37899== For lists of detected and suppressed errors, rerun with: -s
> ==37899== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
> Segmentation fault
> make: *** [Makefile:273: memcheck] Error 139
>
>
>
> On Wed, 6 Apr 2022 at 16:58, Gustaf Neumann <ne...@wu...> wrote:
>
>
>     On 06.04.22 16:46, David Osborne wrote:
>>
>>     On Wed, 6 Apr 2022 at 14:53, Gustaf Neumann <ne...@wu...> wrote:
>>
>>         Hi David,
>>
>>         i will setup a VM for testing in your configuration, but
>>         first i have to
>>         understand, what pt1/pt2 means.
>>
>>     *
>>     *
>>     *Sorry that is just an abbreviation for "part1" and "part2" of a
>>     2 part email.
>>     *
>
>     ok, i thought there is a version called "Debian Buster pt1"....
>     but could not find insights via googling :)
>
>>     *"tcl8.6" debian supplied package version 8.6.9+dfsg-2*
>
>     This seems to be a part of the problem. Tcl 8.6.9 was released in
>     nov 2018 and has
>     probably some issues with UTF-8 which were fixed in later releases.
>
>     i have just now installed NaviServer on a fresh Debian Buster
>     machine using my usual install script [1] (using Tcl 8.6.11) and
>     everything looks ok. It is not unlikely that the problem with
>     ns_strcoll is related, since one has to translate the "internal"
>     UTF-8 to the external variant before calling "strcoll_l()", so,
>     when this step is broken, then there might be some invalid memory
>     around.
>
>     For you, it would the best to use a newer version of Tcl. There
>     are newer Debian packages of Tcl around...
>
>     https://packages.debian.org/search?keywords=tcl
>
>     Is this an option for you?
>
>     Not sure, how NaviServer could address the problem. Deactivating
>     the ns_strcoll command in NaviServer when it is compiled with Tcl
>     8.6.9 or older, is probably no good option, since the
>     UTF-to-external conversion is now all over the place and the
>     problem will pop up at other places. We can consider deactivating
>     the UTF-to-external conversion altogether for older Tcl version
>     (requires several changes, including PostgreSQL driver) ... but
>     the many tests will fail as well, which have to be deactivated as
>     well.
>
>     What do you think?
>
>     -gn
>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

-- 
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"

Re: [naviserver-devel] v4.99.23 Tests failing building on Debian Buster Pt1: encoding_ns_http-1.1

[David O. <da...@qc...>]

Thanks very much,

For versions of Tcl less than 8.6.11 we're failing because it's a new test
exposing an old problem, is that correct?
This would explain why I don't see any test failures after
building v4.99.22 with Tcl 8.6.9 - it's only because 4.99.23 has introduced
the tests and not that 4.99.22 doesn't have the problem.

To test Tcl8.6.11 the easiest way for me is to jump to bullseye (Debian
v11) which provides 8.6.11+dfsg-1

Strangely I seem to get the same ns_strcoll seg fault with that.
But if I remove misc.test temporarily, all other tests pass happily.

# uname -a
Linux ip-172-0-1-190 5.10.0-12-cloud-amd64 #1 SMP Debian 5.10.103-1
(2022-03-07) x86_64 GNU/Linux

# cat /etc/debian_version
11.3

# ls -l /lib/x86_64-linux-gnu/libc.so.6
lrwxrwxrwx 1 root root 12 Mar 17 21:37 /lib/x86_64-linux-gnu/libc.so.6 ->
libc-2.31.so

# apt-cache policy tcl8.6
tcl8.6:
  Installed: 8.6.11+dfsg-1

# git clone https://bitbucket.org/naviserver/naviserver.git
Cloning into 'naviserver'...

# cd naviserver
# git checkout tags/naviserver-4.99.23
Note: switching to 'tags/naviserver-4.99.23'.

# ./autogen.sh --with-tcl=/usr/lib/tcl8.6 --enable-rpath --enable-threads
--enable-symbols
# make

Compiler warning for reference:

gcc   -Wall -fPIC -g -O2
-fdebug-prefix-map=/build/tcl8.6-qxVr7a/tcl8.6-8.6.11+dfsg=.
-fstack-protector-strong -Wformat -Werror=format-security
-fno-unit-at-a-time -pipe -Wdate-time -D_FORTIFY_SOURCE=2 -DNDEBUG
-DSYSTEM_MALLOC -DTCL_NO_DEPRECATED -std=c99 -I../include
-I"/usr/include/tcl8.6"   -DHAVE_CONFIG_H   -c -o tclenv.o tclenv.c
In file included from /usr/include/string.h:495,
                 from ../include/nsthread.h:378,
                 from ../include/ns.h:46,
                 from nsd.h:38,
                 from tclenv.c:37:
In function ‘strncat’,
    inlined from ‘PutEnv’ at tclenv.c:349:13:
/usr/include/x86_64-linux-gnu/bits/string_fortified.h:136:10: warning:
‘__builtin_strncat’ specified bound depends on the length of the source
argument [ ]8;;
https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wstringop-overflow=-Wstringop-overflow=
]8;;]
  136 |   return __builtin___strncat_chk (__dest, __src, __len, __bos
(__dest));
      |
 ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
tclenv.c: In function ‘PutEnv’:
tclenv.c:314:23: note: length computed here
  314 |         valueLength = strlen(value) + 1;
      |                       ^~~~~~~~~~~~~

# make memcheck TESTFLAGS="-verbose start -file misc.test"
---- ns_random-1.1 start
---- ns_fmttime-1.0 start
---- ns_fmttime-1.1 start
---- ns_trim-0.0 start
---- ns_trim-0.1 start
---- ns_trim-0.2 start
---- ns_trim-1.1 start
---- ns_trim-1.2 start
---- ns_trim-1.3 start
---- ns_trim-1.4 start
---- ns_trim-1.5 start
---- ns_trim-2.1 start
---- ns_trim-2.2 start
---- ns_quotehtml start
---- ns_strcoll-1.0.0 start
==37899== Thread 2:
==37899== Invalid read of size 8
==37899==    at 0x49E1361: strcoll_l (strcoll_l.c:260)
==37899==    by 0x48DA9FF: NsTclStrcollObjCmd (tclmisc.c:2802)
==37899==    by 0x4BBC4A1: TclNRRunCallbacks (in /usr/lib/x86_64-linux-gnu/
libtcl8.6.so)
==37899==    by 0x4BBD71F: ??? (in /usr/lib/x86_64-linux-gnu/libtcl8.6.so)
==37899==    by 0x4C794D8: Tcl_FSEvalFileEx (in /usr/lib/x86_64-linux-gnu/
libtcl8.6.so)
==37899==    by 0x4C818AD: Tcl_MainEx (in /usr/lib/x86_64-linux-gnu/
libtcl8.6.so)
==37899==    by 0x4B745AF: NsThreadMain (thread.c:232)
==37899==    by 0x4B75A48: ThreadMain (pthread.c:870)
==37899==    by 0x521BEA6: start_thread (pthread_create.c:477)
==37899==    by 0x4A4DDEE: clone (clone.S:95)
==37899==  Address 0x18 is not stack'd, malloc'd or (recently) free'd
==37899==
==37899==
==37899== Process terminating with default action of signal 11 (SIGSEGV)
==37899==  Access not within mapped region at address 0x18
==37899==    at 0x49E1361: strcoll_l (strcoll_l.c:260)
==37899==    by 0x48DA9FF: NsTclStrcollObjCmd (tclmisc.c:2802)
==37899==    by 0x4BBC4A1: TclNRRunCallbacks (in /usr/lib/x86_64-linux-gnu/
libtcl8.6.so)
==37899==    by 0x4BBD71F: ??? (in /usr/lib/x86_64-linux-gnu/libtcl8.6.so)
==37899==    by 0x4C794D8: Tcl_FSEvalFileEx (in /usr/lib/x86_64-linux-gnu/
libtcl8.6.so)
==37899==    by 0x4C818AD: Tcl_MainEx (in /usr/lib/x86_64-linux-gnu/
libtcl8.6.so)
==37899==    by 0x4B745AF: NsThreadMain (thread.c:232)
==37899==    by 0x4B75A48: ThreadMain (pthread.c:870)
==37899==    by 0x521BEA6: start_thread (pthread_create.c:477)
==37899==    by 0x4A4DDEE: clone (clone.S:95)
==37899==  If you believe this happened as a result of a stack
==37899==  overflow in your program's main thread (unlikely but
==37899==  possible), you can try to increase the size of the
==37899==  main thread stack using the --main-stacksize= flag.
==37899==  The main thread stack size used in this run was 8388608.
==37899==
==37899== HEAP SUMMARY:
==37899==     in use at exit: 12,499,085 bytes in 8,840 blocks
==37899==   total heap usage: 12,059 allocs, 3,219 frees, 29,314,210 bytes
allocated
==37899==
==37899== LEAK SUMMARY:
==37899==    definitely lost: 131 bytes in 1 blocks
==37899==    indirectly lost: 0 bytes in 0 blocks
==37899==      possibly lost: 10,466,879 bytes in 2,978 blocks
==37899==    still reachable: 2,032,075 bytes in 5,861 blocks
==37899==         suppressed: 0 bytes in 0 blocks
==37899== Rerun with --leak-check=full to see details of leaked memory
==37899==
==37899== For lists of detected and suppressed errors, rerun with: -s
==37899== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
Segmentation fault
make: *** [Makefile:273: memcheck] Error 139



On Wed, 6 Apr 2022 at 16:58, Gustaf Neumann <ne...@wu...> wrote:

>
> On 06.04.22 16:46, David Osborne wrote:
>
>
> On Wed, 6 Apr 2022 at 14:53, Gustaf Neumann <ne...@wu...> wrote:
>
>> Hi David,
>>
>> i will setup a VM for testing in your configuration, but first i have to
>> understand, what pt1/pt2 means.
>>
>
>
> *Sorry that is just an abbreviation for "part1" and "part2" of a 2 part
> email. *
>
> ok, i thought there is a version called "Debian Buster pt1".... but could
> not find insights via googling :)
>
>
> *"tcl8.6" debian supplied package version 8.6.9+dfsg-2*
>
> This seems to be a part of the problem. Tcl 8.6.9 was released in nov 2018
> and has
> probably some issues with UTF-8 which were fixed in later releases.
>
> i have just now installed NaviServer on a fresh Debian Buster machine
> using my usual install script [1] (using Tcl 8.6.11) and everything looks
> ok. It is not unlikely that the problem with ns_strcoll is related, since
> one has to translate the "internal" UTF-8 to the external variant before
> calling "strcoll_l()", so, when this step is broken, then there might be
> some invalid memory around.
>
> For you, it would the best to use a newer version of Tcl. There are newer
> Debian packages of Tcl around...
>
>     https://packages.debian.org/search?keywords=tcl
>
> Is this an option for you?
>
> Not sure, how NaviServer could address the problem. Deactivating the
> ns_strcoll command in NaviServer when it is compiled with Tcl 8.6.9 or
> older, is probably no good option, since the UTF-to-external conversion is
> now all over the place and the problem will pop up at other places. We can
> consider deactivating the UTF-to-external conversion altogether for older
> Tcl version (requires several changes, including PostgreSQL driver) ... but
> the many tests will fail as well, which have to be deactivated as well.
>
> What do you think?
>
> -gn
>

Re: [naviserver-devel] v4.99.23 Tests failing building on Debian Buster Pt1: encoding_ns_http-1.1

[Gustaf N. <ne...@wu...>]

On 06.04.22 16:46, David Osborne wrote:
>
> On Wed, 6 Apr 2022 at 14:53, Gustaf Neumann <ne...@wu...> wrote:
>
>     Hi David,
>
>     i will setup a VM for testing in your configuration, but first i
>     have to
>     understand, what pt1/pt2 means.
>
> *
> *
> *Sorry that is just an abbreviation for "part1" and "part2" of a 2 
> part email.
> *

ok, i thought there is a version called "Debian Buster pt1".... but 
could not find insights via googling :)

> *"tcl8.6" debian supplied package version 8.6.9+dfsg-2*

This seems to be a part of the problem. Tcl 8.6.9 was released in nov 
2018 and has
probably some issues with UTF-8 which were fixed in later releases.

i have just now installed NaviServer on a fresh Debian Buster machine 
using my usual install script [1] (using Tcl 8.6.11) and everything 
looks ok. It is not unlikely that the problem with ns_strcoll is 
related, since one has to translate the "internal" UTF-8 to the external 
variant before calling "strcoll_l()", so, when this step is broken, then 
there might be some invalid memory around.

For you, it would the best to use a newer version of Tcl. There are 
newer Debian packages of Tcl around...

https://packages.debian.org/search?keywords=tcl

Is this an option for you?

Not sure, how NaviServer could address the problem. Deactivating the 
ns_strcoll command in NaviServer when it is compiled with Tcl 8.6.9 or 
older, is probably no good option, since the UTF-to-external conversion 
is now all over the place and the problem will pop up at other places. 
We can consider deactivating the UTF-to-external conversion altogether 
for older Tcl version (requires several changes, including PostgreSQL 
driver) ... but the many tests will fail as well, which have to be 
deactivated as well.

What do you think?

-gn

    # cat /etc/debian_version
    10.12
    ...
    [06/Apr/2022:17:06:44][23784.7fa1b0695800][-main:conf-] Notice: nsmain: NaviServer/4.99.23 (tar-4.99.23) starting
    ...

    all.tcl:	Total	1860	Passed	1834	Skipped	26	Failed	0
    Sourced 72 Test Files.
    Number of tests skipped for each constraint:
	20	!usingExternalToUtf
	2	binaryMismatch
	1	copyAliasBug
	2	knownBug
	1	stress

   root@buster:/usr/local/src/naviserver-4.99.23#  uname -a
   Linux buster 4.19.0-9-amd64 #1 SMP Debian 4.19.118-2 (2020-04-29) x86_64 GNU/Linux

   root@buster:/usr/local/src/naviserver-4.99.23#  ls -l /lib/x86_64-linux-gnu/libc.so.6
   lrwxrwxrwx 1 root root 12 Mar 15 22:48 /lib/x86_64-linux-gnu/libc.so.6 -> libc-2.28.so

   root@buster:/usr/local/src/naviserver-4.99.23#  make memcheck TESTFLAGS="-verbose start -file misc.test"
   ...
   ---- ns_trim-1.5 start
   ---- ns_trim-2.1 start
   ---- ns_trim-2.2 start
   ---- ns_quotehtml start
   ---- ns_strcoll-1.0.0 start
   ---- ns_strcoll-1.0.1 start
   ---- ns_strcoll-1.0.2 start
   ---- ns_strcoll-1.1 start
   ---- ns_strcoll-1.2 start


[1] https://github.com/gustafn/install-ns


>     The setup based on install-ns.sh [1]  of the release was tested with:
>
>         macOS 11.6.2, Rocky Linux 8.5, Ubuntu 20.04, OpenBSD 6.9,
>     FreeBSD 13.1
>
>     The situation of strcoll is also platform dependent (depends on
>     version
>     of libc, e.g. the detail behavior is different on *BSD to the tested
>     Linux versions - but identical on the test cases). Maybe we have to
>     deactivate it for some platforms until these contain working versions.
>
>     all the best
>
>     -g
>
>     [1] https://github.com/gustafn/install-ns
>
>     On 06.04.22 13:24, David Osborne wrote:
>     > Hi there,
>     >
>     > We're trying to do a build of the official NaviServer v4.99.23
>     release
>     > (from the sourceforge tarball) on Debian Buster (10.12) but we're
>     > getting some failed tests.
>     >
>     > First one is encoding_ns_http-1.1
>     >
>     > Seems it's serving an Emoji but the expected content-length is
>     wrong
>     > upon receiving it.
>     > I see there's been some discussion about Emoji support since
>     4.99.23
>     > (which I don't fully understand) - not sure if that's relevant
>     here..
>     >
>     > * Test in question:
>     >
>     > test encoding_ns_http-1.1 {
>     >     Send body with ns_return and charset utf-8
>     > } -constraints usingExternalToUtf -setup {
>     >     ns_register_proc GET /encoding {
>     >         ns_return 200 "text/plain; charset=utf-8" "äöü😃"
>     >     }
>     > } -body {
>     >     set result [ns_http run [ns_config test listenurl]/encoding]
>     >     set headers [dict get $result headers]
>     >     list [dict get $result status] \
>     >         [ns_set iget $headers Content-Type] \
>     >         [ns_set iget $headers Content-Length] \
>     >         [dict get $result body]
>     > } -cleanup {
>     >     ns_unregister_op GET /encoding
>     > } -result [list 200 "text/plain; charset=utf-8" 10 "äöü😃"]
>     >
>     >
>     > * Reproduction steps below.
>     >
>     > $ uname -a
>     > Linux ip-172-0-1-61 4.19.0-18-cloud-amd64 #1 SMP Debian 4.19.208-1
>     > (2021-09-29) x86_64 GNU/Linux
>     > $ cat /etc/debian_version
>     > 10.12
>     > $ locale
>     > LANG=C.UTF-8
>     > LANGUAGE=
>     > LC_CTYPE="C.UTF-8"
>     > LC_NUMERIC="C.UTF-8"
>     > LC_TIME="C.UTF-8"
>     > LC_COLLATE="C.UTF-8"
>     > LC_MONETARY="C.UTF-8"
>     > LC_MESSAGES="C.UTF-8"
>     > LC_PAPER="C.UTF-8"
>     > LC_NAME="C.UTF-8"
>     > LC_ADDRESS="C.UTF-8"
>     > LC_TELEPHONE="C.UTF-8"
>     > LC_MEASUREMENT="C.UTF-8"
>     > LC_IDENTIFICATION="C.UTF-8"
>     > LC_ALL=
>     >
>     > $ apt-get install build-essential git automake tcl8.6-dev libssl-dev
>     > $ cd naviserver-4.99.23
>     > $ ./autogen.sh --disable-ipv6 --with-tcl=/usr/lib/tcl8.6
>     > --enable-rpath --enable-threads --enable-symbols
>     > $ make
>     > $ make test
>     > <snip>
>     > ==== encoding_ns_http-1.1 Send body with ns_return and charset
>     utf-8
>     > FAILED
>     > ==== Contents of test case:
>     >
>     >     set result [ns_http run [ns_config test listenurl]/encoding]
>     >     set headers [dict get $result headers]
>     >     list [dict get $result status]  [ns_set iget $headers
>     > Content-Type]  [ns_set iget $headers Content-Length]  [dict get
>     > $result body]
>     >
>     > ---- Result was:
>     > 200 {text/plain; charset=utf-8} 14 äöü😃
>     > ---- Result should have been (exact matching):
>     > 200 {text/plain; charset=utf-8} 10 äöü😃
>     > ==== encoding_ns_http-1.1 FAILED
>     >
>     > Can anyone shed any light?
>     > Regards,
>     > --
>     > David
>
>
>
>     _______________________________________________
>     naviserver-devel mailing list
>     nav...@li...
>     https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>
>
>
> -- 
> David
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

-- 
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"

Re: [naviserver-devel] v4.99.23 Tests failing building on Debian Buster Pt1: encoding_ns_http-1.1

[David O. <da...@qc...>]

Thanks - answers inline....

On Wed, 6 Apr 2022 at 14:53, Gustaf Neumann <ne...@wu...> wrote:

> Hi David,
>
> i will setup a VM for testing in your configuration, but first i have to
> understand, what pt1/pt2 means.
>



*Sorry that is just an abbreviation for "part1" and "part2" of a 2 part
email.I thought it would be confusing reporting both problems in the same
email so I split it (and therefore unintentionally made it more
confusing!).Both problems are occuring on the exact same build.*


> Is it sufficient to set up a Debian Buster with all available updates?
>

*Yes. I used a standard buster **amd64 **image on AWS then ran "apt-get
upgrade" before building NaviServer*


>
> The emoji/UTF-8 problem points to a Tcl problem. What exact version of
> Tcl is used in this installation?
>
>
*"tcl8.6" debian supplied package version 8.6.9+dfsg-2*


> The setup based on install-ns.sh [1]  of the release was tested with:
>
>     macOS 11.6.2, Rocky Linux 8.5, Ubuntu 20.04, OpenBSD 6.9, FreeBSD 13.1
>
> The situation of strcoll is also platform dependent (depends on version
> of libc, e.g. the detail behavior is different on *BSD to the tested
> Linux versions - but identical on the test cases). Maybe we have to
> deactivate it for some platforms until these contain working versions.
>
> all the best
>
> -g
>
> [1] https://github.com/gustafn/install-ns
>
> On 06.04.22 13:24, David Osborne wrote:
> > Hi there,
> >
> > We're trying to do a build of the official NaviServer v4.99.23 release
> > (from the sourceforge tarball) on Debian Buster (10.12) but we're
> > getting some failed tests.
> >
> > First one is encoding_ns_http-1.1
> >
> > Seems it's serving an Emoji but the expected content-length is wrong
> > upon receiving it.
> > I see there's been some discussion about Emoji support since 4.99.23
> > (which I don't fully understand) - not sure if that's relevant here..
> >
> > * Test in question:
> >
> > test encoding_ns_http-1.1 {
> >     Send body with ns_return and charset utf-8
> > } -constraints usingExternalToUtf -setup {
> >     ns_register_proc GET /encoding {
> >         ns_return 200 "text/plain; charset=utf-8" "äöü😃"
> >     }
> > } -body {
> >     set result [ns_http run [ns_config test listenurl]/encoding]
> >     set headers [dict get $result headers]
> >     list [dict get $result status] \
> >         [ns_set iget $headers Content-Type] \
> >         [ns_set iget $headers Content-Length] \
> >         [dict get $result body]
> > } -cleanup {
> >     ns_unregister_op GET /encoding
> > } -result [list 200 "text/plain; charset=utf-8" 10 "äöü😃"]
> >
> >
> > * Reproduction steps below.
> >
> > $ uname -a
> > Linux ip-172-0-1-61 4.19.0-18-cloud-amd64 #1 SMP Debian 4.19.208-1
> > (2021-09-29) x86_64 GNU/Linux
> > $ cat /etc/debian_version
> > 10.12
> > $ locale
> > LANG=C.UTF-8
> > LANGUAGE=
> > LC_CTYPE="C.UTF-8"
> > LC_NUMERIC="C.UTF-8"
> > LC_TIME="C.UTF-8"
> > LC_COLLATE="C.UTF-8"
> > LC_MONETARY="C.UTF-8"
> > LC_MESSAGES="C.UTF-8"
> > LC_PAPER="C.UTF-8"
> > LC_NAME="C.UTF-8"
> > LC_ADDRESS="C.UTF-8"
> > LC_TELEPHONE="C.UTF-8"
> > LC_MEASUREMENT="C.UTF-8"
> > LC_IDENTIFICATION="C.UTF-8"
> > LC_ALL=
> >
> > $ apt-get install build-essential git automake tcl8.6-dev libssl-dev
> > $ cd naviserver-4.99.23
> > $ ./autogen.sh --disable-ipv6 --with-tcl=/usr/lib/tcl8.6
> > --enable-rpath --enable-threads --enable-symbols
> > $ make
> > $ make test
> > <snip>
> > ==== encoding_ns_http-1.1 Send body with ns_return and charset utf-8
> > FAILED
> > ==== Contents of test case:
> >
> >     set result [ns_http run [ns_config test listenurl]/encoding]
> >     set headers [dict get $result headers]
> >     list [dict get $result status]  [ns_set iget $headers
> > Content-Type]  [ns_set iget $headers Content-Length]  [dict get
> > $result body]
> >
> > ---- Result was:
> > 200 {text/plain; charset=utf-8} 14 äöü😃
> > ---- Result should have been (exact matching):
> > 200 {text/plain; charset=utf-8} 10 äöü😃
> > ==== encoding_ns_http-1.1 FAILED
> >
> > Can anyone shed any light?
> > Regards,
> > --
> > David
>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>


-- 
David

Re: [naviserver-devel] v4.99.23 Tests failing building on Debian Buster Pt1: encoding_ns_http-1.1

[Wolfgang W. <wol...@di...>]

Hi!

We are using Debian Buster without any problems.

Here are some details:

cat /etc/debian_version
10.12

dcweb:nscp 1> info patchlevel
8.6.11

dcweb:nscp 1> ns_info patchlevel
4.99.23

This is our configure script:

# --with-openssl=/usr/local assumes a manually installed openssl 
version. For debian buster (10), an entry in
# /etc/ld.so.conf.d/libc.conf for /usr/local/lib64 and calling 
"ldconfig" once is necessary

./configure --enable-64bit --prefix=/usr/local/naviserver 
--with-openssl=/usr/local/ --with-tcl=/usr/local/lib/ --enable-threads


Tcl is self compiled.

Yours,

Wolfgang


Am 06.04.22 um 15:53 schrieb Gustaf Neumann:
> Hi David,
>
> i will setup a VM for testing in your configuration, but first i have 
> to understand, what pt1/pt2 means.
> Is it sufficient to set up a Debian Buster with all available updates?
>
> The emoji/UTF-8 problem points to a Tcl problem. What exact version of 
> Tcl is used in this installation?
>
> The setup based on install-ns.sh [1]  of the release was tested with:
>
>    macOS 11.6.2, Rocky Linux 8.5, Ubuntu 20.04, OpenBSD 6.9, FreeBSD 13.1
>
> The situation of strcoll is also platform dependent (depends on 
> version of libc, e.g. the detail behavior is different on *BSD to the 
> tested Linux versions - but identical on the test cases). Maybe we 
> have to deactivate it for some platforms until these contain working 
> versions.
>
> all the best
>
> -g
>
> [1] https://github.com/gustafn/install-ns
>
> On 06.04.22 13:24, David Osborne wrote:
>> Hi there,
>>
>> We're trying to do a build of the official NaviServer v4.99.23 
>> release (from the sourceforge tarball) on Debian Buster (10.12) but 
>> we're getting some failed tests.
>>
>> First one is encoding_ns_http-1.1
>>
>> Seems it's serving an Emoji but the expected content-length is wrong 
>> upon receiving it.
>> I see there's been some discussion about Emoji support since 4.99.23 
>> (which I don't fully understand) - not sure if that's relevant here..
>>
>> * Test in question:
>>
>> test encoding_ns_http-1.1 {
>>     Send body with ns_return and charset utf-8
>> } -constraints usingExternalToUtf -setup {
>>     ns_register_proc GET /encoding {
>>         ns_return 200 "text/plain; charset=utf-8" "äöü😃"
>>     }
>> } -body {
>>     set result [ns_http run [ns_config test listenurl]/encoding]
>>     set headers [dict get $result headers]
>>     list [dict get $result status] \
>>         [ns_set iget $headers Content-Type] \
>>         [ns_set iget $headers Content-Length] \
>>         [dict get $result body]
>> } -cleanup {
>>     ns_unregister_op GET /encoding
>> } -result [list 200 "text/plain; charset=utf-8" 10 "äöü😃"]
>>
>>
>> * Reproduction steps below.
>>
>> $ uname -a
>> Linux ip-172-0-1-61 4.19.0-18-cloud-amd64 #1 SMP Debian 4.19.208-1 
>> (2021-09-29) x86_64 GNU/Linux
>> $ cat /etc/debian_version
>> 10.12
>> $ locale
>> LANG=C.UTF-8
>> LANGUAGE=
>> LC_CTYPE="C.UTF-8"
>> LC_NUMERIC="C.UTF-8"
>> LC_TIME="C.UTF-8"
>> LC_COLLATE="C.UTF-8"
>> LC_MONETARY="C.UTF-8"
>> LC_MESSAGES="C.UTF-8"
>> LC_PAPER="C.UTF-8"
>> LC_NAME="C.UTF-8"
>> LC_ADDRESS="C.UTF-8"
>> LC_TELEPHONE="C.UTF-8"
>> LC_MEASUREMENT="C.UTF-8"
>> LC_IDENTIFICATION="C.UTF-8"
>> LC_ALL=
>>
>> $ apt-get install build-essential git automake tcl8.6-dev libssl-dev
>> $ cd naviserver-4.99.23
>> $ ./autogen.sh --disable-ipv6 --with-tcl=/usr/lib/tcl8.6 
>> --enable-rpath --enable-threads --enable-symbols
>> $ make
>> $ make test
>> <snip>
>> ==== encoding_ns_http-1.1 Send body with ns_return and charset utf-8 
>> FAILED
>> ==== Contents of test case:
>>
>>     set result [ns_http run [ns_config test listenurl]/encoding]
>>     set headers [dict get $result headers]
>>     list [dict get $result status]  [ns_set iget $headers 
>> Content-Type]  [ns_set iget $headers Content-Length]  [dict get 
>> $result body]
>>
>> ---- Result was:
>> 200 {text/plain; charset=utf-8} 14 äöü😃
>> ---- Result should have been (exact matching):
>> 200 {text/plain; charset=utf-8} 10 äöü😃
>> ==== encoding_ns_http-1.1 FAILED
>>
>> Can anyone shed any light?
>> Regards,
>> -- 
>> David
>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
-- 

*Wolfgang Winkler*
Geschäftsführung
wol...@di...
mobil +43.699.19971172

dc:*büro*
digital concepts Novak Winkler OG
Software & Design
Landstraße 68, 5. Stock, 4020 Linz
www.digital-concepts.com <http://www.digital-concepts.com>
tel +43.732.997117.72
tel +43.699.1997117.2

Firmenbuchnummer: 192003h
Firmenbuchgericht: Landesgericht Linz

Re: [naviserver-devel] v4.99.23 Tests failing building on Debian Buster Pt1: encoding_ns_http-1.1

[Gustaf N. <ne...@wu...>]

Hi David,

i will setup a VM for testing in your configuration, but first i have to 
understand, what pt1/pt2 means.
Is it sufficient to set up a Debian Buster with all available updates?

The emoji/UTF-8 problem points to a Tcl problem. What exact version of 
Tcl is used in this installation?

The setup based on install-ns.sh [1]  of the release was tested with:

    macOS 11.6.2, Rocky Linux 8.5, Ubuntu 20.04, OpenBSD 6.9, FreeBSD 13.1

The situation of strcoll is also platform dependent (depends on version 
of libc, e.g. the detail behavior is different on *BSD to the tested 
Linux versions - but identical on the test cases). Maybe we have to 
deactivate it for some platforms until these contain working versions.

all the best

-g

[1] https://github.com/gustafn/install-ns

On 06.04.22 13:24, David Osborne wrote:
> Hi there,
>
> We're trying to do a build of the official NaviServer v4.99.23 release 
> (from the sourceforge tarball) on Debian Buster (10.12) but we're 
> getting some failed tests.
>
> First one is encoding_ns_http-1.1
>
> Seems it's serving an Emoji but the expected content-length is wrong 
> upon receiving it.
> I see there's been some discussion about Emoji support since 4.99.23 
> (which I don't fully understand) - not sure if that's relevant here..
>
> * Test in question:
>
> test encoding_ns_http-1.1 {
>     Send body with ns_return and charset utf-8
> } -constraints usingExternalToUtf -setup {
>     ns_register_proc GET /encoding {
>         ns_return 200 "text/plain; charset=utf-8" "äöü😃"
>     }
> } -body {
>     set result [ns_http run [ns_config test listenurl]/encoding]
>     set headers [dict get $result headers]
>     list [dict get $result status] \
>         [ns_set iget $headers Content-Type] \
>         [ns_set iget $headers Content-Length] \
>         [dict get $result body]
> } -cleanup {
>     ns_unregister_op GET /encoding
> } -result [list 200 "text/plain; charset=utf-8" 10 "äöü😃"]
>
>
> * Reproduction steps below.
>
> $ uname -a
> Linux ip-172-0-1-61 4.19.0-18-cloud-amd64 #1 SMP Debian 4.19.208-1 
> (2021-09-29) x86_64 GNU/Linux
> $ cat /etc/debian_version
> 10.12
> $ locale
> LANG=C.UTF-8
> LANGUAGE=
> LC_CTYPE="C.UTF-8"
> LC_NUMERIC="C.UTF-8"
> LC_TIME="C.UTF-8"
> LC_COLLATE="C.UTF-8"
> LC_MONETARY="C.UTF-8"
> LC_MESSAGES="C.UTF-8"
> LC_PAPER="C.UTF-8"
> LC_NAME="C.UTF-8"
> LC_ADDRESS="C.UTF-8"
> LC_TELEPHONE="C.UTF-8"
> LC_MEASUREMENT="C.UTF-8"
> LC_IDENTIFICATION="C.UTF-8"
> LC_ALL=
>
> $ apt-get install build-essential git automake tcl8.6-dev libssl-dev
> $ cd naviserver-4.99.23
> $ ./autogen.sh --disable-ipv6 --with-tcl=/usr/lib/tcl8.6 
> --enable-rpath --enable-threads --enable-symbols
> $ make
> $ make test
> <snip>
> ==== encoding_ns_http-1.1 Send body with ns_return and charset utf-8 
> FAILED
> ==== Contents of test case:
>
>     set result [ns_http run [ns_config test listenurl]/encoding]
>     set headers [dict get $result headers]
>     list [dict get $result status]  [ns_set iget $headers 
> Content-Type]  [ns_set iget $headers Content-Length]  [dict get 
> $result body]
>
> ---- Result was:
> 200 {text/plain; charset=utf-8} 14 äöü😃
> ---- Result should have been (exact matching):
> 200 {text/plain; charset=utf-8} 10 äöü😃
> ==== encoding_ns_http-1.1 FAILED
>
> Can anyone shed any light?
> Regards,
> --
> David

[naviserver-devel] v4.99.23 Tests failing building on Debian Buster Pt2: ns_strcoll-1.0.0

[David O. <da...@qc...>]

Still building of the official NaviServer v4.99.23 release on Debian
Buster, we are also seeing some SIGSEGVs starting from ns_strcoll-1.0.0

test ns_strcoll-1.0.0 {ns_strcoll without locale (assuming en_US.UTF-8)} \
    -constraints localeCollate -body {
        return [expr {[ns_strcoll Bär Bor] < 0}]
    } -result 1

Running make memcheck I get the following output for the first signal 11

==19345== Thread 2:
==19345== Invalid read of size 8
==19345==    at 0x5113371: strcoll_l (strcoll_l.c:260)
==19345==    by 0x48D51DA: NsTclStrcollObjCmd (tclmisc.c:2802)
==19345==    by 0x49EDFB6: TclNRRunCallbacks (in /usr/lib/x86_64-linux-gnu/
libtcl8.6.so)
==19345==    by 0x49EF3AE: ??? (in /usr/lib/x86_64-linux-gnu/libtcl8.6.so)
==19345==    by 0x4AA83F7: Tcl_FSEvalFileEx (in /usr/lib/x86_64-linux-gnu/
libtcl8.6.so)
==19345==    by 0x4AB0246: Tcl_MainEx (in /usr/lib/x86_64-linux-gnu/
libtcl8.6.so)
==19345==    by 0x49A527B: NsThreadMain (thread.c:232)
==19345==    by 0x49A64C8: ThreadMain (pthread.c:870)
==19345==    by 0x5256FA2: start_thread (pthread_create.c:486)
==19345==    by 0x5180EFE: clone (clone.S:95)
==19345==  Address 0x18 is not stack'd, malloc'd or (recently) free'd
==19345==
==19345==
==19345== Process terminating with default action of signal 11 (SIGSEGV)
==19345==  Access not within mapped region at address 0x18
==19345==    at 0x5113371: strcoll_l (strcoll_l.c:260)
==19345==    by 0x48D51DA: NsTclStrcollObjCmd (tclmisc.c:2802)
==19345==    by 0x49EDFB6: TclNRRunCallbacks (in /usr/lib/x86_64-linux-gnu/
libtcl8.6.so)
==19345==    by 0x49EF3AE: ??? (in /usr/lib/x86_64-linux-gnu/libtcl8.6.so)
==19345==    by 0x4AA83F7: Tcl_FSEvalFileEx (in /usr/lib/x86_64-linux-gnu/
libtcl8.6.so)
==19345==    by 0x4AB0246: Tcl_MainEx (in /usr/lib/x86_64-linux-gnu/
libtcl8.6.so)
==19345==    by 0x49A527B: NsThreadMain (thread.c:232)
==19345==    by 0x49A64C8: ThreadMain (pthread.c:870)
==19345==    by 0x5256FA2: start_thread (pthread_create.c:486)
==19345==    by 0x5180EFE: clone (clone.S:95)
==19345==  If you believe this happened as a result of a stack
==19345==  overflow in your program's main thread (unlikely but
==19345==  possible), you can try to increase the size of the
==19345==  main thread stack using the --main-stacksize= flag.
==19345==  The main thread stack size used in this run was 8388608.
==19345==
==19345== HEAP SUMMARY:
==19345==     in use at exit: 12,996,807 bytes in 12,198 blocks
==19345==   total heap usage: 40,995 allocs, 28,797 frees, 41,780,348 bytes
allocated
==19345==
==19345== LEAK SUMMARY:
==19345==    definitely lost: 179 bytes in 3 blocks
==19345==    indirectly lost: 0 bytes in 0 blocks
==19345==      possibly lost: 11,145,880 bytes in 6,013 blocks
==19345==    still reachable: 1,850,748 bytes in 6,182 blocks
==19345==         suppressed: 0 bytes in 0 blocks
==19345== Rerun with --leak-check=full to see details of leaked memory
==19345==
==19345== For counts of detected and suppressed errors, rerun with: -v
==19345== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
Segmentation fault
make: *** [Makefile:273: memcheck] Error 139


Can you advise on the best course of action?

Regards,
--
David

[naviserver-devel] v4.99.23 Tests failing building on Debian Buster Pt1: encoding_ns_http-1.1

[David O. <da...@qc...>]

Hi there,

We're trying to do a build of the official NaviServer v4.99.23 release
(from the sourceforge tarball) on Debian Buster (10.12) but we're getting
some failed tests.

First one is encoding_ns_http-1.1

Seems it's serving an Emoji but the expected content-length is wrong upon
receiving it.
I see there's been some discussion about Emoji support since 4.99.23 (which
I don't fully understand) - not sure if that's relevant here..

* Test in question:

test encoding_ns_http-1.1 {
    Send body with ns_return and charset utf-8
} -constraints usingExternalToUtf -setup {
    ns_register_proc GET /encoding {
        ns_return 200 "text/plain; charset=utf-8" "äöü😃"
    }
} -body {
    set result [ns_http run [ns_config test listenurl]/encoding]
    set headers [dict get $result headers]
    list [dict get $result status] \
        [ns_set iget $headers Content-Type] \
        [ns_set iget $headers Content-Length] \
        [dict get $result body]
} -cleanup {
    ns_unregister_op GET /encoding
} -result [list 200 "text/plain; charset=utf-8" 10 "äöü😃"]


* Reproduction steps below.

$ uname -a
Linux ip-172-0-1-61 4.19.0-18-cloud-amd64 #1 SMP Debian 4.19.208-1
(2021-09-29) x86_64 GNU/Linux
$ cat /etc/debian_version
10.12
$ locale
LANG=C.UTF-8
LANGUAGE=
LC_CTYPE="C.UTF-8"
LC_NUMERIC="C.UTF-8"
LC_TIME="C.UTF-8"
LC_COLLATE="C.UTF-8"
LC_MONETARY="C.UTF-8"
LC_MESSAGES="C.UTF-8"
LC_PAPER="C.UTF-8"
LC_NAME="C.UTF-8"
LC_ADDRESS="C.UTF-8"
LC_TELEPHONE="C.UTF-8"
LC_MEASUREMENT="C.UTF-8"
LC_IDENTIFICATION="C.UTF-8"
LC_ALL=

$ apt-get install build-essential git automake tcl8.6-dev libssl-dev
$ cd naviserver-4.99.23
$ ./autogen.sh --disable-ipv6 --with-tcl=/usr/lib/tcl8.6 --enable-rpath
--enable-threads --enable-symbols
$ make
$ make test
<snip>
==== encoding_ns_http-1.1 Send body with ns_return and charset utf-8 FAILED
==== Contents of test case:

    set result [ns_http run [ns_config test listenurl]/encoding]
    set headers [dict get $result headers]
    list [dict get $result status]  [ns_set iget $headers Content-Type]
 [ns_set iget $headers Content-Length]  [dict get $result body]

---- Result was:
200 {text/plain; charset=utf-8} 14 äöü😃
---- Result should have been (exact matching):
200 {text/plain; charset=utf-8} 10 äöü😃
==== encoding_ns_http-1.1 FAILED

Can anyone shed any light?
Regards,
--
David

Re: [naviserver-devel] Issue with certain emojis (unicode/utf-8)

[Gustaf N. <ne...@wu...>]

Just as a short notice: iOS 14.5 (released a few days ago) supports some 
more Unicode 14 characters, iOS 15 is supposed to support all of Unicode 
14.0.
The melting face of Unicode 14 on the test-page on openacs.org (see link 
below) works already.

-gn

On 04.12.21 15:57, Gustaf Neumann wrote:
> It will take some time, until the Emojis from Unicode 14 will be 
> generally available, but when this comes, we should have already 
> everything working in NaviServer and the DB interfaces. I've added a 
> small demo page, one can try when the new clients come out:
>
> https://openacs.org/emojis.tcl
>

[naviserver-devel] emojis and their consequences

[Gustaf N. <ne...@wu...>]

Dear all,

There are more changes related to this problem area:

a) Due to the full support of UTF-8 in the database interface in the 
last release, potential new problems showed up which were hidden so far 
by the mangled Tcl-UTF-8; similarly, problems showed up with 
vulnerability scanners trying to inject invalid UTF-8, causing then some 
extensions (expecting only valid UTF-8) to fatal out (e.g. tDOM). These 
issues were addressed by the stronger input validation changes of the 
last weeks and months since the release.

b) For full emoji support, it is also necessary to support emojis 
specified as numeric entities in HTML markup. The old versions of 
NaviServer were just capable of handling single byte decimal numeric 
entities, now multibyte decimal or hexadecimal numeric entities are 
supported as well (see e.g., in the regression test the mermaids with 
the light and dark skin tones [1]). Since HTML entity interpretation was 
before only available through "ns_striphtml" (which  does also comment 
and tag stripping), I have added the command "ns_unquotehtml" as a 
counterpart to "ns_quotehtml" which just interprets numeric and 
non-numeric entities.

The next release should come out around easter.

all the best

-gn

[1] 
https://bitbucket.org/naviserver/naviserver/commits/b923ad4384529a80ac88cadcadde1947a6413753#Ltests/ns_striphtml.testT369

Re: [naviserver-devel] Fwd: NaviServer Wins an Award from SourceForge

[Brian F. <Bri...@qu...>]

Great news, and thanks to you and your team, Gustaf for this tremendous work.

Brian
________________________________
From: Gustaf Neumann <ne...@wu...>
Sent: Wednesday 2 March 2022 09:23
To: Navidevel <nav...@li...>
Subject: [naviserver-devel] Fwd: NaviServer Wins an Award from SourceForge


This message's attachments contains at least one web link. This is often used for phishing attempts. Please only interact with this attachment if you know its source and that the content is safe. If in doubt, confirm the legitimacy with the sender by phone.


Dear all,

NaviServer has just been recognized with the Community Choice award by SourceForge, which - to my understanding - reflects in essence the number of downloads from sourceforge. As an official winner of the Community Choice Award, we have the permission to use the award badge wherever appropriate. If you are using NaviServer in your products, you might use this information and badge to advertise using an awarded server.

All the best

-gustaf neumann

-------- Forwarded Message --------
Subject:        NaviServer Wins an Award from SourceForge
Date:   Tue, 01 Mar 2022 23:23:35 +0000
From:   SourceForge <no...@so...><mailto:no...@so...>
To:     ne...@wu...<mailto:ne...@wu...>


‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌
[SourceForge] <https://sourceforge.net>

Find, Create, & Publish Open Source Software for free.

Hi gustafn,

Congratulations! NaviServer has just been recognized with a Community Choice award by SourceForge. This honor is awarded only to select projects that have reached significant milestones in terms of downloads and user engagement from the SourceForge community.

This is a big achievement, as your project has qualified for this award out of over 500,000 open source projects on SourceForge. SourceForge sees nearly 30 million users per month looking for, and developing, open source software. This award badge will now appear on your project page, and the award assets can be found in your project admin section.

To recognize NaviServer’s achievement, we’ve awarded you with the Community Choice award, which you can see below: [https://a.fsdn.com/con/img/sandiego/svg/originals/badges/examples/oss-community-choice.png]  Now that NaviServer is an official winner of the Community Choice Award, you have express permission to use the award badge wherever you’d like.

Feel free to proudly display the award on your personal or organizational website, social media, or anywhere else you’d like.

You can get the award badge assets here.<https://sourceforge.net/p/naviserver/admin/files/badges/>

Congrats again on winning and keep doing amazing work because SourceForge and our users appreciate it!

Thanks,

The SourceForge Team

This e-mail was intended for: ne...@wu...<mailto:ne...@wu...>

Unsubscribe from these notifications<https://sourceforge.net/cdn/syndication/unsubscribe?email=ne...@wu...> Privacy policy<https://slashdotmedia.com/privacy-statement/>

PO BOX 2452, La Jolla, CA 92038

[SF Logo] <https://sourceforge.net>

[naviserver-devel] Fwd: NaviServer Wins an Award from SourceForge

[Gustaf N. <ne...@wu...>]

Dear all,

NaviServer has just been recognized with the Community Choice award by 
SourceForge, which - to my understanding - reflects in essence the 
number of downloads from sourceforge. As an official winner of the 
Community Choice Award, we have the permission to use the award badge 
wherever appropriate. If you are using NaviServer in your products, you 
might use this information and badge to advertise using an awarded server.

All the best

-gustaf neumann

-------- Forwarded Message --------
Subject: 	NaviServer Wins an Award from SourceForge
Date: 	Tue, 01 Mar 2022 23:23:35 +0000
From: 	SourceForge <no...@so...>
To: 	ne...@wu...



‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌
SourceForge <https://sourceforge.net> 	

*Find*, *Create*, & *Publish* Open Source Software for *free*.

Hi gustafn,

Congratulations! NaviServer has just been recognized with a Community 
Choice award by SourceForge. This honor is awarded only to select 
projects that have reached significant milestones in terms of downloads 
and user engagement from the SourceForge community.

This is a big achievement, as your project has qualified for this award 
out of over 500,000 open source projects on SourceForge. SourceForge 
sees nearly 30 million users per month looking for, and developing, open 
source software. This award badge will now appear on your project page, 
and the award assets can be found in your project admin section.

To recognize NaviServer’s achievement, we’ve awarded you with the 
Community Choice award, which you can see below: Now that NaviServer is 
an official winner of the Community Choice Award, you have express 
permission to use the award badge wherever you’d like.

Feel free to proudly display the award on your personal or 
organizational website, social media, or anywhere else you’d like.

*You can get the award badge assets here. 
<https://sourceforge.net/p/naviserver/admin/files/badges/>*

Congrats again on winning and keep doing amazing work because 
SourceForge and our users appreciate it!

Thanks,

The SourceForge Team

This e-mail was intended for: ne...@wu...

Unsubscribe from these notifications 
<https://sourceforge.net/cdn/syndication/unsubscribe?email=ne...@wu...> 
Privacy policy <https://slashdotmedia.com/privacy-statement/>

PO BOX 2452, La Jolla, CA 92038

	SF Logo <https://sourceforge.net>

Re: [naviserver-devel] ns_parseurl: small problem

[Gustaf N. <ne...@wu...>]

Dear Wolfgang,

ns_parseurl is intended to parse a compete URL, including scheme, 
authority (host, port),
path, query and fragment

          foo://example.com:8042/over/there?name=ferret#nose
          \_/   \______________/\_________/ \_________/ \__/
           |           |            |            |        |
        scheme     authority       path        query   fragment

The path has to start with a slash, so it tries to parse the scheme, 
when the first
characters of the scheme look ok. When the scheme is not properly 
terminated,
it complains.  In your example, it found the terminating colon.... I 
have now
made the parsing more robust, also in the cases where these are not fully
percent-encoded, while keeping the error cases constant.

We see now the following results

% ns_parseurl "/test/index?url=http://www.test.at"
path test tail index query url=http://www.test.at

% ns_parseurl "index?url=http://www.test.com"
tail index query url=http://www.test.com

% ns_parseurl "/index?url=http://www.test.com"
path {} tail index query url=http://www.test.com

% ns_parseurl "index?url=https//www.test.com"
tail index query url=https//www.test.com

Altogether, we have now 81 tests for the URL parsing...
The changes a committed on Bitbucket.

Let me know, in case you found other questionable cases.

all the best

-gn

On 14.01.22 08:34, Wolfgang Winkler via naviserver-devel wrote:
>
> Hello all!
>
> We encountered a small problem with ns_parseurl:
>
> When parsing
>
> ns_parseurl "/test/index?url=http://www.test.at"
>
> everything works fine.
>
> With
>
> ns_parseurl "index?url=http://www.test.com"
>
> We get "Could not parse URL "index?url=https://www.test.com": invalid 
> scheme"
>
>   * ns_parseurl "/index?url=http://www.test.com"
>   * ns_parseurl "index?url=https//www.test.com" (notice the missing ":")
>
> work as well.
>
> The "url" Parameter is encoded with ns_urlencode. When encoding with JS:
>
> encodeURIComponent("http://www.test.com");
>
> we get
>
> 'http%3A%2F%2Fwww.test.com'
>
> Using this value:
>
> ns_parseurl "index?url=http%3A%2F%2Fwww.test.com"
>
> works. ns_urldecode decodes the value correctly.
>
> Regards,
>
> Wolfgang
>
>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

-- 
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"

[naviserver-devel] ns_parseurl: small problem

[Wolfgang W. <wol...@di...>]

Hello all!

We encountered a small problem with ns_parseurl:

When parsing

ns_parseurl "/test/index?url=http://www.test.at"

everything works fine.

With

ns_parseurl "index?url=http://www.test.com"

We get "Could not parse URL "index?url=https://www.test.com": invalid 
scheme"

  * ns_parseurl "/index?url=http://www.test.com"
  * ns_parseurl "index?url=https//www.test.com" (notice the missing ":")

work as well.

The "url" Parameter is encoded with ns_urlencode. When encoding with JS:

encodeURIComponent("http://www.test.com");

we get

'http%3A%2F%2Fwww.test.com'

Using this value:

ns_parseurl "index?url=http%3A%2F%2Fwww.test.com"

works. ns_urldecode decodes the value correctly.

Regards,

Wolfgang

Re: [naviserver-devel] Naviserver doesn't compile against tcl8.7-alpha5

[Gustaf N. <ne...@wu...>]

Dear Fox,

The function Tcl_PackageInitProc was apparently deprecated recently for 
Tcl 8.7 (TIP #595, [1]).

Therefore, for compiling the released version of NaviServer with Tcl 
8.7a5, remove the flag -DTCL_NO_DEPRECATED from the compiler flags, and 
everything should be fine.

We should probably wait for the release of Tcl 8.7 for switching to the 
non-deprecated function name "Tcl_LibraryInitProc". In the meantime, i 
have remove the -DTCL_NO_DEPRECATED flag from the default compile options.

all the best

-gn

[1] 
https://github.com/tcltk/tcl/commit/06813163a831a73bcdd74a5341bbc547fbb3e342

On 10.01.22 01:51, D.Fox wrote:
> ...
> gmake[1]: Entering directory '/srv/_scrap/naviserver-4.99.23/nsd'
> gcc  -O2 -Wall -fPIC  -pipe  -finput-charset=UTF-8 -DNDEBUG 
> -DSYSTEM_MALLOC -DTCL_NO_DEPRECATED -std=c99 -I../include 
> -I"/srv/tcl/include"   -DHAVE_CONFIG_H   -c -o mo
> dload.o modload.c
...

[naviserver-devel] Naviserver doesn't compile against tcl8.7-alpha5

[D.Fox <un...@cr...>]

Greetings, 
It appears that any version of naviserver won't compile with TCL8.7a5 on FreeBSD13, however with TCL 8.7a3 naviserver does compile. 

The issue I am having is that: 8.7a5 gives me stubs-support, 8.7a3 does not which is required for a package. 
Essentially: tcl8.7a5 gives me stubs, naviserver won't compile, with tcl8.7a3 naviserver compiles, but TCL doesn't give me stubs 

./tclsh8.7a3 
% package require tarray 
This extension requires stubs-support. 

Are there any workarounds to this? Thanks. 

The errors thrown when compiling are below: 

gmake[1]: Entering directory '/srv/_scrap/naviserver-4.99.23/nsd' 
gcc -O2 -Wall -fPIC -pipe -finput-charset=UTF-8 -DNDEBUG -DSYSTEM_MALLOC -DTCL_NO_DEPRECATED -std=c99 -I../include -I"/srv/tcl/include" -DHAVE_CONFIG_H -c -o mo 
dload.o modload.c 
modload.c: In function 'Ns_ModuleLoad': 
modload.c:171:9: error: unknown type name 'Tcl_PackageInitProc'; did you mean 'Tcl_LibraryInitProc'? 
171 | Tcl_PackageInitProc *tclInitProc = NULL, *moduleVersionAddr = NULL; 
| ^~~~~~~~~~~~~~~~~~~ 
| Tcl_LibraryInitProc 
modload.c:190:29: warning: passing argument 5 of 'Tcl_FSLoadFile' from incompatible pointer type [-Wincompatible-pointer-types] 
190 | &tclInitProc, &moduleVersionAddr, &lh, &uPtr); 
| ^~~~~~~~~~~~ 
| | 
| int ** 
In file included from /srv/tcl/include/tcl.h:2404, 
from ../include/nsthread.h:533, 
from ../include/ns.h:46, 
from nsd.h:38, 
from modload.c:37: 
/srv/tcl/include/tclDecls.h:1341:27: note: expected 'int (**)(Tcl_Interp *)' but argument is of type 'int **' 
1341 | Tcl_LibraryInitProc **proc1Ptr, 
| ~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~ 
modload.c:190:43: warning: passing argument 6 of 'Tcl_FSLoadFile' from incompatible pointer type [-Wincompatible-pointer-types] 
190 | &tclInitProc, &moduleVersionAddr, &lh, &uPtr); 
| ^~~~~~~~~~~~~~~~~~ 
| | 
| int ** 
In file included from /srv/tcl/include/tcl.h:2404, 
from ../include/nsthread.h:533, 
from ../include/ns.h:46, 
from nsd.h:38, 
from modload.c:37: 
/srv/tcl/include/tclDecls.h:1342:27: note: expected 'int (**)(Tcl_Interp *)' but argument is of type 'int **' 
1342 | Tcl_LibraryInitProc **proc2Ptr, 
| ~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~ 
gmake[1]: *** [<builtin>: modload.o] Error 1 
gmake[1]: Leaving directory '/srv/_scrap/naviserver-4.99.23/nsd' 
gmake: *** [Makefile:50: all] Error 1 

[naviserver-devel] NaviServer 4.99.23 available

[Gustaf N. <ne...@wu...>]

Dear all,

on sourceforge is the release of NaviServer 4.99.22 [1] available. See 
below for the
list of changes.

The code was tested with Ubuntu 20.04, Rocky Linux 8.5, OpenBSD 6.9 
(clang),
FreeBSD 13.1, macOS 11.6.2 (Intel and M1).

Many thanks to the contributors of this release:

     Antonio Pisano
     Gustaf Neumann
     Maksym Zinchenko
     Oleg Oleinick
     Wolfgang Winkler
     Zoran Vasiljevic

All the best in the new year!

-gustaf neumann

[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.23/

=======================================
NaviServer 4.99.23, released 2021-12-29
=======================================

  70 files changed, 1431 insertions(+), 566 deletions(-)

New Features:

  - Improved hash algorithms for improved security

    The new version supports SCRAM (Salted Challenge Response
    Authentication Mechanism), which is one of the newer recommended
    password hash algorithm to replace the classic salted SHA1
    approaches. The classical hash algorithms become easier to attack
    via high performance hashing hardware (GPUs).  NaviServer supports
    now SCRYPT and SCRAM (when compiled with more recent versions of
    OpenSSL; SCRYPT requires at least OpenSSL 3.0. SCRAM requires OpenSSL 1.1)

    The actual hash function of SCRAM is PBKDF2 [RFC2898] with HMAC as
    the pseudorandom function (PRF) and with dkLen == output length of
    HMAC == output length of the digest function.

    Here is an example of using pbkdf2_hmac for the hash function of
    SCRAM-sha-256. RFC 7677 recommends to use 15K iterations for PBKDF2:

       ::ns_crypto::pbkdf2_hmac \
                -digest sha256 \
                -iterations 15000 \
                -secret $password \
                -salt $salt]

    OpenACS supports already switching to from salted SHA1 to SCAM (or
    SCRYPT) via configuration variable.


  - Better Unicode support, including emojis requiring 4-byte UTF-8 characters.

    Earlier versions of NaviServer and the nsdb* database drivers
    assumed on a few places that Tcl-internal UTF-8 is also valid UTF-8
    for external sources, which is often, but not always true. Now, the
    proper export functions are everywhere called.

    The new code was tested with Emojis up to Unicode 13 (many thanks
    to Wolfgang Winkler)

    This change effects as well the database driver module "nsdbpg".

  - ns_trim enhancements:
    The new option "-prefix ..." can be used to strip a string
    (such as ">> ") from every line starting with it.

  - extended time unit support (added "w" for weeks and "y" for years)

  - Added an experimental global parameter "nocache" to ease to
    experiment with horizontal scaling. As a consequence, "ns_cache
    eval" becomes a dummy operation.

  - Added an experimental command "ns_baseunit"

       ns_baseunit ?-size size? ?-time time?

    Convert from memory units or from time units to its base value
    using the NaviServer internal converters, which are used the same
    way for various commands.  The base unit for a memory size is a
    byte, for a time value a second). This command is necessary to
    provide Tcl-level commands calculating with these units to support
    uniform interfaces (e.g., calculating cache partition sizes base on
    values such as 2MB).

    Either "-size" or "-time" has to be specified.

       % ns_baseunit -size 10KB
       10240

       ns_baseunit -time 2.5h
       9000

  - Added an experimental command "ns_strcoll"

        ns_strcoll ?-locale locale? string1 string2"

    This command compares lexicographically string1 with string2
    according to the current locale collation and returns an integer
    greater than, equal to, or less than 0, depending on whether
    string1 is greater than, equal to, or less than string2. The
    command is necessary in cases, where e.g., the sorting order from
    the database (normally based on local collation) is different from
    default Tcl sorting order to provide a uniform interface with same
    sorting orders. The name is derived from the baseline POSIX
    function call.

    The command is suitable for usage in the lsort command:

       % set l {Bor Bar Bär}
       % lsort -command ns_strcoll $l
       Bar Bär Bor

       % lsort $l
       Bar Bor Bär

    Note that the output of the command depends on correct
    installation of locales on the host. Furthermore, the output of the
    command varies depending on the C library's implementation, which
    might differ for some locales/charsets between platforms.

Performance Improvements:

  - Increase scalability on DB operations by reducing DB pool locks

    On high load servers, the total number of locks and busy locks on
    the DB pools might become quite high. The new code caches these
    statistics in the handles (which are per thread, requiring no
    locks) and transfers the aggregated values on handle closes or
    statistics calls.

  - Set default for "concurrentinterpcreate" to "true" for Tcl 8.6 or
    newer. Versions up to at least Tcl 8.5 are known to crash in case
    two threads create interpreters at the same time. These crashes
    were hard to reproduce, but serializing interpreter creation
    helped. Since all our major servers are running since several years
    without problems with this parameter turned on the default is now
    set to "true", when NaviServer is compiled with Tcl 8.6 or newer.


Bug Fixes:
  - Fixed memory leak in "nsv_dict get" operations.


Documentation improvements:
---------------------------

  - Improved the following man pages:

         doc/src/manual/admin-tuning.man
         doc/src/naviserver/commandlist.man
         doc/src/naviserver/ns_adp_include.man
         doc/src/naviserver/ns_baseunit.man
         doc/src/naviserver/ns_cache.man
         doc/src/naviserver/ns_charsets.man
         doc/src/naviserver/ns_choosecharset.man
         doc/src/naviserver/ns_connchan.man
         doc/src/naviserver/ns_cookie.man
         doc/src/naviserver/ns_cookiecharset.man
         doc/src/naviserver/ns_crypto.man
         doc/src/naviserver/ns_encodingforcharset.man
         doc/src/naviserver/ns_encodingfortype.man
         doc/src/naviserver/ns_formfieldcharset.man
         doc/src/naviserver/ns_ictl.man
         doc/src/naviserver/ns_job.man
         doc/src/naviserver/ns_register.man
         doc/src/naviserver/ns_schedule.man
         doc/src/naviserver/ns_setformencoding.man
         doc/src/naviserver/ns_shutdown.man
         doc/src/naviserver/ns_sleep.man
         doc/src/naviserver/ns_sockcallback.man
         doc/src/naviserver/ns_sockopen.man
         doc/src/naviserver/ns_strcoll.man
         doc/src/naviserver/ns_time.man
         doc/src/naviserver/ns_urlcharset.man
         doc/src/naviserver/ns_valid_utf8.man
         doc/src/naviserver/textutil-cmds.man


Configuration Changes:
----------------------

  -  Ease configuration via environment variables

     This feature is useful to manage many NaviServer instances with
     mostly identical configurations without having to provide multiple
     configuration files (e.g. for Docker setups, or clusters).

     The sample configuration for of OpenACS (openacs-config.tcl)
     contains now a Tcl dictionary with default values:

       set defaultConfig {
          hostname   localhost
          ipaddress  127.0.0.1
          httpport   8000
          httpsport  ""
          server     "openacs"
          serverroot /var/www/$server
          logroot    $serverroot/log/
          homedir    /usr/local/ns
          bindir     $homedir/bin
          db_name    $server
          db_user    $server
          db_host    localhost
          db_port    ""
      }

     These configuration values (keys of the dict) can be overridden by
     environment variables prefixed with "oacs_" followed by the
     parameter name. One can change the default port specified in the
     configuration file for plain HTTP connections by e.g., providing it
     via environment variables:

         oacs_httpport=8101 /usr/local/ns/bin/nsd -i -t ....



Code Changes:
-------------
  - Extended regression test

  - Code Cleanup
      . Do not declare reserved C identifiers

  - Improved comments, fixed typos

  - Marked "ns_set_precision" as deprecated, since there is no
    reason why not setting the Tcl variable ::tcl_precision directly.


Changes in NaviServer Modules:
==============================

  8 files changed, 697 insertions(+), 283 deletions(-)


nsdbpg:
-------

  - Extended added support for UTF-8 including emojis (see above)

  - Added "ns_pg pid /handle/" to return the current backend PID
    without SQL parsing.  This command is about 100x faster than using
    "select pg_backend_pid()" and can be used e.g. for deploying
    prepared statements.

  - Some code cleanup and modernization

  - Added minimal documentation of ns_pg* API in README file

  - Bumped version number to 2.6

nsstats:
--------

  - improved handling of fractional seconds
  - added summary line with total savings to cache statistics
    (to estimate consequences of eliminating this cache)
  - improve bread crums for "mapped" server URLs
  - added user interface for unmapping URLs from connection pools

nsodbc:
-------

  - adjust code with current prototypes
  - fix warnings from static checkers

nsshell:
--------

  - Improved source code documentation
  - Updated PRISM.JS libraries links , to prevent X-Content-Type-Options errors

letsencrypt:
------------

  - Removed obtaining the intermediate cross signed certificate, since this does not seem
    to be required anymore
  - Improved source code documentation

Re: [naviserver-devel] Towards NaviServer 4.99.23

[Gustaf N. <ne...@wu...>]

Dear all,

on sourceforge is now rc2, tested on

    macOS 11.6.2, Rocky Linux 8.5, Ubuntu 20.04, OpenBSD 6.9, FreeBSD 13.1

The handling of collate (as used by the new ns_strcoll function) in 
different
OSes is tricky and surprisingly little consistent. Linux seems to have a 
very
good support for comparing UTF-8 strings, proving case-folding collation 
(probably
using the ICU library (International Components for Unicode [1]), other 
ones behave
quite different. Implementing an extra linkage against the libicu (which is
surprisingly complex) for the other platforms seems currently an overkill,
since most it would be probably just needed for non-Linux systems.
For now, the cases, where strcoll() results differ are deactivated in the
regression test for these systems.

Please test if possible.

all the best

-gn

PS: There seems to be as well differences for some versions of OpenSSL
(e.g. openssl 1.1.1d (broken) vs. openssl 1.1.1k (good))

[1] https://icu.unicode.org/

On 05.12.21 14:13, Gustaf Neumann wrote:
>
> Dear all,
>
> on sourceforge is a release candidate for NaviServer 4.99.23 [1].
>
> The change contains one potentially important fix for a memory
> leak (which was in the code since 1y+, but turned out to be significant
> with a recent OpenACS change) and a couple of new features (including
> the improved Unicode support for e.g. emojis, and crypto improvments
> like SCRAM, which is already incorporated in OpenACS).
>
> Please test if possible.  The release should be in the near future.
>
> Below is a preliminary summary of changes.
>
> all the best
> -gustaf neumann
>
> [1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.23/
> =======================================
> NaviServer 4.99.23, released 2021-12-XX
> =======================================
>
>   60 files changed, 1014 insertions(+), 270 deletions(-)
>
> New Features:
>
>   - Improved hash algorithms for improved security
>
>     The new version supports SCRAM (Salted Challenge Response
>     Authentication Mechanism), which is one of the newer recommended
>     password hash algorithm to replace the classic salted SHA1
>     approaches. The classical hash algorithms become easier to attack
>     via high performance hashing hardware (GPUs).  NaviServer supports
>     now SCRYPT and SCRAM (when compiled with more recent versions of
>     OpenSSL; SCRYPT requires at least OpenSSL 3.0. SCRAM requires OpenSSL 1.1)
>
>     The actual hash function of SCRAM is PBKDF2 [RFC2898] with HMAC as
>     the pseudorandom function (PRF) and with dkLen == output length of
>     HMAC == output length of the digest function.
>      
>     Here is an example of using pbkdf2_hmac for the hash function of
>     SCRAM-sha-256. RFC 7677 recommends to use 15K iterations for PBKDF2:
>      
>        ::ns_crypto::pbkdf2_hmac \
>                 -digest sha256 \
>                 -iterations 15000 \
>                 -secret $password \
>                 -salt $salt]
>                 
>     OpenACS supports already switching to from salted SHA1 to SCAM (or
>     SCRYPT) via configuration variable.
>
>
>   - Better Unicode support, including emojis requiring 4-byte UTF-8 characters.
>
>     Earlier versions of NaviServer and the nsdb* database drivers
>     assumed on a few places that Tcl-internal UTF-8 is also valid UTF-8
>     for external sources, which is often, but not always true. Now, the
>     proper export functions are everywhere called.
>
>     The new code was tested with Emojis up to Unicode 13 (many thanks
>     to Wolfgang Winkler)
>
>     This change effects as well the database driver module "nsdbpg".
>
>   - ns_trim enhancements:
>     The new option "-prefix ..." can be used to strip a string
>     (such as ">> ") from every line starting with it.
>
>   - extended time unit support (added "w" for weeks and "y" for years)
>
>   - Added an experimental global parameter "nocache" to ease to
>     experiment with horizontal scaling. As a consequence, "ns_cache
>     eval" becomes a dummy operation.
>
>   - Added an experimental command "ns_baseunit"
>      
>        ns_baseunit ?-size size? ?-time time?
>      
>     Convert from memory units or from time units to its base value
>     using the NaviServer internal converters, which are used the same
>     way for various commands.  The base unit for a memory size is a
>     byte, for a time value a second). This command is necessary to
>     provide Tcl-level commands calculating with these units to support
>     uniform interfaces (e.g., calculating cache partition sizes base on
>     values such as 2MB).
>      
>     Either "-size" or "-time" has to be specified.
>      
>        % ns_baseunit -size 10KB
>        10240
>      
>        ns_baseunit -time 2.5h
>        9000
>      
>   - Added an experimental command "ns_strcoll"
>   
>         ns_strcoll ?-locale locale? string1 string2"
>      
>     This command compares lexicographically string1 with string2
>     according to the current locale collation and returns an integer
>     greater than, equal to, or less than 0, depending on whether
>     string1 is greater than, equal to, or less than string2. The
>     command is necessary in cases, where e.g., the sorting order from
>     the database (normally based on local collation) is different from
>     default Tcl sorting order to provide a uniform interface with same
>     sorting orders. The name is derived from the baseline POSIX
>     function call.
>      
>     The command is suitable for usage in the lsort command:
>      
>        % set l {Bor Bar Bär}
>        % lsort -command ns_strcoll $l
>        Bar Bär Bor
>      
>        % lsort $l
>        Bar Bor Bär
>
>
> Performance Improvements:
>
>   - Increase scalability on DB operations by reducing DB pool locks
>   
>     On high load servers, the total number of locks and busy locks on
>     the DB pools might become quite high. The new code caches these
>     statistics in the handles (which are per thread, requiring no
>     locks) and transfers the aggregated values on handle closes or
>     statistics calls.
>
>   - Set default for "concurrentinterpcreate" to "true" for Tcl 8.6 or
>     newer. Versions up to at least Tcl 8.5 are known to crash in case
>     two threads create interpreters at the same time. These crashes
>     were hard to reproduce, but serializing interpreter creation
>     helped. Since all our major servers are running since several years
>     without problems with this parameter turned on the default is now
>     set to "true", when NaviServer is compiled with Tcl 8.6 or newer.
>
>
> Bug Fixes:
>   - Fixed memory leak in "nsv_dict get" operations.
>
>   
> Documentation improvements:
> ---------------------------
>
>   - Improved the following man pages:
>
> 	doc/src/manual/admin-tuning.man
> 	doc/src/naviserver/commandlist.man
> 	doc/src/naviserver/ns_adp_include.man
> 	doc/src/naviserver/ns_baseunit.man
> 	doc/src/naviserver/ns_cache.man
> 	doc/src/naviserver/ns_charsets.man
> 	doc/src/naviserver/ns_choosecharset.man
> 	doc/src/naviserver/ns_connchan.man
> 	doc/src/naviserver/ns_cookie.man
> 	doc/src/naviserver/ns_cookiecharset.man
> 	doc/src/naviserver/ns_crypto.man
> 	doc/src/naviserver/ns_encodingforcharset.man
> 	doc/src/naviserver/ns_encodingfortype.man
> 	doc/src/naviserver/ns_formfieldcharset.man
> 	doc/src/naviserver/ns_ictl.man
> 	doc/src/naviserver/ns_job.man
> 	doc/src/naviserver/ns_register.man
> 	doc/src/naviserver/ns_schedule.man
> 	doc/src/naviserver/ns_setformencoding.man
> 	doc/src/naviserver/ns_shutdown.man
> 	doc/src/naviserver/ns_sleep.man
> 	doc/src/naviserver/ns_sockcallback.man
> 	doc/src/naviserver/ns_sockopen.man
> 	doc/src/naviserver/ns_strcoll.man
> 	doc/src/naviserver/ns_time.man
> 	doc/src/naviserver/ns_urlcharset.man
> 	doc/src/naviserver/ns_valid_utf8.man
> 	doc/src/naviserver/textutil-cmds.man
>   
>
> Configuration Changes:
> ----------------------
>
>   -  Ease configuration via environment variables
>
>      This feature is useful to manage many NaviServer instances with
>      mostly identical configurations without having to provide multiple
>      configuration files (e.g. for Docker setups, or clusters).
>      
>      The sample configuration for of OpenACS (openacs-config.tcl)
>      contains now a Tcl dictionary with default values:
>
>        set defaultConfig {
>           hostname   localhost
>           ipaddress  127.0.0.1
>           httpport   8000
>           httpsport  ""
>           server     "openacs"
>           serverroot /var/www/$server
>           logroot    $serverroot/log/
>           homedir    /usr/local/ns
>           bindir     $homedir/bin
>           db_name    $server
>           db_user    $server
>           db_host    localhost
>           db_port    ""
>       }
>
>      These configuration values (keys of the dict) can be overridden by
>      environment variables prefixed with "oacs_" followed by the
>      parameter name. One can change the default port specified in the
>      configuration file for plain HTTP connections by e.g., providing it
>      via environment variables:
>
>          oacs_httpport=8101 /usr/local/ns/bin/nsd -i -t ....
>          
>
> Code Changes:
>
>   - Extended regression test
>      
>   - Code Cleanup
>       . Do not declare reserved C identifiers
>
>   - Improved comments, fixed typos
>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

-- 
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"

Re: [naviserver-devel] Fwd: No notifications with webpush::send

[Gustaf N. <ne...@wu...>]

Part if the problem is that OpenSSL, when configured with the default 
prefix (/usr/local/)
installs its libraries on some platforms into /usr/local/lib64 and not 
into /usr/local/lib.
On these platforms we might have a problem when configuring with

--with-openssl=/usr/local/

I have just now improved the heuristics in the .m4 file (tip version on 
bitbucket),
but i am not completely happy with this, since this potentially 
hardcodes some
Linux conventions. i have to do more cross-platform checks on this 
before the release.

> So far everything seems to work, we only get one warning from the linker:
>
> /usr/bin/ld: warning: libcrypto.so.1.1, needed by 
> /usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/libssl.so, 
> may conflict with libcrypto.so.3
>
This comes, when some files are compiles with different linkage options, 
usually
after continuing with different library settings after an unsuccessful 
compilation attempt.
Normally, a "make clean" + "make" helps.

When using the fresh version form bitbucket, run an "autogen.sh ..." first.

-g

Re: [naviserver-devel] Bug in nsd/binder.c

[Gustaf N. <ne...@wu...>]

Dear Russ,

Many thanks for the report.

The problem seems to occur only, when daemontools are used,
which are used probably mostly in legacy setups (the latest
release of daemontools is according to wikipedia 20 years old [1].
I was not able to reproduce a problem with the recommended
plain systemd setup [2], but nevertheless, reverting the problematic
change seems appropriate,  at least for legacy setups.

Please double-check with the newest version from bitbucket if possible

all the best

-g

[1] https://en.wikipedia.org/wiki/Daemontools
[2] 
https://naviserver.sourceforge.io/n/manual/files/admin-maintenance.html#subsection2

On 12.12.21 02:18, rus...@hi... wrote:
> Several months ago I upgraded to naviserver-4.99.23rc1 and
> initially everything seemed fine, but at some point I noticed
> that if naviserver was restarted the bind port was already taken.
>
> I was using daemontools supervise/svc (started with systemd) to
> start/stop/restart. Initially I blamed systemd since the additional
> process could be controlled with systemctl.
>

[naviserver-devel] Bug in nsd/binder.c

[<rus...@hi...>]

Several months ago I upgraded to naviserver-4.99.23rc1 and
initially everything seemed fine, but at some point I noticed
that if naviserver was restarted the bind port was already taken.

I was using daemontools supervise/svc (started with systemd) to
start/stop/restart. Initially I blamed systemd since the additional
process could be controlled with systemctl. 

This system is Fedora 32 x64 (DigitalOcean droplet), but I have just
verified that the bug shows up on Ubuntu as well.

I used git bisect to locate the exact commit which created the bug,
here's the patch (but see links below):

--- binder.c	2021-08-16 11:59:07.000000000 +0000
+++ binder.c	2021-12-11 06:24:52.543180128 +0000
@@ -1307,12 +1307,10 @@
             (void)ns_sockclose(binderRequest[1]);
             (void)ns_sockclose(binderResponse[0]);
             Binder();
-        } else {
-            /*
-             * Child process.
-             */
-            exit(0);
         }
+
+        exit(0);
+
     } else {
         /*
          * Parent process.

-------------
I believe that the exit(0), placed inside the trailing else allowed
the Binder() to survive. 

Here are my notes (from tags-commits.txt below):
[russell@highfivediet naviserver]$ git bisect --help
[russell@highfivediet naviserver]$ git bisect start
[russell@highfivediet naviserver]$ git bisect bad
[russell@highfivediet naviserver]$ git bisect good naviserver-4.99.17

Bisecting: 698 revisions left to test after this (roughly 10 steps)
STEP1: [65689e7c2cfee84bcf4a9789b4ff14f500e57861] make sure, variable is always initialized
STEP2: [ffeafdebecb999732712c6edf891917ea0eeab4d] Use "for(;;)" instead of "while(1)" like on other places
STEP3: [010daa96412eaf7f02833aa7a9c255ad09b68d8d] fix default temp directory for windows
STEP4: [9a445fcf1b15490728d5803ba3d21df7024eb4c1] Improved compilation cleanness with gcc-11
STEP5: [7909e79e8d41a12dd35e9b28c1afe7dcce9627b0] improved spelling and aligned documentation with code
STEP6: [b70152a504024c4ccf283c2ced9ea6a2a7698511] Fixed Apple M1 problem with double-fork in NsBinder()
STEP7: [51c21ec5bcc4c9c06c6fd1b51b8c27abe5d07bd6] improve variable name ( NaviServer/4.99.22rc1 (naviserver-4.99.20-155-g51c21ec5bcc4+) ) <== BAD
STEP8: [db057401dae34267279f16299b1aee7570240341] styling conventions: use uppercase for hexdigits (NaviServer/4.99.21 (naviserver-4.99.20-149-gdb057401dae3+) ) <== GOOD
STEP9: [7bf2468775ccbbf2191db7b5382624b173332121] fix man page ( NaviServer/4.99.21 (naviserver-4.99.20-152-g7bf2468775cc+) ) <== BAD 
STEP10:[f7740ac62376d99170f904403a9bc50d80824b44] minor cleanup ( NaviServer/4.99.21 (naviserver-4.99.20-151-gf7740ac62376+) ) <== BAD
Bisecting: 0 revisions left to test after this (roughly 0 steps)
STEP11:[8114f6870009985366a65342928062f08cde80fe] Added more comments to double-fork construct in NsForkBinder() ( NaviServer/4.99.21 (naviserver-4.99.20-150-g8114f6870009+) ) <== BAD 

[russell@highfivediet naviserver]$ git bisect bad
8114f6870009985366a65342928062f08cde80fe is the first bad commit
commit 8114f6870009985366a65342928062f08cde80fe
Author: Gustaf Neumann <ne...@wu...>
Date:   Mon Aug 16 13:58:04 2021 +0200

    Added more comments to double-fork construct in NsForkBinder()

 nsd/binder.c | 41 +++++++++++++++++++++++++----------------
 1 file changed, 25 insertions(+), 16 deletions(-)

----- MORE INFO -----

More info generated by git is here: 

https://highfivediet.com/bugs/binder-bug/

The bug doesn't show up in every case, obviously you have to use
the binder switch, but even using a simple command line startup 
didn't create an issue. I have include my startup script at

https://highfivediet.com/bugs/binder-bug/run.txt

Easiest way to test for the bug is to startup and then use
$ ps -axjf and looking for an additional process (same PPID).

On Fedora, this process was connected to pid 1, but on Ubuntu it
was connected to the gnome-terminal-server process.

Otherwise if you kill either of these and then run 
$ ss -ltnp   
the duplicate process is still running on the port (and works fine).

Let me know if more information is needed or if you need other test
run.

Russ

Re: [naviserver-devel] Fwd: No notifications with webpush::send

[Wolfgang W. <wol...@di...>]

This is the solution we came up with:

We downloaded, compiled and installed the latest stable openssl version 
3.0.0.

Then we added /usr/local/lib64 to /etc/ld.so.conf.d/libc.conf

After/sbin/ldconfig Naviserver finds the new openssl version when 
compiling with:

--with-openssl=/usr/local/

So far everything seems to work, we only get one warning from the linker:

/usr/bin/ld: warning: libcrypto.so.1.1, needed by 
/usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/libssl.so, may 
conflict with libcrypto.so.3

Again, thanks for your help,

Wolfgang

Am 09.12.21 um 11:44 schrieb Gustaf Neumann:
>
>
>
>
> -------- Forwarded Message --------
> Subject: 	Re: [naviserver-devel] No notifications with webpush::send
> Date: 	Thu, 9 Dec 2021 11:43:44 +0100
> From: 	Gustaf Neumann <ne...@wu...>
> To: 	Wolfgang Winkler via naviserver-devel 
> <nav...@li...>
>
>
>
>
> On 09.12.21 09:33, Wolfgang Winkler via naviserver-devel wrote:
>>
>> We are using 1.1.1d on our production server, which is a debian buster.
>>
>> bytes {} tag 1e58277931d45f4c593cffbf291b39b7
>
> i can confirm, that with Debian GNU/Linux 10 (buster) and OpenSSL 
> 1.1.1d bytes are empty.
>
> With e.g. Rocky Linux release 8.4 (one successor of CentOS, also 
> conservative), with e.g. 1.1.1g, everything is fine.
>
>
>> I've tried to use 1.1.1k on buster. I installed it with
>>
>> ./config --prefix=/usr/local/openssl && make && make install
>>
>> and compiled naviserver with
>>
>> ./configure 
>> --enable-64bit=true--prefix=/usr/local/naviserver-git--with-openssl=/usr/local/openssl--with-tcl=/usr/local/lib/--enable-threads
>>
>> But naviserver still uses the packaged openssl version:
>> # ldd nsd/nsd
>> libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1
>
>
> There is something starnge  on Buster concerning libraries. I have
> downloaded newest openssl from git, configured + make install, and 
> configured
> Naviserver as usual
>
> $ ./configure --enable-64bit -prefix=/usr/local/ns --with-openssl=/usr/local/
>
> but was surprised that it the version was not picked up for loading.
> After brutally linking the files, everything was fine.
>
> So, there seems to be some load-path that has to be configured for Buster,
> but I am not an expert (and have not time to investigate deeper).
>
> But with this, the right OpenSSL is loaded, encrypt returns non-empty:
>
> $ ln -s /usr/local/lib64/*so* /usr/local/lib/
> $ ldconfig -v
> $ make install
> $ ./nsd/nsd -c -u nsadmin
> [-main:conf-] Notice: OpenSSL 3.1.0-dev  initialized
> ...
> % package require tcltest 2.2
> % namespace import -force ::tcltest::*
> % test aead-1.0 {aead::encrypt} -body {
>      set d [ns_crypto::aead::encrypt string -cipher aes-128-gcm -iv 123456789 -key secret "hello world"]
>      list bytes [string length [dict get $d bytes]] tag [string length [dict get $d tag]]
> } -result {bytes 22 tag 32}
>
>
> I have to rush,
>
> -gn
>
>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
-- 

*Wolfgang Winkler*
Geschäftsführung
wol...@di...
mobil +43.699.19971172

dc:*büro*
digital concepts Novak Winkler OG
Software & Design
Landstraße 68, 5. Stock, 4020 Linz
www.digital-concepts.com <http://www.digital-concepts.com>
tel +43.732.997117.72
tel +43.699.1997117.2

Firmenbuchnummer: 192003h
Firmenbuchgericht: Landesgericht Linz

[naviserver-devel] Fwd: No notifications with webpush::send

[Gustaf N. <ne...@wu...>]

-------- Forwarded Message --------
Subject: 	Re: [naviserver-devel] No notifications with webpush::send
Date: 	Thu, 9 Dec 2021 11:43:44 +0100
From: 	Gustaf Neumann <ne...@wu...>
To: 	Wolfgang Winkler via naviserver-devel 
<nav...@li...>




On 09.12.21 09:33, Wolfgang Winkler via naviserver-devel wrote:
>
> We are using 1.1.1d on our production server, which is a debian buster.
>
> bytes {} tag 1e58277931d45f4c593cffbf291b39b7

i can confirm, that with Debian GNU/Linux 10 (buster) and OpenSSL 1.1.1d 
bytes are empty.

With e.g. Rocky Linux release 8.4 (one successor of CentOS, also 
conservative), with e.g. 1.1.1g, everything is fine.


> I've tried to use 1.1.1k on buster. I installed it with
>
> ./config --prefix=/usr/local/openssl && make && make install
>
> and compiled naviserver with
>
> ./configure 
> --enable-64bit=true--prefix=/usr/local/naviserver-git--with-openssl=/usr/local/openssl--with-tcl=/usr/local/lib/--enable-threads
>
> But naviserver still uses the packaged openssl version:
> # ldd nsd/nsd
> libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1


There is something starnge  on Buster concerning libraries. I have
downloaded newest openssl from git, configured + make install, and 
configured
Naviserver as usual

$ ./configure --enable-64bit -prefix=/usr/local/ns --with-openssl=/usr/local/

but was surprised that it the version was not picked up for loading.
After brutally linking the files, everything was fine.

So, there seems to be some load-path that has to be configured for Buster,
but I am not an expert (and have not time to investigate deeper).

But with this, the right OpenSSL is loaded, encrypt returns non-empty:

$ ln -s /usr/local/lib64/*so* /usr/local/lib/
$ ldconfig -v
$ make install
$ ./nsd/nsd -c -u nsadmin
[-main:conf-] Notice: OpenSSL 3.1.0-dev  initialized
...
% package require tcltest 2.2
% namespace import -force ::tcltest::*
% test aead-1.0 {aead::encrypt} -body {
     set d [ns_crypto::aead::encrypt string -cipher aes-128-gcm -iv 123456789 -key secret "hello world"]
     list bytes [string length [dict get $d bytes]] tag [string length [dict get $d tag]]
} -result {bytes 22 tag 32}


I have to rush,

-gn