naviserver-devel — Developer discussion

[naviserver-devel] Naviserver doesn't compile against tcl8.7-alpha5

[D.Fox <un...@cr...>]

Greetings, 
It appears that any version of naviserver won't compile with TCL8.7a5 on FreeBSD13, however with TCL 8.7a3 naviserver does compile. 

The issue I am having is that: 8.7a5 gives me stubs-support, 8.7a3 does not which is required for a package. 
Essentially: tcl8.7a5 gives me stubs, naviserver won't compile, with tcl8.7a3 naviserver compiles, but TCL doesn't give me stubs 

./tclsh8.7a3 
% package require tarray 
This extension requires stubs-support. 

Are there any workarounds to this? Thanks. 

The errors thrown when compiling are below: 

gmake[1]: Entering directory '/srv/_scrap/naviserver-4.99.23/nsd' 
gcc -O2 -Wall -fPIC -pipe -finput-charset=UTF-8 -DNDEBUG -DSYSTEM_MALLOC -DTCL_NO_DEPRECATED -std=c99 -I../include -I"/srv/tcl/include" -DHAVE_CONFIG_H -c -o mo 
dload.o modload.c 
modload.c: In function 'Ns_ModuleLoad': 
modload.c:171:9: error: unknown type name 'Tcl_PackageInitProc'; did you mean 'Tcl_LibraryInitProc'? 
171 | Tcl_PackageInitProc *tclInitProc = NULL, *moduleVersionAddr = NULL; 
| ^~~~~~~~~~~~~~~~~~~ 
| Tcl_LibraryInitProc 
modload.c:190:29: warning: passing argument 5 of 'Tcl_FSLoadFile' from incompatible pointer type [-Wincompatible-pointer-types] 
190 | &tclInitProc, &moduleVersionAddr, &lh, &uPtr); 
| ^~~~~~~~~~~~ 
| | 
| int ** 
In file included from /srv/tcl/include/tcl.h:2404, 
from ../include/nsthread.h:533, 
from ../include/ns.h:46, 
from nsd.h:38, 
from modload.c:37: 
/srv/tcl/include/tclDecls.h:1341:27: note: expected 'int (**)(Tcl_Interp *)' but argument is of type 'int **' 
1341 | Tcl_LibraryInitProc **proc1Ptr, 
| ~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~ 
modload.c:190:43: warning: passing argument 6 of 'Tcl_FSLoadFile' from incompatible pointer type [-Wincompatible-pointer-types] 
190 | &tclInitProc, &moduleVersionAddr, &lh, &uPtr); 
| ^~~~~~~~~~~~~~~~~~ 
| | 
| int ** 
In file included from /srv/tcl/include/tcl.h:2404, 
from ../include/nsthread.h:533, 
from ../include/ns.h:46, 
from nsd.h:38, 
from modload.c:37: 
/srv/tcl/include/tclDecls.h:1342:27: note: expected 'int (**)(Tcl_Interp *)' but argument is of type 'int **' 
1342 | Tcl_LibraryInitProc **proc2Ptr, 
| ~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~ 
gmake[1]: *** [<builtin>: modload.o] Error 1 
gmake[1]: Leaving directory '/srv/_scrap/naviserver-4.99.23/nsd' 
gmake: *** [Makefile:50: all] Error 1 

[naviserver-devel] NaviServer 4.99.23 available

[Gustaf N. <ne...@wu...>]

Dear all,

on sourceforge is the release of NaviServer 4.99.22 [1] available. See 
below for the
list of changes.

The code was tested with Ubuntu 20.04, Rocky Linux 8.5, OpenBSD 6.9 
(clang),
FreeBSD 13.1, macOS 11.6.2 (Intel and M1).

Many thanks to the contributors of this release:

     Antonio Pisano
     Gustaf Neumann
     Maksym Zinchenko
     Oleg Oleinick
     Wolfgang Winkler
     Zoran Vasiljevic

All the best in the new year!

-gustaf neumann

[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.23/

=======================================
NaviServer 4.99.23, released 2021-12-29
=======================================

  70 files changed, 1431 insertions(+), 566 deletions(-)

New Features:

  - Improved hash algorithms for improved security

    The new version supports SCRAM (Salted Challenge Response
    Authentication Mechanism), which is one of the newer recommended
    password hash algorithm to replace the classic salted SHA1
    approaches. The classical hash algorithms become easier to attack
    via high performance hashing hardware (GPUs).  NaviServer supports
    now SCRYPT and SCRAM (when compiled with more recent versions of
    OpenSSL; SCRYPT requires at least OpenSSL 3.0. SCRAM requires OpenSSL 1.1)

    The actual hash function of SCRAM is PBKDF2 [RFC2898] with HMAC as
    the pseudorandom function (PRF) and with dkLen == output length of
    HMAC == output length of the digest function.

    Here is an example of using pbkdf2_hmac for the hash function of
    SCRAM-sha-256. RFC 7677 recommends to use 15K iterations for PBKDF2:

       ::ns_crypto::pbkdf2_hmac \
                -digest sha256 \
                -iterations 15000 \
                -secret $password \
                -salt $salt]

    OpenACS supports already switching to from salted SHA1 to SCAM (or
    SCRYPT) via configuration variable.


  - Better Unicode support, including emojis requiring 4-byte UTF-8 characters.

    Earlier versions of NaviServer and the nsdb* database drivers
    assumed on a few places that Tcl-internal UTF-8 is also valid UTF-8
    for external sources, which is often, but not always true. Now, the
    proper export functions are everywhere called.

    The new code was tested with Emojis up to Unicode 13 (many thanks
    to Wolfgang Winkler)

    This change effects as well the database driver module "nsdbpg".

  - ns_trim enhancements:
    The new option "-prefix ..." can be used to strip a string
    (such as ">> ") from every line starting with it.

  - extended time unit support (added "w" for weeks and "y" for years)

  - Added an experimental global parameter "nocache" to ease to
    experiment with horizontal scaling. As a consequence, "ns_cache
    eval" becomes a dummy operation.

  - Added an experimental command "ns_baseunit"

       ns_baseunit ?-size size? ?-time time?

    Convert from memory units or from time units to its base value
    using the NaviServer internal converters, which are used the same
    way for various commands.  The base unit for a memory size is a
    byte, for a time value a second). This command is necessary to
    provide Tcl-level commands calculating with these units to support
    uniform interfaces (e.g., calculating cache partition sizes base on
    values such as 2MB).

    Either "-size" or "-time" has to be specified.

       % ns_baseunit -size 10KB
       10240

       ns_baseunit -time 2.5h
       9000

  - Added an experimental command "ns_strcoll"

        ns_strcoll ?-locale locale? string1 string2"

    This command compares lexicographically string1 with string2
    according to the current locale collation and returns an integer
    greater than, equal to, or less than 0, depending on whether
    string1 is greater than, equal to, or less than string2. The
    command is necessary in cases, where e.g., the sorting order from
    the database (normally based on local collation) is different from
    default Tcl sorting order to provide a uniform interface with same
    sorting orders. The name is derived from the baseline POSIX
    function call.

    The command is suitable for usage in the lsort command:

       % set l {Bor Bar Bär}
       % lsort -command ns_strcoll $l
       Bar Bär Bor

       % lsort $l
       Bar Bor Bär

    Note that the output of the command depends on correct
    installation of locales on the host. Furthermore, the output of the
    command varies depending on the C library's implementation, which
    might differ for some locales/charsets between platforms.

Performance Improvements:

  - Increase scalability on DB operations by reducing DB pool locks

    On high load servers, the total number of locks and busy locks on
    the DB pools might become quite high. The new code caches these
    statistics in the handles (which are per thread, requiring no
    locks) and transfers the aggregated values on handle closes or
    statistics calls.

  - Set default for "concurrentinterpcreate" to "true" for Tcl 8.6 or
    newer. Versions up to at least Tcl 8.5 are known to crash in case
    two threads create interpreters at the same time. These crashes
    were hard to reproduce, but serializing interpreter creation
    helped. Since all our major servers are running since several years
    without problems with this parameter turned on the default is now
    set to "true", when NaviServer is compiled with Tcl 8.6 or newer.


Bug Fixes:
  - Fixed memory leak in "nsv_dict get" operations.


Documentation improvements:
---------------------------

  - Improved the following man pages:

         doc/src/manual/admin-tuning.man
         doc/src/naviserver/commandlist.man
         doc/src/naviserver/ns_adp_include.man
         doc/src/naviserver/ns_baseunit.man
         doc/src/naviserver/ns_cache.man
         doc/src/naviserver/ns_charsets.man
         doc/src/naviserver/ns_choosecharset.man
         doc/src/naviserver/ns_connchan.man
         doc/src/naviserver/ns_cookie.man
         doc/src/naviserver/ns_cookiecharset.man
         doc/src/naviserver/ns_crypto.man
         doc/src/naviserver/ns_encodingforcharset.man
         doc/src/naviserver/ns_encodingfortype.man
         doc/src/naviserver/ns_formfieldcharset.man
         doc/src/naviserver/ns_ictl.man
         doc/src/naviserver/ns_job.man
         doc/src/naviserver/ns_register.man
         doc/src/naviserver/ns_schedule.man
         doc/src/naviserver/ns_setformencoding.man
         doc/src/naviserver/ns_shutdown.man
         doc/src/naviserver/ns_sleep.man
         doc/src/naviserver/ns_sockcallback.man
         doc/src/naviserver/ns_sockopen.man
         doc/src/naviserver/ns_strcoll.man
         doc/src/naviserver/ns_time.man
         doc/src/naviserver/ns_urlcharset.man
         doc/src/naviserver/ns_valid_utf8.man
         doc/src/naviserver/textutil-cmds.man


Configuration Changes:
----------------------

  -  Ease configuration via environment variables

     This feature is useful to manage many NaviServer instances with
     mostly identical configurations without having to provide multiple
     configuration files (e.g. for Docker setups, or clusters).

     The sample configuration for of OpenACS (openacs-config.tcl)
     contains now a Tcl dictionary with default values:

       set defaultConfig {
          hostname   localhost
          ipaddress  127.0.0.1
          httpport   8000
          httpsport  ""
          server     "openacs"
          serverroot /var/www/$server
          logroot    $serverroot/log/
          homedir    /usr/local/ns
          bindir     $homedir/bin
          db_name    $server
          db_user    $server
          db_host    localhost
          db_port    ""
      }

     These configuration values (keys of the dict) can be overridden by
     environment variables prefixed with "oacs_" followed by the
     parameter name. One can change the default port specified in the
     configuration file for plain HTTP connections by e.g., providing it
     via environment variables:

         oacs_httpport=8101 /usr/local/ns/bin/nsd -i -t ....



Code Changes:
-------------
  - Extended regression test

  - Code Cleanup
      . Do not declare reserved C identifiers

  - Improved comments, fixed typos

  - Marked "ns_set_precision" as deprecated, since there is no
    reason why not setting the Tcl variable ::tcl_precision directly.


Changes in NaviServer Modules:
==============================

  8 files changed, 697 insertions(+), 283 deletions(-)


nsdbpg:
-------

  - Extended added support for UTF-8 including emojis (see above)

  - Added "ns_pg pid /handle/" to return the current backend PID
    without SQL parsing.  This command is about 100x faster than using
    "select pg_backend_pid()" and can be used e.g. for deploying
    prepared statements.

  - Some code cleanup and modernization

  - Added minimal documentation of ns_pg* API in README file

  - Bumped version number to 2.6

nsstats:
--------

  - improved handling of fractional seconds
  - added summary line with total savings to cache statistics
    (to estimate consequences of eliminating this cache)
  - improve bread crums for "mapped" server URLs
  - added user interface for unmapping URLs from connection pools

nsodbc:
-------

  - adjust code with current prototypes
  - fix warnings from static checkers

nsshell:
--------

  - Improved source code documentation
  - Updated PRISM.JS libraries links , to prevent X-Content-Type-Options errors

letsencrypt:
------------

  - Removed obtaining the intermediate cross signed certificate, since this does not seem
    to be required anymore
  - Improved source code documentation

Re: [naviserver-devel] Towards NaviServer 4.99.23

[Gustaf N. <ne...@wu...>]

Dear all,

on sourceforge is now rc2, tested on

    macOS 11.6.2, Rocky Linux 8.5, Ubuntu 20.04, OpenBSD 6.9, FreeBSD 13.1

The handling of collate (as used by the new ns_strcoll function) in 
different
OSes is tricky and surprisingly little consistent. Linux seems to have a 
very
good support for comparing UTF-8 strings, proving case-folding collation 
(probably
using the ICU library (International Components for Unicode [1]), other 
ones behave
quite different. Implementing an extra linkage against the libicu (which is
surprisingly complex) for the other platforms seems currently an overkill,
since most it would be probably just needed for non-Linux systems.
For now, the cases, where strcoll() results differ are deactivated in the
regression test for these systems.

Please test if possible.

all the best

-gn

PS: There seems to be as well differences for some versions of OpenSSL
(e.g. openssl 1.1.1d (broken) vs. openssl 1.1.1k (good))

[1] https://icu.unicode.org/

On 05.12.21 14:13, Gustaf Neumann wrote:
>
> Dear all,
>
> on sourceforge is a release candidate for NaviServer 4.99.23 [1].
>
> The change contains one potentially important fix for a memory
> leak (which was in the code since 1y+, but turned out to be significant
> with a recent OpenACS change) and a couple of new features (including
> the improved Unicode support for e.g. emojis, and crypto improvments
> like SCRAM, which is already incorporated in OpenACS).
>
> Please test if possible.  The release should be in the near future.
>
> Below is a preliminary summary of changes.
>
> all the best
> -gustaf neumann
>
> [1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.23/
> =======================================
> NaviServer 4.99.23, released 2021-12-XX
> =======================================
>
>   60 files changed, 1014 insertions(+), 270 deletions(-)
>
> New Features:
>
>   - Improved hash algorithms for improved security
>
>     The new version supports SCRAM (Salted Challenge Response
>     Authentication Mechanism), which is one of the newer recommended
>     password hash algorithm to replace the classic salted SHA1
>     approaches. The classical hash algorithms become easier to attack
>     via high performance hashing hardware (GPUs).  NaviServer supports
>     now SCRYPT and SCRAM (when compiled with more recent versions of
>     OpenSSL; SCRYPT requires at least OpenSSL 3.0. SCRAM requires OpenSSL 1.1)
>
>     The actual hash function of SCRAM is PBKDF2 [RFC2898] with HMAC as
>     the pseudorandom function (PRF) and with dkLen == output length of
>     HMAC == output length of the digest function.
>      
>     Here is an example of using pbkdf2_hmac for the hash function of
>     SCRAM-sha-256. RFC 7677 recommends to use 15K iterations for PBKDF2:
>      
>        ::ns_crypto::pbkdf2_hmac \
>                 -digest sha256 \
>                 -iterations 15000 \
>                 -secret $password \
>                 -salt $salt]
>                 
>     OpenACS supports already switching to from salted SHA1 to SCAM (or
>     SCRYPT) via configuration variable.
>
>
>   - Better Unicode support, including emojis requiring 4-byte UTF-8 characters.
>
>     Earlier versions of NaviServer and the nsdb* database drivers
>     assumed on a few places that Tcl-internal UTF-8 is also valid UTF-8
>     for external sources, which is often, but not always true. Now, the
>     proper export functions are everywhere called.
>
>     The new code was tested with Emojis up to Unicode 13 (many thanks
>     to Wolfgang Winkler)
>
>     This change effects as well the database driver module "nsdbpg".
>
>   - ns_trim enhancements:
>     The new option "-prefix ..." can be used to strip a string
>     (such as ">> ") from every line starting with it.
>
>   - extended time unit support (added "w" for weeks and "y" for years)
>
>   - Added an experimental global parameter "nocache" to ease to
>     experiment with horizontal scaling. As a consequence, "ns_cache
>     eval" becomes a dummy operation.
>
>   - Added an experimental command "ns_baseunit"
>      
>        ns_baseunit ?-size size? ?-time time?
>      
>     Convert from memory units or from time units to its base value
>     using the NaviServer internal converters, which are used the same
>     way for various commands.  The base unit for a memory size is a
>     byte, for a time value a second). This command is necessary to
>     provide Tcl-level commands calculating with these units to support
>     uniform interfaces (e.g., calculating cache partition sizes base on
>     values such as 2MB).
>      
>     Either "-size" or "-time" has to be specified.
>      
>        % ns_baseunit -size 10KB
>        10240
>      
>        ns_baseunit -time 2.5h
>        9000
>      
>   - Added an experimental command "ns_strcoll"
>   
>         ns_strcoll ?-locale locale? string1 string2"
>      
>     This command compares lexicographically string1 with string2
>     according to the current locale collation and returns an integer
>     greater than, equal to, or less than 0, depending on whether
>     string1 is greater than, equal to, or less than string2. The
>     command is necessary in cases, where e.g., the sorting order from
>     the database (normally based on local collation) is different from
>     default Tcl sorting order to provide a uniform interface with same
>     sorting orders. The name is derived from the baseline POSIX
>     function call.
>      
>     The command is suitable for usage in the lsort command:
>      
>        % set l {Bor Bar Bär}
>        % lsort -command ns_strcoll $l
>        Bar Bär Bor
>      
>        % lsort $l
>        Bar Bor Bär
>
>
> Performance Improvements:
>
>   - Increase scalability on DB operations by reducing DB pool locks
>   
>     On high load servers, the total number of locks and busy locks on
>     the DB pools might become quite high. The new code caches these
>     statistics in the handles (which are per thread, requiring no
>     locks) and transfers the aggregated values on handle closes or
>     statistics calls.
>
>   - Set default for "concurrentinterpcreate" to "true" for Tcl 8.6 or
>     newer. Versions up to at least Tcl 8.5 are known to crash in case
>     two threads create interpreters at the same time. These crashes
>     were hard to reproduce, but serializing interpreter creation
>     helped. Since all our major servers are running since several years
>     without problems with this parameter turned on the default is now
>     set to "true", when NaviServer is compiled with Tcl 8.6 or newer.
>
>
> Bug Fixes:
>   - Fixed memory leak in "nsv_dict get" operations.
>
>   
> Documentation improvements:
> ---------------------------
>
>   - Improved the following man pages:
>
> 	doc/src/manual/admin-tuning.man
> 	doc/src/naviserver/commandlist.man
> 	doc/src/naviserver/ns_adp_include.man
> 	doc/src/naviserver/ns_baseunit.man
> 	doc/src/naviserver/ns_cache.man
> 	doc/src/naviserver/ns_charsets.man
> 	doc/src/naviserver/ns_choosecharset.man
> 	doc/src/naviserver/ns_connchan.man
> 	doc/src/naviserver/ns_cookie.man
> 	doc/src/naviserver/ns_cookiecharset.man
> 	doc/src/naviserver/ns_crypto.man
> 	doc/src/naviserver/ns_encodingforcharset.man
> 	doc/src/naviserver/ns_encodingfortype.man
> 	doc/src/naviserver/ns_formfieldcharset.man
> 	doc/src/naviserver/ns_ictl.man
> 	doc/src/naviserver/ns_job.man
> 	doc/src/naviserver/ns_register.man
> 	doc/src/naviserver/ns_schedule.man
> 	doc/src/naviserver/ns_setformencoding.man
> 	doc/src/naviserver/ns_shutdown.man
> 	doc/src/naviserver/ns_sleep.man
> 	doc/src/naviserver/ns_sockcallback.man
> 	doc/src/naviserver/ns_sockopen.man
> 	doc/src/naviserver/ns_strcoll.man
> 	doc/src/naviserver/ns_time.man
> 	doc/src/naviserver/ns_urlcharset.man
> 	doc/src/naviserver/ns_valid_utf8.man
> 	doc/src/naviserver/textutil-cmds.man
>   
>
> Configuration Changes:
> ----------------------
>
>   -  Ease configuration via environment variables
>
>      This feature is useful to manage many NaviServer instances with
>      mostly identical configurations without having to provide multiple
>      configuration files (e.g. for Docker setups, or clusters).
>      
>      The sample configuration for of OpenACS (openacs-config.tcl)
>      contains now a Tcl dictionary with default values:
>
>        set defaultConfig {
>           hostname   localhost
>           ipaddress  127.0.0.1
>           httpport   8000
>           httpsport  ""
>           server     "openacs"
>           serverroot /var/www/$server
>           logroot    $serverroot/log/
>           homedir    /usr/local/ns
>           bindir     $homedir/bin
>           db_name    $server
>           db_user    $server
>           db_host    localhost
>           db_port    ""
>       }
>
>      These configuration values (keys of the dict) can be overridden by
>      environment variables prefixed with "oacs_" followed by the
>      parameter name. One can change the default port specified in the
>      configuration file for plain HTTP connections by e.g., providing it
>      via environment variables:
>
>          oacs_httpport=8101 /usr/local/ns/bin/nsd -i -t ....
>          
>
> Code Changes:
>
>   - Extended regression test
>      
>   - Code Cleanup
>       . Do not declare reserved C identifiers
>
>   - Improved comments, fixed typos
>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

-- 
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"

Re: [naviserver-devel] Fwd: No notifications with webpush::send

[Gustaf N. <ne...@wu...>]

Part if the problem is that OpenSSL, when configured with the default 
prefix (/usr/local/)
installs its libraries on some platforms into /usr/local/lib64 and not 
into /usr/local/lib.
On these platforms we might have a problem when configuring with

--with-openssl=/usr/local/

I have just now improved the heuristics in the .m4 file (tip version on 
bitbucket),
but i am not completely happy with this, since this potentially 
hardcodes some
Linux conventions. i have to do more cross-platform checks on this 
before the release.

> So far everything seems to work, we only get one warning from the linker:
>
> /usr/bin/ld: warning: libcrypto.so.1.1, needed by 
> /usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/libssl.so, 
> may conflict with libcrypto.so.3
>
This comes, when some files are compiles with different linkage options, 
usually
after continuing with different library settings after an unsuccessful 
compilation attempt.
Normally, a "make clean" + "make" helps.

When using the fresh version form bitbucket, run an "autogen.sh ..." first.

-g

Re: [naviserver-devel] Bug in nsd/binder.c

[Gustaf N. <ne...@wu...>]

Dear Russ,

Many thanks for the report.

The problem seems to occur only, when daemontools are used,
which are used probably mostly in legacy setups (the latest
release of daemontools is according to wikipedia 20 years old [1].
I was not able to reproduce a problem with the recommended
plain systemd setup [2], but nevertheless, reverting the problematic
change seems appropriate,  at least for legacy setups.

Please double-check with the newest version from bitbucket if possible

all the best

-g

[1] https://en.wikipedia.org/wiki/Daemontools
[2] 
https://naviserver.sourceforge.io/n/manual/files/admin-maintenance.html#subsection2

On 12.12.21 02:18, rus...@hi... wrote:
> Several months ago I upgraded to naviserver-4.99.23rc1 and
> initially everything seemed fine, but at some point I noticed
> that if naviserver was restarted the bind port was already taken.
>
> I was using daemontools supervise/svc (started with systemd) to
> start/stop/restart. Initially I blamed systemd since the additional
> process could be controlled with systemctl.
>

[naviserver-devel] Bug in nsd/binder.c

[<rus...@hi...>]

Several months ago I upgraded to naviserver-4.99.23rc1 and
initially everything seemed fine, but at some point I noticed
that if naviserver was restarted the bind port was already taken.

I was using daemontools supervise/svc (started with systemd) to
start/stop/restart. Initially I blamed systemd since the additional
process could be controlled with systemctl. 

This system is Fedora 32 x64 (DigitalOcean droplet), but I have just
verified that the bug shows up on Ubuntu as well.

I used git bisect to locate the exact commit which created the bug,
here's the patch (but see links below):

--- binder.c	2021-08-16 11:59:07.000000000 +0000
+++ binder.c	2021-12-11 06:24:52.543180128 +0000
@@ -1307,12 +1307,10 @@
             (void)ns_sockclose(binderRequest[1]);
             (void)ns_sockclose(binderResponse[0]);
             Binder();
-        } else {
-            /*
-             * Child process.
-             */
-            exit(0);
         }
+
+        exit(0);
+
     } else {
         /*
          * Parent process.

-------------
I believe that the exit(0), placed inside the trailing else allowed
the Binder() to survive. 

Here are my notes (from tags-commits.txt below):
[russell@highfivediet naviserver]$ git bisect --help
[russell@highfivediet naviserver]$ git bisect start
[russell@highfivediet naviserver]$ git bisect bad
[russell@highfivediet naviserver]$ git bisect good naviserver-4.99.17

Bisecting: 698 revisions left to test after this (roughly 10 steps)
STEP1: [65689e7c2cfee84bcf4a9789b4ff14f500e57861] make sure, variable is always initialized
STEP2: [ffeafdebecb999732712c6edf891917ea0eeab4d] Use "for(;;)" instead of "while(1)" like on other places
STEP3: [010daa96412eaf7f02833aa7a9c255ad09b68d8d] fix default temp directory for windows
STEP4: [9a445fcf1b15490728d5803ba3d21df7024eb4c1] Improved compilation cleanness with gcc-11
STEP5: [7909e79e8d41a12dd35e9b28c1afe7dcce9627b0] improved spelling and aligned documentation with code
STEP6: [b70152a504024c4ccf283c2ced9ea6a2a7698511] Fixed Apple M1 problem with double-fork in NsBinder()
STEP7: [51c21ec5bcc4c9c06c6fd1b51b8c27abe5d07bd6] improve variable name ( NaviServer/4.99.22rc1 (naviserver-4.99.20-155-g51c21ec5bcc4+) ) <== BAD
STEP8: [db057401dae34267279f16299b1aee7570240341] styling conventions: use uppercase for hexdigits (NaviServer/4.99.21 (naviserver-4.99.20-149-gdb057401dae3+) ) <== GOOD
STEP9: [7bf2468775ccbbf2191db7b5382624b173332121] fix man page ( NaviServer/4.99.21 (naviserver-4.99.20-152-g7bf2468775cc+) ) <== BAD 
STEP10:[f7740ac62376d99170f904403a9bc50d80824b44] minor cleanup ( NaviServer/4.99.21 (naviserver-4.99.20-151-gf7740ac62376+) ) <== BAD
Bisecting: 0 revisions left to test after this (roughly 0 steps)
STEP11:[8114f6870009985366a65342928062f08cde80fe] Added more comments to double-fork construct in NsForkBinder() ( NaviServer/4.99.21 (naviserver-4.99.20-150-g8114f6870009+) ) <== BAD 

[russell@highfivediet naviserver]$ git bisect bad
8114f6870009985366a65342928062f08cde80fe is the first bad commit
commit 8114f6870009985366a65342928062f08cde80fe
Author: Gustaf Neumann <ne...@wu...>
Date:   Mon Aug 16 13:58:04 2021 +0200

    Added more comments to double-fork construct in NsForkBinder()

 nsd/binder.c | 41 +++++++++++++++++++++++++----------------
 1 file changed, 25 insertions(+), 16 deletions(-)

----- MORE INFO -----

More info generated by git is here: 

https://highfivediet.com/bugs/binder-bug/

The bug doesn't show up in every case, obviously you have to use
the binder switch, but even using a simple command line startup 
didn't create an issue. I have include my startup script at

https://highfivediet.com/bugs/binder-bug/run.txt

Easiest way to test for the bug is to startup and then use
$ ps -axjf and looking for an additional process (same PPID).

On Fedora, this process was connected to pid 1, but on Ubuntu it
was connected to the gnome-terminal-server process.

Otherwise if you kill either of these and then run 
$ ss -ltnp   
the duplicate process is still running on the port (and works fine).

Let me know if more information is needed or if you need other test
run.

Russ

Re: [naviserver-devel] Fwd: No notifications with webpush::send

[Wolfgang W. <wol...@di...>]

This is the solution we came up with:

We downloaded, compiled and installed the latest stable openssl version 
3.0.0.

Then we added /usr/local/lib64 to /etc/ld.so.conf.d/libc.conf

After/sbin/ldconfig Naviserver finds the new openssl version when 
compiling with:

--with-openssl=/usr/local/

So far everything seems to work, we only get one warning from the linker:

/usr/bin/ld: warning: libcrypto.so.1.1, needed by 
/usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/libssl.so, may 
conflict with libcrypto.so.3

Again, thanks for your help,

Wolfgang

Am 09.12.21 um 11:44 schrieb Gustaf Neumann:
>
>
>
>
> -------- Forwarded Message --------
> Subject: 	Re: [naviserver-devel] No notifications with webpush::send
> Date: 	Thu, 9 Dec 2021 11:43:44 +0100
> From: 	Gustaf Neumann <ne...@wu...>
> To: 	Wolfgang Winkler via naviserver-devel 
> <nav...@li...>
>
>
>
>
> On 09.12.21 09:33, Wolfgang Winkler via naviserver-devel wrote:
>>
>> We are using 1.1.1d on our production server, which is a debian buster.
>>
>> bytes {} tag 1e58277931d45f4c593cffbf291b39b7
>
> i can confirm, that with Debian GNU/Linux 10 (buster) and OpenSSL 
> 1.1.1d bytes are empty.
>
> With e.g. Rocky Linux release 8.4 (one successor of CentOS, also 
> conservative), with e.g. 1.1.1g, everything is fine.
>
>
>> I've tried to use 1.1.1k on buster. I installed it with
>>
>> ./config --prefix=/usr/local/openssl && make && make install
>>
>> and compiled naviserver with
>>
>> ./configure 
>> --enable-64bit=true--prefix=/usr/local/naviserver-git--with-openssl=/usr/local/openssl--with-tcl=/usr/local/lib/--enable-threads
>>
>> But naviserver still uses the packaged openssl version:
>> # ldd nsd/nsd
>> libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1
>
>
> There is something starnge  on Buster concerning libraries. I have
> downloaded newest openssl from git, configured + make install, and 
> configured
> Naviserver as usual
>
> $ ./configure --enable-64bit -prefix=/usr/local/ns --with-openssl=/usr/local/
>
> but was surprised that it the version was not picked up for loading.
> After brutally linking the files, everything was fine.
>
> So, there seems to be some load-path that has to be configured for Buster,
> but I am not an expert (and have not time to investigate deeper).
>
> But with this, the right OpenSSL is loaded, encrypt returns non-empty:
>
> $ ln -s /usr/local/lib64/*so* /usr/local/lib/
> $ ldconfig -v
> $ make install
> $ ./nsd/nsd -c -u nsadmin
> [-main:conf-] Notice: OpenSSL 3.1.0-dev  initialized
> ...
> % package require tcltest 2.2
> % namespace import -force ::tcltest::*
> % test aead-1.0 {aead::encrypt} -body {
>      set d [ns_crypto::aead::encrypt string -cipher aes-128-gcm -iv 123456789 -key secret "hello world"]
>      list bytes [string length [dict get $d bytes]] tag [string length [dict get $d tag]]
> } -result {bytes 22 tag 32}
>
>
> I have to rush,
>
> -gn
>
>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
-- 

*Wolfgang Winkler*
Geschäftsführung
wol...@di...
mobil +43.699.19971172

dc:*büro*
digital concepts Novak Winkler OG
Software & Design
Landstraße 68, 5. Stock, 4020 Linz
www.digital-concepts.com <http://www.digital-concepts.com>
tel +43.732.997117.72
tel +43.699.1997117.2

Firmenbuchnummer: 192003h
Firmenbuchgericht: Landesgericht Linz

[naviserver-devel] Fwd: No notifications with webpush::send

[Gustaf N. <ne...@wu...>]

-------- Forwarded Message --------
Subject: 	Re: [naviserver-devel] No notifications with webpush::send
Date: 	Thu, 9 Dec 2021 11:43:44 +0100
From: 	Gustaf Neumann <ne...@wu...>
To: 	Wolfgang Winkler via naviserver-devel 
<nav...@li...>




On 09.12.21 09:33, Wolfgang Winkler via naviserver-devel wrote:
>
> We are using 1.1.1d on our production server, which is a debian buster.
>
> bytes {} tag 1e58277931d45f4c593cffbf291b39b7

i can confirm, that with Debian GNU/Linux 10 (buster) and OpenSSL 1.1.1d 
bytes are empty.

With e.g. Rocky Linux release 8.4 (one successor of CentOS, also 
conservative), with e.g. 1.1.1g, everything is fine.


> I've tried to use 1.1.1k on buster. I installed it with
>
> ./config --prefix=/usr/local/openssl && make && make install
>
> and compiled naviserver with
>
> ./configure 
> --enable-64bit=true--prefix=/usr/local/naviserver-git--with-openssl=/usr/local/openssl--with-tcl=/usr/local/lib/--enable-threads
>
> But naviserver still uses the packaged openssl version:
> # ldd nsd/nsd
> libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1


There is something starnge  on Buster concerning libraries. I have
downloaded newest openssl from git, configured + make install, and 
configured
Naviserver as usual

$ ./configure --enable-64bit -prefix=/usr/local/ns --with-openssl=/usr/local/

but was surprised that it the version was not picked up for loading.
After brutally linking the files, everything was fine.

So, there seems to be some load-path that has to be configured for Buster,
but I am not an expert (and have not time to investigate deeper).

But with this, the right OpenSSL is loaded, encrypt returns non-empty:

$ ln -s /usr/local/lib64/*so* /usr/local/lib/
$ ldconfig -v
$ make install
$ ./nsd/nsd -c -u nsadmin
[-main:conf-] Notice: OpenSSL 3.1.0-dev  initialized
...
% package require tcltest 2.2
% namespace import -force ::tcltest::*
% test aead-1.0 {aead::encrypt} -body {
     set d [ns_crypto::aead::encrypt string -cipher aes-128-gcm -iv 123456789 -key secret "hello world"]
     list bytes [string length [dict get $d bytes]] tag [string length [dict get $d tag]]
} -result {bytes 22 tag 32}


I have to rush,

-gn

Re: [naviserver-devel] No notifications with webpush::send

[Gustaf N. <ne...@wu...>]

On 09.12.21 09:33, Wolfgang Winkler via naviserver-devel wrote:
>
> We are using 1.1.1d on our production server, which is a debian buster.
>
> bytes {} tag 1e58277931d45f4c593cffbf291b39b7

i can confirm, that with Debian GNU/Linux 10 (buster) and OpenSSL 1.1.1d 
bytes are empty.

With e.g. Rocky Linux release 8.4 (one successor of CentOS, also 
conservative), with e.g. 1.1.1g, everything is fine.


> I've tried to use 1.1.1k on buster. I installed it with
>
> ./config --prefix=/usr/local/openssl && make && make install
>
> and compiled naviserver with
>
> ./configure 
> --enable-64bit=true--prefix=/usr/local/naviserver-git--with-openssl=/usr/local/openssl--with-tcl=/usr/local/lib/--enable-threads
>
> But naviserver still uses the packaged openssl version:
> # ldd nsd/nsd
> libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1


There is something starnge  on Buster concerning libraries. I have
downloaded newest openssl from git, configured + make install, and 
configured
Naviserver as usual

$ ./configure --enable-64bit -prefix=/usr/local/ns --with-openssl=/usr/local/

but was surprised that it the version was not picked up for loading.
After brutally linking the files, everything was fine.

So, there seems to be some load-path that has to be configured for Buster,
but I am not an expert (and have not time to investigate deeper).

But with this, the right OpenSSL is loaded, encrypt returns non-empty:

$ ln -s /usr/local/lib64/*so* /usr/local/lib/
$ ldconfig -v
$ make install
$ ./nsd/nsd -c -u nsadmin
[-main:conf-] Notice: OpenSSL 3.1.0-dev  initialized
...
% package require tcltest 2.2
% namespace import -force ::tcltest::*
% test aead-1.0 {aead::encrypt} -body {
     set d [ns_crypto::aead::encrypt string -cipher aes-128-gcm -iv 123456789 -key secret "hello world"]
     list bytes [string length [dict get $d bytes]] tag [string length [dict get $d tag]]
} -result {bytes 22 tag 32}


I have to rush,

-gn

Re: [naviserver-devel] No notifications with webpush::send

[Wolfgang W. <wol...@di...>]

We are using 1.1.1d on our production server, which is a debian buster.

Here we get the result

bytes {} tag 1e58277931d45f4c593cffbf291b39b7

On our newer system with openssl 1.1.1k

bytes dbf5a0d98c8b32e1b6dccc tag 81294002f4a5df64c8a44e1bcc07a6cb

The call was:

ns_crypto::aead::encrypt string -cipher aes-128-gcm -iv 123456789 -key 
secret "hello world"


I've tried to use 1.1.1k on buster. I installed it with

./config --prefix=/usr/local/openssl && make && make install

and compiled naviserver with

./configure 
--enable-64bit=true--prefix=/usr/local/naviserver-git--with-openssl=/usr/local/openssl--with-tcl=/usr/local/lib/--enable-threads

But naviserver still uses the packaged openssl version:
# ldd nsd/nsd
libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1

Regards,

Wolfgang


Am 07.12.21 um 17:16 schrieb Gustaf Neumann:
>
>
> On 07.12.21 14:40, Wolfgang Winkler via naviserver-devel wrote:
>> We also tried the demo here:
>>
>> https://openacs.org/webpush-demo/webpush-demo.tcl
>>
>> The empty message works ("Naviserver rocks!"), but with a message, we 
>> get an error:
>>
>> "could not derive EC point from provided key could not derive EC 
>> point from provided key "
>>
> you are right, something seems broken at least at openacs.org. The 
> regression test of nswebpush fails at openacs.org  (but succeeds on my 
> local machine). ... so maybe there is a version dependency.
>
> What version of OpenSSL are you using?
>
> -gn
>
> PS: it will take probably until the weekend to get some time in a 
> block to looking this.
>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
-- 

*Wolfgang Winkler*
Geschäftsführung
wol...@di...
mobil +43.699.19971172

dc:*büro*
digital concepts Novak Winkler OG
Software & Design
Landstraße 68, 5. Stock, 4020 Linz
www.digital-concepts.com <http://www.digital-concepts.com>
tel +43.732.997117.72
tel +43.699.1997117.2

Firmenbuchnummer: 192003h
Firmenbuchgericht: Landesgericht Linz

Re: [naviserver-devel] No notifications with webpush::send

[Gustaf N. <ne...@wu...>]

On 07.12.21 14:40, Wolfgang Winkler via naviserver-devel wrote:
> We also tried the demo here:
>
> https://openacs.org/webpush-demo/webpush-demo.tcl
>
> The empty message works ("Naviserver rocks!"), but with a message, we 
> get an error:
>
> "could not derive EC point from provided key could not derive EC point 
> from provided key "
>
you are right, something seems broken at least at openacs.org. The 
regression test of nswebpush fails at openacs.org  (but succeeds on my 
local machine). ... so maybe there is a version dependency.

What version of OpenSSL are you using?

-gn

PS: it will take probably until the weekend to get some time in a block 
to looking this.

[naviserver-devel] No notifications with webpush::send

[Wolfgang W. <wol...@di...>]

Hi list!

We are trying to send webpush messages with the naviserver webpush moule 
(webpush::send). It used to work fine a few months ago.

Now we have the following problem: When sending an empty push message with:

setmessage{}
setret [webpush::send \
            -subscription $subscription_list\
            -data $message\
            -claim [listsub mailto:${claim_email}] \
            -privateKeyPem [dictget $channelprivate_key_path] \
            -mode aesgcm \
            -timeout 3\
            -ttl $ttl\
        ]

we get a proper desktop notification with "No message!" body.

When we try to enter a message, the call returns with a status of 201, 
but no notification is shown.

We also tried the demo here:

https://openacs.org/webpush-demo/webpush-demo.tcl

The empty message works ("Naviserver rocks!"), but with a message, we 
get an error:

"could not derive EC point from provided key could not derive EC point 
from provided key "

After a bit of code exploration, we noticed, that the call to 
"::ns_crypto::aead::encrypt"
in webpush::encrypt returns an empty result for "bytes".

So we've tried the examples from

https://naviserver.sourceforge.io/n/naviserver/files/ns_crypto.html#1

which get the same result.

ns_crypto::aead::encrypt string -cipher aes-128-gcm -iv 123456789 -key 
secret "hello world"

returns

bytes {} tag ecb7f31df9e12b6118c1d8fcd5b7534a

instead of

bytes fa260f97eae35e3e3df0b7 tag 93654f78fd189b559c091acb410a0040

Any insights would be highly appreciated.

Regards,

Wolfgang
-- 

*Wolfgang Winkler*
Geschäftsführung
wol...@di...
mobil +43.699.19971172

dc:*büro*
digital concepts Novak Winkler OG
Software & Design
Landstraße 68, 5. Stock, 4020 Linz
www.digital-concepts.com <http://www.digital-concepts.com>
tel +43.732.997117.72
tel +43.699.1997117.2

Firmenbuchnummer: 192003h
Firmenbuchgericht: Landesgericht Linz

[naviserver-devel] Towards NaviServer 4.99.23

[Gustaf N. <ne...@wu...>]

Dear all,

on sourceforge is a release candidate for NaviServer 4.99.23 [1].

The change contains one potentially important fix for a memory
leak (which was in the code since 1y+, but turned out to be significant
with a recent OpenACS change) and a couple of new features (including
the improved Unicode support for e.g. emojis, and crypto improvments
like SCRAM, which is already incorporated in OpenACS).

Please test if possible.  The release should be in the near future.

Below is a preliminary summary of changes.

all the best
-gustaf neumann

[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.23/

=======================================
NaviServer 4.99.23, released 2021-12-XX
=======================================

  60 files changed, 1014 insertions(+), 270 deletions(-)

New Features:

  - Improved hash algorithms for improved security

    The new version supports SCRAM (Salted Challenge Response
    Authentication Mechanism), which is one of the newer recommended
    password hash algorithm to replace the classic salted SHA1
    approaches. The classical hash algorithms become easier to attack
    via high performance hashing hardware (GPUs).  NaviServer supports
    now SCRYPT and SCRAM (when compiled with more recent versions of
    OpenSSL; SCRYPT requires at least OpenSSL 3.0. SCRAM requires OpenSSL 1.1)

    The actual hash function of SCRAM is PBKDF2 [RFC2898] with HMAC as
    the pseudorandom function (PRF) and with dkLen == output length of
    HMAC == output length of the digest function.
     
    Here is an example of using pbkdf2_hmac for the hash function of
    SCRAM-sha-256. RFC 7677 recommends to use 15K iterations for PBKDF2:
     
       ::ns_crypto::pbkdf2_hmac \
                -digest sha256 \
                -iterations 15000 \
                -secret $password \
                -salt $salt]
                
    OpenACS supports already switching to from salted SHA1 to SCAM (or
    SCRYPT) via configuration variable.


  - Better Unicode support, including emojis requiring 4-byte UTF-8 characters.

    Earlier versions of NaviServer and the nsdb* database drivers
    assumed on a few places that Tcl-internal UTF-8 is also valid UTF-8
    for external sources, which is often, but not always true. Now, the
    proper export functions are everywhere called.

    The new code was tested with Emojis up to Unicode 13 (many thanks
    to Wolfgang Winkler)

    This change effects as well the database driver module "nsdbpg".

  - ns_trim enhancements:
    The new option "-prefix ..." can be used to strip a string
    (such as ">> ") from every line starting with it.

  - extended time unit support (added "w" for weeks and "y" for years)

  - Added an experimental global parameter "nocache" to ease to
    experiment with horizontal scaling. As a consequence, "ns_cache
    eval" becomes a dummy operation.

  - Added an experimental command "ns_baseunit"
     
       ns_baseunit ?-size size? ?-time time?
     
    Convert from memory units or from time units to its base value
    using the NaviServer internal converters, which are used the same
    way for various commands.  The base unit for a memory size is a
    byte, for a time value a second). This command is necessary to
    provide Tcl-level commands calculating with these units to support
    uniform interfaces (e.g., calculating cache partition sizes base on
    values such as 2MB).
     
    Either "-size" or "-time" has to be specified.
     
       % ns_baseunit -size 10KB
       10240
     
       ns_baseunit -time 2.5h
       9000
     
  - Added an experimental command "ns_strcoll"
  
        ns_strcoll ?-locale locale? string1 string2"
     
    This command compares lexicographically string1 with string2
    according to the current locale collation and returns an integer
    greater than, equal to, or less than 0, depending on whether
    string1 is greater than, equal to, or less than string2. The
    command is necessary in cases, where e.g., the sorting order from
    the database (normally based on local collation) is different from
    default Tcl sorting order to provide a uniform interface with same
    sorting orders. The name is derived from the baseline POSIX
    function call.
     
    The command is suitable for usage in the lsort command:
     
       % set l {Bor Bar Bär}
       % lsort -command ns_strcoll $l
       Bar Bär Bor
     
       % lsort $l
       Bar Bor Bär


Performance Improvements:

  - Increase scalability on DB operations by reducing DB pool locks
  
    On high load servers, the total number of locks and busy locks on
    the DB pools might become quite high. The new code caches these
    statistics in the handles (which are per thread, requiring no
    locks) and transfers the aggregated values on handle closes or
    statistics calls.

  - Set default for "concurrentinterpcreate" to "true" for Tcl 8.6 or
    newer. Versions up to at least Tcl 8.5 are known to crash in case
    two threads create interpreters at the same time. These crashes
    were hard to reproduce, but serializing interpreter creation
    helped. Since all our major servers are running since several years
    without problems with this parameter turned on the default is now
    set to "true", when NaviServer is compiled with Tcl 8.6 or newer.


Bug Fixes:
  - Fixed memory leak in "nsv_dict get" operations.

  
Documentation improvements:
---------------------------

  - Improved the following man pages:

	doc/src/manual/admin-tuning.man
	doc/src/naviserver/commandlist.man
	doc/src/naviserver/ns_adp_include.man
	doc/src/naviserver/ns_baseunit.man
	doc/src/naviserver/ns_cache.man
	doc/src/naviserver/ns_charsets.man
	doc/src/naviserver/ns_choosecharset.man
	doc/src/naviserver/ns_connchan.man
	doc/src/naviserver/ns_cookie.man
	doc/src/naviserver/ns_cookiecharset.man
	doc/src/naviserver/ns_crypto.man
	doc/src/naviserver/ns_encodingforcharset.man
	doc/src/naviserver/ns_encodingfortype.man
	doc/src/naviserver/ns_formfieldcharset.man
	doc/src/naviserver/ns_ictl.man
	doc/src/naviserver/ns_job.man
	doc/src/naviserver/ns_register.man
	doc/src/naviserver/ns_schedule.man
	doc/src/naviserver/ns_setformencoding.man
	doc/src/naviserver/ns_shutdown.man
	doc/src/naviserver/ns_sleep.man
	doc/src/naviserver/ns_sockcallback.man
	doc/src/naviserver/ns_sockopen.man
	doc/src/naviserver/ns_strcoll.man
	doc/src/naviserver/ns_time.man
	doc/src/naviserver/ns_urlcharset.man
	doc/src/naviserver/ns_valid_utf8.man
	doc/src/naviserver/textutil-cmds.man
  

Configuration Changes:
----------------------

  -  Ease configuration via environment variables

     This feature is useful to manage many NaviServer instances with
     mostly identical configurations without having to provide multiple
     configuration files (e.g. for Docker setups, or clusters).
     
     The sample configuration for of OpenACS (openacs-config.tcl)
     contains now a Tcl dictionary with default values:

       set defaultConfig {
          hostname   localhost
          ipaddress  127.0.0.1
          httpport   8000
          httpsport  ""
          server     "openacs"
          serverroot /var/www/$server
          logroot    $serverroot/log/
          homedir    /usr/local/ns
          bindir     $homedir/bin
          db_name    $server
          db_user    $server
          db_host    localhost
          db_port    ""
      }

     These configuration values (keys of the dict) can be overridden by
     environment variables prefixed with "oacs_" followed by the
     parameter name. One can change the default port specified in the
     configuration file for plain HTTP connections by e.g., providing it
     via environment variables:

         oacs_httpport=8101 /usr/local/ns/bin/nsd -i -t ....
         

Code Changes:

  - Extended regression test
     
  - Code Cleanup
      . Do not declare reserved C identifiers

  - Improved comments, fixed typos

Re: [naviserver-devel] Issue with certain emojis (unicode/utf-8)

[Gustaf N. <ne...@wu...>]

Dear all,

It will take some time, until the Emojis from Unicode 14 will be 
generally available, but when this comes, we should have already 
everything working in NaviServer and the DB interfaces. I've added a 
small demo page, one can try when the new clients come out:

https://openacs.org/emojis.tcl

One interesting part is the grapheme cluster (like e.g. 👩‍👩‍👧‍👦), 
which is made up of the following unicode graphemes:

    WOMAN👩 ZWJ WOMAN👩 ZWJ GIRL👧 ZWJ BOY👦

where ZWJ are zero-width joiners. One can enter these e.g. via

   set x \ud83d\udc69\u200d\ud83d\udc69\u200d\ud83d\udc67\u200d\ud83d\udc66

into current Tcl. When just passing such string through Tcl, everything 
seems fine. I would not be surprised that  "string length", "string 
range" etc can lead to unexpected results, but is is quite fun to 
decompose this emoji with Tcl:

    % set x \ud83d\udc69\u200d\ud83d\udc69\u200d\ud83d\udc67\u200d\ud83d\udc66
    👩‍👩‍👧‍👦
    % string range $x 0 1
    👩
    % string range $x 6 7
    👧
    % string range $x 9 10
    👦

AFIKT, eveything is fine with NaviServer in this respect.

Concerning Unicode 14:

Android 12L contains support for Emojis from Unicode 14. Google 
announced Android 12L in October 2021, less than one month after the 
stable release of Android 12. 12L is expected in early 2022 [2].

According to [3] iOS 15.0 will not include Unicode 14 emojis. Support 
for Emoji 14.0 on Apple platforms is expected in the first half of 2022 
(probably in iOS 16).

all the best

-g

[1] https://9to5google.com/2021/10/27/android-12l-unicode-14/
[2] https://developer.android.com/about/versions/12/12L/summary
[3] https://emojipedia.org/apple/


On 26.11.21 10:40, Wolfgang Winkler via naviserver-devel wrote:
>
> Hi!
>
> We've testet the encoding now extensively. All Emojis up to 13 
> <https://emojipedia.org/unicode-13.0/> are handled correctly, 
> including database storage and retrieving, tdom and form handling.
>
> Version 14 emojis <https://emojipedia.org/unicode-14.0/> are not 
> supported by any of the browsers we've testet, but don't throw errors. 
> It seems we are save for future updates.
>
> Wolfgang
>
> Am 18.11.21 um 18:24 schrieb Gustaf Neumann:
>>
>> Dear all
>>
>> On bitbucket is now an update (see change log message below) that 
>> introduces support of UTF-8 characters using up to 4 bytes (with Tcl 
>> 8.6). It should work as well with 6 byte UTF when Tcl 8.7 is properly 
>> compiled (by setting TCL_UTF_MAX).
>>
>> One can now use e.g. emoticons in SQL queries
>>
>>      db_0or1row ... {select 1 from cr_items where name = '😈'}
>>
>> or as values of bind variables
>>
>>      set x 😈
>>      db_0or1row ... {select 1 from cr_items where name = :x}
>>
>> ... but not as names of bind variables (these have the same 
>> restricted syntax than before
>> (in essence no funny characters).
>>
>> The code is already running at openacs.org.
>>
>> all the best
>>
>> -gn
>>
>>
>> Added support for UTF-8 characters up to 4 bytes
>>
>> These changes add proper export of UTF-8 for Unicode symbols taking up
>> to 4 bytes. For the western world the biggest interest is probably for
>> emoticons. The change is implemented with performance in mind. The
>> proper encoded byte-strings are cached in Tcl_Objs, such that only the
>> values for bind-vars (which have probably different values per call)
>> have to be recoded at call time. This should keep the performance
>> penalty small (we see on some of our servers in day-average 1500 SQL
>> operations per second, peaks at >10K).
>>
>> The names of bind variables follow still the same rules as before (no
>> emoticons as variable names).
>>
>> On 16.11.21 16:39, Wolfgang Winkler via naviserver-devel wrote:
>>
>>>>> the fix worked, thank you Gustaf! But we still have a problem with 
>>>>> emojis when writing them to the database. The error we get is:
>>>>>
>>>>> Database operation "dml" failed (exception ERROR, "ERROR:  invalid 
>>>>> byte sequence for encoding "UTF8": 0xf0 0x9f
>>>>>
>>
>>
>>
>> _______________________________________________
>> naviserver-devel mailing list
>> nav...@li...
>> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> -- 
>
> *Wolfgang Winkler*
> Geschäftsführung
> wol...@di...
> mobil +43.699.19971172
>
> dc:*büro*
> digital concepts Novak Winkler OG
> Software & Design
> Landstraße 68, 5. Stock, 4020 Linz
> www.digital-concepts.com <http://www.digital-concepts.com>
> tel +43.732.997117.72
> tel +43.699.1997117.2
>
> Firmenbuchnummer: 192003h
> Firmenbuchgericht: Landesgericht Linz
>
>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

-- 
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"

Re: [naviserver-devel] Issue with certain emojis (unicode/utf-8)

[Wolfgang W. <wol...@di...>]

Hi!

We've testet the encoding now extensively. All Emojis up to 13 
<https://emojipedia.org/unicode-13.0/> are handled correctly, including 
database storage and retrieving, tdom and form handling.

Version 14 emojis <https://emojipedia.org/unicode-14.0/> are not 
supported by any of the browsers we've testet, but don't throw errors. 
It seems we are save for future updates.

Wolfgang

Am 18.11.21 um 18:24 schrieb Gustaf Neumann:
>
> Dear all
>
> On bitbucket is now an update (see change log message below) that 
> introduces support of UTF-8 characters using up to 4 bytes (with Tcl 
> 8.6). It should work as well with 6 byte UTF when Tcl 8.7 is properly 
> compiled (by setting TCL_UTF_MAX).
>
> One can now use e.g. emoticons in SQL queries
>
>      db_0or1row ... {select 1 from cr_items where name = '😈'}
>
> or as values of bind variables
>
>      set x 😈
>      db_0or1row ... {select 1 from cr_items where name = :x}
>
> ... but not as names of bind variables (these have the same restricted 
> syntax than before
> (in essence no funny characters).
>
> The code is already running at openacs.org.
>
> all the best
>
> -gn
>
>
> Added support for UTF-8 characters up to 4 bytes
>
> These changes add proper export of UTF-8 for Unicode symbols taking up
> to 4 bytes. For the western world the biggest interest is probably for
> emoticons. The change is implemented with performance in mind. The
> proper encoded byte-strings are cached in Tcl_Objs, such that only the
> values for bind-vars (which have probably different values per call)
> have to be recoded at call time. This should keep the performance
> penalty small (we see on some of our servers in day-average 1500 SQL
> operations per second, peaks at >10K).
>
> The names of bind variables follow still the same rules as before (no
> emoticons as variable names).
>
> On 16.11.21 16:39, Wolfgang Winkler via naviserver-devel wrote:
>
>>>> the fix worked, thank you Gustaf! But we still have a problem with 
>>>> emojis when writing them to the database. The error we get is:
>>>>
>>>> Database operation "dml" failed (exception ERROR, "ERROR:  invalid 
>>>> byte sequence for encoding "UTF8": 0xf0 0x9f
>>>>
>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
-- 

*Wolfgang Winkler*
Geschäftsführung
wol...@di...
mobil +43.699.19971172

dc:*büro*
digital concepts Novak Winkler OG
Software & Design
Landstraße 68, 5. Stock, 4020 Linz
www.digital-concepts.com <http://www.digital-concepts.com>
tel +43.732.997117.72
tel +43.699.1997117.2

Firmenbuchnummer: 192003h
Firmenbuchgericht: Landesgericht Linz

Re: [naviserver-devel] Issue with certain emojis (unicode/utf-8)

[Gustaf N. <ne...@wu...>]

Dear all

On bitbucket is now an update (see change log message below) that 
introduces support of UTF-8 characters using up to 4 bytes (with Tcl 
8.6). It should work as well with 6 byte UTF when Tcl 8.7 is properly 
compiled (by setting TCL_UTF_MAX).

One can now use e.g. emoticons in SQL queries

     db_0or1row ... {select 1 from cr_items where name = '😈'}

or as values of bind variables

     set x 😈
     db_0or1row ... {select 1 from cr_items where name = :x}

... but not as names of bind variables (these have the same restricted 
syntax than before
(in essence no funny characters).

The code is already running at openacs.org.

all the best

-gn


Added support for UTF-8 characters up to 4 bytes

These changes add proper export of UTF-8 for Unicode symbols taking up
to 4 bytes. For the western world the biggest interest is probably for
emoticons. The change is implemented with performance in mind. The
proper encoded byte-strings are cached in Tcl_Objs, such that only the
values for bind-vars (which have probably different values per call)
have to be recoded at call time. This should keep the performance
penalty small (we see on some of our servers in day-average 1500 SQL
operations per second, peaks at >10K).

The names of bind variables follow still the same rules as before (no
emoticons as variable names).

On 16.11.21 16:39, Wolfgang Winkler via naviserver-devel wrote:

>>> the fix worked, thank you Gustaf! But we still have a problem with 
>>> emojis when writing them to the database. The error we get is:
>>>
>>> Database operation "dml" failed (exception ERROR, "ERROR:  invalid 
>>> byte sequence for encoding "UTF8": 0xf0 0x9f
>>>

Re: [naviserver-devel] Issue with certain emojis (unicode/utf-8)

[Wolfgang W. <wol...@di...>]

We've a workaround which is probably much worse than yours, but working. 
We are using nsdbpg, I've not tried the query with the nsdbi interface.

Am 16.11.21 um 17:06 schrieb Gustaf Neumann:
>
> Funny enough, i same a very similar problem today and provide a local 
> fix for this. I am not happy with this fix since it is rather costly, 
> so i would like to work on this more before committing. However, today 
> and tomorrow i am fully booked with urgent items, so don't expect a 
> fix for this before the weekend.
>
> -g
>
> PS: i assume, you are using the nsdbpg driver.
>
> On 16.11.21 16:39, Wolfgang Winkler via naviserver-devel wrote:
>> the fix worked, thank you Gustaf! But we still have a problem with 
>> emojis when writing them to the database. The error we get is:
>>
>> Database operation "dml" failed (exception ERROR, "ERROR:  invalid 
>> byte sequence for encoding "UTF8": 0xf0 0x9f 0x98 0xff
>>
>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
-- 

*Wolfgang Winkler*
Geschäftsführung
wol...@di...
mobil +43.699.19971172

dc:*büro*
digital concepts Novak Winkler OG
Software & Design
Landstraße 68, 5. Stock, 4020 Linz
www.digital-concepts.com <http://www.digital-concepts.com>
tel +43.732.997117.72
tel +43.699.1997117.2

Firmenbuchnummer: 192003h
Firmenbuchgericht: Landesgericht Linz

Re: [naviserver-devel] Issue with certain emojis (unicode/utf-8)

[Gustaf N. <ne...@wu...>]

Funny enough, i same a very similar problem today and provide a local 
fix for this. I am not happy with this fix since it is rather costly, so 
i would like to work on this more before committing. However, today and 
tomorrow i am fully booked with urgent items, so don't expect a fix for 
this before the weekend.

-g

PS: i assume, you are using the nsdbpg driver.

On 16.11.21 16:39, Wolfgang Winkler via naviserver-devel wrote:
> the fix worked, thank you Gustaf! But we still have a problem with 
> emojis when writing them to the database. The error we get is:
>
> Database operation "dml" failed (exception ERROR, "ERROR:  invalid 
> byte sequence for encoding "UTF8": 0xf0 0x9f 0x98 0xff
>

Re: [naviserver-devel] Issue with certain emojis (unicode/utf-8)

[Wolfgang W. <wol...@di...>]

Hello all,

the fix worked, thank you Gustaf! But we still have a problem with 
emojis when writing them to the database. The error we get is:

Database operation "dml" failed (exception ERROR, "ERROR:  invalid byte 
sequence for encoding "UTF8": 0xf0 0x9f 0x98 0xff

when trying to write the emoji to a TEXT or VARCHAR field in the 
database. Inserting the same string in the database console works as 
expected. When we read the string and reinsert it, it also works 
flawlessly.

We've compared the two strings, wrote them to files and compared them 
with a hex reader, converted them with tcl "encoding convertto" and 
iconv, all with no luck.

We are using postgres 12 and the nsdbpg module with 
naviserver-4.99.22-16-g67adf3c34710+

Here is the test case:

In the database console:

CREATE TABLE test (
     idx SERIAL,
     txt TEXT
);

INSERT INTO test (txt) VALUES ('<smiley>😃</smiley>');

In the naviserver console or in a script:

# V1: working
set db [ns_db gethandle]
set sql "SELECT txt FROM test WHERE idx=1"
set selection [ns_db 1row $db $sql]
set str [ns_set value $selection 0]
set sql "INSERT INTO test (txt) VALUES ('$str')"
ns_db dml $db $sql
ns_db releasehandle $db

# V2: not working
set db [ns_db gethandle]
set sql "INSERT INTO test (txt) VALUES ('<smiley>😃</smiley>')"
ns_db dml $db $sql
ns_db releasehandle $db

With nscp, pasting the string of V2 already shows a wrong string in the log:

Notice: nscp:  3: set sql "INSERT INTO test (txt) VALUES 
('<smiley>�������</smiley>')"


Whereas V1 works (the smiley is not printed here, but works in the console):

Notice: nscp:  5: puts $str
<smiley></smiley>

Any help is greatly appreciated!

Wolfgang Winkler

Am 09.11.21 um 09:36 schrieb Gustaf Neumann:
> Dear all,
>
> The situation is trickier than someone might hope.  Aside of the Tcl 
> version dependencies (as Brian pointed out), Tcl before 8.7 do not 
> support TCL_UTF_MAX with longer multi-byte sequences than 4 (see Tcl 
> TIP 389), which are also mostly relevant for some newer emojis. So, 
> for full emoji support, Tcl 8.7 with the proper compilation options is 
> needed.
>
> Anyhow, in the case of Wolfgang's the "Smiling Face with Open Mouth" 
> we have just a 4-byte UTF-8 character, which is supported by 
> out-of-the-box Tcl 8.6. However, this emoji is represented 
> Tcl-internally as a 6-byte sequence. Since NaviServer wrongly assumed 
> that Tcl-internal representations are also accepted as external 
> representations, a conversion step was omitted for utf-8 (which is not 
> always true).
>
> In the tip version of NaviServer on Bitbucket, this optimization is 
> now removed, the examples work as expected, the regression test is 
> extended for this case.
>
> Many thanks to Wolfgang for the good bug report.
>
> -g
>
>
>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
-- 

*Wolfgang Winkler*
Geschäftsführung
wol...@di...
mobil +43.699.19971172

dc:*büro*
digital concepts Novak Winkler OG
Software & Design
Landstraße 68, 5. Stock, 4020 Linz
www.digital-concepts.com <http://www.digital-concepts.com>
tel +43.732.997117.72
tel +43.699.1997117.2

Firmenbuchnummer: 192003h
Firmenbuchgericht: Landesgericht Linz

Re: [naviserver-devel] Issue with certain emojis (unicode/utf-8)

[Gustaf N. <ne...@wu...>]

Dear all,

The situation is trickier than someone might hope.  Aside of the Tcl 
version dependencies (as Brian pointed out), Tcl before 8.7 do not 
support TCL_UTF_MAX with longer multi-byte sequences than 4 (see Tcl TIP 
389), which are also mostly relevant for some newer emojis. So, for full 
emoji support, Tcl 8.7 with the proper compilation options is needed.

Anyhow, in the case of Wolfgang's the "Smiling Face with Open Mouth" we 
have just a 4-byte UTF-8 character, which is supported by out-of-the-box 
Tcl 8.6. However, this emoji is represented Tcl-internally as a 6-byte 
sequence. Since NaviServer wrongly assumed that Tcl-internal 
representations are also accepted as external representations, a 
conversion step was omitted for utf-8 (which is not always true).

In the tip version of NaviServer on Bitbucket, this optimization is now 
removed, the examples work as expected, the regression test is extended 
for this case.

Many thanks to Wolfgang for the good bug report.

-g

Re: [naviserver-devel] Issue with certain emojis (unicode/utf-8)

[Brian F. <Bri...@qu...>]

Hi Wolfgang

I wonder if it's a TCL issue? For example, I see this discussion has some mentions of problems with emojis in some TCL versions https://core.tcl-lang.org/tips/doc/trunk/tip/600.md

Brian

________________________________
From: Wolfgang Winkler via naviserver-devel <nav...@li...>
Sent: Monday 8 November 2021 11:09
To: nav...@li... <nav...@li...>
Cc: Wolfgang Winkler <wol...@di...>
Subject: [naviserver-devel] Issue with certain emojis (unicode/utf-8)


This message's attachments contains at least one web link. This is often used for phishing attempts. Please only interact with this attachment if you know its source and that the content is safe. If in doubt, confirm the legitimacy with the sender by phone.


Dear all!

We face some issues with certain UTF-8 characters, e.g. this one: https://emojipedia.org/grinning-face-with-big-eyes/.

When we set "charset" to "utf-8", everything works as expected, except the output of various emojis.

set str "Test smiley: 😃<br>öüäÖÄÜß<br>"
ns_return 200 "text/html; charset=unicode" $str; return

returns the smiley correctly, whereas

set str "Test smiley: 😃<br>öüäÖÄÜß<br>"
ns_return 200 "text/html; charset=utf-8" $str; return

returns

Test smiley: ������
öüäÖÄÜß

So I tried to set the charset to "unicode". This works for some files and not for others, especially not for javascript files.

This are the parameters in the config section:

ns_section "ns/parameters"
 ....
 #ns_param   HackContentType true
 ns_param   HackContentType false
 ns_param   OutputCharset   $charset
 ns_param   URLCharset      $charset


We also tried with nscp and the tclsh:

nscp Input:

puts "😃"

Log Output:

Notice: nscp:  1: puts "�������"

The nscp telnet client does not return to the prompt.

Tclsh works as expected:

tclsh
% puts "😃"
😃


Tcl version is 8.6.11, naviserver 4.99.22 running on Debian 10.11.

Has anybody encountered and solved a similiar issue?

Thanks,

Wolfgang Winkler

--

Wolfgang Winkler
Geschäftsführung
wol...@di...<mailto:wol...@di...>
mobil +43.699.19971172

dc:büro
digital concepts Novak Winkler OG
Software & Design
Landstraße 68, 5. Stock, 4020 Linz
www.digital-concepts.com<http://www.digital-concepts.com>
tel +43.732.997117.72
tel +43.699.1997117.2

Firmenbuchnummer: 192003h
Firmenbuchgericht: Landesgericht Linz

[https://www.digital-concepts.com/cu/digitalconcepts2016/images/logo_digitalconcepts2016.png]

[naviserver-devel] Issue with certain emojis (unicode/utf-8)

[Wolfgang W. <wol...@di...>]

Dear all!

We face some issues with certain UTF-8 characters, e.g. this one: 
https://emojipedia.org/grinning-face-with-big-eyes/.

When we set "charset" to "utf-8", everything works as expected, except 
the output of various emojis.

setstr "Test smiley: 😃<br>öüäÖÄÜß<br>"
ns_return 200"text/html; charset=unicode"$str; return

returns the smiley correctly, whereas

setstr "Test smiley: 😃<br>öüäÖÄÜß<br>"
ns_return 200"text/html; charset=utf-8"$str; return

returns

Test smiley: ������
öüäÖÄÜß

So I tried to set the charset to "unicode". This works for some files 
and not for others, especially not for javascript files.

This are the parameters in the config section:

ns_section "ns/parameters"
  ....
  #ns_param   HackContentType true
  ns_param   HackContentType false
  ns_param   OutputCharset $charset
  ns_param   URLCharset $charset

We also tried with nscp and the tclsh:

nscp Input:

puts "😃"

Log Output:

Notice: nscp:  1: puts "�������"

The nscp telnet client does not return to the prompt.

Tclsh works as expected:

tclsh
% puts "😃"
😃

Tcl version is 8.6.11, naviserver 4.99.22 running on Debian 10.11.

Has anybody encountered and solved a similiar issue?

Thanks,

Wolfgang Winkler

-- 

*Wolfgang Winkler*
Geschäftsführung
wol...@di...
mobil +43.699.19971172

dc:*büro*
digital concepts Novak Winkler OG
Software & Design
Landstraße 68, 5. Stock, 4020 Linz
www.digital-concepts.com <http://www.digital-concepts.com>
tel +43.732.997117.72
tel +43.699.1997117.2

Firmenbuchnummer: 192003h
Firmenbuchgericht: Landesgericht Linz

Re: [naviserver-devel] Buffer overflow detected

[Gustaf N. <ne...@wu...>]

There are many possible reasons for the crash, i just mentioned one 
possibility which i have seen in the past. You have to check with gdb as 
mentioned below to find the reason in your case.

If it is indeed the select() problem, then it might be due to a bug in 
your Tcl code (e.g. missing close) or not (really requiring > 1024 open 
file descriptors).

-g

On 06.11.21 19:43, Maksym Zinchenko wrote:
> Thank you very much for explaining Gustaf, at least I know now that 
> somewhere in my Tcl code something is wrong. I'm using only 2 C 
> modules/packages.
>
> On Fri, Nov 5, 2021 at 7:37 AM Gustaf Neumann <ne...@wu...> wrote:
>
>     Dear Maksym,
>
>     These kind of errors should never happen, but when this happen,
>     this can
>     come from NaviServer or Tcl or some C modules/package loaded.
>
>     The last time i saw this kind of crash in a NaviServer
>     environment, it
>     was triggered from Tcl, where a select() was tried in a situation
>     where
>     more than 1024 file descriptors were open (stay away from async
>     Tcl I/O
>     operations and use NaviServer built-in features ... e.g. ns_http
>     instead
>     of tcllib http). The number of open file descriptors of NaviServer
>     can
>     be monitored e.g. by the munin-plugins [2,3].
>
>     To debug this situation,
>     - make sure to compile Tcl and NaviServer with debugging symbols
>     (i.e.
>     with compiler flag "-g" [1]),
>     - make sure, core dumps are enabled, and when you have a core,
>     - use "gdb /usr/local/ns/bin/nsd YOURCORE" to see exactly, where this
>     happened.
>
>     all the best
>
>     -g
>
>     [1] https://openacs.org/forums/message-view?message_id=5537675
>     [2] https://github.com/gustafn/munin-plugins-ns
>     [3]
>     https://openacs.org/munin/localdomain/localhost.localdomain/naviserver_openacs_lsof.html
>
>     On 04.11.21 21:17, Maksym Zinchenko wrote:
>     > Hi every now and then my server crashes with the last message in
>     the
>     > log file *** Buffer overflow detected *** I would really
>     appreciate if
>     > someone can point me in the right direction of debugging and figure
>     > out what's going on.
>     > Thank you
>     > Maksym Zinchenko
>

Re: [naviserver-devel] Buffer overflow detected

[Maksym Z. <siq...@gm...>]

Thank you very much for explaining Gustaf, at least I know now that
somewhere in my Tcl code something is wrong. I'm using only 2 C
modules/packages.

On Fri, Nov 5, 2021 at 7:37 AM Gustaf Neumann <ne...@wu...> wrote:

> Dear Maksym,
>
> These kind of errors should never happen, but when this happen, this can
> come from NaviServer or Tcl or some C modules/package loaded.
>
> The last time i saw this kind of crash in a NaviServer environment, it
> was triggered from Tcl, where a select() was tried in a situation where
> more than 1024 file descriptors were open (stay away from async Tcl I/O
> operations and use NaviServer built-in features ... e.g. ns_http instead
> of tcllib http). The number of open file descriptors of NaviServer can
> be monitored e.g. by the munin-plugins [2,3].
>
> To debug this situation,
> - make sure to compile Tcl and NaviServer with debugging symbols (i.e.
> with compiler flag "-g" [1]),
> - make sure, core dumps are enabled, and when you have a core,
> - use "gdb /usr/local/ns/bin/nsd YOURCORE" to see exactly, where this
> happened.
>
> all the best
>
> -g
>
> [1] https://openacs.org/forums/message-view?message_id=5537675
> [2] https://github.com/gustafn/munin-plugins-ns
> [3]
>
> https://openacs.org/munin/localdomain/localhost.localdomain/naviserver_openacs_lsof.html
>
> On 04.11.21 21:17, Maksym Zinchenko wrote:
> > Hi every now and then my server crashes with the last message in the
> > log file *** Buffer overflow detected *** I would really appreciate if
> > someone can point me in the right direction of debugging and figure
> > out what's going on.
> > Thank you
> > Maksym Zinchenko
>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>

Re: [naviserver-devel] Buffer overflow detected

[Gustaf N. <ne...@wu...>]

Dear Maksym,

These kind of errors should never happen, but when this happen, this can 
come from NaviServer or Tcl or some C modules/package loaded.

The last time i saw this kind of crash in a NaviServer environment, it 
was triggered from Tcl, where a select() was tried in a situation where 
more than 1024 file descriptors were open (stay away from async Tcl I/O 
operations and use NaviServer built-in features ... e.g. ns_http instead 
of tcllib http). The number of open file descriptors of NaviServer can 
be monitored e.g. by the munin-plugins [2,3].

To debug this situation,
- make sure to compile Tcl and NaviServer with debugging symbols (i.e. 
with compiler flag "-g" [1]),
- make sure, core dumps are enabled, and when you have a core,
- use "gdb /usr/local/ns/bin/nsd YOURCORE" to see exactly, where this 
happened.

all the best

-g

[1] https://openacs.org/forums/message-view?message_id=5537675
[2] https://github.com/gustafn/munin-plugins-ns
[3] 
https://openacs.org/munin/localdomain/localhost.localdomain/naviserver_openacs_lsof.html

On 04.11.21 21:17, Maksym Zinchenko wrote:
> Hi every now and then my server crashes with the last message in the 
> log file *** Buffer overflow detected *** I would really appreciate if 
> someone can point me in the right direction of debugging and figure 
> out what's going on.
> Thank you
> Maksym Zinchenko