naviserver-devel — Developer discussion

Re: [naviserver-devel] No notifications with webpush::send

[Gustaf N. <ne...@wu...>]

On 09.12.21 09:33, Wolfgang Winkler via naviserver-devel wrote:
>
> We are using 1.1.1d on our production server, which is a debian buster.
>
> bytes {} tag 1e58277931d45f4c593cffbf291b39b7

i can confirm, that with Debian GNU/Linux 10 (buster) and OpenSSL 1.1.1d 
bytes are empty.

With e.g. Rocky Linux release 8.4 (one successor of CentOS, also 
conservative), with e.g. 1.1.1g, everything is fine.


> I've tried to use 1.1.1k on buster. I installed it with
>
> ./config --prefix=/usr/local/openssl && make && make install
>
> and compiled naviserver with
>
> ./configure 
> --enable-64bit=true--prefix=/usr/local/naviserver-git--with-openssl=/usr/local/openssl--with-tcl=/usr/local/lib/--enable-threads
>
> But naviserver still uses the packaged openssl version:
> # ldd nsd/nsd
> libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1


There is something starnge  on Buster concerning libraries. I have
downloaded newest openssl from git, configured + make install, and 
configured
Naviserver as usual

$ ./configure --enable-64bit -prefix=/usr/local/ns --with-openssl=/usr/local/

but was surprised that it the version was not picked up for loading.
After brutally linking the files, everything was fine.

So, there seems to be some load-path that has to be configured for Buster,
but I am not an expert (and have not time to investigate deeper).

But with this, the right OpenSSL is loaded, encrypt returns non-empty:

$ ln -s /usr/local/lib64/*so* /usr/local/lib/
$ ldconfig -v
$ make install
$ ./nsd/nsd -c -u nsadmin
[-main:conf-] Notice: OpenSSL 3.1.0-dev  initialized
...
% package require tcltest 2.2
% namespace import -force ::tcltest::*
% test aead-1.0 {aead::encrypt} -body {
     set d [ns_crypto::aead::encrypt string -cipher aes-128-gcm -iv 123456789 -key secret "hello world"]
     list bytes [string length [dict get $d bytes]] tag [string length [dict get $d tag]]
} -result {bytes 22 tag 32}


I have to rush,

-gn

Re: [naviserver-devel] No notifications with webpush::send

[Wolfgang W. <wol...@di...>]

We are using 1.1.1d on our production server, which is a debian buster.

Here we get the result

bytes {} tag 1e58277931d45f4c593cffbf291b39b7

On our newer system with openssl 1.1.1k

bytes dbf5a0d98c8b32e1b6dccc tag 81294002f4a5df64c8a44e1bcc07a6cb

The call was:

ns_crypto::aead::encrypt string -cipher aes-128-gcm -iv 123456789 -key 
secret "hello world"


I've tried to use 1.1.1k on buster. I installed it with

./config --prefix=/usr/local/openssl && make && make install

and compiled naviserver with

./configure 
--enable-64bit=true--prefix=/usr/local/naviserver-git--with-openssl=/usr/local/openssl--with-tcl=/usr/local/lib/--enable-threads

But naviserver still uses the packaged openssl version:
# ldd nsd/nsd
libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1

Regards,

Wolfgang


Am 07.12.21 um 17:16 schrieb Gustaf Neumann:
>
>
> On 07.12.21 14:40, Wolfgang Winkler via naviserver-devel wrote:
>> We also tried the demo here:
>>
>> https://openacs.org/webpush-demo/webpush-demo.tcl
>>
>> The empty message works ("Naviserver rocks!"), but with a message, we 
>> get an error:
>>
>> "could not derive EC point from provided key could not derive EC 
>> point from provided key "
>>
> you are right, something seems broken at least at openacs.org. The 
> regression test of nswebpush fails at openacs.org  (but succeeds on my 
> local machine). ... so maybe there is a version dependency.
>
> What version of OpenSSL are you using?
>
> -gn
>
> PS: it will take probably until the weekend to get some time in a 
> block to looking this.
>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
-- 

*Wolfgang Winkler*
Geschäftsführung
wol...@di...
mobil +43.699.19971172

dc:*büro*
digital concepts Novak Winkler OG
Software & Design
Landstraße 68, 5. Stock, 4020 Linz
www.digital-concepts.com <http://www.digital-concepts.com>
tel +43.732.997117.72
tel +43.699.1997117.2

Firmenbuchnummer: 192003h
Firmenbuchgericht: Landesgericht Linz

Re: [naviserver-devel] No notifications with webpush::send

[Gustaf N. <ne...@wu...>]

On 07.12.21 14:40, Wolfgang Winkler via naviserver-devel wrote:
> We also tried the demo here:
>
> https://openacs.org/webpush-demo/webpush-demo.tcl
>
> The empty message works ("Naviserver rocks!"), but with a message, we 
> get an error:
>
> "could not derive EC point from provided key could not derive EC point 
> from provided key "
>
you are right, something seems broken at least at openacs.org. The 
regression test of nswebpush fails at openacs.org  (but succeeds on my 
local machine). ... so maybe there is a version dependency.

What version of OpenSSL are you using?

-gn

PS: it will take probably until the weekend to get some time in a block 
to looking this.

[naviserver-devel] No notifications with webpush::send

[Wolfgang W. <wol...@di...>]

Hi list!

We are trying to send webpush messages with the naviserver webpush moule 
(webpush::send). It used to work fine a few months ago.

Now we have the following problem: When sending an empty push message with:

setmessage{}
setret [webpush::send \
            -subscription $subscription_list\
            -data $message\
            -claim [listsub mailto:${claim_email}] \
            -privateKeyPem [dictget $channelprivate_key_path] \
            -mode aesgcm \
            -timeout 3\
            -ttl $ttl\
        ]

we get a proper desktop notification with "No message!" body.

When we try to enter a message, the call returns with a status of 201, 
but no notification is shown.

We also tried the demo here:

https://openacs.org/webpush-demo/webpush-demo.tcl

The empty message works ("Naviserver rocks!"), but with a message, we 
get an error:

"could not derive EC point from provided key could not derive EC point 
from provided key "

After a bit of code exploration, we noticed, that the call to 
"::ns_crypto::aead::encrypt"
in webpush::encrypt returns an empty result for "bytes".

So we've tried the examples from

https://naviserver.sourceforge.io/n/naviserver/files/ns_crypto.html#1

which get the same result.

ns_crypto::aead::encrypt string -cipher aes-128-gcm -iv 123456789 -key 
secret "hello world"

returns

bytes {} tag ecb7f31df9e12b6118c1d8fcd5b7534a

instead of

bytes fa260f97eae35e3e3df0b7 tag 93654f78fd189b559c091acb410a0040

Any insights would be highly appreciated.

Regards,

Wolfgang
-- 

*Wolfgang Winkler*
Geschäftsführung
wol...@di...
mobil +43.699.19971172

dc:*büro*
digital concepts Novak Winkler OG
Software & Design
Landstraße 68, 5. Stock, 4020 Linz
www.digital-concepts.com <http://www.digital-concepts.com>
tel +43.732.997117.72
tel +43.699.1997117.2

Firmenbuchnummer: 192003h
Firmenbuchgericht: Landesgericht Linz

[naviserver-devel] Towards NaviServer 4.99.23

[Gustaf N. <ne...@wu...>]

Dear all,

on sourceforge is a release candidate for NaviServer 4.99.23 [1].

The change contains one potentially important fix for a memory
leak (which was in the code since 1y+, but turned out to be significant
with a recent OpenACS change) and a couple of new features (including
the improved Unicode support for e.g. emojis, and crypto improvments
like SCRAM, which is already incorporated in OpenACS).

Please test if possible.  The release should be in the near future.

Below is a preliminary summary of changes.

all the best
-gustaf neumann

[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.23/

=======================================
NaviServer 4.99.23, released 2021-12-XX
=======================================

  60 files changed, 1014 insertions(+), 270 deletions(-)

New Features:

  - Improved hash algorithms for improved security

    The new version supports SCRAM (Salted Challenge Response
    Authentication Mechanism), which is one of the newer recommended
    password hash algorithm to replace the classic salted SHA1
    approaches. The classical hash algorithms become easier to attack
    via high performance hashing hardware (GPUs).  NaviServer supports
    now SCRYPT and SCRAM (when compiled with more recent versions of
    OpenSSL; SCRYPT requires at least OpenSSL 3.0. SCRAM requires OpenSSL 1.1)

    The actual hash function of SCRAM is PBKDF2 [RFC2898] with HMAC as
    the pseudorandom function (PRF) and with dkLen == output length of
    HMAC == output length of the digest function.
     
    Here is an example of using pbkdf2_hmac for the hash function of
    SCRAM-sha-256. RFC 7677 recommends to use 15K iterations for PBKDF2:
     
       ::ns_crypto::pbkdf2_hmac \
                -digest sha256 \
                -iterations 15000 \
                -secret $password \
                -salt $salt]
                
    OpenACS supports already switching to from salted SHA1 to SCAM (or
    SCRYPT) via configuration variable.


  - Better Unicode support, including emojis requiring 4-byte UTF-8 characters.

    Earlier versions of NaviServer and the nsdb* database drivers
    assumed on a few places that Tcl-internal UTF-8 is also valid UTF-8
    for external sources, which is often, but not always true. Now, the
    proper export functions are everywhere called.

    The new code was tested with Emojis up to Unicode 13 (many thanks
    to Wolfgang Winkler)

    This change effects as well the database driver module "nsdbpg".

  - ns_trim enhancements:
    The new option "-prefix ..." can be used to strip a string
    (such as ">> ") from every line starting with it.

  - extended time unit support (added "w" for weeks and "y" for years)

  - Added an experimental global parameter "nocache" to ease to
    experiment with horizontal scaling. As a consequence, "ns_cache
    eval" becomes a dummy operation.

  - Added an experimental command "ns_baseunit"
     
       ns_baseunit ?-size size? ?-time time?
     
    Convert from memory units or from time units to its base value
    using the NaviServer internal converters, which are used the same
    way for various commands.  The base unit for a memory size is a
    byte, for a time value a second). This command is necessary to
    provide Tcl-level commands calculating with these units to support
    uniform interfaces (e.g., calculating cache partition sizes base on
    values such as 2MB).
     
    Either "-size" or "-time" has to be specified.
     
       % ns_baseunit -size 10KB
       10240
     
       ns_baseunit -time 2.5h
       9000
     
  - Added an experimental command "ns_strcoll"
  
        ns_strcoll ?-locale locale? string1 string2"
     
    This command compares lexicographically string1 with string2
    according to the current locale collation and returns an integer
    greater than, equal to, or less than 0, depending on whether
    string1 is greater than, equal to, or less than string2. The
    command is necessary in cases, where e.g., the sorting order from
    the database (normally based on local collation) is different from
    default Tcl sorting order to provide a uniform interface with same
    sorting orders. The name is derived from the baseline POSIX
    function call.
     
    The command is suitable for usage in the lsort command:
     
       % set l {Bor Bar Bär}
       % lsort -command ns_strcoll $l
       Bar Bär Bor
     
       % lsort $l
       Bar Bor Bär


Performance Improvements:

  - Increase scalability on DB operations by reducing DB pool locks
  
    On high load servers, the total number of locks and busy locks on
    the DB pools might become quite high. The new code caches these
    statistics in the handles (which are per thread, requiring no
    locks) and transfers the aggregated values on handle closes or
    statistics calls.

  - Set default for "concurrentinterpcreate" to "true" for Tcl 8.6 or
    newer. Versions up to at least Tcl 8.5 are known to crash in case
    two threads create interpreters at the same time. These crashes
    were hard to reproduce, but serializing interpreter creation
    helped. Since all our major servers are running since several years
    without problems with this parameter turned on the default is now
    set to "true", when NaviServer is compiled with Tcl 8.6 or newer.


Bug Fixes:
  - Fixed memory leak in "nsv_dict get" operations.

  
Documentation improvements:
---------------------------

  - Improved the following man pages:

	doc/src/manual/admin-tuning.man
	doc/src/naviserver/commandlist.man
	doc/src/naviserver/ns_adp_include.man
	doc/src/naviserver/ns_baseunit.man
	doc/src/naviserver/ns_cache.man
	doc/src/naviserver/ns_charsets.man
	doc/src/naviserver/ns_choosecharset.man
	doc/src/naviserver/ns_connchan.man
	doc/src/naviserver/ns_cookie.man
	doc/src/naviserver/ns_cookiecharset.man
	doc/src/naviserver/ns_crypto.man
	doc/src/naviserver/ns_encodingforcharset.man
	doc/src/naviserver/ns_encodingfortype.man
	doc/src/naviserver/ns_formfieldcharset.man
	doc/src/naviserver/ns_ictl.man
	doc/src/naviserver/ns_job.man
	doc/src/naviserver/ns_register.man
	doc/src/naviserver/ns_schedule.man
	doc/src/naviserver/ns_setformencoding.man
	doc/src/naviserver/ns_shutdown.man
	doc/src/naviserver/ns_sleep.man
	doc/src/naviserver/ns_sockcallback.man
	doc/src/naviserver/ns_sockopen.man
	doc/src/naviserver/ns_strcoll.man
	doc/src/naviserver/ns_time.man
	doc/src/naviserver/ns_urlcharset.man
	doc/src/naviserver/ns_valid_utf8.man
	doc/src/naviserver/textutil-cmds.man
  

Configuration Changes:
----------------------

  -  Ease configuration via environment variables

     This feature is useful to manage many NaviServer instances with
     mostly identical configurations without having to provide multiple
     configuration files (e.g. for Docker setups, or clusters).
     
     The sample configuration for of OpenACS (openacs-config.tcl)
     contains now a Tcl dictionary with default values:

       set defaultConfig {
          hostname   localhost
          ipaddress  127.0.0.1
          httpport   8000
          httpsport  ""
          server     "openacs"
          serverroot /var/www/$server
          logroot    $serverroot/log/
          homedir    /usr/local/ns
          bindir     $homedir/bin
          db_name    $server
          db_user    $server
          db_host    localhost
          db_port    ""
      }

     These configuration values (keys of the dict) can be overridden by
     environment variables prefixed with "oacs_" followed by the
     parameter name. One can change the default port specified in the
     configuration file for plain HTTP connections by e.g., providing it
     via environment variables:

         oacs_httpport=8101 /usr/local/ns/bin/nsd -i -t ....
         

Code Changes:

  - Extended regression test
     
  - Code Cleanup
      . Do not declare reserved C identifiers

  - Improved comments, fixed typos

Re: [naviserver-devel] Issue with certain emojis (unicode/utf-8)

[Gustaf N. <ne...@wu...>]

Dear all,

It will take some time, until the Emojis from Unicode 14 will be 
generally available, but when this comes, we should have already 
everything working in NaviServer and the DB interfaces. I've added a 
small demo page, one can try when the new clients come out:

https://openacs.org/emojis.tcl

One interesting part is the grapheme cluster (like e.g. 👩‍👩‍👧‍👦), 
which is made up of the following unicode graphemes:

    WOMAN👩 ZWJ WOMAN👩 ZWJ GIRL👧 ZWJ BOY👦

where ZWJ are zero-width joiners. One can enter these e.g. via

   set x \ud83d\udc69\u200d\ud83d\udc69\u200d\ud83d\udc67\u200d\ud83d\udc66

into current Tcl. When just passing such string through Tcl, everything 
seems fine. I would not be surprised that  "string length", "string 
range" etc can lead to unexpected results, but is is quite fun to 
decompose this emoji with Tcl:

    % set x \ud83d\udc69\u200d\ud83d\udc69\u200d\ud83d\udc67\u200d\ud83d\udc66
    👩‍👩‍👧‍👦
    % string range $x 0 1
    👩
    % string range $x 6 7
    👧
    % string range $x 9 10
    👦

AFIKT, eveything is fine with NaviServer in this respect.

Concerning Unicode 14:

Android 12L contains support for Emojis from Unicode 14. Google 
announced Android 12L in October 2021, less than one month after the 
stable release of Android 12. 12L is expected in early 2022 [2].

According to [3] iOS 15.0 will not include Unicode 14 emojis. Support 
for Emoji 14.0 on Apple platforms is expected in the first half of 2022 
(probably in iOS 16).

all the best

-g

[1] https://9to5google.com/2021/10/27/android-12l-unicode-14/
[2] https://developer.android.com/about/versions/12/12L/summary
[3] https://emojipedia.org/apple/


On 26.11.21 10:40, Wolfgang Winkler via naviserver-devel wrote:
>
> Hi!
>
> We've testet the encoding now extensively. All Emojis up to 13 
> <https://emojipedia.org/unicode-13.0/> are handled correctly, 
> including database storage and retrieving, tdom and form handling.
>
> Version 14 emojis <https://emojipedia.org/unicode-14.0/> are not 
> supported by any of the browsers we've testet, but don't throw errors. 
> It seems we are save for future updates.
>
> Wolfgang
>
> Am 18.11.21 um 18:24 schrieb Gustaf Neumann:
>>
>> Dear all
>>
>> On bitbucket is now an update (see change log message below) that 
>> introduces support of UTF-8 characters using up to 4 bytes (with Tcl 
>> 8.6). It should work as well with 6 byte UTF when Tcl 8.7 is properly 
>> compiled (by setting TCL_UTF_MAX).
>>
>> One can now use e.g. emoticons in SQL queries
>>
>>      db_0or1row ... {select 1 from cr_items where name = '😈'}
>>
>> or as values of bind variables
>>
>>      set x 😈
>>      db_0or1row ... {select 1 from cr_items where name = :x}
>>
>> ... but not as names of bind variables (these have the same 
>> restricted syntax than before
>> (in essence no funny characters).
>>
>> The code is already running at openacs.org.
>>
>> all the best
>>
>> -gn
>>
>>
>> Added support for UTF-8 characters up to 4 bytes
>>
>> These changes add proper export of UTF-8 for Unicode symbols taking up
>> to 4 bytes. For the western world the biggest interest is probably for
>> emoticons. The change is implemented with performance in mind. The
>> proper encoded byte-strings are cached in Tcl_Objs, such that only the
>> values for bind-vars (which have probably different values per call)
>> have to be recoded at call time. This should keep the performance
>> penalty small (we see on some of our servers in day-average 1500 SQL
>> operations per second, peaks at >10K).
>>
>> The names of bind variables follow still the same rules as before (no
>> emoticons as variable names).
>>
>> On 16.11.21 16:39, Wolfgang Winkler via naviserver-devel wrote:
>>
>>>>> the fix worked, thank you Gustaf! But we still have a problem with 
>>>>> emojis when writing them to the database. The error we get is:
>>>>>
>>>>> Database operation "dml" failed (exception ERROR, "ERROR:  invalid 
>>>>> byte sequence for encoding "UTF8": 0xf0 0x9f
>>>>>
>>
>>
>>
>> _______________________________________________
>> naviserver-devel mailing list
>> nav...@li...
>> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> -- 
>
> *Wolfgang Winkler*
> Geschäftsführung
> wol...@di...
> mobil +43.699.19971172
>
> dc:*büro*
> digital concepts Novak Winkler OG
> Software & Design
> Landstraße 68, 5. Stock, 4020 Linz
> www.digital-concepts.com <http://www.digital-concepts.com>
> tel +43.732.997117.72
> tel +43.699.1997117.2
>
> Firmenbuchnummer: 192003h
> Firmenbuchgericht: Landesgericht Linz
>
>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

-- 
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"

Re: [naviserver-devel] Issue with certain emojis (unicode/utf-8)

[Wolfgang W. <wol...@di...>]

Hi!

We've testet the encoding now extensively. All Emojis up to 13 
<https://emojipedia.org/unicode-13.0/> are handled correctly, including 
database storage and retrieving, tdom and form handling.

Version 14 emojis <https://emojipedia.org/unicode-14.0/> are not 
supported by any of the browsers we've testet, but don't throw errors. 
It seems we are save for future updates.

Wolfgang

Am 18.11.21 um 18:24 schrieb Gustaf Neumann:
>
> Dear all
>
> On bitbucket is now an update (see change log message below) that 
> introduces support of UTF-8 characters using up to 4 bytes (with Tcl 
> 8.6). It should work as well with 6 byte UTF when Tcl 8.7 is properly 
> compiled (by setting TCL_UTF_MAX).
>
> One can now use e.g. emoticons in SQL queries
>
>      db_0or1row ... {select 1 from cr_items where name = '😈'}
>
> or as values of bind variables
>
>      set x 😈
>      db_0or1row ... {select 1 from cr_items where name = :x}
>
> ... but not as names of bind variables (these have the same restricted 
> syntax than before
> (in essence no funny characters).
>
> The code is already running at openacs.org.
>
> all the best
>
> -gn
>
>
> Added support for UTF-8 characters up to 4 bytes
>
> These changes add proper export of UTF-8 for Unicode symbols taking up
> to 4 bytes. For the western world the biggest interest is probably for
> emoticons. The change is implemented with performance in mind. The
> proper encoded byte-strings are cached in Tcl_Objs, such that only the
> values for bind-vars (which have probably different values per call)
> have to be recoded at call time. This should keep the performance
> penalty small (we see on some of our servers in day-average 1500 SQL
> operations per second, peaks at >10K).
>
> The names of bind variables follow still the same rules as before (no
> emoticons as variable names).
>
> On 16.11.21 16:39, Wolfgang Winkler via naviserver-devel wrote:
>
>>>> the fix worked, thank you Gustaf! But we still have a problem with 
>>>> emojis when writing them to the database. The error we get is:
>>>>
>>>> Database operation "dml" failed (exception ERROR, "ERROR:  invalid 
>>>> byte sequence for encoding "UTF8": 0xf0 0x9f
>>>>
>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
-- 

*Wolfgang Winkler*
Geschäftsführung
wol...@di...
mobil +43.699.19971172

dc:*büro*
digital concepts Novak Winkler OG
Software & Design
Landstraße 68, 5. Stock, 4020 Linz
www.digital-concepts.com <http://www.digital-concepts.com>
tel +43.732.997117.72
tel +43.699.1997117.2

Firmenbuchnummer: 192003h
Firmenbuchgericht: Landesgericht Linz

Re: [naviserver-devel] Issue with certain emojis (unicode/utf-8)

[Gustaf N. <ne...@wu...>]

Dear all

On bitbucket is now an update (see change log message below) that 
introduces support of UTF-8 characters using up to 4 bytes (with Tcl 
8.6). It should work as well with 6 byte UTF when Tcl 8.7 is properly 
compiled (by setting TCL_UTF_MAX).

One can now use e.g. emoticons in SQL queries

     db_0or1row ... {select 1 from cr_items where name = '😈'}

or as values of bind variables

     set x 😈
     db_0or1row ... {select 1 from cr_items where name = :x}

... but not as names of bind variables (these have the same restricted 
syntax than before
(in essence no funny characters).

The code is already running at openacs.org.

all the best

-gn


Added support for UTF-8 characters up to 4 bytes

These changes add proper export of UTF-8 for Unicode symbols taking up
to 4 bytes. For the western world the biggest interest is probably for
emoticons. The change is implemented with performance in mind. The
proper encoded byte-strings are cached in Tcl_Objs, such that only the
values for bind-vars (which have probably different values per call)
have to be recoded at call time. This should keep the performance
penalty small (we see on some of our servers in day-average 1500 SQL
operations per second, peaks at >10K).

The names of bind variables follow still the same rules as before (no
emoticons as variable names).

On 16.11.21 16:39, Wolfgang Winkler via naviserver-devel wrote:

>>> the fix worked, thank you Gustaf! But we still have a problem with 
>>> emojis when writing them to the database. The error we get is:
>>>
>>> Database operation "dml" failed (exception ERROR, "ERROR:  invalid 
>>> byte sequence for encoding "UTF8": 0xf0 0x9f
>>>

Re: [naviserver-devel] Issue with certain emojis (unicode/utf-8)

[Wolfgang W. <wol...@di...>]

We've a workaround which is probably much worse than yours, but working. 
We are using nsdbpg, I've not tried the query with the nsdbi interface.

Am 16.11.21 um 17:06 schrieb Gustaf Neumann:
>
> Funny enough, i same a very similar problem today and provide a local 
> fix for this. I am not happy with this fix since it is rather costly, 
> so i would like to work on this more before committing. However, today 
> and tomorrow i am fully booked with urgent items, so don't expect a 
> fix for this before the weekend.
>
> -g
>
> PS: i assume, you are using the nsdbpg driver.
>
> On 16.11.21 16:39, Wolfgang Winkler via naviserver-devel wrote:
>> the fix worked, thank you Gustaf! But we still have a problem with 
>> emojis when writing them to the database. The error we get is:
>>
>> Database operation "dml" failed (exception ERROR, "ERROR:  invalid 
>> byte sequence for encoding "UTF8": 0xf0 0x9f 0x98 0xff
>>
>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
-- 

*Wolfgang Winkler*
Geschäftsführung
wol...@di...
mobil +43.699.19971172

dc:*büro*
digital concepts Novak Winkler OG
Software & Design
Landstraße 68, 5. Stock, 4020 Linz
www.digital-concepts.com <http://www.digital-concepts.com>
tel +43.732.997117.72
tel +43.699.1997117.2

Firmenbuchnummer: 192003h
Firmenbuchgericht: Landesgericht Linz

Re: [naviserver-devel] Issue with certain emojis (unicode/utf-8)

[Gustaf N. <ne...@wu...>]

Funny enough, i same a very similar problem today and provide a local 
fix for this. I am not happy with this fix since it is rather costly, so 
i would like to work on this more before committing. However, today and 
tomorrow i am fully booked with urgent items, so don't expect a fix for 
this before the weekend.

-g

PS: i assume, you are using the nsdbpg driver.

On 16.11.21 16:39, Wolfgang Winkler via naviserver-devel wrote:
> the fix worked, thank you Gustaf! But we still have a problem with 
> emojis when writing them to the database. The error we get is:
>
> Database operation "dml" failed (exception ERROR, "ERROR:  invalid 
> byte sequence for encoding "UTF8": 0xf0 0x9f 0x98 0xff
>

Re: [naviserver-devel] Issue with certain emojis (unicode/utf-8)

[Wolfgang W. <wol...@di...>]

Hello all,

the fix worked, thank you Gustaf! But we still have a problem with 
emojis when writing them to the database. The error we get is:

Database operation "dml" failed (exception ERROR, "ERROR:  invalid byte 
sequence for encoding "UTF8": 0xf0 0x9f 0x98 0xff

when trying to write the emoji to a TEXT or VARCHAR field in the 
database. Inserting the same string in the database console works as 
expected. When we read the string and reinsert it, it also works 
flawlessly.

We've compared the two strings, wrote them to files and compared them 
with a hex reader, converted them with tcl "encoding convertto" and 
iconv, all with no luck.

We are using postgres 12 and the nsdbpg module with 
naviserver-4.99.22-16-g67adf3c34710+

Here is the test case:

In the database console:

CREATE TABLE test (
     idx SERIAL,
     txt TEXT
);

INSERT INTO test (txt) VALUES ('<smiley>😃</smiley>');

In the naviserver console or in a script:

# V1: working
set db [ns_db gethandle]
set sql "SELECT txt FROM test WHERE idx=1"
set selection [ns_db 1row $db $sql]
set str [ns_set value $selection 0]
set sql "INSERT INTO test (txt) VALUES ('$str')"
ns_db dml $db $sql
ns_db releasehandle $db

# V2: not working
set db [ns_db gethandle]
set sql "INSERT INTO test (txt) VALUES ('<smiley>😃</smiley>')"
ns_db dml $db $sql
ns_db releasehandle $db

With nscp, pasting the string of V2 already shows a wrong string in the log:

Notice: nscp:  3: set sql "INSERT INTO test (txt) VALUES 
('<smiley>�������</smiley>')"


Whereas V1 works (the smiley is not printed here, but works in the console):

Notice: nscp:  5: puts $str
<smiley></smiley>

Any help is greatly appreciated!

Wolfgang Winkler

Am 09.11.21 um 09:36 schrieb Gustaf Neumann:
> Dear all,
>
> The situation is trickier than someone might hope.  Aside of the Tcl 
> version dependencies (as Brian pointed out), Tcl before 8.7 do not 
> support TCL_UTF_MAX with longer multi-byte sequences than 4 (see Tcl 
> TIP 389), which are also mostly relevant for some newer emojis. So, 
> for full emoji support, Tcl 8.7 with the proper compilation options is 
> needed.
>
> Anyhow, in the case of Wolfgang's the "Smiling Face with Open Mouth" 
> we have just a 4-byte UTF-8 character, which is supported by 
> out-of-the-box Tcl 8.6. However, this emoji is represented 
> Tcl-internally as a 6-byte sequence. Since NaviServer wrongly assumed 
> that Tcl-internal representations are also accepted as external 
> representations, a conversion step was omitted for utf-8 (which is not 
> always true).
>
> In the tip version of NaviServer on Bitbucket, this optimization is 
> now removed, the examples work as expected, the regression test is 
> extended for this case.
>
> Many thanks to Wolfgang for the good bug report.
>
> -g
>
>
>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
-- 

*Wolfgang Winkler*
Geschäftsführung
wol...@di...
mobil +43.699.19971172

dc:*büro*
digital concepts Novak Winkler OG
Software & Design
Landstraße 68, 5. Stock, 4020 Linz
www.digital-concepts.com <http://www.digital-concepts.com>
tel +43.732.997117.72
tel +43.699.1997117.2

Firmenbuchnummer: 192003h
Firmenbuchgericht: Landesgericht Linz

Re: [naviserver-devel] Issue with certain emojis (unicode/utf-8)

[Gustaf N. <ne...@wu...>]

Dear all,

The situation is trickier than someone might hope.  Aside of the Tcl 
version dependencies (as Brian pointed out), Tcl before 8.7 do not 
support TCL_UTF_MAX with longer multi-byte sequences than 4 (see Tcl TIP 
389), which are also mostly relevant for some newer emojis. So, for full 
emoji support, Tcl 8.7 with the proper compilation options is needed.

Anyhow, in the case of Wolfgang's the "Smiling Face with Open Mouth" we 
have just a 4-byte UTF-8 character, which is supported by out-of-the-box 
Tcl 8.6. However, this emoji is represented Tcl-internally as a 6-byte 
sequence. Since NaviServer wrongly assumed that Tcl-internal 
representations are also accepted as external representations, a 
conversion step was omitted for utf-8 (which is not always true).

In the tip version of NaviServer on Bitbucket, this optimization is now 
removed, the examples work as expected, the regression test is extended 
for this case.

Many thanks to Wolfgang for the good bug report.

-g

Re: [naviserver-devel] Issue with certain emojis (unicode/utf-8)

[Brian F. <Bri...@qu...>]

Hi Wolfgang

I wonder if it's a TCL issue? For example, I see this discussion has some mentions of problems with emojis in some TCL versions https://core.tcl-lang.org/tips/doc/trunk/tip/600.md

Brian

________________________________
From: Wolfgang Winkler via naviserver-devel <nav...@li...>
Sent: Monday 8 November 2021 11:09
To: nav...@li... <nav...@li...>
Cc: Wolfgang Winkler <wol...@di...>
Subject: [naviserver-devel] Issue with certain emojis (unicode/utf-8)


This message's attachments contains at least one web link. This is often used for phishing attempts. Please only interact with this attachment if you know its source and that the content is safe. If in doubt, confirm the legitimacy with the sender by phone.


Dear all!

We face some issues with certain UTF-8 characters, e.g. this one: https://emojipedia.org/grinning-face-with-big-eyes/.

When we set "charset" to "utf-8", everything works as expected, except the output of various emojis.

set str "Test smiley: 😃<br>öüäÖÄÜß<br>"
ns_return 200 "text/html; charset=unicode" $str; return

returns the smiley correctly, whereas

set str "Test smiley: 😃<br>öüäÖÄÜß<br>"
ns_return 200 "text/html; charset=utf-8" $str; return

returns

Test smiley: ������
öüäÖÄÜß

So I tried to set the charset to "unicode". This works for some files and not for others, especially not for javascript files.

This are the parameters in the config section:

ns_section "ns/parameters"
 ....
 #ns_param   HackContentType true
 ns_param   HackContentType false
 ns_param   OutputCharset   $charset
 ns_param   URLCharset      $charset


We also tried with nscp and the tclsh:

nscp Input:

puts "😃"

Log Output:

Notice: nscp:  1: puts "�������"

The nscp telnet client does not return to the prompt.

Tclsh works as expected:

tclsh
% puts "😃"
😃


Tcl version is 8.6.11, naviserver 4.99.22 running on Debian 10.11.

Has anybody encountered and solved a similiar issue?

Thanks,

Wolfgang Winkler

--

Wolfgang Winkler
Geschäftsführung
wol...@di...<mailto:wol...@di...>
mobil +43.699.19971172

dc:büro
digital concepts Novak Winkler OG
Software & Design
Landstraße 68, 5. Stock, 4020 Linz
www.digital-concepts.com<http://www.digital-concepts.com>
tel +43.732.997117.72
tel +43.699.1997117.2

Firmenbuchnummer: 192003h
Firmenbuchgericht: Landesgericht Linz

[https://www.digital-concepts.com/cu/digitalconcepts2016/images/logo_digitalconcepts2016.png]

[naviserver-devel] Issue with certain emojis (unicode/utf-8)

[Wolfgang W. <wol...@di...>]

Dear all!

We face some issues with certain UTF-8 characters, e.g. this one: 
https://emojipedia.org/grinning-face-with-big-eyes/.

When we set "charset" to "utf-8", everything works as expected, except 
the output of various emojis.

setstr "Test smiley: 😃<br>öüäÖÄÜß<br>"
ns_return 200"text/html; charset=unicode"$str; return

returns the smiley correctly, whereas

setstr "Test smiley: 😃<br>öüäÖÄÜß<br>"
ns_return 200"text/html; charset=utf-8"$str; return

returns

Test smiley: ������
öüäÖÄÜß

So I tried to set the charset to "unicode". This works for some files 
and not for others, especially not for javascript files.

This are the parameters in the config section:

ns_section "ns/parameters"
  ....
  #ns_param   HackContentType true
  ns_param   HackContentType false
  ns_param   OutputCharset $charset
  ns_param   URLCharset $charset

We also tried with nscp and the tclsh:

nscp Input:

puts "😃"

Log Output:

Notice: nscp:  1: puts "�������"

The nscp telnet client does not return to the prompt.

Tclsh works as expected:

tclsh
% puts "😃"
😃

Tcl version is 8.6.11, naviserver 4.99.22 running on Debian 10.11.

Has anybody encountered and solved a similiar issue?

Thanks,

Wolfgang Winkler

-- 

*Wolfgang Winkler*
Geschäftsführung
wol...@di...
mobil +43.699.19971172

dc:*büro*
digital concepts Novak Winkler OG
Software & Design
Landstraße 68, 5. Stock, 4020 Linz
www.digital-concepts.com <http://www.digital-concepts.com>
tel +43.732.997117.72
tel +43.699.1997117.2

Firmenbuchnummer: 192003h
Firmenbuchgericht: Landesgericht Linz

Re: [naviserver-devel] Buffer overflow detected

[Gustaf N. <ne...@wu...>]

There are many possible reasons for the crash, i just mentioned one 
possibility which i have seen in the past. You have to check with gdb as 
mentioned below to find the reason in your case.

If it is indeed the select() problem, then it might be due to a bug in 
your Tcl code (e.g. missing close) or not (really requiring > 1024 open 
file descriptors).

-g

On 06.11.21 19:43, Maksym Zinchenko wrote:
> Thank you very much for explaining Gustaf, at least I know now that 
> somewhere in my Tcl code something is wrong. I'm using only 2 C 
> modules/packages.
>
> On Fri, Nov 5, 2021 at 7:37 AM Gustaf Neumann <ne...@wu...> wrote:
>
>     Dear Maksym,
>
>     These kind of errors should never happen, but when this happen,
>     this can
>     come from NaviServer or Tcl or some C modules/package loaded.
>
>     The last time i saw this kind of crash in a NaviServer
>     environment, it
>     was triggered from Tcl, where a select() was tried in a situation
>     where
>     more than 1024 file descriptors were open (stay away from async
>     Tcl I/O
>     operations and use NaviServer built-in features ... e.g. ns_http
>     instead
>     of tcllib http). The number of open file descriptors of NaviServer
>     can
>     be monitored e.g. by the munin-plugins [2,3].
>
>     To debug this situation,
>     - make sure to compile Tcl and NaviServer with debugging symbols
>     (i.e.
>     with compiler flag "-g" [1]),
>     - make sure, core dumps are enabled, and when you have a core,
>     - use "gdb /usr/local/ns/bin/nsd YOURCORE" to see exactly, where this
>     happened.
>
>     all the best
>
>     -g
>
>     [1] https://openacs.org/forums/message-view?message_id=5537675
>     [2] https://github.com/gustafn/munin-plugins-ns
>     [3]
>     https://openacs.org/munin/localdomain/localhost.localdomain/naviserver_openacs_lsof.html
>
>     On 04.11.21 21:17, Maksym Zinchenko wrote:
>     > Hi every now and then my server crashes with the last message in
>     the
>     > log file *** Buffer overflow detected *** I would really
>     appreciate if
>     > someone can point me in the right direction of debugging and figure
>     > out what's going on.
>     > Thank you
>     > Maksym Zinchenko
>

Re: [naviserver-devel] Buffer overflow detected

[Maksym Z. <siq...@gm...>]

Thank you very much for explaining Gustaf, at least I know now that
somewhere in my Tcl code something is wrong. I'm using only 2 C
modules/packages.

On Fri, Nov 5, 2021 at 7:37 AM Gustaf Neumann <ne...@wu...> wrote:

> Dear Maksym,
>
> These kind of errors should never happen, but when this happen, this can
> come from NaviServer or Tcl or some C modules/package loaded.
>
> The last time i saw this kind of crash in a NaviServer environment, it
> was triggered from Tcl, where a select() was tried in a situation where
> more than 1024 file descriptors were open (stay away from async Tcl I/O
> operations and use NaviServer built-in features ... e.g. ns_http instead
> of tcllib http). The number of open file descriptors of NaviServer can
> be monitored e.g. by the munin-plugins [2,3].
>
> To debug this situation,
> - make sure to compile Tcl and NaviServer with debugging symbols (i.e.
> with compiler flag "-g" [1]),
> - make sure, core dumps are enabled, and when you have a core,
> - use "gdb /usr/local/ns/bin/nsd YOURCORE" to see exactly, where this
> happened.
>
> all the best
>
> -g
>
> [1] https://openacs.org/forums/message-view?message_id=5537675
> [2] https://github.com/gustafn/munin-plugins-ns
> [3]
>
> https://openacs.org/munin/localdomain/localhost.localdomain/naviserver_openacs_lsof.html
>
> On 04.11.21 21:17, Maksym Zinchenko wrote:
> > Hi every now and then my server crashes with the last message in the
> > log file *** Buffer overflow detected *** I would really appreciate if
> > someone can point me in the right direction of debugging and figure
> > out what's going on.
> > Thank you
> > Maksym Zinchenko
>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>

Re: [naviserver-devel] Buffer overflow detected

[Gustaf N. <ne...@wu...>]

Dear Maksym,

These kind of errors should never happen, but when this happen, this can 
come from NaviServer or Tcl or some C modules/package loaded.

The last time i saw this kind of crash in a NaviServer environment, it 
was triggered from Tcl, where a select() was tried in a situation where 
more than 1024 file descriptors were open (stay away from async Tcl I/O 
operations and use NaviServer built-in features ... e.g. ns_http instead 
of tcllib http). The number of open file descriptors of NaviServer can 
be monitored e.g. by the munin-plugins [2,3].

To debug this situation,
- make sure to compile Tcl and NaviServer with debugging symbols (i.e. 
with compiler flag "-g" [1]),
- make sure, core dumps are enabled, and when you have a core,
- use "gdb /usr/local/ns/bin/nsd YOURCORE" to see exactly, where this 
happened.

all the best

-g

[1] https://openacs.org/forums/message-view?message_id=5537675
[2] https://github.com/gustafn/munin-plugins-ns
[3] 
https://openacs.org/munin/localdomain/localhost.localdomain/naviserver_openacs_lsof.html

On 04.11.21 21:17, Maksym Zinchenko wrote:
> Hi every now and then my server crashes with the last message in the 
> log file *** Buffer overflow detected *** I would really appreciate if 
> someone can point me in the right direction of debugging and figure 
> out what's going on.
> Thank you
> Maksym Zinchenko

[naviserver-devel] Buffer overflow detected

[Maksym Z. <siq...@gm...>]

Hi every now and then my server crashes with the last message in the log
file *** Buffer overflow detected *** I would really appreciate if someone
can point me in the right direction of debugging and figure out what's
going on.
Thank you
Maksym Zinchenko

[naviserver-devel] Snit error

[Maksym Z. <siq...@gm...>]

Hi, I'm getting this kind of error in my log file. What do they mean:

: Error: can't read "selfns": no such variable
:    can't read "selfns": no such variable
:        while executing
:    "list upvar 1 ${selfns}::$varname $varname"
:        (procedure "::snit::RT.variable" line 5)
:        invoked from within
:    "variable Snit_typemethodInfo"
:        (in namespace eval "::snit::window" script line 2)
:        invoked from within
:    "namespace eval ::snit::window {
:     variable Snit_typemethodInfo
:    array set Snit_typemethodInfo {info {0 {::snit::RT.typemethod.info %t}
{}} create {0 {::..."
:        invoked from within
:    "ns_ictl update"
:        (procedure "ns_cleanup" line 8)
:        invoked from within
:    "ns_cleanup"
:        while executing callback
:    ns:tcltrace ns_cleanup
:    (context: trace proc)
:    (context: trace deallocate) line 1

Thank you
Maksym Zinchenko

Re: [naviserver-devel] Capturing the URL query without the key?

[Gustaf N. <ne...@wu...>]

Dear David,

This is just solved in the case, the query parameters consists of just 
one value. The command [ns_conn query] gives you the raw values of all 
parameters as passed to the server.

In order the get the keys, values, ... of all query parameters, get the 
ns_set of the decoded query parameters via [ns_conn form], and from 
that  you can get the list of keys, values, or whatever. See below for 
an example

all the best

-gn


========================================== query-parameters.tcl
ns_return 200 text/plain [subst {
     ns_conn query = '[ns_conn query]'
     ns_conn form  = '[ns_conn form]'
     
     array         = '[ns_set array [ns_conn form]]'
     keys          = '[ns_set keys [ns_conn form]]'
     values        = '[ns_set values [ns_conn form]]'
}]
==========================================



========================================== output from /query-parameters.tcl?x=1&y=2%203&zzzz
     ns_conn query = 'x=1&y=2%203&zzzz'
     ns_conn form  = 't7'
     
     array         = 'x 1 y {2 3} zzzz {}'
     keys          = 'x y zzzz'
     values        = '1 {2 3} {}'
==========================================

On 05.10.21 23:37, D.Fox wrote:
> Sorry for the inconvenience.
>
> [ns_conn query] is what I'm looking for!
>
> All solved. thanks.
>
> ------------------------------------------------------------------------
> *From: *"D.Fox" <un...@cr...>
> *To: *"naviserver-devel" <nav...@li...>
> *Sent: *Tuesday, October 5, 2021 10:31:54 PM
> *Subject: *Capturing the URL query without the key?
>
> Hi, I've scanned through the manual. I am aware of ns_queryget which 
> works fine if your using key/value.
>
> URL/?Someone=Somewhere
> [ns_queryget Someone] would return "Somewhere"
>
> However I am looking to just capture just the key.
>
> Example:
> URL/?00000000000
> Capturing "00000000000"
>
> Is this possible and if so, could someone point me in the right direction?

[naviserver-devel] Capturing the URL query without the key?

[D.Fox <un...@cr...>]

Hi, I've scanned through the manual. I am aware of ns_queryget which works fine if your using key/value. 

URL/? Someone=Somewhere 
[ns_queryget Someone] would return "Somewhere" 

However I am looking to just capture just the key. 

Example: 
URL/?00000000000 
Capturing " 00000000000" 

Is this possible and if so, could someone point me in the right direction? 

Th anks. 
David 

Re: [naviserver-devel] Capturing the URL query without the key?

[D.Fox <un...@cr...>]

Sorry for the inconvenience. 

[ns_conn query] is what I'm looking for! 

All solved. thanks. 


From: "D.Fox" <un...@cr...> 
To: "naviserver-devel" <nav...@li...> 
Sent: Tuesday, October 5, 2021 10:31:54 PM 
Subject: Capturing the URL query without the key? 

Hi, I've scanned through the manual. I am aware of ns_queryget which works fine if your using key/value. 

URL/?Someone=Somewhere 
[ns_queryget Someone] would return "Somewhere" 

However I am looking to just capture just the key. 

Example: 
URL/?00000000000 
Capturing "00000000000" 

Is this possible and if so, could someone point me in the right direction? 

Thanks. 
David 

Re: [naviserver-devel] Fatal: received fatal signal 11

[Gustaf N. <ne...@wu...>]

On 25.09.21 18:10, Maksym Zinchenko wrote:
> Hello everyone, 2 latest commits are not letting me start naviserver.
sorry for the omission. tip version should be fixed now. I'll be back in 
Vienna on monday in case i have overseen something... -g

[naviserver-devel] Fatal: received fatal signal 11

[Maksym Z. <siq...@gm...>]

Hello everyone, 2 latest commits are not letting me start naviserver.
It just crashes without any log with message Fatal: received fatal signal
11.

This one works fine: e11e014

Cheers. Maksym Zinchenko

[naviserver-devel] naviserver and sql-relay

[Bernard D. <bdr...@gm...>]

Hi

Forgive me if this has been considered before or if it is inappropriate.

I've been looking at sql-relay which has a Tcl library and has drivers for
a group of databases which Naviserver does not support (e.g. Firebird, DB2,
Informix, etc).

I am testing sql-relay as a drop-in replacement for ns_db or nsdbi.
http://sqlrelay.sourceforge.net/index.html  And whilst sql-relay doesn't
necessarily add much benefit for the Naviserver/Postgres combination, in
addition to providing other db drivers, the use of sql-relay can provide a
standard connection pooling/filtering/routing system that would mean
Naviserver could be integrated with far more databases. The addition of
sql-relay may thus increase the scalability of database-backed websites.

I realise it's a new complexity and that in itself is probably unwelcome
but I thought I'd just raise it as something that might be of interest.
Sql-relay has been around for about 20 years.

Regards, Bernard