naviserver-devel — Developer discussion

[naviserver-devel] NaviServer 4.99.22 available

[Gustaf N. <ne...@wu...>]

Dear all,

on sourceforge is the release of NaviServer 4.99.22 [1] available

The code was tested with Ubuntu 20.04, Rocky Linux 8.4, OpenBSD 6.9 
(clang),
FreeBSD 13.0-CURRENT, macOS 11.5.2 (Intel and M1).

The following people have contributed to this release:

     Gustaf Neumann
     Ibrahim Tannir
     Oleg Oleinick
     Zoran Vasiljevic

Below is a the summary of changes.

all the best
-gustaf neumann

[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.22/

=======================================
NaviServer 4.99.22, released 2021-08-25
=======================================

  93 files changed, 4216 insertions(+), 1465 deletions(-)

New Features:
-------------

  - Added support for macOS machines with the M1 processor.

  - ns_http improvements:
    . Added proxy handling for [ns_http]
    . Added handling of charsets in ns_http get requests (for non-binary MIME types)

  - Extended default mime-types:
    . Added "jpeg/xl" to the default MIME types (specified in IANA provisional list)

  - Better handling with invalid UTF-8 (also relevant for security reasons)
    . new function "ns_valid_utf8" to check whether a byte-array contains
      a valid UTF-8 byte sequence

    . Check internally the validity of UTF-8 and complain in the system
      log when invalid characters are encountered. Note that Tcl
      transforms sometimes invalid UTF-8 into valid UTF-8 (e.g., URL
      /x%c3.) such that the exact analysis of potential attacks can
      become complex.

    . Proper encoded URLs contain just a subset of 7-bit US ASCII. The
      general problem of handling UTF-8 in request headers is actually
      quite complex and spread over several RFCs. The current HTTP
      specification (RFC 7230) states that "Parsing an HTTP message as
      a stream of Unicode characters, without regard for the specific
      encoding, creates security vulnerabilities due to the varying
      ways that string processing libraries handle invalid multibyte
      character sequences that contain the octet LF (%x0A)." ... older
      RFCs allowed explicitly ISO8859-1.  However, all recent browsers
      code properly encode UTF-8 in the URLs.  So, in theory, a server
      could reject uncoded "binary" data, and assume this data is
      coming from hacking attacks. However, this change just adds a
      warning for these cases.

    . UTF-8 validity checking is performed for URLs in percent-encoded
      and in non-encoded form

  - "ns_parseurl" reform: make URL conformant with RFC 3986

     Previously, the parsing of URLs as performed by "ns_parseurl" was
     more driven by heuristics than by standards. The new version
     parses now URLs according to RFC 3986 (checking as well for valid
     characters in user-defined URL components when the new option
     "-strict" is used). Most internal usages of Ns_HttpParseHost() are
     non-strict to provide good backwards compatibility.  The function
     now parses as well the userinfo in the authority. (authority =
     [userinfo "@"] host [":" port]).

  - Added command: "ns_parsehostport ?-strict? string"

    This command implements a subset of "ns_parseurl" by just trying to
    parse the provided string into "host" and "port" (when
    available). The command handles also the IP literal convention as
    specified in RFC 3986 for parsing IPv6 addresses with ports.

  - Added automated reloading of server certificates when SIGHUP is
    received. While in previous version it was necessary to restart the
    server, when certificates were renewed (e.g., via letsencrypt), the
    new version reloads certificates when it receives a SIGHUP signal.


Bug Fixes:
----------

  - Make sure to nul-terminate IPv4 portion in V4MAPPED addresses.

  - Fixed a potential race condition on peer and proxy IP address,
    where on e.g., pipelined requests the request structure is already
    reused in a new request, while the old connection is used for
    logging. This could result in incorrect peer addresses in the
    access.log

  - More precise execution of scheduled procs

    Background: previously, the scheduling of repeated scheduled procs
    was based on the last finish time. This has the consequence that
    the execution time will drift away more and more from a starting
    time, depending on the runtime of certain jobs. If one has e.g., a
    service that should run every minute, some of these minutes might
    be skipped by the cumulative drift on a long running server. New
    repeated scheduled procs on the original scheduled time rather than
    on last finish time.

  - Fixed Ns_StrTrimRight() to avoid damaging of UTF-8 characters

    Background: Up to now, NaviServer was using "CHARTYPE(space, c)" to
    determine, which characters can be trimmed at the end of a string.
    Unfortunately, there exists characters, which are classified as
    "space", but which can be trailing bytes with different semantics
    in multibyte UTF-8 characters (e.g., 0x85). When these bytes are
    stripped the result are invalid UTF-8 characters.

  - Added support for handling potentially negative time values when
    Ns_GetTime() is non-monotonous

    Background: The time of Ns_GetTime() is determined by
    gettimeofday() is therefore potentially affected by discontinuous
    jumps in the system time (e.g., if the system administrator
    manually changes the system time).

  - nsproxy: Disambiguate the name of the helper command with the name of the module

    NaviServer could get confused between the helper command "nsproxy"
    and the shared object of the module (called nsproxy.so). Since for
    module-loading, the suffix is optional (to support multi-platform
    config files), there as a potential confusion. The helper command
    is now called "nsproxy-helper".

  - Fixed list of charsets as returned by "ns_charsets".

    Previously, "ns_charsets" returned just the mapped charsets (where
    the name of the charset as defined in Tcl and by IANA differs). So
    e.g., "utf8" was not reported back. Now, the full is reported
    back. Additionally, more recent mappings were added.

  - Bugfix for "ns_http connect": Linux sometimes returned error (false positive)

    Sometimes "ns_http run SOMEHOST:PORT" returned under Linux an error
    of the form "can't connect to SOMEHOST port PORT: operation now in
    progress".  This problem could have occurred on connections to hosts
    where the DNS entries have multiple IP addresses associated. This
    error only showed up on the first connection attempt (e.g., after a
    server restart, or on no connections to this host for e.g., a few
    hours), all later attempts with identical parameters worked without
    problems. It turned out that sometimes - while working through the
    associated IP addresses - the call "getsockopt(sock, SOL_SOCKET,
    SO_ERROR, ...)"  retrieved errno 113 <No route to host> from the
    socket, maybe related with routing table lookup.

  - Improved log messages concerning limit of number of open files

  - Improved handling of running out of memory when creating threads
    from Tcl

  - Fixed potential race condition in logging during shutdown

  - Fix potential bug in openssl.m4 (could check for files on a wrong path)


Documentation improvements:
---------------------------

  - Improved the following man pages

         doc/src/manual/admin-config.man
         doc/src/manual/admin-maintenance.man
         doc/src/manual/admin-tuning.man
         doc/src/manual/c-driverdb.man
         doc/src/manual/main-history.man
         doc/src/naviserver/commandlist.man
         doc/src/naviserver/ns_cache.man
         doc/src/naviserver/ns_conn.man
         doc/src/naviserver/ns_crypto.man
         doc/src/naviserver/ns_getcontent.man
         doc/src/naviserver/ns_http.man
         doc/src/naviserver/ns_locationproc.man
         doc/src/naviserver/ns_log.man
         doc/src/naviserver/ns_parseurl.man
         doc/src/naviserver/ns_schedule.man
         doc/src/naviserver/ns_sendmail.man
         doc/src/naviserver/ns_urlspace.man
         doc/src/naviserver/ns_write.man
         nsssl/doc/mann/nsssl.man

  - Added examples for "ns_getcontent" to the manual pages

  - Improved sample configuration files:

    . Added a section for the sample config files how to use the
      letsencrypt NaviServer module
    . Updated cipher configurations as recommended by Mozilla in
      sample configuration files.



Code Changes:
-------------
  - Improved naming of functions
  - OpenSSL: aligned code with current snapshot of OpenSSL 3.0* (3.0.0-beta3-dev)
  - Aligned stubbed functions with Linux prototypes (use "restrict" keyword)

  - Extended regression test
     . Improved setup in tests for testing with private keys
     . Added testing for application/json with UTF-8 charset vis ns_http
     . Fixed handling of ns_hostbyaddr under macOS
     . Added 90 additional tests

  - Code Cleanup
     . Improved security by reducing usage of "ns_mktemp":
       use on the Tcl level "file tempfile ..." (introduced in Tcl 8.6)
       instead of "ns_mktemp" whenever possible.
     . Do not require to have tcllib package "try" installed when using
       Tcl 8.6 or newer
     . Use also in test cases reentrant version of localtime()
     . Reduced (harmless) data races
     . Fixed issued found by facebook infer 1.1.0
     - Avoided passing NULL after the last typed argument to a variadic function

  - Added ability to pass "CFLAGS_OPTIMIZE=..." to "make"
    (eases build specific optimization)

  - Improved comments, fixed typos

Changes in modules:

--- nsdbsqlite ---
  ChangeLog    | 18 ------------------
  nsdbsqlite.c | 12 ++++++------
  2 files changed, 6 insertions(+), 24 deletions(-)
--- nsdbpg ---
  README   | 66 ++++++++++++++++++++++++-----------------------
  dbpg.h   | 10 ++++++++
  nsdbpg.c | 90 +++++++++++++++++++++++++++++++++++++++++++---------------------
  3 files changed, 105 insertions(+), 61 deletions(-)
--- nsdbmysql ---
  nsdbmysql.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
--- nsocaml ---
  nsocaml.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
--- nssmtpd ---
  nssmtpd.c | 16 ++++++++++------
  1 file changed, 10 insertions(+), 6 deletions(-)
--- nsdns ---
  nsdns.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
--- nsfortune ---
  nsfortune.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
--- nsicmp ---
  nsicmp.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
--- nsudp ---
  nsudp.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
--- nsaccess ---
  nsaccess.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
--- nschartdir ---
  nschartdir.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
--- nsexample ---
  nsexample.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
--- nszlib ---
  ChangeLog | 8 --------
  nszlib.c  | 4 ++--
  2 files changed, 2 insertions(+), 10 deletions(-)
--- nsaspell ---
  nsaspell.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
--- nsimap ---
  nsimap.c | 6 +++---
  1 file changed, 3 insertions(+), 3 deletions(-)
--- nstftpd ---
  nstftpd.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
--- nssyslogd ---
  nssyslogd.c | 8 ++++----
  1 file changed, 4 insertions(+), 4 deletions(-)
--- nsphp ---
  nsphp.c | 22 ++++++++++++++--------
  1 file changed, 14 insertions(+), 8 deletions(-)
--- nsstats ---
  nsstats.tcl | 38 ++++++++++++++++++++------------------
  1 file changed, 20 insertions(+), 18 deletions(-)
--- nsauthpam ---
  nsauthpam.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
--- nsmemcache ---
  nsmemcache.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
--- nsvfs ---
  ChangeLog | 4 ----
  nsvfs.c   | 4 ++--
  2 files changed, 2 insertions(+), 6 deletions(-)
--- nsdbi ---
  doc/src/mann/nsdbi.man |  61 ++--
  init.c                 | 122 ++++---
  nsdbi.h                |  12 +-
  tclcmds.c              | 856 ++++++++++++++++++++++++++-----------------------
  4 files changed, 569 insertions(+), 482 deletions(-)
--- nsloopctl ---
  nsloopctl.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
--- websocket ---
  websocket-procs.tcl | 5 ++---
  1 file changed, 2 insertions(+), 3 deletions(-)
--- revproxy ---
  README             |   4 +-
  revproxy-procs.tcl | 128 ++++++++++++++++++++++++++++++++++++++++++++++-------
  2 files changed, 115 insertions(+), 17 deletions(-)
--- letsencrypt ---
  Makefile              |   7 +-
  README                |  38 +-
  letsencrypt-procs.tcl | 934 ++++++++++++++++++++++++++++++++++++++++++++++++++
  letsencrypt.tcl       | 879 +----------------------------------------------
  4 files changed, 984 insertions(+), 874 deletions(-)
--- nsldap ---
  nsldap.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)

Re: [naviserver-devel] percent-encoded characters in URL

[Gustaf N. <ne...@wu...>]

On 25.08.21 11:07, Wolfgang Winkler wrote:
> Thank you for your answer. I've missed the corresponding RFC 3986 part 
> you quoted.
>
> We have a client who, for some reason, got an error, when the @ sign 
> of emails in the query section was not escaped. It has been solved on 
> his side, but I was not sure what the more standard compliant version 
> was. I've tried it myself with various libraries and clients and none 
> hat a problem with the NaviServer version. They might use an outdated 
> java library somewhere.
>
One potential problem might be, if there is a literal comparison of the 
result of JavaScript encodeURIComponent() with the result of 
ns_urlencode, or some sloppy parser.... Would it be of help for you, 
when we would add one more encoding table (e.g. named "js") to 
ns_urlencode to produce the same result as encodeURIComponent() ?

> BTW: I love the nsv dicts and the ns_crypto stuff.

Great, many thanks for the feedback

-g

Re: [naviserver-devel] ns_cache question

[Gustaf N. <ne...@wu...>]

Dear Maksym,

On 24.08.21 02:39, Maksym Zinchenko wrote:

> But lately, user logs in start working and randomly redirected back to 
> the login screen because data in db is not the same as in cache,

when you have values from the DB cached, then whenever the DB changes, 
you have to flush the cache values. Accessing the DB via "ns_cache_eval" 
as mentioned in my last mail is the right thing, since it will rebuild 
the cached values when needed.

When you see this effect just lately, make sure that the cache is large 
enough (check this via the nsstats cache page). A few releases ago, the 
memory computation (which previously did not account for the cache keys) 
was changed, which might lead to a higher nominal cache size value.

> If you know any session package or code example I would appreciate 
> tremendously 

We are using everywhere OpenACS, which has its own session management, 
which is non-trivial, since it tries to avoid hacking, implements 
session refreshes, etc. One other source is Vlads ossweb [1] (which i 
have not looked into, but Vlad produces usually high quality code).

Hope this helps

-gn

[1] https://github.com/vseryakov/ossweb

Re: [naviserver-devel] percent-encoded characters in URL

[Gustaf N. <ne...@wu...>]

Dear Wolfgang,

according to RFC 3986, the at-sign should not be encoded, neither in the 
path segments, nor in the query components.
For querycomponents the right parameter for ns_urlencode is

       ns_urlencode -part query a@b

The full query parameter pair should be encoded with

       set pair [ns_urlencode -part query $key]=[ns_urlencode -part query $value]

RFC 3986 mentions explicitly the at-sign here:

       pchar       = unreserved / pct-encoded / sub-delims / ":" / "@"
       query       = *( pchar / "/" / "?" )

The JavaScript definition is rather vague about the meaning of a URI 
component; the closest thing in the RFC are path segments and query 
components. Since these two kind of components are defined differently 
in RFC 3986, it is clear that encodeURIComponent() does not follow the 
definition in RFC 3986. JavaScript has actually its own definition of 
"Universal Resource Identifier Character Classes" (see Annex 7 in [1]), 
which encodes more than necessary (probably JavaScript does not want to 
release a new version whenever the RFC is updated). The document states 
as well that the syntax of Uniform Resource Identifiers is based upon 
RFC 2396 (sect 18.2.6.1.2)

    /This syntax of Uniform Resource Identifiers is based upon RFC 2396
    and does not reflect the more recent RFC 3986 which replaces RFC
    2396. A formal description and implementation of UTF-8 is given in
    RFC 3629./

On an other place (section B.2.1.1, definition of escape), it states:

    /The encoding is partly based on the encoding described in RFC 1738,
    but the entire encoding specified in this standard is described
    above without regard to the contents of RFC 1738. This encoding does
    not reflect changes to RFC 1738 made by RFC 3986./

An encoding-set agnostic percent decoder decodes everything, so this 
works in practice. No recent web software should have problems with 
standard-compliant URIs, as produced by NaviServer.

Why are you asking?

all the best
-gn

[1] 
https://262.ecma-international.org/9.0/#sec-universal-resource-identifier-character-classes
On 25.08.21 08:21, Wolfgang Winkler via naviserver-devel wrote:
>
> Dear List!
>
> When using ns_urlencode, I've noticed, that the "@" sign will not be 
> percent encoded, unless "-part oauth1" is stated:
>
> ns_urlencode te...@te...
> te...@te...
>
> ns_urlencode -part oauth1 te...@te...
> test%40test.com
>
> What is the correct way to encode URL params, e.g. in
>
> http://test.com/register/login?email=te...@te...
>
> In Javascript (Chrome + Firefox)
>
> encodeURIComponent("te...@te...");
>
> yields
>
> "test%40test.com"
>
> This is should be the RFC for this topic:
>
> https://datatracker.ietf.org/doc/html/rfc3986#section-2.3
>
> Yours,
>
> Wolfgang
>
> -- 
>
> *Wolfgang Winkler*
> Geschäftsführung
> wol...@di...
> mobil +43.699.19971172
>
> dc:*büro*
> digital concepts Novak Winkler OG
> Software & Design
> Landstraße 68, 5. Stock, 4020 Linz
> www.digital-concepts.com <http://www.digital-concepts.com>
> tel +43.732.997117.72
> tel +43.699.1997117.2
>
> Firmenbuchnummer: 192003h
> Firmenbuchgericht: Landesgericht Linz
>
>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

-- 
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"

[naviserver-devel] percent-encoded characters in URL

[Wolfgang W. <wol...@di...>]

Dear List!

When using ns_urlencode, I've noticed, that the "@" sign will not be 
percent encoded, unless "-part oauth1" is stated:

ns_urlencode te...@te...
te...@te...

ns_urlencode -part oauth1 te...@te...
test%40test.com

What is the correct way to encode URL params, e.g. in

http://test.com/register/login?email=te...@te...

In Javascript (Chrome + Firefox)

encodeURIComponent("te...@te...");

yields

"test%40test.com"

This is should be the RFC for this topic:

https://datatracker.ietf.org/doc/html/rfc3986#section-2.3 
<https://datatracker.ietf.org/doc/html/rfc3986#section-2.3>

Yours,

Wolfgang

-- 

*Wolfgang Winkler*
Geschäftsführung
wol...@di...
mobil +43.699.19971172

dc:*büro*
digital concepts Novak Winkler OG
Software & Design
Landstraße 68, 5. Stock, 4020 Linz
www.digital-concepts.com <http://www.digital-concepts.com>
tel +43.732.997117.72
tel +43.699.1997117.2

Firmenbuchnummer: 192003h
Firmenbuchgericht: Landesgericht Linz

Re: [naviserver-devel] ns_cache question

[Maksym Z. <siq...@gm...>]

Thank you, for explanation Gustaf.
Error must be somewhere in my code. What I'm trying to accomplish is create
session. Ive got this code from somewhere on the internet, I don’t even
know where. And modified it little bit. Basically, Im creating session for
each virtual server like: $domain_sessions, if user logs in I’m creating
uuid as cache key and as a cache value tcl dict with info related to user,
such as username, real name etc. I have a persistence function which saves
this data to db table, updates session timestamp etc.
But lately, user logs in start working and randomly redirected back to the
login screen because data in db is not the same as in cache, so I'm a
little bit lost, still trying to figure out what’s going on.
If you know any session package or code example I would appreciate
tremendously

On Mon, Aug 23, 2021 at 7:57 AM Gustaf Neumann <ne...@wu...> wrote:

> Dear Maksym,
>
> a cache is a collection of entries with certain properties, not to be
> confused with a cache entry. The call "ns_cache_exists" checks, whether the
> cache exists (independent of the fact whether the cache has entries or
> not). The expiration is for cache entries.
>
> You can get some overview via the NaviServer "nsstats" module. For example
> on openacs.org the following caches are defined. The table is sorted by
> the caches saving the most per request. You see under "Entries" the number
> of entries per cache.
>
>
> By clicking on the details view of a cache, on can see the entries and
> per-entry statistics. Here is the page from the "xotcl_object_cache-1",
> where one can see that certain entries are substantially more often reused
> from the cache than others.
>
>
> Maybe you are interested whether or not an entry is cached. Actually, this
> kind of query is rather discouraged, since it is a source for race
> conditions. Consider the following code:
>
> ============================================== BAD
> [1]  set entry foo
> [2]  if {[... in cache $entry ...]} {
> [3]     return [... get from cache $entry ...]
> [4]  } else {
> [5]     set value [... compute something with $entry ...]
> [6]     ... save in cache $entry value
> [7]  }
> ==============================================
>
> In this "BAD" snippet, many things can go wrong in a multi-threaded
> environment, where also other requests might massage at the same time
> the same entries, etc. For example it might be the case that the call
> in [3] fails, when the entry found in [2] is expired/deleted/... between
> [2] and [3] (we have in NaviServer real concurrency, two threads might
> run on different cores really simultaneously. Similarly, in line [6], the
> entry might have been set already by some other thread.
>
> ============================================== GOOD
>   set entry foo
>   return [... cache eval $entry {
>      set value [... compute something with $entry ...]
>    }]
> ==============================================
>
>
> So, it is intentional not to encourage the coding style in small steps as
> in "BAD".
> But of course, one can query the the cache entries via "ns_cache_keys
> $cache_name"
>
> Hope this helps
>
> -g
>
> On 22.08.21 22:44, Maksym Zinchenko wrote:
>
> Hello, I have some question about ns_cache. Im creating cache with
> command: ns_cache_create -timeout 1800 -expires 1800 max 5MB When I do
> ns_cache_exists max right away, it shows 1 If i wait a little bit, lets say
> 5 min, not 30 min as difined (1800 are seconds right?) it shows 0, cache
> doesnt exists, if i run command again it shows 1 again. I dont know whats
> going on, may be you can advise me. Thank you
> Maksym Zinchenko
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>

Re: [naviserver-devel] ns_cache question

[Gustaf N. <ne...@wu...>]

Dear Maksym,

a cache is a collection of entries with certain properties, not to be 
confused with a cache entry. The call "ns_cache_exists" checks, whether 
the cache exists (independent of the fact whether the cache has entries 
or not). The expiration is for cache entries.

You can get some overview via the NaviServer "nsstats" module. For 
example on openacs.org the following caches are defined. The table is 
sorted by the caches saving the most per request. You see under 
"Entries" the number of entries per cache.


By clicking on the details view of a cache, on can see the entries and 
per-entry statistics. Here is the page from the "xotcl_object_cache-1", 
where one can see that certain entries are substantially more often 
reused from the cache than others.


Maybe you are interested whether or not an entry is cached. Actually, 
this kind of query is rather discouraged, since it is a source for race 
conditions. Consider the following code:

============================================== BAD
[1]  set entry foo
[2]  if {[... in cache $entry ...]} {
[3]     return [... get from cache $entry ...]
[4]  } else {
[5]     set value [... compute something with $entry ...]
[6]     ... save in cache $entry value
[7]  }
==============================================

In this "BAD" snippet, many things can go wrong in a multi-threaded
environment, where also other requests might massage at the same time
the same entries, etc. For example it might be the case that the call
in [3] fails, when the entry found in [2] is expired/deleted/... between
[2] and [3] (we have in NaviServer real concurrency, two threads might
run on different cores really simultaneously. Similarly, in line [6], the
entry might have been set already by some other thread.

============================================== GOOD
   set entry foo
   return [... cache eval $entry {
      set value [... compute something with $entry ...]
    }]
==============================================

So, it is intentional not to encourage the coding style in small steps 
as in "BAD".
But of course, one can query the the cache entries via "ns_cache_keys 
$cache_name"

Hope this helps

-g


On 22.08.21 22:44, Maksym Zinchenko wrote:
> Hello, I have some question about ns_cache. Im creating cache with 
> command: ns_cache_create -timeout 1800 -expires 1800 max 5MB When I do 
> ns_cache_exists max right away, it shows 1 If i wait a little bit, 
> lets say 5 min, not 30 min as difined (1800 are seconds right?) it 
> shows 0, cache doesnt exists, if i run command again it shows 1 again. 
> I dont know whats going on, may be you can advise me. Thank you
> Maksym Zinchenko

[naviserver-devel] ns_cache question

[Maksym Z. <siq...@gm...>]

 Hello, I have some question about ns_cache. Im creating cache with
command: ns_cache_create -timeout 1800 -expires 1800 max 5MB When I do
ns_cache_exists max right away, it shows 1 If i wait a little bit, lets say
5 min, not 30 min as difined (1800 are seconds right?) it shows 0, cache
doesnt exists, if i run command again it shows 1 again. I dont know whats
going on, may be you can advise me. Thank you
Maksym Zinchenko

[naviserver-devel] Towards NaviServer 4.99.22

[Gustaf N. <ne...@wu...>]

Dear all,

on sourceforge is a release candidate for NaviServer 4.99.22 [1].

I have tested the code with Ubuntu 20.04, Rocky Linux 8.4, OpenBSD 6.9,
FreeBSD 13.0-CURRENT, macOS 11.5.1 Intel, macOS 11.5.2 M1
Please test if possible.  The release should be in about one week.

Below is a preliminary summary of changes.

all the best
-gustaf neumann

[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.22/


=======================================
NaviServer 4.99.22, released 2021-08-XX
=======================================

  84 files changed, 3905 insertions(+), 1423 deletions(-)

New Features:
-------------

  - Added support for macOS machines with the M1 processor.

  - ns_http improvements:
    . Added proxy handling for [ns_http]
    . Added handling of charsets in ns_http get requests (for non-binary MIME types)

  - Extended default mime-types:
    . Added "jpeg/xl" to the default mimetypes (specified in IANA provisional list)

  - Better handling with invalid UTF-8 (also relevant for security reasons)
    . new function "ns_valid_utf8" to check whether a byte-array contains
      a valid UTF-8 byte sequence
      
    . Check internally the validity of UTF-8 and complain in the system
      log when invalid characters are encountered. Note that Tcl
      transforms sometimes invalid UTF-8 into valid UTF-8 (e.g. URL
      /x%c3.) such that the exact analysis of potential attacks can
      become complex.

    . Proper encoded URLs contain just a subset of 7-bit US ASCII. The
      general problem of handling UTF-8 in request headers is actually
      quite complex and spread over several RFCs. The current HTTP
      specification (RFC 7230) states that "Parsing an HTTP message as
      a stream of Unicode characters, without regard for the specific
      encoding, creates security vulnerabilities due to the varying
      ways that string processing libraries handle invalid multibyte
      character sequences that contain the octet LF (%x0A)." ... older
      RFCs allowed explicitly ISO8859-1.  However, all recent browsers
      code properly encode UTF-8 in the URLs.  So, in theory, a server
      could reject uncoded "binary" data, and assume this data is
      coming from hacking attacks. However, this change just adds a
      warning for these cases.

    . UTF-8 validity checking is performed for URLs in percent-encoded
      and in non-encoded form

  - "ns_parseurl" reform: make URL conformant with RFC 3986

     Previously, the parsing of URLs as performed by "ns_parseurl" was
     more driven by heuristics than by standards. The new version
     parses now URLs according to RFC 3986 (checking as well for valid
     characters in user-defined URL components when the new option
     "-strict" is used). Most internal usages of Ns_HttpParseHost() are
     non-strict to provide good backwards compatibility.  The function
     now parses as well the userinfo in the authority. (authority =
     [userinfo "@"] host [":" port]).

  - Added command: "ns_parsehostport ?-strict? string"
  
    This command implements a subset of "ns_parseurl" by just trying to
    parse the provided string into "host" and "port" (when
    available). The command handles also the IP literal convention as
    specified in RFC 3986 for parsing IPv6 addresses with ports.

  - Added automated reloading of server certificates when SIGHUP is
    received. While in previous version it was necessary to restart the
    server, when certificates were renewed (e.g. via letsencrypt), the
    new version reloads certificates when it receives a SIGHUP signal.
  
  
Bug Fixes:
----------

  - Make sure to nul-terminate IPv4 portion in V4MAPPED addresses.
     
  - Fixed a potential race condition on peer and proxy IP address,
    where on e.g. pipelined requests the request structure is already
    reused in a new request, while the old connection is used for
    logging. This could result in incorrect peer addresses in the
    access.log

  - More precise execution of scheduled procs
     
    Background: previously, the scheduling of repeated scheduled procs
    was based on the last finish time. This has the consequence that
    the execution time will drift away more and more from a starting
    time, depending on the runtime of certain jobs. If one has e.g. a
    service that should run every minute, some of these minutes might
    be skipped by the cumulative drift on a long running server. New
    repeated scheduled procs on the original scheduled time rather than
    on last finish time.

  - Fixed Ns_StrTrimRight() to avoid damaging of UTF-8 characters
     
    Background: Up to now, NaviServer was using "CHARTYPE(space, c)" to
    determine, which characters can be trimmed at the end of a string.
    Unfortunately, there exists characters, which are classified as
    "space", but which can be trailing bytes with different semantics
    in multibyte UTF-8 characters (e.g. 0x85). When these bytes are
    stripped the result are invalid UTF-8 characters.

  - Added support for handling potentially negative time values when
    Ns_GetTime() is non-monotonous
     
    Background: The time of Ns_GetTime() is determined by
    gettimeofday() is therefore potentially affected by discontinuous
    jumps in the system time (e.g., if the system administrator
    manually changes the system time).

  - nsproxy: Disambiguate the name of the helper cmd with the name of the module
     
    NaviServer could get confused between the helper command "nsproxy"
    and the shared object of the module (called nsproxy.so). Since for
    module-loading, the suffix is optional (to support multi-platform
    config files), there as a potential confusion. The helper command
    is now called "nsproxy-helper".

  - Fixed list of charsets as returned by "ns_charsets".

    Previously, "ns_charsets" returned just the mapped charsets (where
    the name of the charset as defined in Tcl and by IANA differs). So
    e.g. "utf8" was not reported back. Now, the full is reported
    back. Additionally, more recent mappings were added.

  - Bugfix for "ns_http connect": Linux sometimes returned error (false positive)
     
    Sometimes "ns_http run SOMEHOST:PORT" returned under Linux an error
    of the form "can't connect to SOMEHOST port PORT: operation now in
    progress".  This problem could have occurred on connections to hosts
    where the DNS entries have multiple IP addresses associated. This
    error only showed up on the first connection attempt (e.g. after a
    server restart, or on no connections to this host for e.g. a few
    hours), all later attempts with identical parameters worked without
    problems. It turned out that sometimes - while working through the
    associated IP addresses - the call "getsockopt(sock, SOL_SOCKET,
    SO_ERROR, ...)"  retrieved errno 113 <No route to host> from the
    socket, maybe related with routing table lookup.

  - Improved log messages concerning limit of number of open files
  
  - Improved handling of running out of memory when creating threads
    from Tcl
    
  - Fixed potential race condition in logging during shutdown

  - Fix potential bug in openssl.m4 (could check for files on a wrong path)


Documentation improvements:
---------------------------

  - Improved the following man pages

         doc/src/manual/admin-config.man
         doc/src/naviserver/commandlist.man
         doc/src/naviserver/ns_conn.man
         doc/src/naviserver/ns_crypto.man
         doc/src/naviserver/ns_getcontent.man
         doc/src/naviserver/ns_http.man
         doc/src/naviserver/ns_parsehostport.man
         doc/src/naviserver/ns_parseurl.man
         doc/src/naviserver/ns_schedule.man
         doc/src/naviserver/ns_valid_utf8.man
         nsssl/doc/mann/nsssl.man
        
  - Added examples for "ns_getcontent" to the manual pages

  - Improved sample configuration files:
  
    . Added a section for the sample config files how to use the
      letsencrypt NaviServer module
    . Updated cipher configurations as recommended by Mozilla in
      sample configuration files.



Code Changes:
-------------
  - Improved naming of functions
  - OpenSSL: aligned code with current snapshot of OpenSSL 3.0*
  - Aligned stubbed functions with Linux prototypes (use "restrict" keyword)
  
  - Extended regression test
     . improved setup in tests for testing with private keys
     . added testing for application/json with UTF-8 charset vis ns_http
     . fixed handling of ns_hostbyaddr under macOS
     . added 90 additional tests
    
  - Code Cleanup
     . Reduced usage of "ns_mktemp": use on the Tcl level "file
       tempfile ..." (introduced in Tcl 8.6) instead whenever possible).
     . Do not require to have tcllib package "try" installed when using
       Tcl 8.6 or newer
     . Use also in test cases reentrant version of localtime()
     . Reduced (harmless) data races
     . Fixed issued found by facebook infer 1.1.0
     - Avoided passing NULL after the last typed argument to a variadic function

  - Improved comments, fixed typos


The following people have contributed to this release:

     Gustaf Neumann
     Oleg Oleinick
     Ibrahim Tannir
     Zoran Vasiljevic



Changes in modules:

--- nsdbsqlite ---
  ChangeLog    | 18 ------------------
  nsdbsqlite.c | 12 ++++++------
  2 files changed, 6 insertions(+), 24 deletions(-)
--- nsdbpg ---
  README   | 66 ++++++++++++++++++++++++-----------------------
  dbpg.h   | 10 ++++++++
  nsdbpg.c | 90 +++++++++++++++++++++++++++++++++++++++++++---------------------
  3 files changed, 105 insertions(+), 61 deletions(-)
--- nsdbmysql ---
  nsdbmysql.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
--- nsocaml ---
  nsocaml.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
--- nssmtpd ---
  nssmtpd.c | 16 ++++++++++------
  1 file changed, 10 insertions(+), 6 deletions(-)
--- nsdns ---
  nsdns.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
--- nsfortune ---
  nsfortune.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
--- nsicmp ---
  nsicmp.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
--- nsudp ---
  nsudp.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
--- nsaccess ---
  nsaccess.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
--- nschartdir ---
  nschartdir.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
--- nsexample ---
  nsexample.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
--- nszlib ---
  ChangeLog | 8 --------
  nszlib.c  | 4 ++--
  2 files changed, 2 insertions(+), 10 deletions(-)
--- nsaspell ---
  nsaspell.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
--- nsimap ---
  nsimap.c | 6 +++---
  1 file changed, 3 insertions(+), 3 deletions(-)
--- nstftpd ---
  nstftpd.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
--- nssyslogd ---
  nssyslogd.c | 8 ++++----
  1 file changed, 4 insertions(+), 4 deletions(-)
--- nsphp ---
  nsphp.c | 22 ++++++++++++++--------
  1 file changed, 14 insertions(+), 8 deletions(-)
--- nsstats ---
  nsstats.tcl | 38 ++++++++++++++++++++------------------
  1 file changed, 20 insertions(+), 18 deletions(-)
--- nsauthpam ---
  nsauthpam.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
--- nsmemcache ---
  nsmemcache.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
--- nsvfs ---
  ChangeLog | 4 ----
  nsvfs.c   | 4 ++--
  2 files changed, 2 insertions(+), 6 deletions(-)
--- nsdbi ---
  doc/src/mann/nsdbi.man |  61 ++--
  init.c                 | 122 ++++---
  nsdbi.h                |  12 +-
  tclcmds.c              | 856 ++++++++++++++++++++++++++-----------------------
  4 files changed, 569 insertions(+), 482 deletions(-)
--- nsloopctl ---
  nsloopctl.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
--- websocket ---
  websocket-procs.tcl | 5 ++---
  1 file changed, 2 insertions(+), 3 deletions(-)
--- revproxy ---
  README             |   4 +-
  revproxy-procs.tcl | 128 ++++++++++++++++++++++++++++++++++++++++++++++-------
  2 files changed, 115 insertions(+), 17 deletions(-)
--- letsencrypt ---
  Makefile              |   7 +-
  README                |  38 +-
  letsencrypt-procs.tcl | 934 ++++++++++++++++++++++++++++++++++++++++++++++++++
  letsencrypt.tcl       | 879 +----------------------------------------------
  4 files changed, 984 insertions(+), 874 deletions(-)
--- nsldap ---
  nsldap.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)


         

-- 
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"

Re: [naviserver-devel] Warning: private key load error: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt

[THORPE M. <ta...@me...>]

Hi Gustav,

Well put. Me too.

Thank you for the explanation re DH Params. 

All is well.

Thank you.

Thorpe

> On Jul 28, 2021, at 14:00, Gustaf Neumann <ne...@wu...> wrote:
> 
> Hi Thorpe,
> 
> On 27.07.21 14:42, THORPE MAYES via naviserver-devel wrote:
>> This issue surfaced when I was creating a new ssl certificate. The old certificate expires at the end of the month. 
>> The old certificate continued to work after I updated naviserver (with openssl 1.1k). The new certificate did not work.
>> The old certificate has the Diffie-Hellmand key exchange applied.
>> I applied the DH key exchange via openssl: openssl dhparam 2048 >> server.pem
>> The version of openssl installed at the time predated 1.1k. I believe that it was 1.1.i; it may have been an earlier version.
>> 
>> I decided to not apply the DH Param to the new certificate. The new certificate now works. 
> Short answer: glad it works now. 
> 
> Specifying DH Params in the .pem file should not be needed in newer versions of OpenSSL.
> 
> The DH parameters should not hurt in general either. I checked the (letsencrypt) certificate used on openacs.org. The pemfile contains DH parameters and has the following structure:
> -----BEGIN PRIVATE KEY-----
> ...
> -----END PRIVATE KEY-----
> -----BEGIN CERTIFICATE-----
> ...
> -----END CERTIFICATE-----
> -----BEGIN CERTIFICATE-----
> ...
> -----END CERTIFICATE-----
> -----BEGIN CERTIFICATE-----
> ...
> -----END CERTIFICATE-----
> -----BEGIN CERTIFICATE-----
> ...
> -----END CERTIFICATE-----
> -----BEGIN DH PARAMETERS-----
> ...
> -----END DH PARAMETERS-----
> and everything works fine there.
> 
> Handling of DH parameter is a moving target in OpenSSL (and clones). In newer versions, OpenSSL defines SSL_CTX_set_dh_auto(), such that DH PARAMETERS are not needed anymore in the PEM file. Some of the this code was backported to earlier version of OpenSSL. When NaviServer is compiled with OpenSSL 1.1.* or newer, it uses this function. So, i am not completely sure, what the problem is you had, but version-wise, DH PARAMETERS should not be necessary in your setup.
> 
> I've also updated the log message of the server when certificate loading fails, ... since these messages were also confusing for me.
> all the best
> 
> -g
> 
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] Warning: private key load error: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt

[Gustaf N. <ne...@wu...>]

Hi Thorpe,

On 27.07.21 14:42, THORPE MAYES via naviserver-devel wrote:
> This issue surfaced when I was creating a new ssl certificate. The old 
> certificate expires at the end of the month.
> The old certificate continued to work after I updated naviserver (with 
> openssl 1.1k). The new certificate did not work.
> The old certificate has the Diffie-Hellmand key exchange applied.
> I applied the DH key exchange via openssl: openssl dhparam 2048 >> 
> server.pem
> The version of openssl installed at the time predated 1.1k. I believe 
> that it was 1.1.i; it may have been an earlier version.
>
> I decided to not apply the DH Param to the new certificate. The new 
> certificate now works.

Short answer: glad it works now.

Specifying DH Params in the .pem file should not be needed in newer 
versions of OpenSSL.

The DH parameters should not hurt in general either. I checked the 
(letsencrypt) certificate used on openacs.org. The pemfile contains DH 
parameters and has the following structure:

-----BEGIN PRIVATE KEY-----
...
-----END PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
-----BEGIN DH PARAMETERS-----
...
-----END DH PARAMETERS-----

and everything works fine there.

Handling of DH parameter is a moving target in OpenSSL (and clones). In 
newer versions, OpenSSL defines SSL_CTX_set_dh_auto(), such that DH 
PARAMETERS are not needed anymore in the PEM file. Some of the this code 
was backported to earlier version of OpenSSL. When NaviServer is 
compiled with OpenSSL 1.1.* or newer, it uses this function. So, i am 
not completely sure, what the problem is you had, but version-wise, DH 
PARAMETERS should not be necessary in your setup.

I've also updated the log message of the server when certificate loading 
fails, ... since these messages were also confusing for me.

all the best

-g

Re: [naviserver-devel] Warning: private key load error: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt

[THORPE M. <ta...@me...>]

Hi Gustav,

Thank you for your response.

Here is what I have in the nsssl section of the config file:
ns_param   certificate   ${serverdir}/modules/openssl/server.pem
      ns_param   address       $address
      ns_param   port          443
      ns_param   ciphers      "ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!RC4"
      ns_param   ciphersuites "TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256"
      ns_param   protocols    "!SSLv2:!SSLv3:!TLSv1.0:!TLSv1.1"
      ns_param   OCSPstapling   on
      ns_param   verify         0
      ns_param   maxinput      [expr 10 * 1024 * 1024]
    
      ns_param   extraheaders {
         Strict-Transport-Security "max-age=31536000; includeSubDomains"
         X-Frame-Options SAMEORIGIN
         X-Content-Type-Options nosniff
      }

${serverdir) is defined as /usr/local/ns/servers/mealdeliverysoftware

set homedir                /usr/local/ns
set servername             “mealdeliverysoftware"
set serverdir              ${homedir}/servers/${servername}


This issue surfaced when I was creating a new ssl certificate. The old certificate expires at the end of the month. 

The old certificate continued to work after I updated naviserver (with openssl 1.1k). The new certificate did not work.

The old certificate has the Diffie-Hellmand key exchange applied.

I applied the DH key exchange via openssl: openssl dhparam 2048 >> server.pem

The version of openssl installed at the time predated 1.1k. I believe that it was 1.1.i; it may have been an earlier version.

I decided to not apply the DH Param to the new certificate. The new certificate now works. 

Thank you for your responsiveness and your help.

Best regards,

Thorpe

> On Jul 27, 2021, at 03:15, Gustaf Neumann <ne...@wu...> wrote:
> 
> Hi Thorpe.
> 
> NaviServer 4.99.21 is not released yet... but nevertheless, it is supposed to work (and is in use e.g. on openacs.org and on many more sites).
> 
> What is your configuration line for the certificate?
> It looks like NaviServer is passing the the path 
> 
>  ns/server/mealdeliverysoftware/module/nsssl//usr/local/ns/servers/mealdeliverysoftware/modules/openssl/server.pem
> 
> to OpenSSL, but it should pass probably
> 
>  /usr/local/ns/servers/mealdeliverysoftware/modules/openssl/server.pem
> 
> -g
> 
> On 24.07.21 14:43, THORPE MAYES via naviserver-devel wrote:
>> Hi,
>> 
>> I have updated to naviserver-4-99.21
>> 
>> I get this error when starting the server:
>> Notice: OpenSSL OpenSSL 1.1.1k  25 Mar 2021 initialized
>> Notice: load certificate from <ns/server/mealdeliverysoftware/module/nsssl//usr/local/ns/servers/mealdeliverysoftware/modules/openssl/server.pem>
>> ...
>> Warning: private key load error: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt
>> Error: nsssl: init error: No such file or directory
>> Error: modload: /usr/local/ns/bin/nsssl.so: Ns_ModuleInit returned: -1
>> Fatal: modload: failed to load module '/usr/local/ns/bin/nsssl.so’
>> 
>> I checked the server.pem file:
>> openssl rsa -inform PEM -in server.pem -check -noout 
>> Enter pass phrase for server.pem:
>> RSA key ok
>> 
>> I am stuck.
>> 
>> Does anyone have any insight re this issue?
>> 
>> Thank you.
>> 
>> Thorpe
>> 
>> 
>> 
>> _______________________________________________
>> naviserver-devel mailing list
>> nav...@li... <mailto:nav...@li...>
>> https://lists.sourceforge.net/lists/listinfo/naviserver-devel <https://lists.sourceforge.net/lists/listinfo/naviserver-devel>
> -- 
> Univ.Prof. Dr. Gustaf Neumann
> Head of the Institute of Information Systems and New Media
> of Vienna University of Economics and Business
> Program Director of MSc "Information Systems"
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] Warning: private key load error: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt

[Gustaf N. <ne...@wu...>]

Hi Thorpe.

NaviServer 4.99.21 is not released yet... but nevertheless, it is 
supposed to work (and is in use e.g. on openacs.org and on many more sites).

What is your configuration line for the certificate?
It looks like NaviServer is passing the the path

  ns/server/mealdeliverysoftware/module/nsssl//usr/local/ns/servers/mealdeliverysoftware/modules/openssl/server.pem

to OpenSSL, but it should pass probably

  /usr/local/ns/servers/mealdeliverysoftware/modules/openssl/server.pem

-g

On 24.07.21 14:43, THORPE MAYES via naviserver-devel wrote:
> Hi,
>
> I have updated to naviserver-4-99.21
>
> I get this error when starting the server:
> Notice: OpenSSL OpenSSL 1.1.1k  25 Mar 2021 initialized
> Notice: load certificate from 
> <ns/server/mealdeliverysoftware/module/nsssl//usr/local/ns/servers/mealdeliverysoftware/modules/openssl/server.pem>
> ...
> Warning: private key load error: error:06065064:digital envelope 
> routines:EVP_DecryptFinal_ex:bad decrypt
> Error: nsssl: init error: No such file or directory
> Error: modload: /usr/local/ns/bin/nsssl.so: Ns_ModuleInit returned: -1
> Fatal: modload: failed to load module '/usr/local/ns/bin/nsssl.so’
>
> I checked the server.pem file:
> openssl rsa -inform PEM -in server.pem -check -noout
> Enter pass phrase for server.pem:
> RSA key ok
>
> I am stuck.
>
> Does anyone have any insight re this issue?
>
> Thank you.
>
> Thorpe
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

-- 
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"

[naviserver-devel] Warning: private key load error: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt

[THORPE M. <ta...@me...>]

Hi,

I have updated to naviserver-4-99.21

I get this error when starting the server:
Notice: OpenSSL OpenSSL 1.1.1k  25 Mar 2021 initialized
Notice: load certificate from <ns/server/mealdeliverysoftware/module/nsssl//usr/local/ns/servers/mealdeliverysoftware/modules/openssl/server.pem>
...
Warning: private key load error: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt
Error: nsssl: init error: No such file or directory
Error: modload: /usr/local/ns/bin/nsssl.so: Ns_ModuleInit returned: -1
Fatal: modload: failed to load module '/usr/local/ns/bin/nsssl.so’

I checked the server.pem file:
openssl rsa -inform PEM -in server.pem -check -noout 
Enter pass phrase for server.pem:
RSA key ok

I am stuck.

Does anyone have any insight re this issue?

Thank you.

Thorpe

Re: [naviserver-devel] ns_charsets

[Gustaf N. <ne...@wu...>]

On 11.07.21 20:00, Gustaf Neumann wrote:
>
> When i look at the IANA page, i see several entries there, which are 
> not in the naviserver default table. So, one should check, what's 
> feasible to be added.
>
Dear all,

i've worked through the IANA charset definitions [1], especially through 
the "Preferred MIME Name" and "Name" fields and bring the default 
charset definitions of NaviServer up-to-date. In addition i have added 
the charset names which are identical in the IANA definitions and Tcl 
the list returned by [ns_charset] to reduce complexity and potential 
confusions. It would be good if someone working with many charsets to 
have a second look at the list. The changes are on bitbucket.

all the best

-g

[1] https://www.iana.org/assignments/character-sets/character-sets.xml



index f9fd791b..836338a7 100644
--- a/nsd/encoding.c
+++ b/nsd/encoding.c
@@ -90,9 +90,30 @@ static const struct {
      const char  *charset;
      const char  *name;
  } builtinChar[] = {
+    { "gb18030",             "cp936" },
+    { "gb_2312-80",          "gb2312" },
+    { "ibm437",              "cp437" },
+    { "ibm775",              "cp775" },
+    { "ibm850",              "cp850" },
+    { "ibm852",              "cp852" },
+    { "ibm855",              "cp855" },
+    { "ibm857",              "cp857" },
+    { "ibm860",              "cp860" },
+    { "ibm861",              "cp861" },
+    { "ibm862",              "cp862" },
+    { "ibm863",              "cp863" },
+    { "ibm864",              "cp864" },
+    { "ibm865",              "cp865" },
+    { "ibm866",              "cp866" },
+    { "ibm869",              "cp869" },
      { "iso-2022-jp",         "iso2022-jp" },
      { "iso-2022-kr",         "iso2022-kr" },
      { "iso-8859-1",          "iso8859-1" },
+    { "iso-8859-10",         "iso8859-10" },
+    { "iso-8859-13",         "iso8859-13" },
+    { "iso-8859-14",         "iso8859-14" },
+    { "iso-8859-15",         "iso8859-15" },
+    { "iso-8859-16",         "iso8859-16" },
      { "iso-8859-2",          "iso8859-2" },
      { "iso-8859-3",          "iso8859-3" },
      { "iso-8859-4",          "iso8859-4" },
@@ -101,6 +122,8 @@ static const struct {
      { "iso-8859-7",          "iso8859-7" },
      { "iso-8859-8",          "iso8859-8" },
      { "iso-8859-9",          "iso8859-9" },
+    { "jis_x0201",           "jis0201" },
+    { "jis_x0212-1990",      "jis0212" },
      { "korean",              "ksc5601" },
      { "ksc_5601",            "ksc5601" },
      { "mac",                 "macRoman" },
@@ -140,6 +163,8 @@ static const struct {
      { "windows-1256",        "cp1256" },
      { "windows-1257",        "cp1257" },
      { "windows-1258",        "cp1258" },
+    { "windows-31j",         "cp932" },
+    { "windows-874",         "cp874" },
      { "x-mac",               "macRoman" },
      { "x-mac-centeuro",      "macCentEuro" },
      { "x-mac-centraleupore", "macCentEuro" },
@@ -154,6 +179,24 @@ static const struct {
      { "x-mac-turkish",       "macTurkish" },
      { "x-mac-ukraine",       "macUkraine" },
      { "x-macintosh",         "macRoman" },
+
+    /*
+     * The following entries are strictly speaking not needed, since the
+     * IANA name is identical with the Tcl charset name. We add these to
+     * be able to return full set of supported IANA charsets via
+     * [ns_charset].
+     *
+     * See:https://www.iana.org/assignments/character-sets/character-sets.xml
+     */
+    { "big5",                "big5" },
+    { "euc-jp",              "euc-jp" },
+    { "euc-kr",              "euc-kr" },
+    { "gb2312",              "gb2312" },
+    { "koi8-r",              "koi8-r" },
+    { "koi8-u",              "koi8-u" },
+    { "tis-620",             "tis-620" },
+    { "utf-8",               "utf-8" },
+
      { NULL, NULL }
  };

Re: [naviserver-devel] ns_charsets

[Maksym Z. <siq...@gm...>]

Thank you for clarifications Gustaf, i'll check what's going on with my
code.

On Sun, Jul 11, 2021 at 5:00 PM Gustaf Neumann <ne...@wu...> wrote:

>
> On 11.07.21 14:14, Maksym Zinchenko wrote:
>
> Hello,I'm having trouble with encodings, or maybe I don't understand
> something.
> My first question is: when I run *"ns_charsets*" command it will not
> return UTF-8 encoding in the list, why and what does it mean?
>
> Dear Maksym,
>
> "ns_charsets" is a code that was not touched since a very long time. In
> essence, it provides a mapping between "official" charset names and
> tcl-encodings. In general charsets can have multiple names for the the same
> thing (preferred MIME name, official name, aliases)
>
> For example the preferred mime charset "iso-8859-1" is mapped to the
> Tcl_Encoding named "iso8859-1". If a charset name is not returned via
> "ns_charsets", it is used literally. So, the charset "utf-8" is mapped the
> the Tcl_encoding with the same name. The charset mapping can be extended
> via the configuration file (section "ns/charsets").... so in essence,
> "ns_charset" just returns entries which require special mappings.
>
> When i look at the IANA page, i see several entries there, which are not
> in the naviserver default table. So, one should check, what's feasible to
> be added.
>
> The Second problem is: I'm trying to *"ns_return"* XML UTF-8 data from my
> RESTFull API responder.
> ns_return 200 "application/xml; charset=utf-8" [dict get $response data]
>
> But firefox complains about "XML Parsing Error: not well-formed" it will
> get stuck at portuguese characters, Response header in firefox dev console
> shows: "Content-Type
> application/xml; charset=utf-8", what I'm doing wrong?
>
> Probably, the "data" element of dict "response" contains binary data (a
> Tcl byte array).
> I can see nothing wrong in NaviServer, check below for a minimal test
> setup.
>
> all the best
>
> -gn
>
> ============================================================================================== xml-responder.tcl
> ns_return 200 "application/xml; charset=utf-8" <root><name>Motörhead</name></root>\n
> ==============================================================================================
>
> Testing:
>
> $ curl -k -i  https://localhost:8443/xml-responder.tcl
> HTTP/1.1 200 OK
> Server: NaviServer/4.99.21
> Date: Sun, 11 Jul 2021 17:51:50 GMT
> Content-Type: application/xml; charset=utf-8
> Content-Length: 37
> Connection: keep-alive
> X-Content-Type-Options: nosniff
> X-XSS-Protection: 1; mode=block
> Referrer-Policy: strict-origin
>
> <root><name>Motörhead</name></root>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>

Re: [naviserver-devel] ns_charsets

[Gustaf N. <ne...@wu...>]

On 11.07.21 14:14, Maksym Zinchenko wrote:
> Hello,I'm having trouble with encodings, or maybe I don't understand 
> something.
> My first question is: when I run *"ns_charsets*" command it will not 
> return UTF-8 encoding in the list, why and what does it mean?

Dear Maksym,

"ns_charsets" is a code that was not touched since a very long time. In 
essence, it provides a mapping between "official" charset names and 
tcl-encodings. In general charsets can have multiple names for the the 
same thing (preferred MIME name, official name, aliases)

For example the preferred mime charset "iso-8859-1" is mapped to the 
Tcl_Encoding named "iso8859-1". If a charset name is not returned via 
"ns_charsets", it is used literally. So, the charset "utf-8" is mapped 
the the Tcl_encoding with the same name. The charset mapping can be 
extended via the configuration file (section "ns/charsets").... so in 
essence, "ns_charset" just returns entries which require special mappings.

When i look at the IANA page, i see several entries there, which are not 
in the naviserver default table. So, one should check, what's feasible 
to be added.

> The Second problem is: I'm trying to *"ns_return"* XML UTF-8 data from 
> my RESTFull API responder.
> ns_return 200 "application/xml; charset=utf-8" [dict get $response data]
>
> But firefox complains about "XML Parsing Error: not well-formed" it 
> will get stuck at portuguese characters, Response header in firefox 
> dev console shows: "Content-Type
> application/xml; charset=utf-8", what I'm doing wrong?
Probably, the "data" element of dict "response" contains binary data (a 
Tcl byte array).
I can see nothing wrong in NaviServer, check below for a minimal test 
setup.

all the best

-gn

============================================================================================== xml-responder.tcl
ns_return 200 "application/xml; charset=utf-8" <root><name>Motörhead</name></root>\n
==============================================================================================

Testing:

$ curl -k -ihttps://localhost:8443/xml-responder.tcl
HTTP/1.1 200 OK
Server: NaviServer/4.99.21
Date: Sun, 11 Jul 2021 17:51:50 GMT
Content-Type: application/xml; charset=utf-8
Content-Length: 37
Connection: keep-alive
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Referrer-Policy: strict-origin

<root><name>Motörhead</name></root>

[naviserver-devel] ns_charsets

[Maksym Z. <siq...@gm...>]

Hello,I'm having trouble with encodings, or maybe I don't understand
something.
My first question is: when I run *"ns_charsets*" command it will not return
UTF-8 encoding in the list, why and what does it mean?
The Second problem is: I'm trying to *"ns_return"* XML UTF-8 data from my
RESTFull API responder.
ns_return 200 "application/xml; charset=utf-8" [dict get $response data]

But firefox complains about "XML Parsing Error: not well-formed" it will
get stuck at portuguese characters, Response header in firefox dev console
shows: "Content-Type
application/xml; charset=utf-8", what I'm doing wrong?

[naviserver-devel] Naviserver failing to compile with TCL9 Alpha

[D.Fox <un...@cr...>]

Hey, so -- I am aware this is all bleeding edge and really unsupported however just throwing this in the mailing list for awareness that at the moment Naviserver does not compile with TCL9a3. If you would like to submit a bug report, I am happy to do so. 

TCL version: TCL9a3 
OS version: FreeBSD 14 
Naviserver version: 4.99.21 

g.c: In function 'NsInitLog': 
log.c:263:22: warning: passing argument 1 of 'Tcl_SetPanicProc' makes '__attribute__((noreturn))' qualified function pointer from unqualified [-Wdiscarded-qualifiers] 
263 | Tcl_SetPanicProc(Panic); 
| ^~~~~ 
In file included from ../include/nsthread.h:529, 
from ../include/ns.h:46, 
from nsd.h:38, 
from log.c:37: 
/srv/scrap/tcl9.0a3/generic/tcl.h:2198:37: note: expected '__attribute__((noreturn)) void (*)(const char *, ...)' but argument is of type 'void (*)(const char *, ...)' 
2198 | TCL_NORETURN1 Tcl_PanicProc *panicProc); 
| ~~~~~~~~~~~~~~~^~~~~~~~~ 
gcc -O2 -DNDEBUG -DSYSTEM_MALLOC -Wall -fPIC -pipe -finput-charset=UTF-8 -DSYSTEM_MALLOC -DTCL_NO_DEPRECATED -std=c99 -I../include -I"/srv/scrap/tcl9.0a3/generic" -DHAVE_CONFIG_H -c -o mimetypes.o mimetypes.c 
gcc -O2 -DNDEBUG -DSYSTEM_MALLOC -Wall -fPIC -pipe -finput-charset=UTF-8 -DSYSTEM_MALLOC -DTCL_NO_DEPRECATED -std=c99 -I../include -I"/srv/scrap/tcl9.0a3/generic" -DHAVE_CONFIG_H -c -o modload.o modload.c 
modload.c: In function 'Ns_ModuleLoad': 
modload.c:171:9: error: unknown type name 'Tcl_PackageInitProc'; did you mean 'Tcl_LibraryInitProc'? 
171 | Tcl_PackageInitProc *tclInitProc = NULL, *moduleVersionAddr = NULL; 
| ^~~~~~~~~~~~~~~~~~~ 
| Tcl_LibraryInitProc 
modload.c:190:29: warning: passing argument 5 of 'Tcl_FSLoadFile' from incompatible pointer type [-Wincompatible-pointer-types] 
190 | &tclInitProc, &moduleVersionAddr, &lh, &uPtr); 
| ^~~~~~~~~~~~ 
| | 
| int ** 
In file included from /srv/scrap/tcl9.0a3/generic/tcl.h:2254, 
from ../include/nsthread.h:529, 
from ../include/ns.h:46, 
from nsd.h:38, 
from modload.c:37: 
/srv/scrap/tcl9.0a3/generic/tclDecls.h:1153:27: note: expected 'int (**)(Tcl_Interp *)' but argument is of type 'int **' 
1153 | Tcl_LibraryInitProc **proc1Ptr, 
| ~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~ 
modload.c:190:43: warning: passing argument 6 of 'Tcl_FSLoadFile' from incompatible pointer type [-Wincompatible-pointer-types] 
190 | &tclInitProc, &moduleVersionAddr, &lh, &uPtr); 
| ^~~~~~~~~~~~~~~~~~ 
| | 
| int ** 
In file included from /srv/scrap/tcl9.0a3/generic/tcl.h:2254, 
from ../include/nsthread.h:529, 
from ../include/ns.h:46, 
from nsd.h:38, 
from modload.c:37: 
/srv/scrap/tcl9.0a3/generic/tclDecls.h:1154:27: note: expected 'int (**)(Tcl_Interp *)' but argument is of type 'int **' 
1154 | Tcl_LibraryInitProc **proc2Ptr, 
| ~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~ 
gmake[1]: *** [<builtin>: modload.o] Error 1 
gmake[1]: Leaving directory '/srv/scrap/naviserver-4.99.21/nsd' 
gmake: *** [Makefile:50: all] Error 1 

If I make the changes of Tcl_PackageInitProc within modload.c to Tcl_PackageInitProc I get the following error: 

roc.c:180:1: error: conflicting types for 'FuncptrKey' 
180 | FuncptrKey(Tcl_HashTable *UNUSED(tablePtr), void *keyPtr) 

Regards, 
D Fox 

[naviserver-devel] ns_getcontent UTF-8

[Maksym Z. <siq...@gm...>]

Hello everyone, I have a strange issue with UTF-8. Im sending AJAX request
to Naviserver with defined UTF-8, this is part in my JS script.

function send_POST(url, postedData, callbackf) {
$.ajaxSetup({
contentType: "application/json; charset=UTF-8"
});

I can see in console that value really UTF-8 encoded:

add_category_modal{ "name": "Colchões" }

On server Im using "ns_getcontent":

if {[set ct [ns_getcontent -as_file false]] ne ""} {
puts $ct
}

And it returns not UTF-8 content

May be its a bug?

Thank you

Re: [naviserver-devel] Content-Type application/json request

[Gustaf N. <ne...@wu...>]

good to know. i've added this as an example to the manual pages in the 
repository... -gn

On 01.05.21 16:35, Maksym Zinchenko wrote:
> Thank you. that's exactly what I need

Re: [naviserver-devel] Content-Type application/json request

[Maksym Z. <siq...@gm...>]

Thank you. that's exactly what I need

On Fri, Apr 30, 2021 at 6:42 PM Gustaf Neumann <ne...@wu...> wrote:

> Dear Maksym,
>
> one can access the content of a PUT/POST request via "ns_getcontent".
> Just get the content and parse content e.g. into a dict.
>
> -gn
>
> ============================
> if {[ns_conn method] in {PUT POST}
>     && [ns_set iget [ns_conn headers] Content-Type] eq "application/json"
> } {
>     package req json
>
>     set dict [json::json2dict [ns_getcontent -as_file false]]
>     ns_return 200 text/plain $dict\n
>
> } else {
>     ad_return_complaint 1 "unsupported HTTP method: [ns_conn method]"
> }
> ============================
>
> Test:
> curl --header "Content-Type: application/json" --request POST --data '{"username":"xyz","password":"xyz"}' http://localhost:8100/json-receiver.tcl
>
>
>
>
> On 30.04.21 19:14, Maksym Zinchenko wrote:
>
> Hello, maybe it's obvious but I'm a little bit stuck in here.
> When I'm doing POST requests to Naviserver with Content-Type:
> application/x-www-form-urlencoded, I can use [ns_conn files] [ns_getform]
> to access data.
> How do I access Content-Type: application/json requests? For example if
> I'm sending JSON like that:
> {
>     "action": "list",
>     "path": "/public_html"
> }
> How do I get those values? What about Content-Type: application/xml or
> text/plain?
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>

Re: [naviserver-devel] Content-Type application/json request

[Gustaf N. <ne...@wu...>]

Dear Maksym,

one can access the content of a PUT/POST request via "ns_getcontent".
Just get the content and parse content e.g. into a dict.

-gn

============================
if {[ns_conn method] in {PUT POST}
     && [ns_set iget [ns_conn headers] Content-Type] eq "application/json"
} {
     package req json

     set dict [json::json2dict [ns_getcontent -as_file false]]
     ns_return 200 text/plain $dict\n
     
} else {
     ad_return_complaint 1 "unsupported HTTP method: [ns_conn method]"
}
============================

Test:
curl --header "Content-Type: application/json" --request POST --data '{"username":"xyz","password":"xyz"}'http://localhost:8100/json-receiver.tcl




On 30.04.21 19:14, Maksym Zinchenko wrote:
> Hello, maybe it's obvious but I'm a little bit stuck in here.
> When I'm doing POST requests to Naviserver with Content-Type: 
> application/x-www-form-urlencoded, I can use [ns_conn files] 
> [ns_getform] to access data.
> How do I access Content-Type: application/json requests? For example 
> if I'm sending JSON like that:
> {
>     "action": "list",
>     "path": "/public_html"
> }
> How do I get those values? What about Content-Type: application/xml or 
> text/plain|?|

[naviserver-devel] Content-Type application/json request

[Maksym Z. <siq...@gm...>]

Hello, maybe it's obvious but I'm a little bit stuck in here.
When I'm doing POST requests to Naviserver with Content-Type:
application/x-www-form-urlencoded, I can use [ns_conn files] [ns_getform]
to access data.
How do I access Content-Type: application/json requests? For example if I'm
sending JSON like that:
{
    "action": "list",
    "path": "/public_html"
}
How do I get those values? What about Content-Type: application/xml or
text/plain?

Re: [naviserver-devel] Problem with new pthread code: Resource temporarily unavailable

[Oscar R. F. <oro...@vr...>]

Dear Gustaf,

Thank you very much for your help.

It does work perfectly, therefore it is configuration related.

I'll work with my setup and double check everything in the config
file.

When I pinpoint the problem I'll post the solution to close the
thread.

Best regards,
Óscar

-----Mensaje original-----
De: Gustaf Neumann <ne...@wu...>
Responder a: nav...@li...
Para: nav...@li...
Asunto: Re: [naviserver-devel] Problem with new pthread code:
Resource temporarily unavailable
Fecha: Tue, 27 Apr 2021 12:12:36 +0200


  
    
  
  
    Just a quick followup: I've tested what I've suggested below with
      Debian buster (== Debian stable)

    

        root@buster# uname -a    Linux buster 4.19.0-13-amd64 #1 SMP
Debian 4.19.160-2 (2020-11-28) x86_64 GNU/Linux
    root@buster# lsb_release -d    Description:    Debian GNU/Linux
10 (buster)
    and everything looks ok 

      

      all the best

    -gn

    

    On 27.04.21 11:52, Gustaf Neumann
      wrote:

    

    
> Dear
>       Oscar,
>       
> 
>       
> 
>       Can you make please a quick check:
>       
> 
>       when you run [1] the installation says at the end:
>       
> 
>       
> 
>          You can now run plain NaviServer by typing the following
>       command:
>       
> 
>          sudo /usr/local/ns/bin/nsd -f -u nsadmin -g nsadmin -t
>       /usr/local/ns/conf/nsd-config.tcl
>       
> 
>       
> 
>       When you start this instance of naviserver, does it work?
>       
> 
>       If you, we know the libraries on your machine are fine, the
>       problem is
>       
> 
>       either your installation process or the used configuration
> file.
>       
> 
>       
> 
>       all the best
>       
> 
>       
> 
>       -gn
>       
> 
>       
> 
>       [1] https://github.com/gustafn/install-ns
>       
> 
>     

    

  

_______________________________________________naviserver-devel
mailing lis...@li...
https://lists.sourceforge.net/lists/listinfo/naviserver-devel