naviserver-devel — Developer discussion

[naviserver-devel] "connection already closed" vs "connection socket is detached" and the "rejectalreadyclosedconn" parameter

[Brian F. <bri...@ai...>]

Hi

this discussion here https://openacs.org/forums/message-view?message_id=5659253 suggests that the "rejectalreadyclosedconn" parameter will prevent the "connection socket is detached" message from showing in the logs. However, this doesn't appear to work (tried in 4.99.28). Looking at "NsConnRequire" in conn.c, it looks like "reject_already_closed_connection" is only used in the "connection already closed" case. https://bitbucket.org/naviserver/naviserver/src/b21b6d75a8f2ff41cd00dd40cb4c143d9a8a55ce/nsd/conn.c#lines-2748

It would be good to be able to turn off the "connection socket is detached" (understanding of course that we should in the meantime resolve the source of those messages). Have I misunderstood something or is this not possible?

thanks
Brian

[naviserver-devel] smtp log file

[Gustaf N. <ne...@wu...>]

Dear NaviServer users,

the updated version of nsstats supports now viewing smtp log files in a 
similar way to the HTTP client log files (see for an example the 
presentation of the last EuroTcl/OpenACS conference [1]). The SMTP sent 
log visualizes the performance, frequency and the result codes for the 
SMTP send commands, when the nssmptd module is used.

See below for a sample configuration of for the nssmtpd module with 
logging configured.

all the best

-gn


ns_section "ns/server/$server/module/nssmtpd" {
     ns_param port $smtpdport
     ns_param address $nssmtpd_address ;# 127.0.0.1
     ns_param relay localhost:25
     ns_param spamd localhost
     ns_param initproc smtpd::init
     ns_param rcptproc smtpd::rcpt
     ns_param dataproc smtpd::data
     ns_param errorproc smtpd::error
     ns_param relaydomains "localhost"
     ns_param localdomains "localhost"

     ns_param logging on ;# default: off
     ns_param logfile ${logroot}/smtpsend.log
     ns_param logrollfmt %Y-%m-%d ;# format appended to log filename
     #ns_param logmaxbackup 100 ;# 10, max number of backup log files
     #ns_param logroll true ;# true, should server log files automatically
     #ns_param logrollonsignal true ;# false, perform roll on a sighup
     #ns_param logrollhour 0 ;# 0, specify at which hour to roll
}


[1] 
https://openacs.org/conf2023/info/download/file/openacs-conf-2023-naviserver.pdf

Re: [naviserver-devel] NX and Naviserver init

[Maksym Z. <siq...@gm...>]

Hi Gustaf, thank you for your explanation and link.

On Fri, Sep 15, 2023 at 9:19 AM Gustaf Neumann <ne...@wu...> wrote:

> Dear Maksym,
> On 15.09.23 01:09, Maksym Zinchenko wrote:
>
> Hello, I've been struggling with a problem for a few days now. First I
> thought it's something with my understanding of NX, but I think its has to
> do something with Naviserver.
>
> actually, the problem is not with NaviServer, but it is rooted in the
> XOTcl/NX serializer. When a NaviServer blueprint is created, NaviServer
> performs "::Serializer all" to serialize the current workspace. Actually,
> it does not serialize the full workspace, since the workspace contains as
> well the NX internal class definitions. Therefore, the serializer has
> "ignore patterns", including everything from, e.g., the "::ns"f namespace
> [1].
>
> When objects are created with "new", these are named by default
> "::nsf::__#1", "::nsf::__#2". These names match the ignore pattern, and
> therefore, these names are excluded from the blueprint.
>
> If you create the objects in your code with the "-childof" flag (see
> below), you can place these outside the "::nsf" namespace, and everything
> should be fine
>
> Hope this helps
>
> -g
>
>
> [1]
> https://github.com/gustafn/nsf/blob/master/library/serialize/serializer.tcl#L868
>
>
>     nx::Class create B {
>         :method init {} {
>             ns_log notice "Im class [self]"
>         }
>     }
>
>     nx::Class create C {
>         :method init {} {
>             ns_log notice "Im class [self]"
>         }
>     }
>
>     nx::Class create A {
>         :property -accessor public inBObj:object
>         :property -accessor public inCObj:object
>
>         :method init {} {
>             set :inBObj [B new -childof [::nsf::current object]]
>             set :inCObj [C new -childof [::nsf::current object]]
>         }
>
>         :public method myObjs {} {
>             ns_log notice  "My B obj is: [${:inBObj} info name]"
>             ns_log notice  "My C obj is: [${:inCObj} info name]"
>         }
>     }
>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>

Re: [naviserver-devel] NX and Naviserver init

[Gustaf N. <ne...@wu...>]

Dear Maksym,

On 15.09.23 01:09, Maksym Zinchenko wrote:
> Hello, I've been struggling with a problem for a few days now. First I 
> thought it's something with my understanding of NX, but I think its 
> has to do something with Naviserver.

actually, the problem is not with NaviServer, but it is rooted in the 
XOTcl/NX serializer. When a NaviServer blueprint is created, NaviServer 
performs "::Serializer all" to serialize the current workspace. 
Actually, it does not serialize the full workspace, since the workspace 
contains as well the NX internal class definitions. Therefore, the 
serializer has "ignore patterns", including everything from, e.g., the 
"::ns"f namespace [1].

When objects are created with "new", these are named by default 
"::nsf::__#1", "::nsf::__#2". These names match the ignore pattern, and 
therefore, these names are excluded from the blueprint.

If you create the objects in your code with the "-childof" flag (see 
below), you can place these outside the "::nsf" namespace, and 
everything should be fine

Hope this helps

-g


[1] 
https://github.com/gustafn/nsf/blob/master/library/serialize/serializer.tcl#L868


     nx::Class create B {
         :method init {} {
             ns_log notice "Im class [self]"
         }
     }

     nx::Class create C {
         :method init {} {
             ns_log notice "Im class [self]"
         }
     }

     nx::Class create A {
         :property -accessor public inBObj:object
         :property -accessor public inCObj:object

         :method init {} {
             set :inBObj [B new -childof [::nsf::current object]]
             set :inCObj [C new -childof [::nsf::current object]]
         }

         :public method myObjs {} {
             ns_log notice  "My B obj is: [${:inBObj} info name]"
             ns_log notice  "My C obj is: [${:inCObj} info name]"
         }
     }

[naviserver-devel] NX and Naviserver init

[Maksym Z. <siq...@gm...>]

Hello, I've been struggling with a problem for a few days now. First I
thought it's something with my understanding of NX, but I think its has to
do something with Naviserver.
I have a global library named "oodz" in my tcl folder, insideI have some
subfolders with my NX classes. I have an init.tcl file inside. According to
documentation this "init.tcl" file executed first. First thing I do is
looping through my subfolders and sourcing my NX Classes. It works fine as
expected. Next Im creating some instances of classes I need inside
"init.tcl"
I made a simple example, to show: I have an NX class like that:

nx::Class create b {
> :method init {} {
> :say
> }
>
> :public method say {} {
> puts "Im class B"
> }
> }
>
> nx::Class create c {
> :method init {} {
> :say
> }
>
> :public method say {} {
> puts "Im class C"
> }
> }
>
> nx::Class create a {
> :property -accessor public inBObj:object
> :property -accessor public inCObj:object
>
> :method init {} {
> set :inBObj [b new]
> set :inCObj [c new]
> }
>
> :public method myObjs {} {
> puts "My B obj is: [${:inBObj} info name]"
> puts "My C obj is: [${:inCObj} info name]"
> }
> }
>

In my "init.tcl" I have line: a create aObj

Here's what's happenin:
1. I can see in the server log that aObj was created, I see line:
Im class B, Im class C

2. If I open nsshell console, and run aObj cget:
aObj cget -inBObj
::nsf::__#3
aObj cget -inCObj
::nsf::__#4

3. But when I try to run myObjs, I get an error:
aObj myObjs
invalid command name "::nsf::__#3"

The same code works fine in tcl shell, is it related to Naviserver? Or am I
doing something wrong with NX?

Thank you

Re: [naviserver-devel] NaviServer 4.99.28 available

[Gustaf N. <ne...@wu...>]

Dear all,

Part of the announcement of NaviServer 4.99.28 was the update of the 
nsdbbdb module (Berkley DB driver via nsdb). I did a few tests about its 
performance, that might interest a few here.

The test of this module configured with lmdb is quite impressive (see 
below). For comparison, there are also accesses to nsv, ns_cache and 
similar included. All timings are from my notebook (Apple Silicon M1). 
This is just a micro-benchmark and measures buffered access through the 
NaviServer DB driver infrastructure. More detailed information about 
LMDB in comparison to Berkley DB concerning large Applications and more 
micro-benchmarks is in [1]

all the best

-gb

183 ns nsv_set foo x 1; time {nsv_get foo x} 100000 213 ns time 
{ns_cache_eval ns:memoize 1 {set x 1}} 100000 253 ns time {array set x 
{a 1 b 2 c 3}} 100000 308 ns ns_urlspace set -key foo1 /*.adp A; time 
{ns_urlspace get -key foo1 /static/test.adp} 100000 >>> 511 ns set ::db 
[ns_db gethandle lmdb]; time {ns_db 0or1row $::db "GET key1"} 100000 
3633 ns time {parameter::get -package_id [ad_conn subsite_id] -parameter 
DefaultMaster -default "x"} 100000 36893 ns time {xo::dc get_value 
dbqd..qn {select title from acs_objects where object_id=221}} 100000 
42523 ns time {::mongo::collection::query $::mongoColl {name string 
Gustaf}} 100000
Times are in nano seconds, the "slowest" is here the access to MongoDB (via nsf package) and PostgreSQL.
The PostgreSQL access takes 36 micro seconds, this means, one can run ~27K of those in a second.
But with LMDB it is possible to gets nearly 2 Mio DB accesses per second.

[1]http://www.lmdb.tech/media/20130329-devox-MDB.pdf

[naviserver-devel] mobile device friendlyness and sourceforge problems

[Gustaf N. <ne...@wu...>]

Dear all,

For all people having troubles to download the new release NaviServer 
from sourceforge: there seems to be some outages today:

https://downdetector.com/status/sourceforge/

One has to patient, i am sure the good guys on sourceforge working in 
the background are busy right now.

As mentioned a couple of times, fixes are currently added to the 
release/4.99 branch, and new features are added to the main branch, 
which has the goal to be released, when Tcl 9 is out.

All push operations on NaviServer trigger a regression test, so the code 
quality can always be checked on:

https://github.com/nm-wu/naviserver-mirror/actions

Today, I've committed a change to improve the mobile friendliness of 
NaviServer in the main branch, which makes the output readable for 
humans on mobile devices. This is not terribly important, since every 
larger application that cares provides it own templating, but 
nevertheless, sometime the results of "ns_returnnotice" pop up, such as 
e.g. in the directory listings. The new version is based on a flexible 
design based on flexbox. See below for a comparison (two screenshots for 
a display on a recent iPhone).

all the best

-gn

PS: On the longer range, we should provide a fully templated approach 
for "ns_returnnotice"

old:

new





-- 
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"

[naviserver-devel] NaviServer 4.99.28 available

[Gustaf N. <ne...@wu...>]

Dear all,

I am glad to announce that the release of NaviServer 4.99.28 is 
available at SourceForge [1].  This release is a pure bug-fix and 
maintenance release, which fixes a bug annoying for some OpenACS users.
See below for a summary of the changes.

All the best!

-gustaf neumann

[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.28/

=======================================
NaviServer 4.99.28, released 2023-09-06
=======================================

  49 files changed, 851 insertions(+), 328 deletions(-)

New Features:
-------------
   - None

Bug Fixes:
----------

   - Fixed incorrect list quoting for "filter" in urlspace
     introspection. This lead to display problems, e.g. in the nsstats
     module.

   - Fixed naming Inconsistency: Unified the term "fastpath" across
     configuration, API-interface, and documentation.

   - Fixed potential crash in Ns_SetIUpdateSz(): Fixed a bug that could
     lead to a crash when updating the content-type header field with a
     lowercase field name via the old C API.

     Many thanks to Brian Fenton for reporting the problem and his help
     for debugging it.

Code Maintenance:
-----------------

   - New API Function Ns_PlainUrlPath: Checks if a given URL path is
     "plain," meaning it doesn't contain a query or fragment. Useful
     for data validation in configuration files and Tcl API calls.

   - Improved Validation when registering request handler. Previously,
     it was possible to register request handlers for paths containing
     query variables and fragments, which were never matched.

   - Added Missing "-server" Argument: The "-server" argument was added
     to ns_unregister_op and ns_unregister_url2file, allowing for more
     centralized management (e.g. from the nsstats module).

   - Added a more Portable Way of Printing NULL Pointer: Improved
     portability for printing NULL pointers across different compilers,
     which could lead to false positives in the regression test.

   - Extended and Simplified Test Cases: Several commits were made to
     improve and extend test cases across different modules and
     scenarios.

   - Improved and corrected documentation, including styling changes,
     bug fixes, and providing more detailed information.


Modules:
--------
The following list contains the most important changes:

  - Extending nsstats:

    * Introduced templating to improve style-ability
    * Provided a more ,odern GUI based on the CSS classes from w3schools
    * Added management of request handlers
    * Added management of url2file mappings

  - Rework of module nsdbbdb:
     * properly initializing mutex variables (many thanks to John Buckman)
     * support for LMDB (Lightning Memory-Mapped Database)
     * Improved configurability for Makefile
     * Modernization of C code (better following style guide)
     * Added debug support via Debug(bdb)

      Build nsdbbdb with Berkeley DB 6.2.32 (results in nsdbbdb.so)
         make DBINCLUDE=/opt/local/include/db62  DBLIB=/opt/local/lib/db62/

      Build nsdbbdb with LMDB (Lightning Memory-Mapped Database) (results in nsdblmdb.so)
         make LMDB=1 DBINCLUDE=/opt/local/include DBLIB=/opt/local/lib/

Re: [naviserver-devel] “worker process pool” feature?

[Gustaf N. <ne...@wu...>]

On 05.09.23 14:40, John at Decent wrote:
> Gustav, I think my confusion is as to how this works. As it's not 
> documented, one needs to figure it out from the source.

the man page 
https://naviserver.sourceforge.io/n/nsproxy/files/ns_proxy.html is not a 
perfect documentation, but this is what we have now, ... and this is 
certainly more than reading the source. Improvements are always welcome!

> I can't really tell what each ns_proxy::exec does if the executed 
> process doesn't exit.

a worker process is potentially started by "ns_proxy get ..." but is 
influenced by the definition of the proxy pool (e.g. "maxworkers", ...).

> Does it reuse it for the next request? It seems so, but I'm not clear.

by "request" you probably mean the next invocation of "nsproxy eval". It 
is reused, but as before, the reuse is controlled by the pool definition 
(e.g. "idletimeout", "maxruns", ...)

> If it does reuse the still-running process, how does it tell it’s the 
> same executable?

the name of the executable (per default 
/usr/local/ns/bin/nsproxy-helper) is defined via config file or via  
"nsproxy configure ... -exec XXX". The running instances are managed via 
the handles. If you want to have different executables, define different 
nsproxy pools.

-g

Re: [naviserver-devel] =?utf-8?Q?=E2=80=9Cworker_?=process =?utf-8?Q?pool=E2=80=9D_?=feature?

[John at D. <jo...@de...>]

Gustav, I think my confusion is as to how this works. As it's not documented, one needs to figure it out from the source.

I can't really tell what each ns_proxy::exec does if the executed process doesn't exit. 

Does it reuse it for the next request? It seems so, but I'm not clear.  If it does reuse the still-running process, how does it tell it’s the same executable?

-john

Re: [naviserver-devel] “worker process pool” feature?

[Gustaf N. <ne...@wu...>]

On 05.09.23 11:39, Gustaf Neumann wrote:
> On 05.09.23 09:17, John at Decent wrote:
>>
>> However, won’t that cause the external process to get reloaded every 
>> time I call proxy::exec ?
> see
>
> https://github.com/openacs/openacs-core/blob/oacs-5-10/packages/acs-tcl/tcl/proxy-procs.tcl#L74-L83
>
> in the link i have sent in my last mail
>
Maybe there is some confusion with the term "reloaded"
Check also the "-maxuns" parameter...
Or do you want to have - technically speaking - one fork per request? ....

-g

Re: [naviserver-devel] “worker process pool” feature?

[Gustaf N. <ne...@wu...>]

On 05.09.23 09:17, John at Decent wrote:
> Thanks Gustaf.
>
> I knew about ns_proxy, but as it is documented as being a proxy to 
> external Tcl processes, I didn’t think to use it as a general process 
> pool.
>
> The part I was missing was
> > proxy::exec
>
> However, won’t that cause the external process to get reloaded every 
> time I call proxy::exec ?
see

https://github.com/openacs/openacs-core/blob/oacs-5-10/packages/acs-tcl/tcl/proxy-procs.tcl#L74-L83

in the link i have sent in my last mail

-g

Re: [naviserver-devel] =?utf-8?Q?=E2=80=9Cworker_?=process =?utf-8?Q?pool=E2=80=9D_?=feature?

[John at D. <jo...@de...>]

I'm currently looking into bringing FastCGI support to Naviserver. In principle, FastCGI is specified to support a pool of backends. For your case, one would have to implement a FastCGI server, maybe directly in undroidwish, which does the marshaling to the respective Tcl scripts.  Would this suit your requirements?

Yes, FastCGI is very similar to what I’m describing, and should work.



However, I’m thinking a bit more about ns_proxy and wondering if:

- a given Tcl interpreter launches a new process via exec, connects to its STDIN/STDOUT

- feeds the input to the process, returns its output

- but leaves the process running

- so that next time a request comes in, the proxy proc first checks to see if it has a running “exec” external process, and if so, just re-uses it.



If that works, then it’s very similar to FastCGI, and likely easier to implement, as it’s all Tcl, and not much of it.  The code from

https://github.com/openacs/openacs-core/blob/oacs-5-10/packages/acs-tcl/tcl/proxy-procs.tcl

would just need to be modified a bit.

Re: [naviserver-devel] “worker process pool” feature?

[Georg L. <jor...@ma...>]

Hello,

I'm currently looking into bringing FastCGI support to Naviserver. In 
principle, FastCGI is specified to support a pool of backends. For your 
case, one would have to implement a FastCGI server, maybe directly in 
undroidwish, which does the marshaling to the respective Tcl scripts.  
Would this suit your requirements?

Best Regards,

   Georg

On 9/5/23 08:10, John at Decent wrote:
> I was talking to Christian Werner about how to have undroidwish work 
> well with Naviserver, allowing Tcl scripts to run in undroidwish, and 
> return their results to a calling Naviserver page.  Why do this? 
> Perhaps undroidwish loads libraries that are not (easily) available 
> under naviserver?
>
> I think the best way is to have a naviserver module that implements a 
> “worker pool of processes”.  The module would launch X numbers of this 
> external process (say, undroidwish), controlled via STDOUT/STDIN, and 
> dispatch requests free members of the pool, returning the result to 
> the caller.  This would be an efficient way to have slow-loading Unix 
> processes brought into Naviserver efficiently.
>
> For some reason, I imagined that this feature existed already, years 
> ago, but I can’t find a mention of it.  Certainly, there is a lot in 
> Naviserver that resembles this idea already.
>
> -john
>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

-- 
MagmaSoft e.U. (FN 601643w)
Inhaber: Ing. Georg Lehner
Kolpingstraße 15, A-4020 Linz
https://magma-soft.at/
mailto:jo...@ma...

Re: [naviserver-devel] =?utf-8?Q?=E2=80=9Cworker_?=process =?utf-8?Q?pool=E2=80=9D_?=feature?

[John at D. <jo...@de...>]

Thanks Gustaf. 

I knew about ns_proxy, but as it is documented as being a proxy to external Tcl processes, I didn’t think to use it as a general process pool.

The part I was missing was
> proxy::exec 

However, won’t that cause the external process to get reloaded every time I call proxy::exec ?

If I’m running a fairly heavy Unix process, this doesn’t seem efficient.

Each run of the external unix process would cause a new run of exec, and thus a new process loading:
> set return_string [ns_proxy eval $handle [list ::exec {*}$exec_flags {*}$call]]

Re: [naviserver-devel] “worker process pool” feature?

[Gustaf N. <ne...@wu...>]

On 05.09.23 08:10, John at Decent wrote:
> I think the best way is to have a naviserver module that implements a 
> “worker pool of processes”.  The module would launch X numbers of this 
> external process (say, undroidwish), controlled via STDOUT/STDIN, and 
> dispatch requests free members of the pool, returning the result to 
> the caller.  This would be an efficient way to have slow-loading Unix 
> processes brought into Naviserver efficiently.

The NaviServer module nsproxy does exactly this, i.e. running processes 
in the background talking to NaviServer via pipes, providing queues, 
cancellation, ..... The processes, which are executed via the nsproxy 
module, are the "nsproxy-helpers", which are executing the incoming Tcl 
commands and return the results back via pipe, such that for a caller, 
it looks like a local "exec". The connection to the workers is 
established via handles.

See [1] for an example, how nsproxy is used for implemented "exec" in 
NaviServer, see [2] for the nsproxy API and configuration options.

-g

[1] 
https://github.com/openacs/openacs-core/blob/oacs-5-10/packages/acs-tcl/tcl/proxy-procs.tcl
[2] https://naviserver.sourceforge.io/n/nsproxy/files/ns_proxy.html

[naviserver-devel] =?utf-8?Q?=E2=80=9Cworker_?=process =?utf-8?Q?pool=E2=80=9D_?=feature?

[John at D. <jo...@de...>]

I was talking to Christian Werner about how to have undroidwish work well with Naviserver, allowing Tcl scripts to run in undroidwish, and return their results to a calling Naviserver page.  Why do this? Perhaps undroidwish loads libraries that are not (easily) available under naviserver?  

I think the best way is to have a naviserver module that implements a “worker pool of processes”.  The module would launch X numbers of this external process (say, undroidwish), controlled via STDOUT/STDIN, and dispatch requests free members of the pool, returning the result to the caller.  This would be an efficient way to have slow-loading Unix processes brought into Naviserver efficiently.

For some reason, I imagined that this feature existed already, years ago, but I can’t find a mention of it.  Certainly, there is a lot in Naviserver that resembles this idea already.

-john

Re: [naviserver-devel] Crashing on all versions >4.99.24 on Ubuntu

[Brian F. <bri...@ai...>]

Hi Gustaf

thanks for the explanation of the cause of the issue.

I have forked the docker image from oupfiz5, and changed it to run 4.99.27, which was how I discovered the issue. That Dockerfile relies on the existence of the official naviserver-"${NS_VERSION}".tar.gz and naviserver-"${NS_MODULES_VERSION}"-modules.tar.gz files, so I can't just easily patch in this fix.

We have no deadline for a new release. For now, we can continue to use 4.99.24 in our testing, but obviously we would like to explain to clients why we're not running the latest version. If we can tell them e.g. there will soon be a new official release which includes the ns_set bugfix, that will keep them happy.

cheers
Brian

________________________________
From: Gustaf Neumann <ne...@wu...>
Sent: Friday 18 August 2023 7:51 pm
To: nav...@li... <nav...@li...>
Subject: Re: [naviserver-devel] Crashing on all versions >4.99.24 on Ubuntu



> FYI, also the "ns_set put" crashes the server e.g. if I run this in /ds/shell:


it is clear. The problem were not the "ns_set update" or the "ns_set put" commands itself, but the fact, that the key "content-type" was in lower-case and the normalization of the output headers was causing the crash. Some ages ago, it was possible to set two different header values with upper and lower case keys, leading to some unspecified behaviors in browsers.



On 18.08.23 15:55, Brian Fenton wrote:
Hi again Gustaf

Great news, this seems to have worked according to all my tests. I am assuming that I built this correctly with the "version_ns=GIT" parameter:
version_ns=GIT ns_modules="nsconf nsstats" with_postgres=0 with_postgres_driver=0 bash install-ns.sh build


by your parameter list, you will get the newest version from git.... which is the unreleased version of NaviServer 5. If one specifies the branch as well, different versions can be installed from git. E.g.,

    ... version_ns=GIT git_branch_ns=release/4.99 ....


installs the newest version from the release/4.99 branch, what you probably want.
But one can use as well arbitrary tags. For example

    ... version_ns=GIT git_branch_ns=naviserver-4.99.16  ....


will checkout the released version of NaviServer 4.99.16 via git.


Do you have an idea when this will be included in an official Naviserver release? Our Docker build relies on the official releases, so it would be good to have a timeframe on that.


When you use the docker image from oupfiz5, these are still based on 4.99.24 (released on 2022-06-14, a year ago).


Anyhow, when do you need a release?


all the best

-g


thanks again for the amazing support,
Brian

________________________________
From: Gustaf Neumann <ne...@wu...><mailto:ne...@wu...>
Sent: Friday 18 August 2023 12:30 pm
To: nav...@li...<mailto:nav...@li...> <nav...@li...><mailto:nav...@li...>
Subject: Re: [naviserver-devel] Crashing on all versions >4.99.24 on Ubuntu

Hi Brian,

With your input, I could locate the source of the problem and fix this
in the repository. It was an ns_set but, that could only happen with the
output headers, when the code normalized the capitalization of the
header fields. There is now a test for this in the regression test suite
of NaviServer.

Many thanks for your patience and input!

-g

https://bitbucket.org/naviserver/naviserver/commits/4f8cf8a548bc60f88756a30f33f6a5b589fc6997





_______________________________________________
naviserver-devel mailing list
nav...@li...<mailto:nav...@li...>
https://lists.sourceforge.net/lists/listinfo/naviserver-devel




_______________________________________________
naviserver-devel mailing list
nav...@li...<mailto:nav...@li...>
https://lists.sourceforge.net/lists/listinfo/naviserver-devel


--
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"

[naviserver-devel] updates for nsstats

[Gustaf N. <ne...@wu...>]

Dear all,

i've added some improvements to the nsstats module on Bitbucket,
such as

- Basic ability to style appearance:
   * switched from hard-coded styling to a page template based on ADP
   * added optional ADP template based on the CSS classes from
https://www.w3schools.com/w3css/
   * When the optional ADP template is not available, fall back
     to the old styling.

- Structured main menu
   * use a hierarchy of commands instead of the flat alphabetical list
   * hierarchy is used in the menu-bar of the new template

- New page for registered request handlers. This should improve
   transparency, how requests are resolved.

Below are a few screenshots of a minimal installation (plain NaviServer)

All the best
-gustaf


a) start page of nsstats with the new template
- showing menubar (makes old index page mostly obsolete)
- shows "process" page per default
- additional line per server for Request handlers (here 15 handlers are 
registered)


b) request handlers page (with the new template)
- provide a filter for HTTP methods
- provide button for unregistering the handler
- showing registered ADP pages, Tcl pages, fastpath pages and CGI pages.


c) start page without the new template

Re: [naviserver-devel] Crashing on all versions >4.99.24 on Ubuntu

[Gustaf N. <ne...@wu...>]

 > FYI, also the "ns_set put" crashes the server e.g. if I run this in 
/ds/shell:


it is clear. The problem were not the "ns_set update" or the "ns_set 
put" commands itself, but the fact, that the key "content-type" was in 
lower-case and the normalization of the output headers was causing the 
crash. Some ages ago, it was possible to set two different header values 
with upper and lower case keys, leading to some unspecified behaviors in 
browsers.



On 18.08.23 15:55, Brian Fenton wrote:
> Hi again Gustaf
>
> Great news, this seems to have worked according to all my tests. I am 
> assuming that I built this correctly with the "version_ns=GIT" parameter:
> version_ns=GIT ns_modules="nsconf nsstats" with_postgres=0 
> with_postgres_driver=0 bash install-ns.sh build


by your parameter list, you will get the newest version from git.... 
which is the unreleased version of NaviServer 5. If one specifies the 
branch as well, different versions can be installed from git. E.g.,

     ... version_ns=GIT git_branch_ns=release/4.99 ....

installs the newest version from the release/4.99 branch, what you 
probably want.
But one can use as well arbitrary tags. For example

     ... version_ns=GIT git_branch_ns=naviserver-4.99.16  ....

will checkout the released version of NaviServer 4.99.16 via git.


> Do you have an idea when this will be included in an official 
> Naviserver release? Our Docker build relies on the official releases, 
> so it would be good to have a timeframe on that.


When you use the docker image from oupfiz5, these are still based on 
4.99.24 (released on 2022-06-14, a year ago).


Anyhow, when do you need a release?


all the best

-g


>
> thanks again for the amazing support,
> Brian
>
> ------------------------------------------------------------------------
> *From:* Gustaf Neumann <ne...@wu...>
> *Sent:* Friday 18 August 2023 12:30 pm
> *To:* nav...@li... 
> <nav...@li...>
> *Subject:* Re: [naviserver-devel] Crashing on all versions >4.99.24 on 
> Ubuntu
> Hi Brian,
>
> With your input, I could locate the source of the problem and fix this
> in the repository. It was an ns_set but, that could only happen with the
> output headers, when the code normalized the capitalization of the
> header fields. There is now a test for this in the regression test suite
> of NaviServer.
>
> Many thanks for your patience and input!
>
> -g
>
> https://bitbucket.org/naviserver/naviserver/commits/4f8cf8a548bc60f88756a30f33f6a5b589fc6997 
> <https://bitbucket.org/naviserver/naviserver/commits/4f8cf8a548bc60f88756a30f33f6a5b589fc6997>
>
>
>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel 
> <https://lists.sourceforge.net/lists/listinfo/naviserver-devel>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

-- 
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"

Re: [naviserver-devel] Crashing on all versions >4.99.24 on Ubuntu

[Brian F. <bri...@ai...>]

Hi again Gustaf

Great news, this seems to have worked according to all my tests. I am assuming that I built this correctly with the "version_ns=GIT" parameter:
version_ns=GIT ns_modules="nsconf nsstats" with_postgres=0 with_postgres_driver=0 bash install-ns.sh build

Do you have an idea when this will be included in an official Naviserver release? Our Docker build relies on the official releases, so it would be good to have a timeframe on that.

thanks again for the amazing support,
Brian

________________________________
From: Gustaf Neumann <ne...@wu...>
Sent: Friday 18 August 2023 12:30 pm
To: nav...@li... <nav...@li...>
Subject: Re: [naviserver-devel] Crashing on all versions >4.99.24 on Ubuntu

Hi Brian,

With your input, I could locate the source of the problem and fix this
in the repository. It was an ns_set but, that could only happen with the
output headers, when the code normalized the capitalization of the
header fields. There is now a test for this in the regression test suite
of NaviServer.

Many thanks for your patience and input!

-g

https://bitbucket.org/naviserver/naviserver/commits/4f8cf8a548bc60f88756a30f33f6a5b589fc6997





_______________________________________________
naviserver-devel mailing list
nav...@li...
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] Crashing on all versions >4.99.24 on Ubuntu

[Brian F. <bri...@ai...>]

Wow, that's a quick turnaround, thank you so much Gustaf! I will test out your fix.

FYI, also the "ns_set put" crashes the server e.g. if I run this in /ds/shell:
ns_set put [ad_conn outputheaders] content-type "text/html"

many thanks
Brian

________________________________
From: Gustaf Neumann <ne...@wu...>
Sent: Friday 18 August 2023 12:30 pm
To: nav...@li... <nav...@li...>
Subject: Re: [naviserver-devel] Crashing on all versions >4.99.24 on Ubuntu

Hi Brian,

With your input, I could locate the source of the problem and fix this
in the repository. It was an ns_set but, that could only happen with the
output headers, when the code normalized the capitalization of the
header fields. There is now a test for this in the regression test suite
of NaviServer.

Many thanks for your patience and input!

-g

https://bitbucket.org/naviserver/naviserver/commits/4f8cf8a548bc60f88756a30f33f6a5b589fc6997





_______________________________________________
naviserver-devel mailing list
nav...@li...
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] Crashing on all versions >4.99.24 on Ubuntu

[Gustaf N. <ne...@wu...>]

Hi Brian,

With your input, I could locate the source of the problem and fix this 
in the repository. It was an ns_set but, that could only happen with the 
output headers, when the code normalized the capitalization of the 
header fields. There is now a test for this in the regression test suite 
of NaviServer.

Many thanks for your patience and input!

-g

https://bitbucket.org/naviserver/naviserver/commits/4f8cf8a548bc60f88756a30f33f6a5b589fc6997

Re: [naviserver-devel] Crashing on all versions >4.99.24 on Ubuntu

[Brian F. <bri...@ai...>]

Hi Gustaf

thanks to your help, I was able to track down the source of the problem: it was a TCL/ADP pair that generates some Javascript. This gets called in the blank master and so was breaking on every page request. Once I removed it from the blank master I was able to access the site. After looking into further, I discovered a few more places where the same problem occurs: a few more Javascript pages generated from TCL/ADP pairs and also a file download TCL page.

What all of these have in common are calls like this - removing these lines prevents the crash in each case:
 ns_set update [ns_conn outputheaders] content-type "application/javascript"
 ns_set update [ns_conn outputheaders] content-type "$mime_type"

I created a simple test case which reproduces the problem on our OpenACS system i.e browsing to this page immediately crashes Naviserver.
However the same test case runs fine without crashing on the simple nsd-config.tcl system.
It also runs fine on a fresh OpenACS Oracle installation - this again suggests an issue specific to our application.

I'm not sure how much deeper I can go here - would be grateful for your thoughts.

This is my test case - if I create a file called test.adp and browse to it, Naviserver crashes.
<html>
<body>
   Welcome to NaviServer
   <%=[ns_info patchlevel]%> under
   <%=[set . "$::tcl_platform(os) $::tcl_platform(osVersion)"]%>
   <%=[ns_set update [ns_conn outputheaders] content-type "text/html"]%>
 </body>
</html>

regards
Brian

________________________________
From: Gustaf Neumann <ne...@wu...>
Sent: Wednesday 16 August 2023 7:31 pm
To: Navidevel <nav...@li...>
Subject: Re: [naviserver-devel] Crashing on all versions >4.99.24 on Ubuntu

On 16.08.23 15:37, Brian Fenton wrote:
We don't issue any "ns_set cleanup" ourselves, but when I added your recommendation of tracing "ns_set", I saw plenty in the logs. The docs say that "This command is autoamtically executed by ns_cleanup, which runs after every request, freeing all sets created via ns_set", so I presume that is where the calls come from.

Hi Brian,

Yes, it is clear from the log that the crash happens in the automatic cleanup, but I just wanted to make sure, that the application has not tried the same earlier - this would be a good place to start debugging.

Is there something in particular I should be looking for in the trace output? There is an enormous amount of information the logs.

see below


I ran another backtrace and this time the error was a little bit different
as i said before, the messages one gets from the operating system are not always helpful to find the source of the problem.

("error: Cannot access memory at address 0x2" - see below).

This is actually not a message of the operating system, but a message from gdb about an unused C-structure, which is here not relevant.

... but we are getting closer.  The crash happens on an dynamic ns_set (in yesterday's mail in "d6").

#11 0x00007ffff7f3e119 in NsTclSetObjCmd
(clientData=0x7fffdc035570, interp=0x7fffdc005250, objc=2,
objv=0x7fffdc245650) at tclset.c:330

        key = 0x7fffdc3aa070 "d6"

In this case, one has to watch out for operations on "d6". You might ask "What is a dynamic ns_set?".

Background: ns_sets might be

- C-only structures, or
- Tcl-exposed structures, which might be
   * dynamic (i.e., freed after every request, the Tcl-name of these starts with a "d"), or
   * static (the structures will be reused).

The Tcl-exposed ns_sets are "entered" (NaviServer terminology) into Tcl interpreters. It might be the case, that you do not find anything interesting in the log concerning the crashing ns_set (here "d6") in the Tcl trace output, ... since the ns_sets are as well manipulated from C.

In case, you find nothing revealing on "d6", I've added more debugging output for C-level ns_set operations. Get the newest version from the branch "release/4.99", activate debug output on Debug(nsset)

    ns_logctl severity Debug(nsset) on

and then you will see some more output, like e.g. the following:

[16/Aug/2023:19:53:13.016627][-conn:oacs-head:default:0:11-] Debug(nsset): Ns_SetPut 0x600002845600 [29] key 'item_id' value '1066' size -1
[16/Aug/2023:19:53:13.016630][-conn:oacs-head:default:0:11-] Debug(nsset): Ns_SetPut 0x600002845600 [30] key 'revision_id' value '163776' size -1
[16/Aug/2023:19:53:13.016633][-conn:oacs-head:default:0:11-] Debug(nsset): Ns_SetPut 0x600002845600 [31] key 'publish_date' value '2021-11-13 13:35:27.423904+01' size -1
[16/Aug/2023:19:53:13.016637][-conn:oacs-head:default:0:11-] Debug(nsset): Ns_SetPut 0x600002845600 [32] key 'modifying_user' value '' size -1
[16/Aug/2023:19:53:13.016640][-conn:oacs-head:default:0:11-] Debug(nsset): Ns_SetPut 0x600002845600 [33] key 'last_modified' value '2017-08-08 13:26:49.138414+02' size -1
[16/Aug/2023:19:53:13.016644][-conn:oacs-head:default:0:11-] Debug(nsset): Ns_SetPut 0x600002845600 [34] key 'modifying_ip' value '::1' size -1
[16/Aug/2023:19:53:13.016647][-conn:oacs-head:default:0:11-] Debug(nsset): Ns_SetPut 0x600002845600 [35] key 'modifying_user' value '704' size -1
[16/Aug/2023:19:53:13.016651][-conn:oacs-head:default:0:11-] Debug(nsset): Ns_SetPut 0x600002845600 [36] key 'parent_id' value '1064' size -1
[16/Aug/2023:19:53:13.016655][-conn:oacs-head:default:0:11-] Debug(nsset): EnterSet 0x600002845600 with name 'd15'
[16/Aug/2023:19:53:13.016678][-conn:oacs-head:default:0:11-] Notice: trace: ns_set array d15

...
[16/Aug/2023:20:04:31.283342][-conn:oacs-head:default:0:11-] Debug(nsset): ns_set cleanup key <d15> dynamic 1
[16/Aug/2023:20:04:31.283344][-conn:oacs-head:default:0:11-] Debug(nsset): Ns_SetFree 0x600002845600 'db': elements 37
[16/Aug/2023:20:04:31.283346][-conn:oacs-head:default:0:11-] Debug(nsset): ... 0: key <object_type> value <::xowiki::Form>
[16/Aug/2023:20:04:31.283349][-conn:oacs-head:default:0:11-] Debug(nsset): ... 1: key <package_id> value <1060>
[16/Aug/2023:20:04:31.283351][-conn:oacs-head:default:0:11-] Debug(nsset): ... 2: key <creation_user> value <704>
...



In this example output, there is a bunch of Ns_SetPut() operations on a set "0x600002845600"  which is entered then to Tcl as set "d15".... somewhere later it is freed via "ns_set cleanup".

Let me know if you need some help on getting/compiling a branch of NaviServer from Bitbucket.

all the best

-g

Re: [naviserver-devel] Crashing on all versions >4.99.24 on Ubuntu

[Gustaf N. <ne...@wu...>]

On 16.08.23 15:37, Brian Fenton wrote:
> We don't issue any "ns_set cleanup" ourselves, but when I added your 
> recommendation of tracing "ns_set", I saw plenty in the logs. The docs 
> say that "This command is autoamtically executed by ns_cleanup, which 
> runs after every request, freeing all sets created via *ns_set*", so I 
> presume that is where the calls come from.

Hi Brian,

Yes, it is clear from the log that the crash happens in the automatic 
cleanup, but I just wanted to make sure, that the application has not 
tried the same earlier - this would be a good place to start debugging.

> Is there something in particular I should be looking for in the trace 
> output? There is an enormous amount of information the logs.

see below


> I ran another backtrace and this time the error was a little bit 
> different
as i said before, the messages one gets from the operating system are 
not always helpful to find the source of the problem.

> ("error: Cannot access memory at address 0x2" - see below).

This is actually not a message of the operating system, but a message 
from gdb about an unused C-structure, which is here not relevant.

... but we are getting closer.  The crash happens on an dynamic ns_set 
(in yesterday's mail in "d6").

#11 0x00007ffff7f3e119 in NsTclSetObjCmd
(clientData=0x7fffdc035570, interp=0x7fffdc005250, objc=2,
objv=0x7fffdc245650) at tclset.c:330

         key = 0x7fffdc3aa070 "d6"

In this case, one has to watch out for operations on "d6". You might ask 
"What is a dynamic ns_set?".

Background: ns_sets might be

- C-only structures, or
- Tcl-exposed structures, which might be
    * dynamic (i.e., freed after every request, the Tcl-name of these 
starts with a "d"), or
    * static (the structures will be reused).

The Tcl-exposed ns_sets are "entered" (NaviServer terminology) into Tcl 
interpreters. It might be the case, that you do not find anything 
interesting in the log concerning the crashing ns_set (here "d6") in the 
Tcl trace output, ... since the ns_sets are as well manipulated from C.

In case, you find nothing revealing on "d6", I've added more debugging 
output for C-level ns_set operations. Get the newest version from the 
branch "release/4.99", activate debug output on Debug(nsset)

     ns_logctl severity Debug(nsset) on

and then you will see some more output, like e.g. the following:

[16/Aug/2023:19:53:13.016627][-conn:oacs-head:default:0:11-] Debug(nsset): Ns_SetPut 0x600002845600 [29] key 'item_id' value '1066' size -1
[16/Aug/2023:19:53:13.016630][-conn:oacs-head:default:0:11-] Debug(nsset): Ns_SetPut 0x600002845600 [30] key 'revision_id' value '163776' size -1
[16/Aug/2023:19:53:13.016633][-conn:oacs-head:default:0:11-] Debug(nsset): Ns_SetPut 0x600002845600 [31] key 'publish_date' value '2021-11-13 13:35:27.423904+01' size -1
[16/Aug/2023:19:53:13.016637][-conn:oacs-head:default:0:11-] Debug(nsset): Ns_SetPut 0x600002845600 [32] key 'modifying_user' value '' size -1
[16/Aug/2023:19:53:13.016640][-conn:oacs-head:default:0:11-] Debug(nsset): Ns_SetPut 0x600002845600 [33] key 'last_modified' value '2017-08-08 13:26:49.138414+02' size -1
[16/Aug/2023:19:53:13.016644][-conn:oacs-head:default:0:11-] Debug(nsset): Ns_SetPut 0x600002845600 [34] key 'modifying_ip' value '::1' size -1
[16/Aug/2023:19:53:13.016647][-conn:oacs-head:default:0:11-] Debug(nsset): Ns_SetPut 0x600002845600 [35] key 'modifying_user' value '704' size -1
[16/Aug/2023:19:53:13.016651][-conn:oacs-head:default:0:11-] Debug(nsset): Ns_SetPut 0x600002845600 [36] key 'parent_id' value '1064' size -1
[16/Aug/2023:19:53:13.016655][-conn:oacs-head:default:0:11-] Debug(nsset): EnterSet 0x600002845600 with name 'd15'
[16/Aug/2023:19:53:13.016678][-conn:oacs-head:default:0:11-] Notice: trace: ns_set array d15

...
[16/Aug/2023:20:04:31.283342][-conn:oacs-head:default:0:11-] Debug(nsset): ns_set cleanup key <d15> dynamic 1
[16/Aug/2023:20:04:31.283344][-conn:oacs-head:default:0:11-] Debug(nsset): Ns_SetFree 0x600002845600 'db': elements 37
[16/Aug/2023:20:04:31.283346][-conn:oacs-head:default:0:11-] Debug(nsset): ... 0: key <object_type> value <::xowiki::Form>
[16/Aug/2023:20:04:31.283349][-conn:oacs-head:default:0:11-] Debug(nsset): ... 1: key <package_id> value <1060>
[16/Aug/2023:20:04:31.283351][-conn:oacs-head:default:0:11-] Debug(nsset): ... 2: key <creation_user> value <704>
...

In this example output, there is a bunch of Ns_SetPut() operations on a 
set "0x600002845600"  which is entered then to Tcl as set "d15".... 
somewhere later it is freed via "ns_set cleanup".

Let me know if you need some help on getting/compiling a branch of 
NaviServer from Bitbucket.

all the best

-g