naviserver-devel — Developer discussion

Re: [naviserver-devel] test ns_config-7.4.1, Stack around the variable 'filter' was corrupted

[Gustaf N. <ne...@wu...>]

On 07.11.23 22:56, Andrew Piskorski wrote:
> On the current NaviServer head, that problem has gone away!  The
> ns_config-7.4.1 test now runs fine for me on Windows.

You see: time heals all wounds! -g

Re: [naviserver-devel] A selection of messages appearing in our error logs

[Gustaf N. <ne...@wu...>]

Hi Brian,


In general, we try to follow with the logging severities the guideline 
[1]. "Notice" is informational, "warning" is something 
unexpected/unwanted (hitting some limits, got some data that looks like 
an attack, ...), "error" is something which should be looked at by an 
engineer.


     [1] https://openacs.org/xowiki/logging-conventions


below, I comment inline.



On 07.11.23 18:28, Brian Fenton wrote:
> Hi
>
> with the goal of improving signal-to-noise in our error logs, and also 
> to increase my understanding of Naviserver internals, I have a few 
> questions about various different messages we are seeing regularly in 
> our logs.
>
>  1. [06/Nov/2023:14:58:08][77320.7f7b9a7fd640][-driver:nsssl:0-]
>     Notice: ... sockAccept accepted 2 connections
>     this is by far the most common entry in our logs - we see a LOT of
>     these. Are they merely informative or indicative of some tuning
>     action we could take? I don't see any way to disable them. I would
>     love to remove these.
>

This indicates some stress situation of the server, where multiple 
requests come in at the same time. The value of 2 is not of a concern, 
if you see values like 10 or 20, you should consider a change in your 
configuration (e.g., configuring a second driver thread). It would be 
possible to define a configuration value for the network driver to set 
the threshold when the notice is written to the log file.


> 1.
>
> 2.
>     [06/Nov/2023:14:57:28][77320.7f7b63fff640][-conn:openacs:default:6:6292-]
>     Notice: ns_cache create entry collision cache util_memoize key
>     '__util_memoize_installed_p', no timeout
>     We also see a quite a few of these, always for
>     '__util_memoize_installed_p', and always with 'no timeout'. From
>     looking at cache.c I see  that this means that Naviserver failed
>     to create the cache entry. I checked the 'util_memoize' cache
>     utilization in case the cache was full, but it's quite low. What
>     other cause could there be for this?
>
This means that concurrent operations are executed to obtain the same 
cache value, and that the second request (here 
conn:openacs:default:6:6292) has to wait until the locking process has 
finished. The problem can become serious, when the locking process 
"hangs", i.e. taking a very long time to finish. This means, that more 
and more of caching requests for the same entry will pile up, 
potentially until a restart, since no timeout was specified. The warning 
was introduced to give the developer a hint, why suddenly the server is 
hanging (before the message was introduced, the cause was very hard to 
determine).


This message tells as well something else:
* It looks as if the installation is based on a very old version of 
openacs-core (including in particular acs-tcl) installed. At least since 
9 years, the code has changed to perform this test just at startup. 
Upgrade to recent versions will improve the performance and increase 
security.


* The message tells me as well that the application has a stress on the 
util-memoize cache. Old versions of OpenACS used this "util-memoize" 
cache as a kitchen-sink cache for everything. I've seen instances having 
300K + entries in this cache. The size itself is not the problem, but 
there were/are many operations in OpenACS that obtain all keys from the 
cache. It means that a lock is created, a list with 300K+ entries is 
created and handed to the interpreter before the cache is released. In 
the meantime, *all* caching requests for this cache have to be halted, 
... causing in high load situations degraded performance and limited 
concurrency. I would not be surprised, if you see high lock times on 
this cache.


There are caches aside of the util-memoize cache, where cache collisions 
might happen. See e.g. [2] for more background. Cache partitioning is in 
this situation the instrument for scaling.


    [2] https://openacs.org/xowiki/cache-size-tuning


> 1.
>  2. These 5 appear usually as a group:
>     [07/Nov/2023:13:46:07][38.7f352cff9640][-conn:openacs:default:3:4088-]
>     Notice: exiting: exceeded max connections per thread
>     [07/Nov/2023:13:46:07][38.7f3525b84640][-driver:nsssl:0-] Notice:
>     NsEnsureRunningConnectionThreads wantCreate 1 waiting 1 idle 0
>     current 2
>     [07/Nov/2023:13:46:07][38.7f352d7fa640][-conn:openacs:default:5:0-]
>     Notice: start update interpreter openacs to epoch 1, concurrent 1
>     [07/Nov/2023:13:46:08][38.7f352d7fa640][-conn:openacs:default:5:0-]
>     Notice: update interpreter openacs to epoch 1 done, trace none,
>     time 0.373006 secs concurrent 1
>     [07/Nov/2023:13:46:08][38.7f352d7fa640][-conn:openacs:default:5:0-]
>     Notice: thread initialized (0.388878 secs)
>     We see these pretty regularly, over 50 per day on  a  local
>     development server.  "maxconnections" is set to the default value,
>     which is 100, I believe.  "maxthreads" and "minthreads" are also
>     set to default values. What could  be causing these so regularly
>     on a system with 1 logged-in user?
>

The message tells that a connection thread has reached its end of live, 
as defined by the "connsperthread" parameter. When this happens, the 
thread is stopped, and depending on the configuration parameters, a new 
thread is created maybe immediately or on demand. The reason for 
stopping connection threads from time to time is to clean up some 
interpreter specific data, which was accumulated by some applications. 
With the modern versions of OpenACS and with well-behaved packages, this 
is not an issue, the value can be increased substantially.


The relevant counter does not depend on the number of different users, 
but on the number of requests handled by a particular connection thread.


> 1.
>
> 2.
>      These 3 are related to each other,  I presume:
>     [07/Nov/2023:15:36:24][38.7f3536b86640][-conn:openacs:default:28:28082-]
>     Notice: nsdb: closing old handle in pool 'pool1'
>     [07/Nov/2023:15:45:39][38.7f3536b86640][-conn:openacs:default:31:31287-]
>     Notice: nsdb: closing idle handle in pool 'pool1'
>     [07/Nov/2023:15:45:39][38.7f3536b86640][-conn:openacs:default:31:31287-]
>     Notice: dbdrv: opening database 'nsoracle:192.168.1.1/ORACLE'
>     These are quite frequent too. Am I correct in thinking that we can
>     tune these with pool "maxidle" and "maxopen" params mentioned here
>     https://naviserver.sourceforge.io/n/nsdb/files/ns_db.html ? Is
>     there any reason not to turn off the automatic closing feature?
>
There might be some reasons for keeping the number of db connections low 
(e.g. many servers use the same database with a limited number of client 
connections, etc.), since at least 20 years there is no technical reason 
for the common db drivers not to turn off the automatic closing feature. 
Reconnecting to the database can be costly, when the db is running on a 
different host and/or the connection is via HTTPS.


hope this helps a little


all the best

-g

[naviserver-devel] A selection of messages appearing in our error logs

[Brian F. <bri...@ai...>]

Hi

with the goal of improving signal-to-noise in our error logs, and also to increase my understanding of Naviserver internals, I have a few questions about various different messages we are seeing regularly in our logs.


  1.  [06/Nov/2023:14:58:08][77320.7f7b9a7fd640][-driver:nsssl:0-] Notice: ... sockAccept accepted 2 connections
this is by far the most common entry in our logs - we see a LOT of these. Are they merely informative or indicative of some tuning action we could take? I don't see any way to disable them. I would love to remove these.

  2.
[06/Nov/2023:14:57:28][77320.7f7b63fff640][-conn:openacs:default:6:6292-] Notice: ns_cache create entry collision cache util_memoize key '__util_memoize_installed_p', no timeout
We also see a quite a few of these, always for  '__util_memoize_installed_p', and always with 'no timeout'. From looking at cache.c I see  that this means that Naviserver failed to create the cache entry. I checked the 'util_memoize' cache utilization in case the cache was full, but it's quite low. What other cause could there be for this?

  3.  These 5 appear usually as a group:
[07/Nov/2023:13:46:07][38.7f352cff9640][-conn:openacs:default:3:4088-] Notice: exiting: exceeded max connections per thread
[07/Nov/2023:13:46:07][38.7f3525b84640][-driver:nsssl:0-] Notice: NsEnsureRunningConnectionThreads wantCreate 1 waiting 1 idle 0 current 2
[07/Nov/2023:13:46:07][38.7f352d7fa640][-conn:openacs:default:5:0-] Notice: start update interpreter openacs to epoch 1, concurrent 1
[07/Nov/2023:13:46:08][38.7f352d7fa640][-conn:openacs:default:5:0-] Notice: update interpreter openacs to epoch 1 done, trace none, time 0.373006 secs concurrent 1
[07/Nov/2023:13:46:08][38.7f352d7fa640][-conn:openacs:default:5:0-] Notice: thread initialized (0.388878 secs)
We see these pretty regularly, over 50 per day on  a  local development server.  "maxconnections" is set to the  default value, which is 100, I believe.  "maxthreads" and "minthreads" are also set to default values. What could  be causing these so regularly on a system with 1 logged-in user?

  4.
 These 3 are related to each other,  I presume:
[07/Nov/2023:15:36:24][38.7f3536b86640][-conn:openacs:default:28:28082-] Notice: nsdb: closing old handle in pool 'pool1'
[07/Nov/2023:15:45:39][38.7f3536b86640][-conn:openacs:default:31:31287-] Notice: nsdb: closing idle handle in pool 'pool1'
[07/Nov/2023:15:45:39][38.7f3536b86640][-conn:openacs:default:31:31287-] Notice: dbdrv: opening database 'nsoracle:192.168.1.1/ORACLE'
These are quite frequent too. Am I correct in thinking that we can tune these with pool "maxidle" and "maxopen" params mentioned here https://naviserver.sourceforge.io/n/nsdb/files/ns_db.html ? Is there any reason not to turn off the automatic closing feature?

many thanks in advance for any tips
Brian


  1.

Re: [naviserver-devel] test ns_config-7.4.1, Stack around the variable 'filter' was corrupted

[Andrew P. <at...@pi...>]

On Wed, Mar 15, 2023 at 03:57:40PM -0400, Andrew Piskorski wrote:
> On Tue, Mar 14, 2023 at 04:44:21PM -0400, Andrew Piskorski wrote:
> 
> >   Debug Error!
> >   Program:  C:\web\ns-fork-pub\naviserver\nsd\libnsd.dll
> >   Run-Time Check Failure #2 - Stack around the variable 'filter' was corrupted.
> > 
> > I never see the usual "all.tcl" test summary output.  Maybe nsd is
> > hitting the above "Debug Error!" before getting there?
> 
> On Windows, running the "ns_config.test" tests triggers that one.  All
> the tests through ns_config-7.4.0, pass, then it stops with no further
> output.  Yep, the next test, ns_config-7.4.1, is sufficient to trigger
> the problem all by itself.

On the current NaviServer head, that problem has gone away!  The
ns_config-7.4.1 test now runs fine for me on Windows.

-- 
Andrew Piskorski <at...@pi...>

Re: [naviserver-devel] NaviServer 4.99.29 available

[Gustaf N. <ne...@wu...>]

Hi Brian,


The proper solution is not to skip the error message, but to skip the 
full connection output operations in error situations. Since this 
involves many commands (from ns_return*, ns_write, ns_cookie* ...) this 
is a larger change. I will look into this in the next days. Should be 
doable with moderate effort.


all the best

-g

PS: bitbucket recovery still takes time, i am now at >15 email messages 
with support from support of atlassian and/or bitbucket.


On 06.11.23 15:11, Brian Fenton wrote:
> Hi Gustaf
>
> my apologies, I hadn't realised that silencing the log would lead to 
> different behaviour. If this is a bigger job than expected, please 
> feel free to revert back to previous version. It's a nice-to-have 
> feature for us.
>
> The attached script reproduces the issue when the parameter is set to 
> false.
>
> thanks
> Brian
>
> ------------------------------------------------------------------------
> *From:* Gustaf Neumann <ne...@wu...>
> *Sent:* Monday 6 November 2023 1:59 pm
> *To:* nav...@li... 
> <nav...@li...>
> *Subject:* Re: [naviserver-devel] NaviServer 4.99.29 available
>
> Hi Brian,
>
>
> as stated several times, the right action is to fix your script (as 
> you did) rather than "silencing" NaviServer. I am not surprised, that 
> attempts to write on detached connections can lead to error conditions 
> on several occasions (generating errors avoids this).
>
>
> But since we offer this silencing parameter, i do agree, the crashing 
> is harsh. If you could send a short script triggering the problem it 
> would help to work on such cases.
>
>
> all the best
>
> -g
>
>
> On 06.11.23 14:25, Brian Fenton wrote:
>> Hi Gustaf
>>
>> I just built and ran some tests on the "rejectalreadyclosedconn" 
>> parameter to see how it handles code that triggers the "connection 
>> socket is detached" error.
>>
>>  If I set "rejectalreadyclosedconn"to false, and browse to a page 
>> that triggers the "connection socket is detached" error, Naviserver 
>> crashes with the following error message:
>>
>> [06/Nov/2023:13:13:21][39.7f3489fb9640][-conn:openacs:default:1:30-] 
>> Warning: NsWriterQueue: called without sockPtr size 414 bufs 1 flags 
>> 1030431 stream 000000 chan (nil) fd -1
>> [06/Nov/2023:13:13:21][39.7f3489fb9640][-conn:openacs:default:1:30-] 
>> Fatal: received fatal signal 11
>>
>> If I then fix the code that was triggering the "connection socket is 
>> detached" error, by adding the missing "return" after the offending 
>> "ad_returnredirect", everything works fine.
>>
>> Let me know if you need more info to help reproduce this.
>> thanks,
>> Brian
>>
>> ------------------------------------------------------------------------
>> *From:* Gustaf Neumann <ne...@wu...> <mailto:ne...@wu...>
>> *Sent:* Thursday 2 November 2023 2:30 pm
>> *To:* Navidevel <nav...@li...> 
>> <mailto:nav...@li...>
>> *Subject:* [naviserver-devel] NaviServer 4.99.29 available
>> Dear all,
>>
>> I am glad to announce that the release of NaviServer 4.99.29 is 
>> available at SourceForge [1].  This release is a pure bug-fix and 
>> maintenance release, which fixes a potentiall serious memory leak 
>> when working with PostgreSQL and large text contents. Furthermore, 
>> the release contains a small enhancement as requested by Brian not 
>> very long ago on this list.
>>
>> See below for a summary of the changes.
>>
>> So far, these changes are only available on sourceforge, since i have 
>> lost write access to the repository at bitbucket. The people on 
>> Atlassian seem to have changed some account types, and - on to of 
>> this - they annouced via the Blog post on September 27, 2023, that 
>> the billing model changed (where they also refer to the 
>> "unified-user-management"). It took me a while to figure out, what 
>> happened. The blog post states:
>>
>>     /From October 31st, 2023, Bitbucket Cloud will begin counting all
>>     workspace members as a billable user. ....
>>
>>     Free plans: If you're on a free plan and your billable user count
>>     is higher than 5 as per the new definition of billable user, all
>>     repositories in your workspace will become read-only until you
>>     remove some users or upgrade your workspace to a paid plan./
>>
>> It seems that the users of the "naviserver" group are now counted as 
>> "billable users", and it contains 19 users. According to support, we 
>> have to reduce this number to 5, otherwise nobody will be able to 
>> commit anything.
>>
>> Due to the ability with PRs, i think the reduction will be possible 
>> without too much loss in functionality. If nobody objects, i will go 
>> back in history and reduce the number of commit-member based on the 
>> most recent direct commits. I hope, that non of the "old members" 
>> will be offended by this. One other option would be to upgrade to a 
>> paid plain - but i am not sure, who is gonna pay for this.
>>
>> All the best!
>>
>> -gustaf neumann
>>
>> [1] 
>> https://sourceforge.net/projects/naviserver/files/naviserver/4.99.29/ 
>> <https://sourceforge.net/projects/naviserver/files/naviserver/4.99.29/>
>> [2] https://bitbucket.org/blog/billing-model-change 
>> <https://bitbucket.org/blog/billing-model-change>
>>
>> =======================================
>> NaviServer 4.99.29, released 2023-11-01
>> =======================================
>>
>>   37 files changed, 261 insertions(+), 132 deletions(-)
>>
>> New Features:
>> -------------
>>   - Eased configuration of simple setups
>>     * don't require to specify a "defaultserver" when a single
>>       server is in use.
>>
>>     * reduce warnings for per-server network drivers. This is a not
>>       recommended but possible configuration, global network drivers
>>       should be used.
>>
>>   - The configuration option "rejectalreadyclosedconn", which warns
>>     about attempts to send data to the web client at times, when the
>>     connection is not available anymore, is now applied on closed and
>>     detached connections. Before it was only applied on closed
>>     connections, causing potentially many warnings for legacy
>>     applications.
>>
>> Bug Fixes:
>> ----------
>>
>>    - Fixed a potential memory leak introduced two releases ago (in
>>      4.99.27).
>>
>>    - Fixed a potential compilation problem with glibc 2.38 or newer
>>      (released 31 Jul 2023)
>>
>>    - Fixed reloading of certificates for mass virtual hosting
>>
>> Code Maintenance:
>> -----------------
>>
>>    - fixed typos
>>    - fixed enum/int conversion flagged by gcc13
>>
>>
>> Modules:
>> --------
>> The following list contains the most important changes:
>>
>>   - module nsdbpg:
>>     fixed memory leak (see above).

Re: [naviserver-devel] NaviServer 4.99.29 available

[Brian F. <bri...@ai...>]

Hi Gustaf

my apologies, I hadn't realised that silencing the log would lead to different behaviour. If this is a bigger job than expected, please feel free to revert back to previous version. It's a nice-to-have feature for us.

The attached script reproduces the issue when the parameter is set to false.

thanks
Brian

________________________________
From: Gustaf Neumann <ne...@wu...>
Sent: Monday 6 November 2023 1:59 pm
To: nav...@li... <nav...@li...>
Subject: Re: [naviserver-devel] NaviServer 4.99.29 available


Hi Brian,


as stated several times, the right action is to fix your script (as you did) rather than "silencing" NaviServer. I am not surprised, that attempts to write on detached connections can lead to error conditions on several occasions (generating errors avoids this).


But since we offer this silencing parameter, i do agree, the crashing is harsh. If you could send a short script triggering the problem it would help to work on such cases.


all the best

-g


On 06.11.23 14:25, Brian Fenton wrote:
Hi Gustaf

I just built and ran some tests on the "rejectalreadyclosedconn" parameter to see how it handles code that triggers the "connection socket is detached" error.

 If I set "rejectalreadyclosedconn" to false, and browse to a page that triggers the "connection socket is detached" error, Naviserver crashes with the following error message:

[06/Nov/2023:13:13:21][39.7f3489fb9640][-conn:openacs:default:1:30-] Warning: NsWriterQueue: called without sockPtr size 414 bufs 1 flags 1030431 stream 000000 chan (nil) fd -1
[06/Nov/2023:13:13:21][39.7f3489fb9640][-conn:openacs:default:1:30-] Fatal: received fatal signal 11

If I then fix the code that was triggering the "connection socket is detached" error, by adding the missing "return" after the offending "ad_returnredirect", everything works fine.

Let me know if you need more info to help reproduce this.
thanks,
Brian

________________________________
From: Gustaf Neumann <ne...@wu...><mailto:ne...@wu...>
Sent: Thursday 2 November 2023 2:30 pm
To: Navidevel <nav...@li...><mailto:nav...@li...>
Subject: [naviserver-devel] NaviServer 4.99.29 available

Dear all,

I am glad to announce that the release of NaviServer 4.99.29 is available at SourceForge [1].  This release is a pure bug-fix and maintenance release, which fixes a potentiall serious memory leak when working with PostgreSQL and large text contents. Furthermore, the release contains a small enhancement as requested by Brian not very long ago on this list.

See below for a summary of the changes.

So far, these changes are only available on sourceforge, since i have lost write access to the repository at bitbucket. The people on Atlassian seem to have changed some account types, and - on to of this - they annouced via the Blog post on September 27, 2023, that the billing model changed (where they also refer to the "unified-user-management"). It took me a while to figure out, what happened. The blog post states:

>From October 31st, 2023, Bitbucket Cloud will begin counting all workspace members as a billable user. ....

Free plans: If you're on a free plan and your billable user count is higher than 5 as per the new definition of billable user, all repositories in your workspace will become read-only until you remove some users or upgrade your workspace to a paid plan.

It seems that the users of the "naviserver" group are now counted as "billable users", and it contains 19 users. According to support, we have to reduce this number to 5, otherwise nobody will be able to commit anything.

Due to the ability with PRs, i think the reduction will be possible without too much loss in functionality. If nobody objects, i will go back in history and reduce the number of commit-member based on the most recent direct commits. I hope, that non of the "old members" will be offended by this. One other option would be to upgrade to a paid plain - but i am not sure, who is gonna pay for this.

All the best!

-gustaf neumann

[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.29/
[2] https://bitbucket.org/blog/billing-model-change


=======================================
NaviServer 4.99.29, released 2023-11-01
=======================================

 37 files changed, 261 insertions(+), 132 deletions(-)

New Features:
-------------
 - Eased configuration of simple setups
   * don't require to specify a "defaultserver" when a single
     server is in use.

   * reduce warnings for per-server network drivers. This is a not
     recommended but possible configuration, global network drivers
     should be used.

 - The configuration option "rejectalreadyclosedconn", which warns
   about attempts to send data to the web client at times, when the
   connection is not available anymore, is now applied on closed and
   detached connections. Before it was only applied on closed
   connections, causing potentially many warnings for legacy
   applications.

Bug Fixes:
----------

  - Fixed a potential memory leak introduced two releases ago (in
    4.99.27).

  - Fixed a potential compilation problem with glibc 2.38 or newer
    (released 31 Jul 2023)

  - Fixed reloading of certificates for mass virtual hosting

Code Maintenance:
-----------------

  - fixed typos
  - fixed enum/int conversion flagged by gcc13


Modules:
--------
The following list contains the most important changes:

 - module nsdbpg:
   fixed memory leak (see above).

Re: [naviserver-devel] NaviServer 4.99.29 available

[Gustaf N. <ne...@wu...>]

Hi Brian,


as stated several times, the right action is to fix your script (as you 
did) rather than "silencing" NaviServer. I am not surprised, that 
attempts to write on detached connections can lead to error conditions 
on several occasions (generating errors avoids this).


But since we offer this silencing parameter, i do agree, the crashing is 
harsh. If you could send a short script triggering the problem it would 
help to work on such cases.


all the best

-g


On 06.11.23 14:25, Brian Fenton wrote:
> Hi Gustaf
>
> I just built and ran some tests on the "rejectalreadyclosedconn" 
> parameter to see how it handles code that triggers the "connection 
> socket is detached" error.
>
>  If I set "rejectalreadyclosedconn"to false, and browse to a page that 
> triggers the "connection socket is detached" error, Naviserver crashes 
> with the following error message:
>
> [06/Nov/2023:13:13:21][39.7f3489fb9640][-conn:openacs:default:1:30-] 
> Warning: NsWriterQueue: called without sockPtr size 414 bufs 1 flags 
> 1030431 stream 000000 chan (nil) fd -1
> [06/Nov/2023:13:13:21][39.7f3489fb9640][-conn:openacs:default:1:30-] 
> Fatal: received fatal signal 11
>
> If I then fix the code that was triggering the "connection socket is 
> detached" error, by adding the missing "return" after the offending 
> "ad_returnredirect", everything works fine.
>
> Let me know if you need more info to help reproduce this.
> thanks,
> Brian
>
> ------------------------------------------------------------------------
> *From:* Gustaf Neumann <ne...@wu...>
> *Sent:* Thursday 2 November 2023 2:30 pm
> *To:* Navidevel <nav...@li...>
> *Subject:* [naviserver-devel] NaviServer 4.99.29 available
> Dear all,
>
> I am glad to announce that the release of NaviServer 4.99.29 is 
> available at SourceForge [1].  This release is a pure bug-fix and 
> maintenance release, which fixes a potentiall serious memory leak when 
> working with PostgreSQL and large text contents. Furthermore, the 
> release contains a small enhancement as requested by Brian not very 
> long ago on this list.
>
> See below for a summary of the changes.
>
> So far, these changes are only available on sourceforge, since i have 
> lost write access to the repository at bitbucket. The people on 
> Atlassian seem to have changed some account types, and - on to of this 
> - they annouced via the Blog post on September 27, 2023, that the 
> billing model changed (where they also refer to the 
> "unified-user-management"). It took me a while to figure out, what 
> happened. The blog post states:
>
>     /From October 31st, 2023, Bitbucket Cloud will begin counting all
>     workspace members as a billable user. ....
>
>     Free plans: If you're on a free plan and your billable user count
>     is higher than 5 as per the new definition of billable user, all
>     repositories in your workspace will become read-only until you
>     remove some users or upgrade your workspace to a paid plan./
>
> It seems that the users of the "naviserver" group are now counted as 
> "billable users", and it contains 19 users. According to support, we 
> have to reduce this number to 5, otherwise nobody will be able to 
> commit anything.
>
> Due to the ability with PRs, i think the reduction will be possible 
> without too much loss in functionality. If nobody objects, i will go 
> back in history and reduce the number of commit-member based on the 
> most recent direct commits. I hope, that non of the "old members" will 
> be offended by this. One other option would be to upgrade to a paid 
> plain - but i am not sure, who is gonna pay for this.
>
> All the best!
>
> -gustaf neumann
>
> [1] 
> https://sourceforge.net/projects/naviserver/files/naviserver/4.99.29/ 
> <https://sourceforge.net/projects/naviserver/files/naviserver/4.99.29/>
> [2] https://bitbucket.org/blog/billing-model-change 
> <https://bitbucket.org/blog/billing-model-change>
>
> =======================================
> NaviServer 4.99.29, released 2023-11-01
> =======================================
>
>   37 files changed, 261 insertions(+), 132 deletions(-)
>
> New Features:
> -------------
>   - Eased configuration of simple setups
>     * don't require to specify a "defaultserver" when a single
>       server is in use.
>
>     * reduce warnings for per-server network drivers. This is a not
>       recommended but possible configuration, global network drivers
>       should be used.
>
>   - The configuration option "rejectalreadyclosedconn", which warns
>     about attempts to send data to the web client at times, when the
>     connection is not available anymore, is now applied on closed and
>     detached connections. Before it was only applied on closed
>     connections, causing potentially many warnings for legacy
>     applications.
>
> Bug Fixes:
> ----------
>
>    - Fixed a potential memory leak introduced two releases ago (in
>      4.99.27).
>
>    - Fixed a potential compilation problem with glibc 2.38 or newer
>      (released 31 Jul 2023)
>
>    - Fixed reloading of certificates for mass virtual hosting
>
> Code Maintenance:
> -----------------
>
>    - fixed typos
>    - fixed enum/int conversion flagged by gcc13
>
>
> Modules:
> --------
> The following list contains the most important changes:
>
>   - module nsdbpg:
>     fixed memory leak (see above).

Re: [naviserver-devel] NaviServer 4.99.29 available

[Brian F. <bri...@ai...>]

Hi Gustaf

I just built and ran some tests on the "rejectalreadyclosedconn" parameter to see how it handles code that triggers the "connection socket is detached" error.

 If I set "rejectalreadyclosedconn" to false, and browse to a page that triggers the "connection socket is detached" error, Naviserver crashes with the following error message:

[06/Nov/2023:13:13:21][39.7f3489fb9640][-conn:openacs:default:1:30-] Warning: NsWriterQueue: called without sockPtr size 414 bufs 1 flags 1030431 stream 000000 chan (nil) fd -1
[06/Nov/2023:13:13:21][39.7f3489fb9640][-conn:openacs:default:1:30-] Fatal: received fatal signal 11

If I then fix the code that was triggering the "connection socket is detached" error, by adding the missing "return" after the offending "ad_returnredirect", everything works fine.

Let me know if you need more info to help reproduce this.
thanks,
Brian

________________________________
From: Gustaf Neumann <ne...@wu...>
Sent: Thursday 2 November 2023 2:30 pm
To: Navidevel <nav...@li...>
Subject: [naviserver-devel] NaviServer 4.99.29 available

Dear all,

I am glad to announce that the release of NaviServer 4.99.29 is available at SourceForge [1].  This release is a pure bug-fix and maintenance release, which fixes a potentiall serious memory leak when working with PostgreSQL and large text contents. Furthermore, the release contains a small enhancement as requested by Brian not very long ago on this list.

See below for a summary of the changes.

So far, these changes are only available on sourceforge, since i have lost write access to the repository at bitbucket. The people on Atlassian seem to have changed some account types, and - on to of this - they annouced via the Blog post on September 27, 2023, that the billing model changed (where they also refer to the "unified-user-management"). It took me a while to figure out, what happened. The blog post states:

>From October 31st, 2023, Bitbucket Cloud will begin counting all workspace members as a billable user. ....

Free plans: If you're on a free plan and your billable user count is higher than 5 as per the new definition of billable user, all repositories in your workspace will become read-only until you remove some users or upgrade your workspace to a paid plan.

It seems that the users of the "naviserver" group are now counted as "billable users", and it contains 19 users. According to support, we have to reduce this number to 5, otherwise nobody will be able to commit anything.

Due to the ability with PRs, i think the reduction will be possible without too much loss in functionality. If nobody objects, i will go back in history and reduce the number of commit-member based on the most recent direct commits. I hope, that non of the "old members" will be offended by this. One other option would be to upgrade to a paid plain - but i am not sure, who is gonna pay for this.

All the best!

-gustaf neumann

[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.29/
[2] https://bitbucket.org/blog/billing-model-change


=======================================
NaviServer 4.99.29, released 2023-11-01
=======================================

 37 files changed, 261 insertions(+), 132 deletions(-)

New Features:
-------------
 - Eased configuration of simple setups
   * don't require to specify a "defaultserver" when a single
     server is in use.

   * reduce warnings for per-server network drivers. This is a not
     recommended but possible configuration, global network drivers
     should be used.

 - The configuration option "rejectalreadyclosedconn", which warns
   about attempts to send data to the web client at times, when the
   connection is not available anymore, is now applied on closed and
   detached connections. Before it was only applied on closed
   connections, causing potentially many warnings for legacy
   applications.

Bug Fixes:
----------

  - Fixed a potential memory leak introduced two releases ago (in
    4.99.27).

  - Fixed a potential compilation problem with glibc 2.38 or newer
    (released 31 Jul 2023)

  - Fixed reloading of certificates for mass virtual hosting

Code Maintenance:
-----------------

  - fixed typos
  - fixed enum/int conversion flagged by gcc13


Modules:
--------
The following list contains the most important changes:

 - module nsdbpg:
   fixed memory leak (see above).

Re: [naviserver-devel] NaviServer 4.99.29 available

[Andrew P. <at...@pi...>]

On Thu, Nov 02, 2023 at 03:30:43PM +0100, Gustaf Neumann wrote:

> So far, these changes are only available on sourceforge, since i have 
> lost write access to the repository at bitbucket.

Wow.  Yes, please remove whatever group members are necessary to
restore write access, and let us know when the public NaviServer Git
repository is again up to date.  (As you say, we can always send you
patches via forks in our personal repositories on Bitbucket, if
necessary.)

-- 
Andrew Piskorski <at...@pi...>

Re: [naviserver-devel] NaviServer 4.99.29 available

[Brian F. <bri...@ai...>]

Dear Gustaf

thank you for including my requested enhancement -  a pleasant surprise!

I have a few more thoughts and questions about reducing the volume of log entries - I will put that together and send it on in the near future.

thanks again
Brian

________________________________
From: Gustaf Neumann <ne...@wu...>
Sent: Thursday 2 November 2023 2:30 pm
To: Navidevel <nav...@li...>
Subject: [naviserver-devel] NaviServer 4.99.29 available

Dear all,

I am glad to announce that the release of NaviServer 4.99.29 is available at SourceForge [1].  This release is a pure bug-fix and maintenance release, which fixes a potentiall serious memory leak when working with PostgreSQL and large text contents. Furthermore, the release contains a small enhancement as requested by Brian not very long ago on this list.

See below for a summary of the changes.

So far, these changes are only available on sourceforge, since i have lost write access to the repository at bitbucket. The people on Atlassian seem to have changed some account types, and - on to of this - they annouced via the Blog post on September 27, 2023, that the billing model changed (where they also refer to the "unified-user-management"). It took me a while to figure out, what happened. The blog post states:

>From October 31st, 2023, Bitbucket Cloud will begin counting all workspace members as a billable user. ....

Free plans: If you're on a free plan and your billable user count is higher than 5 as per the new definition of billable user, all repositories in your workspace will become read-only until you remove some users or upgrade your workspace to a paid plan.

It seems that the users of the "naviserver" group are now counted as "billable users", and it contains 19 users. According to support, we have to reduce this number to 5, otherwise nobody will be able to commit anything.

Due to the ability with PRs, i think the reduction will be possible without too much loss in functionality. If nobody objects, i will go back in history and reduce the number of commit-member based on the most recent direct commits. I hope, that non of the "old members" will be offended by this. One other option would be to upgrade to a paid plain - but i am not sure, who is gonna pay for this.

All the best!

-gustaf neumann


[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.29/

[2] https://bitbucket.org/blog/billing-model-change


=======================================
NaviServer 4.99.29, released 2023-11-01
=======================================

 37 files changed, 261 insertions(+), 132 deletions(-)

New Features:
-------------
 - Eased configuration of simple setups
   * don't require to specify a "defaultserver" when a single
     server is in use.

   * reduce warnings for per-server network drivers. This is a not
     recommended but possible configuration, global network drivers
     should be used.

 - The configuration option "rejectalreadyclosedconn", which warns
   about attempts to send data to the web client at times, when the
   connection is not available anymore, is now applied on closed and
   detached connections. Before it was only applied on closed
   connections, causing potentially many warnings for legacy
   applications.

Bug Fixes:
----------

  - Fixed a potential memory leak introduced two releases ago (in
    4.99.27).

  - Fixed a potential compilation problem with glibc 2.38 or newer
    (released 31 Jul 2023)

  - Fixed reloading of certificates for mass virtual hosting

Code Maintenance:
-----------------

  - fixed typos
  - fixed enum/int conversion flagged by gcc13


Modules:
--------
The following list contains the most important changes:

 - module nsdbpg:
   fixed memory leak (see above).

[naviserver-devel] NaviServer 4.99.29 available

[Gustaf N. <ne...@wu...>]

Dear all,

I am glad to announce that the release of NaviServer 4.99.29 is 
available at SourceForge [1].  This release is a pure bug-fix and 
maintenance release, which fixes a potentiall serious memory leak when 
working with PostgreSQL and large text contents. Furthermore, the 
release contains a small enhancement as requested by Brian not very long 
ago on this list.

See below for a summary of the changes.

So far, these changes are only available on sourceforge, since i have 
lost write access to the repository at bitbucket. The people on 
Atlassian seem to have changed some account types, and - on to of this - 
they annouced via the Blog post on September 27, 2023, that the billing 
model changed (where they also refer to the "unified-user-management"). 
It took me a while to figure out, what happened. The blog post states:

    /From October 31st, 2023, Bitbucket Cloud will begin counting all
    workspace members as a billable user. ....

    Free plans: If you're on a free plan and your billable user count is
    higher than 5 as per the new definition of billable user, all
    repositories in your workspace will become read-only until you
    remove some users or upgrade your workspace to a paid plan./

It seems that the users of the "naviserver" group are now counted as 
"billable users", and it contains 19 users. According to support, we 
have to reduce this number to 5, otherwise nobody will be able to commit 
anything.

Due to the ability with PRs, i think the reduction will be possible 
without too much loss in functionality. If nobody objects, i will go 
back in history and reduce the number of commit-member based on the most 
recent direct commits. I hope, that non of the "old members" will be 
offended by this. One other option would be to upgrade to a paid plain - 
but i am not sure, who is gonna pay for this.

All the best!

-gustaf neumann

[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.29/
[2] https://bitbucket.org/blog/billing-model-change

=======================================
NaviServer 4.99.29, released 2023-11-01
=======================================

  37 files changed, 261 insertions(+), 132 deletions(-)

New Features:
-------------
  - Eased configuration of simple setups
    * don't require to specify a "defaultserver" when a single
      server is in use.

    * reduce warnings for per-server network drivers. This is a not
      recommended but possible configuration, global network drivers
      should be used.

  - The configuration option "rejectalreadyclosedconn", which warns
    about attempts to send data to the web client at times, when the
    connection is not available anymore, is now applied on closed and
    detached connections. Before it was only applied on closed
    connections, causing potentially many warnings for legacy
    applications.

Bug Fixes:
----------

   - Fixed a potential memory leak introduced two releases ago (in
     4.99.27).

   - Fixed a potential compilation problem with glibc 2.38 or newer
     (released 31 Jul 2023)

   - Fixed reloading of certificates for mass virtual hosting

Code Maintenance:
-----------------

   - fixed typos
   - fixed enum/int conversion flagged by gcc13


Modules:
--------
The following list contains the most important changes:

  - module nsdbpg:
    fixed memory leak (see above).

[naviserver-devel] Proposal: config file boolean parameter "showserverheader" to show/hide "Server:" response header

[Brian F. <bri...@ai...>]

Hi all

Recently during a client security audit, the "Server: NaviServer/4.99.28" response header was flagged as an issue. The client has asked us to remove the header, if possible.
The RFC suggests that the "Server: " header is optional, so I believe this should be OK to remove. https://www.rfc-editor.org/rfc/rfc7231#section-7.4.2

We would like to propose a new config file boolean parameter "showserverheader" with default true.

Ns_ConnConstructHeaders in return.c could then check this parameter before outputting the "Server: " header e.g. something like this:

    if (Ns_ConfigBool(path, "showserverheader", NS_TRUE) == NS_TRUE) {
      Ns_DStringVarAppend(dsPtr, "Server: ", Ns_InfoServerName(), "/", Ns_InfoServerVersion(), "\r\n");    
    }

Thoughts? Alternatives?

thanks
Brian

Re: [naviserver-devel] "connection already closed" vs "connection socket is detached" and the "rejectalreadyclosedconn" parameter

[Brian F. <bri...@ai...>]

Hi Gustaf

thanks for the explanation and the commit. I got confused because the discussion on https://openacs.org/forums/message-view?message_id=5659253 was about "connection socket is detached", when as you say that distinction came later.

This kind of parameter is handy with clients running older versions of our application where it's not feasible to immediately upgrade them. With the parameter, we could theoretically reduce the noise in the error logs, while the client awaits an upgrade.

thanks as always
Brian
________________________________
From: Gustaf Neumann <ne...@wu...>
Sent: Saturday 30 September 2023 11:08 am
To: nav...@li... <nav...@li...>
Subject: Re: [naviserver-devel] "connection already closed" vs "connection socket is detached" and the "rejectalreadyclosedconn" parameter

Hi Brian,

The parameter "rejectalreadyclosedconn" does what it is supposed to do
(controls error messages, when someone tries to write to a connection,
which was already closed). The parameter was introduced at a time,
before the distinction between closed and detached connections was
introduced in the code. So, I read your request to extend the meaning of
the parameter to cover detached connections as well.

 > Have I misunderstood something or is this not possible?

There is no hard reason rendering this as impossible. One could either
introduce (a) another parameter "rejectalreadydetachedconn" or (b) make
"rejectalreadyclosedconn" slightly misleading from the name to cover as
well the detached cases.

Since the parameter is just for legacy applications, which did not care
about the semantics, when someone tries to write to a connection which
is not available anymore (leading to potentially complex debugging
questions), the updated code [1] follows the (b) approach. I see the
potential misuse of the parameter to encourage lazy coding practices,
but i do understand as well code-maintainers who see large amounts of
errors in their log files on busy sites, not having the resources right
now to address these...

-gn

[1]
https://bitbucket.org/naviserver/naviserver/commits/725cf9475fe78f495918ed4549bf434cf2daaf96





_______________________________________________
naviserver-devel mailing list
nav...@li...
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] "connection already closed" vs "connection socket is detached" and the "rejectalreadyclosedconn" parameter

[Gustaf N. <ne...@wu...>]

On 29.09.23 12:56, Brian Fenton wrote:
> Hi
>
> this discussion here 
> https://openacs.org/forums/message-view?message_id=5659253 suggests 
> that the "rejectalreadyclosedconn" parameter will prevent the 
> "connection socket is detached" message from showing in the logs. 
> However, this doesn't appear to work (tried in 4.99.28). Looking at 
> "NsConnRequire" in conn.c, it looks like 
> "reject_already_closed_connection" is only used in the "connection 
> already closed" case. 
> https://bitbucket.org/naviserver/naviserver/src/b21b6d75a8f2ff41cd00dd40cb4c143d9a8a55ce/nsd/conn.c#lines-2748
>
> It would be good to be able to turn off the "connection socket is 
> detached" (understanding of course that we should in the meantime 
> resolve the source of those messages). Have I misunderstood something 
> or is this not possible?
>
> thanks
> Brian
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

-- 
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"

Re: [naviserver-devel] "connection already closed" vs "connection socket is detached" and the "rejectalreadyclosedconn" parameter

[Gustaf N. <ne...@wu...>]

Hi Brian,

The parameter "rejectalreadyclosedconn" does what it is supposed to do 
(controls error messages, when someone tries to write to a connection, 
which was already closed). The parameter was introduced at a time, 
before the distinction between closed and detached connections was 
introduced in the code. So, I read your request to extend the meaning of 
the parameter to cover detached connections as well.

 > Have I misunderstood something or is this not possible?

There is no hard reason rendering this as impossible. One could either 
introduce (a) another parameter "rejectalreadydetachedconn" or (b) make 
"rejectalreadyclosedconn" slightly misleading from the name to cover as 
well the detached cases.

Since the parameter is just for legacy applications, which did not care 
about the semantics, when someone tries to write to a connection which 
is not available anymore (leading to potentially complex debugging 
questions), the updated code [1] follows the (b) approach. I see the 
potential misuse of the parameter to encourage lazy coding practices, 
but i do understand as well code-maintainers who see large amounts of 
errors in their log files on busy sites, not having the resources right 
now to address these...

-gn

[1] 
https://bitbucket.org/naviserver/naviserver/commits/725cf9475fe78f495918ed4549bf434cf2daaf96

[naviserver-devel] "connection already closed" vs "connection socket is detached" and the "rejectalreadyclosedconn" parameter

[Brian F. <bri...@ai...>]

Hi

this discussion here https://openacs.org/forums/message-view?message_id=5659253 suggests that the "rejectalreadyclosedconn" parameter will prevent the "connection socket is detached" message from showing in the logs. However, this doesn't appear to work (tried in 4.99.28). Looking at "NsConnRequire" in conn.c, it looks like "reject_already_closed_connection" is only used in the "connection already closed" case. https://bitbucket.org/naviserver/naviserver/src/b21b6d75a8f2ff41cd00dd40cb4c143d9a8a55ce/nsd/conn.c#lines-2748

It would be good to be able to turn off the "connection socket is detached" (understanding of course that we should in the meantime resolve the source of those messages). Have I misunderstood something or is this not possible?

thanks
Brian

[naviserver-devel] smtp log file

[Gustaf N. <ne...@wu...>]

Dear NaviServer users,

the updated version of nsstats supports now viewing smtp log files in a 
similar way to the HTTP client log files (see for an example the 
presentation of the last EuroTcl/OpenACS conference [1]). The SMTP sent 
log visualizes the performance, frequency and the result codes for the 
SMTP send commands, when the nssmptd module is used.

See below for a sample configuration of for the nssmtpd module with 
logging configured.

all the best

-gn


ns_section "ns/server/$server/module/nssmtpd" {
     ns_param port $smtpdport
     ns_param address $nssmtpd_address ;# 127.0.0.1
     ns_param relay localhost:25
     ns_param spamd localhost
     ns_param initproc smtpd::init
     ns_param rcptproc smtpd::rcpt
     ns_param dataproc smtpd::data
     ns_param errorproc smtpd::error
     ns_param relaydomains "localhost"
     ns_param localdomains "localhost"

     ns_param logging on ;# default: off
     ns_param logfile ${logroot}/smtpsend.log
     ns_param logrollfmt %Y-%m-%d ;# format appended to log filename
     #ns_param logmaxbackup 100 ;# 10, max number of backup log files
     #ns_param logroll true ;# true, should server log files automatically
     #ns_param logrollonsignal true ;# false, perform roll on a sighup
     #ns_param logrollhour 0 ;# 0, specify at which hour to roll
}


[1] 
https://openacs.org/conf2023/info/download/file/openacs-conf-2023-naviserver.pdf

Re: [naviserver-devel] NX and Naviserver init

[Maksym Z. <siq...@gm...>]

Hi Gustaf, thank you for your explanation and link.

On Fri, Sep 15, 2023 at 9:19 AM Gustaf Neumann <ne...@wu...> wrote:

> Dear Maksym,
> On 15.09.23 01:09, Maksym Zinchenko wrote:
>
> Hello, I've been struggling with a problem for a few days now. First I
> thought it's something with my understanding of NX, but I think its has to
> do something with Naviserver.
>
> actually, the problem is not with NaviServer, but it is rooted in the
> XOTcl/NX serializer. When a NaviServer blueprint is created, NaviServer
> performs "::Serializer all" to serialize the current workspace. Actually,
> it does not serialize the full workspace, since the workspace contains as
> well the NX internal class definitions. Therefore, the serializer has
> "ignore patterns", including everything from, e.g., the "::ns"f namespace
> [1].
>
> When objects are created with "new", these are named by default
> "::nsf::__#1", "::nsf::__#2". These names match the ignore pattern, and
> therefore, these names are excluded from the blueprint.
>
> If you create the objects in your code with the "-childof" flag (see
> below), you can place these outside the "::nsf" namespace, and everything
> should be fine
>
> Hope this helps
>
> -g
>
>
> [1]
> https://github.com/gustafn/nsf/blob/master/library/serialize/serializer.tcl#L868
>
>
>     nx::Class create B {
>         :method init {} {
>             ns_log notice "Im class [self]"
>         }
>     }
>
>     nx::Class create C {
>         :method init {} {
>             ns_log notice "Im class [self]"
>         }
>     }
>
>     nx::Class create A {
>         :property -accessor public inBObj:object
>         :property -accessor public inCObj:object
>
>         :method init {} {
>             set :inBObj [B new -childof [::nsf::current object]]
>             set :inCObj [C new -childof [::nsf::current object]]
>         }
>
>         :public method myObjs {} {
>             ns_log notice  "My B obj is: [${:inBObj} info name]"
>             ns_log notice  "My C obj is: [${:inCObj} info name]"
>         }
>     }
>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>

Re: [naviserver-devel] NX and Naviserver init

[Gustaf N. <ne...@wu...>]

Dear Maksym,

On 15.09.23 01:09, Maksym Zinchenko wrote:
> Hello, I've been struggling with a problem for a few days now. First I 
> thought it's something with my understanding of NX, but I think its 
> has to do something with Naviserver.

actually, the problem is not with NaviServer, but it is rooted in the 
XOTcl/NX serializer. When a NaviServer blueprint is created, NaviServer 
performs "::Serializer all" to serialize the current workspace. 
Actually, it does not serialize the full workspace, since the workspace 
contains as well the NX internal class definitions. Therefore, the 
serializer has "ignore patterns", including everything from, e.g., the 
"::ns"f namespace [1].

When objects are created with "new", these are named by default 
"::nsf::__#1", "::nsf::__#2". These names match the ignore pattern, and 
therefore, these names are excluded from the blueprint.

If you create the objects in your code with the "-childof" flag (see 
below), you can place these outside the "::nsf" namespace, and 
everything should be fine

Hope this helps

-g


[1] 
https://github.com/gustafn/nsf/blob/master/library/serialize/serializer.tcl#L868


     nx::Class create B {
         :method init {} {
             ns_log notice "Im class [self]"
         }
     }

     nx::Class create C {
         :method init {} {
             ns_log notice "Im class [self]"
         }
     }

     nx::Class create A {
         :property -accessor public inBObj:object
         :property -accessor public inCObj:object

         :method init {} {
             set :inBObj [B new -childof [::nsf::current object]]
             set :inCObj [C new -childof [::nsf::current object]]
         }

         :public method myObjs {} {
             ns_log notice  "My B obj is: [${:inBObj} info name]"
             ns_log notice  "My C obj is: [${:inCObj} info name]"
         }
     }

[naviserver-devel] NX and Naviserver init

[Maksym Z. <siq...@gm...>]

Hello, I've been struggling with a problem for a few days now. First I
thought it's something with my understanding of NX, but I think its has to
do something with Naviserver.
I have a global library named "oodz" in my tcl folder, insideI have some
subfolders with my NX classes. I have an init.tcl file inside. According to
documentation this "init.tcl" file executed first. First thing I do is
looping through my subfolders and sourcing my NX Classes. It works fine as
expected. Next Im creating some instances of classes I need inside
"init.tcl"
I made a simple example, to show: I have an NX class like that:

nx::Class create b {
> :method init {} {
> :say
> }
>
> :public method say {} {
> puts "Im class B"
> }
> }
>
> nx::Class create c {
> :method init {} {
> :say
> }
>
> :public method say {} {
> puts "Im class C"
> }
> }
>
> nx::Class create a {
> :property -accessor public inBObj:object
> :property -accessor public inCObj:object
>
> :method init {} {
> set :inBObj [b new]
> set :inCObj [c new]
> }
>
> :public method myObjs {} {
> puts "My B obj is: [${:inBObj} info name]"
> puts "My C obj is: [${:inCObj} info name]"
> }
> }
>

In my "init.tcl" I have line: a create aObj

Here's what's happenin:
1. I can see in the server log that aObj was created, I see line:
Im class B, Im class C

2. If I open nsshell console, and run aObj cget:
aObj cget -inBObj
::nsf::__#3
aObj cget -inCObj
::nsf::__#4

3. But when I try to run myObjs, I get an error:
aObj myObjs
invalid command name "::nsf::__#3"

The same code works fine in tcl shell, is it related to Naviserver? Or am I
doing something wrong with NX?

Thank you

Re: [naviserver-devel] NaviServer 4.99.28 available

[Gustaf N. <ne...@wu...>]

Dear all,

Part of the announcement of NaviServer 4.99.28 was the update of the 
nsdbbdb module (Berkley DB driver via nsdb). I did a few tests about its 
performance, that might interest a few here.

The test of this module configured with lmdb is quite impressive (see 
below). For comparison, there are also accesses to nsv, ns_cache and 
similar included. All timings are from my notebook (Apple Silicon M1). 
This is just a micro-benchmark and measures buffered access through the 
NaviServer DB driver infrastructure. More detailed information about 
LMDB in comparison to Berkley DB concerning large Applications and more 
micro-benchmarks is in [1]

all the best

-gb

183 ns nsv_set foo x 1; time {nsv_get foo x} 100000 213 ns time 
{ns_cache_eval ns:memoize 1 {set x 1}} 100000 253 ns time {array set x 
{a 1 b 2 c 3}} 100000 308 ns ns_urlspace set -key foo1 /*.adp A; time 
{ns_urlspace get -key foo1 /static/test.adp} 100000 >>> 511 ns set ::db 
[ns_db gethandle lmdb]; time {ns_db 0or1row $::db "GET key1"} 100000 
3633 ns time {parameter::get -package_id [ad_conn subsite_id] -parameter 
DefaultMaster -default "x"} 100000 36893 ns time {xo::dc get_value 
dbqd..qn {select title from acs_objects where object_id=221}} 100000 
42523 ns time {::mongo::collection::query $::mongoColl {name string 
Gustaf}} 100000
Times are in nano seconds, the "slowest" is here the access to MongoDB (via nsf package) and PostgreSQL.
The PostgreSQL access takes 36 micro seconds, this means, one can run ~27K of those in a second.
But with LMDB it is possible to gets nearly 2 Mio DB accesses per second.

[1]http://www.lmdb.tech/media/20130329-devox-MDB.pdf

[naviserver-devel] mobile device friendlyness and sourceforge problems

[Gustaf N. <ne...@wu...>]

Dear all,

For all people having troubles to download the new release NaviServer 
from sourceforge: there seems to be some outages today:

https://downdetector.com/status/sourceforge/

One has to patient, i am sure the good guys on sourceforge working in 
the background are busy right now.

As mentioned a couple of times, fixes are currently added to the 
release/4.99 branch, and new features are added to the main branch, 
which has the goal to be released, when Tcl 9 is out.

All push operations on NaviServer trigger a regression test, so the code 
quality can always be checked on:

https://github.com/nm-wu/naviserver-mirror/actions

Today, I've committed a change to improve the mobile friendliness of 
NaviServer in the main branch, which makes the output readable for 
humans on mobile devices. This is not terribly important, since every 
larger application that cares provides it own templating, but 
nevertheless, sometime the results of "ns_returnnotice" pop up, such as 
e.g. in the directory listings. The new version is based on a flexible 
design based on flexbox. See below for a comparison (two screenshots for 
a display on a recent iPhone).

all the best

-gn

PS: On the longer range, we should provide a fully templated approach 
for "ns_returnnotice"

old:

new





-- 
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"

[naviserver-devel] NaviServer 4.99.28 available

[Gustaf N. <ne...@wu...>]

Dear all,

I am glad to announce that the release of NaviServer 4.99.28 is 
available at SourceForge [1].  This release is a pure bug-fix and 
maintenance release, which fixes a bug annoying for some OpenACS users.
See below for a summary of the changes.

All the best!

-gustaf neumann

[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.28/

=======================================
NaviServer 4.99.28, released 2023-09-06
=======================================

  49 files changed, 851 insertions(+), 328 deletions(-)

New Features:
-------------
   - None

Bug Fixes:
----------

   - Fixed incorrect list quoting for "filter" in urlspace
     introspection. This lead to display problems, e.g. in the nsstats
     module.

   - Fixed naming Inconsistency: Unified the term "fastpath" across
     configuration, API-interface, and documentation.

   - Fixed potential crash in Ns_SetIUpdateSz(): Fixed a bug that could
     lead to a crash when updating the content-type header field with a
     lowercase field name via the old C API.

     Many thanks to Brian Fenton for reporting the problem and his help
     for debugging it.

Code Maintenance:
-----------------

   - New API Function Ns_PlainUrlPath: Checks if a given URL path is
     "plain," meaning it doesn't contain a query or fragment. Useful
     for data validation in configuration files and Tcl API calls.

   - Improved Validation when registering request handler. Previously,
     it was possible to register request handlers for paths containing
     query variables and fragments, which were never matched.

   - Added Missing "-server" Argument: The "-server" argument was added
     to ns_unregister_op and ns_unregister_url2file, allowing for more
     centralized management (e.g. from the nsstats module).

   - Added a more Portable Way of Printing NULL Pointer: Improved
     portability for printing NULL pointers across different compilers,
     which could lead to false positives in the regression test.

   - Extended and Simplified Test Cases: Several commits were made to
     improve and extend test cases across different modules and
     scenarios.

   - Improved and corrected documentation, including styling changes,
     bug fixes, and providing more detailed information.


Modules:
--------
The following list contains the most important changes:

  - Extending nsstats:

    * Introduced templating to improve style-ability
    * Provided a more ,odern GUI based on the CSS classes from w3schools
    * Added management of request handlers
    * Added management of url2file mappings

  - Rework of module nsdbbdb:
     * properly initializing mutex variables (many thanks to John Buckman)
     * support for LMDB (Lightning Memory-Mapped Database)
     * Improved configurability for Makefile
     * Modernization of C code (better following style guide)
     * Added debug support via Debug(bdb)

      Build nsdbbdb with Berkeley DB 6.2.32 (results in nsdbbdb.so)
         make DBINCLUDE=/opt/local/include/db62  DBLIB=/opt/local/lib/db62/

      Build nsdbbdb with LMDB (Lightning Memory-Mapped Database) (results in nsdblmdb.so)
         make LMDB=1 DBINCLUDE=/opt/local/include DBLIB=/opt/local/lib/

Re: [naviserver-devel] “worker process pool” feature?

[Gustaf N. <ne...@wu...>]

On 05.09.23 14:40, John at Decent wrote:
> Gustav, I think my confusion is as to how this works. As it's not 
> documented, one needs to figure it out from the source.

the man page 
https://naviserver.sourceforge.io/n/nsproxy/files/ns_proxy.html is not a 
perfect documentation, but this is what we have now, ... and this is 
certainly more than reading the source. Improvements are always welcome!

> I can't really tell what each ns_proxy::exec does if the executed 
> process doesn't exit.

a worker process is potentially started by "ns_proxy get ..." but is 
influenced by the definition of the proxy pool (e.g. "maxworkers", ...).

> Does it reuse it for the next request? It seems so, but I'm not clear.

by "request" you probably mean the next invocation of "nsproxy eval". It 
is reused, but as before, the reuse is controlled by the pool definition 
(e.g. "idletimeout", "maxruns", ...)

> If it does reuse the still-running process, how does it tell it’s the 
> same executable?

the name of the executable (per default 
/usr/local/ns/bin/nsproxy-helper) is defined via config file or via  
"nsproxy configure ... -exec XXX". The running instances are managed via 
the handles. If you want to have different executables, define different 
nsproxy pools.

-g

Re: [naviserver-devel] =?utf-8?Q?=E2=80=9Cworker_?=process =?utf-8?Q?pool=E2=80=9D_?=feature?

[John at D. <jo...@de...>]

Gustav, I think my confusion is as to how this works. As it's not documented, one needs to figure it out from the source.

I can't really tell what each ns_proxy::exec does if the executed process doesn't exit. 

Does it reuse it for the next request? It seems so, but I'm not clear.  If it does reuse the still-running process, how does it tell it’s the same executable?

-john