naviserver-devel — Developer discussion

[naviserver-devel] NaviServer 4.99.30 available

[Gustaf N. <gus...@wu...>]

Dear all,

I am glad to announce that the release of NaviServer 4.99.30 is 
available at SourceForge [1].  This release contains a small improvement 
to reduce the log file verbosity on busy sites (many thanks to Brian for 
the suggestion) and contains minor fixes for certain configurations, 
improved timing information under error conditions, and improved support 
for cluster configurations for OpenACS.

As discussed before, the repository of NaviServer was moved from 
Bitbucket to GitHub [2]. The plan is to empty the Bitbucket repository 
in the near future. Please adjust your install/update scripts to point 
to the new link. The installation scripts from [3] have been adjusted to 
GitHub as well.

All the best!

-gustaf neumann

[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.30/
[2] https://github.com/orgs/naviserver-project/repositories
[3] https://github.com/gustafn/install-ns

=======================================
NaviServer 4.99.30, released 2024-01-26
=======================================

  35 files changed, 700 insertions(+), 333 deletions(-)

New Features:
-------------

     - Global Parameter "sockacceptlog" (Commit a8b5f33c):

       Introduced a new global parameter "sockacceptlog" to define a
       threshold for logging, when to inform the user about a high number
       of incoming connections in a single accept callback.

Bug Fixes:
----------

     - Compilation Issues (Commit 0b72baa7): Fixed compilation issues
       when compiled without OpenSSL or zlib support.

     - Meaningful End-Times on Connection Errors (Commit 929d55be):
       Ensured meaningful end-times are provided during connection
       errors in ns_http.

     - Rework of "rejectalreadyclosedconn" (Commit f0963834): Reworked
       the "rejectalreadyclosedconn" for safer operations and logging.

     - Shutdown Callbacks and Spoolers (Commit 0f0ea439): Change
       shutdown order to let shutdown callbacks use spoolers. This is
       beneficial for cluster mode in OpenACS, where other cluster
       nodes should be be informed about the shutdown of a node (Used
       by OpenACS).

Changes in Sample Configuration Files:
--------------------------------------

     - Oracle Startup Configurable (Commit 1ee7fef): Made startup
       configuration for Oracle databases adjustable in
       openacs-config.tcl.

     - Use Uppercase Memory Units (Commit 2f96dc62): Standardized
       memory units to uppercase in openacs-config.tcl.

     - Removed Custom Setting for "connsperthread" (Commit 823b68b2):
       Removed the custom setting for "connsperthread" in
       openacs-config.tcl to avoid misconceptions.

     - Backported nsd-config.tcl and openacs-config.tcl from main
       branch to improve maintainability

Misc Improvements:
------------------

     - Library Initialization After Command Line Processing (Commit
       fc3ee12b): Moved library initialization to occur after command
       line processing to avoid potential unexpected output.

     - Timeout Information in Warning Messages (Commit 12fe7805):
       Enhanced warning messages to include timeout value information.

     - Improved Robustness of Regression Rest (Commit bcc309716)

     - Compatibility Improvements:
       * Improved Forward Compatibility for Tcl9 (Commit d9e15fb6)
       * Improved Cross Compilation for certain ARM Processors in .m4
         setup (Commit e1071fcf)
       * Improved Include Path for macOS (Commit 120dcafce): Updated
         the include path for cppcheck under newer macOS versions.

[naviserver-devel] Detailed memory statistics via nsstats module

[Gustaf N. <ne...@wu...>]

Dear all,

The newest version of nsstats in the repository can now report detailed 
memory statistics, when
a) NaviServer and Tcl were compiled with SYSTEM_MALLOC, and
b) tcmalloc (the memory allocator library from Google's perftools) was 
preloaded

In this case, the memory statistics (from the "Memory" pull down menu) 
provides detailed statistics, which can answer some questions about 
memory growth, memory fragmentation, and the memory release policy of 
the memory allocator library. It answers, e.g., the question, whether 
the allocator library does not want to release memory, or if it can't 
release memory.

Below is a screenshot from openacs.org. Note, that one needs both, the 
newest version of the nsstats module and a recent version of NaviServer 
(main branch), which implements the integration with tcmalloc.

A detailed description of the output can be found here:
https://github.com/google/tcmalloc/blob/master/docs/stats.md

All the best
-g


LD_PRELOAD=/opt/local/lib/libtcmalloc.9.10.5.dylib  /usr/local/ns/bin/nsd -f -t /usr/local/ns/openacs-5.10.tcl -u nsadmin -g nsadmin

Re: [naviserver-devel] Compile error for nsdbpg

[Gustaf N. <ne...@wu...>]

Dear Wolfgang,

You have to compile the newest version of nsdbpg either against

- the newest version of NaviServer from the branch "main", or against
- the newest version of NaviServer from the branch "release/4.99"

General rule: Take either the releases with matching version number from 
sourceforge, or take the newest versions of NaviServer and the modules 
from github (typically unreleased versions).

The following combinations are tested on the latest code

all the best

-g

Tested environments for NaviServer 4.99 (branch release/4.99)

           - os: ubuntu-latest
             compiler: gcc-10
             tcltag: core-8-6-13
             nsf_version: 2.3.0
             tdom_version: 0.9.1

           - os: ubuntu-latest
             compiler: gcc-11
             tcltag: core-8-7-a5
             ns_modules: nsdbpg nsdbi nsdbipg nsudp nscoap nssmtpd
             nsf_version: 2.4.0
             tdom_version: 0.9.3

Tested environments for NaviServer 5 (branch main)

           - os: ubuntu-latest
             compiler: gcc-10
             tcltag: core-8-6-13
             nsf_version: 2.3.0
             tdom_version: 0.9.1

           - os: ubuntu-latest
             compiler: gcc-12
             tcltag: core-8-7-a5
             ns_modules: nsdbpg nsdbi nsdbipg nsudp nscoap nssmtpd
             nsf_version: 2.4.0
             tdom_version: 0.9.3

           - os: ubuntu-latest
             compiler: gcc-12
             tcltag: main
             ns_modules: nsdbpg nsdbi nsdbipg nsudp nscoap nssmtpd nsloopctl
             nsf_version: HEAD
             tdom_version: 0.9.3


On 22.11.23 08:18, Wolfgang Winkler via naviserver-devel wrote:
>
> Hi all!
>
> When compiling the current version of nsdbg from
>
> https://github.com/naviserver-project/nsdbpg/tree/main
>
> I get the error:
>
> tclcmds.c:1635:37:error: expected ‘)’ before ‘PRITcl_Size’
>         sprintf(segment_pos, "%d, %"PRITcl_Size", '%s')",
> ^~~~~~~~~~~~
> )
> I locally changed the line 1635 to the old version, which works.
>
> -        sprintf(segment_pos, "%d, %" PRITcl_Size ", '%s')",
> -                segment, segment_len, out_buf);
> +sprintf(segment_pos, "%d, %d, '%s')", segment, segment_len, out_buf);
>
> Regards,
>
> wiwo
>

[naviserver-devel] Compile error for nsdbpg

[Wolfgang W. <wol...@di...>]

Hi all!

When compiling the current version of nsdbg from

https://github.com/naviserver-project/nsdbpg/tree/main

I get the error:

tclcmds.c:1635:37:error: expected ‘)’ before ‘PRITcl_Size’
         sprintf(segment_pos, "%d, %"PRITcl_Size", '%s')",
^~~~~~~~~~~~
)
I locally changed the line 1635 to the old version, which works.

-        sprintf(segment_pos, "%d, %" PRITcl_Size ", '%s')",
-                segment, segment_len, out_buf);
+sprintf(segment_pos, "%d, %d, '%s')", segment, segment_len, out_buf);

Regards,

wiwo


-- 

*Wolfgang Winkler*
Geschäftsführung
wol...@di...
mobil +43.699.19971172

dc:*büro*
digital concepts Novak Winkler OG
Software & Design
Landstraße 68, 5. Stock, 4020 Linz
www.digital-concepts.com <http://www.digital-concepts.com>
tel +43.732.997117.72
tel +43.699.1997117.2

Firmenbuchnummer: 192003h
Firmenbuchgericht: Landesgericht Linz

Re: [naviserver-devel] NaviServer 4.99.29 available

[Gustaf N. <ne...@wu...>]

Hi Brian,


There is no plan to change the policy of NaviServer on sourceforge. We 
will continue to use sourceforge for NaviServer releases in the form of 
tar files, along side with tar files for the matching modules. Github is 
for developer, code-inspection, PRs, etc.


all the best

-g


On 20.11.23 16:21, Brian Fenton wrote:
> Dear Gustaf
>
> thanks for this good work. Github does appear to be superior to 
> Bitbucket on many fronts.
>
> Will the tar.gz files for each release continue to be published to 
> https://sourceforge.net/projects/naviserver/files/naviserver/ ?
>
> best wishes
> Brian

Re: [naviserver-devel] NaviServer 4.99.29 available

[Brian F. <bri...@ai...>]

Dear Gustaf

thanks for this good work. Github does appear to be superior to Bitbucket on many fronts.

Will the tar.gz files for each release continue to be published to https://sourceforge.net/projects/naviserver/files/naviserver/ ?

best wishes
Brian

________________________________
From: Gustaf Neumann <ne...@wu...>
Sent: Sunday 19 November 2023 5:17 pm
To: nav...@li... <nav...@li...>
Subject: Re: [naviserver-devel] NaviServer 4.99.29 available


Dear all,

This is a follow-up to my own mail. The problem due to the changes of the billing rules on Bitbucket (see below) are somewhat sorted out, I have again access to the repository, but many of the former contributors are deleted from the "naviserver" group to get it functioning. There was also a change with "managed accounts", which made the problem for me even more complicated. In total, I count 31 emails with Atlassian support to sort these things out. Anyhow, these changes are signs that Atlassian does not have a high priority on open source projects with free accounts.

We have discussed this with Zoran, and we came to the conclusion that it is the best to move the NaviServer repositories to GitHub

   https://github.com/orgs/naviserver-project/repositories

We have now on GitHub an organization "naviserver-project" which contains now the 54 sub-repositories. This is essentially the same structure we had before. We also have a certain hope, that the move to GitHub will rather improve the visibility of NaviServer. All the newest commits were done on these new repositories. The plan is to move completely to GitHub and to delete the repositories on Bitbucket to avoid confusion.

Please adjust your install/update scripts to point to the new link.

All the best

-g

On 02.11.23 15:30, I wrote:
So far, these changes are only available on sourceforge, since i have lost write access to the repository at bitbucket. The people on Atlassian seem to have changed some account types, and - on to of this - they annouced via the Blog post on September 27, 2023, that the billing model changed (where they also refer to the "unified-user-management"). It took me a while to figure out, what happened. The blog post states:
>From October 31st, 2023, Bitbucket Cloud will begin counting all workspace members as a billable user. ....

Free plans: If you're on a free plan and your billable user count is higher than 5 as per the new definition of billable user, all repositories in your workspace will become read-only until you remove some users or upgrade your workspace to a paid plan.

It seems that the users of the "naviserver" group are now counted as "billable users", and it contains 19 users. According to support, we have to reduce this number to 5, otherwise nobody will be able to commit anything.

Due to the ability with PRs, i think the reduction will be possible without too much loss in functionality. If nobody objects, i will go back in history and reduce the number of commit-member based on the most recent direct commits. I hope, that non of the "old members" will be offended by this. One other option would be to upgrade to a paid plain - but i am not sure, who is gonna pay for this.

Re: [naviserver-devel] NaviServer 4.99.29 available

[Gustaf N. <ne...@wu...>]

Dear all,

This is a follow-up to my own mail. The problem due to the changes of 
the billing rules on Bitbucket (see below) are somewhat sorted out, I 
have again access to the repository, but many of the former contributors 
are deleted from the "naviserver" group to get it functioning. There was 
also a change with "managed accounts", which made the problem for me 
even more complicated. In total, I count 31 emails with Atlassian 
support to sort these things out. Anyhow, these changes are signs that 
Atlassian does not have a high priority on open source projects with 
free accounts.

We have discussed this with Zoran, and we came to the conclusion that it 
is the best to move the NaviServer repositories to GitHub

https://github.com/orgs/naviserver-project/repositories

We have now on GitHub an organization "naviserver-project" which 
contains now the 54 sub-repositories. This is essentially the same 
structure we had before. We also have a certain hope, that the move to 
GitHub will rather improve the visibility of NaviServer. All the newest 
commits were done on these new repositories. The plan is to move 
completely to GitHub and to delete the repositories on Bitbucket to 
avoid confusion.

Please adjust your install/update scripts to point to the new link.

All the best

-g

On 02.11.23 15:30, I wrote:
> So far, these changes are only available on sourceforge, since i have 
> lost write access to the repository at bitbucket. The people on 
> Atlassian seem to have changed some account types, and - on to of this 
> - they annouced via the Blog post on September 27, 2023, that the 
> billing model changed (where they also refer to the 
> "unified-user-management"). It took me a while to figure out, what 
> happened. The blog post states:
>
>     /From October 31st, 2023, Bitbucket Cloud will begin counting all
>     workspace members as a billable user. ....
>
>     Free plans: If you're on a free plan and your billable user count
>     is higher than 5 as per the new definition of billable user, all
>     repositories in your workspace will become read-only until you
>     remove some users or upgrade your workspace to a paid plan./
>
> It seems that the users of the "naviserver" group are now counted as 
> "billable users", and it contains 19 users. According to support, we 
> have to reduce this number to 5, otherwise nobody will be able to 
> commit anything.
>
> Due to the ability with PRs, i think the reduction will be possible 
> without too much loss in functionality. If nobody objects, i will go 
> back in history and reduce the number of commit-member based on the 
> most recent direct commits. I hope, that non of the "old members" will 
> be offended by this. One other option would be to upgrade to a paid 
> plain - but i am not sure, who is gonna pay for this.
>

Re: [naviserver-devel] A selection of messages appearing in our error logs

[Brian F. <bri...@ai...>]

I enabled 2 driverthreads each on both nsssl and nssock on our Docker system, and I think it's working ,based on these driver stats:

thread nsssl:1 module nsssl received 5K spooled 0 partial 5K errors 0
thread nsssl:0 module nsssl received 5K spooled 0 partial 5K errors 0
thread nssock:1 module nssock received 135 spooled 0 partial 135 errors 0
thread nssock:0 module nssock received 136 spooled 0 partial 136 errors 0

Still seeing a huge amount of these in the log though:
[10/Nov/2023:16:08:08][37.7fb2857fa640][-driver:nsssl:0-] Notice: ... sockAccept accepted 2 connections
[10/Nov/2023:16:08:16][37.7fb286ffd640][-driver:nsssl:1-] Notice: ... sockAccept accepted 2 connections
[10/Nov/2023:16:08:17][37.7fb2857fa640][-driver:nsssl:0-] Notice: ... sockAccept accepted 2 connections
[10/Nov/2023:16:08:21][37.7fb286ffd640][-driver:nsssl:1-] Notice: ... sockAccept accepted 2 connections
[10/Nov/2023:16:08:26][37.7fb2857fa640][-driver:nsssl:0-] Notice: ... sockAccept accepted 2 connections

I'll leave it running anyway, and see how it goes.

thanks!
Brian
________________________________
From: Gustaf Neumann <ne...@wu...>
Sent: Thursday 9 November 2023 3:27 pm
To: nav...@li... <nav...@li...>
Subject: Re: [naviserver-devel] A selection of messages appearing in our error logs



On 08.11.23 15:42, Brian Fenton wrote:
Also regarding configuring a second driver thread - just to be clear are you referring to this mail you sent back in 2016 https://sourceforge.net/p/naviserver/mailman/message/35502664/  i.e. enable reuseport and set driverthreads = 2? It's unclear to me if this has benefit when running Naviserver in a Docker container - definitely beyond my level of comprehension. See https://github.com/moby/moby/issues/7536#issuecomment-1039443599 for discussion of SO_REUSEPORT and Docker.

SO_REUSEPORT allows multiple sockets on the same host to bind to the same port at the same time, therefore, multiple driver threads can listen at the same time at the same port. I do not see, why this should not be possible in a container, ... but i am not a container expert. You can try a configuration with two driver threads, if it does not reduce the number of entries, then undo it.


On a configuration with e.g. two spooler threads, it should look like the following, distributing the requests more or less evenly


   thread nsssl:1 module nsssl received 890K spooled 422 partial 437K errors 0
   thread nsssl:0 module nsssl received 935K spooled 27K partial 491K errors 0


If the load distribution does not work on your docker machine (only one driver thread is doing all the work), then a configuration with two driver threads is indeed useless. Since the driver threads will work on different cores, this improving concurrency and scalability. Giving cores something to do is always preferable. But also, when one is defining two driver threads for the same port, don't expect that the messages will go away completely.


3. "exiting: exceeded max connections per thread" - actually we used to have "connsperthread" set to 10000, but I reduced to 1000 when we moved to Linux, as I noticed that was the value in latest OpenACS config, even though the default is still 10000 https://bitbucket.org/naviserver/naviserver/src/main/openacs-config.tcl  Is 1000 the recommended value for OpenACS?

The value of 1000 is probably a "traditional" value, which is forgiving for sloppy developers not cleaning up their garbage after a request. I have commented out the line in the repository such that users don't draw wrong conclusions.


-g

Re: [naviserver-devel] A selection of messages appearing in our error logs

[Gustaf N. <ne...@wu...>]

On 08.11.23 15:42, Brian Fenton wrote:
> Also regarding configuring a second driver thread - just to be clear 
> are you referring to this mail you sent back in 2016 
> https://sourceforge.net/p/naviserver/mailman/message/35502664/ i.e. 
> enable reuseport and set driverthreads = 2? It's unclear to me if this 
> has benefit when running Naviserver in a Docker container - definitely 
> beyond my level of comprehension. See 
> https://github.com/moby/moby/issues/7536#issuecomment-1039443599 for 
> discussion of SO_REUSEPORT and Docker.

SO_REUSEPORT allows multiple sockets on the same host to bind to the 
same port at the same time, therefore, multiple driver threads can 
listen at the same time at the same port. I do not see, why this should 
not be possible in a container, ... but i am not a container expert. You 
can try a configuration with two driver threads, if it does not reduce 
the number of entries, then undo it.


On a configuration with e.g. two spooler threads, it should look like 
the following, distributing the requests more or less evenly


    thread nsssl:1 module nsssl received 890K spooled 422 partial 437K 
errors 0
    thread nsssl:0 module nsssl received 935K spooled 27K partial 491K 
errors 0


If the load distribution does not work on your docker machine (only one 
driver thread is doing all the work), then a configuration with two 
driver threads is indeed useless. Since the driver threads will work on 
different cores, this improving concurrency and scalability. Giving 
cores something to do is always preferable. But also, when one is 
defining two driver threads for the same port, don't expect that the 
messages will go away completely.


> 3. "exiting: exceeded max connections per thread" - actually we used 
> to have "connsperthread" set to 10000, but I reduced to 1000 when we 
> moved to Linux, as I noticed that was the value in latest OpenACS 
> config, even though the default is still 10000 
> https://bitbucket.org/naviserver/naviserver/src/main/openacs-config.tcl 
> Is 1000 the recommended value for OpenACS?

The value of 1000 is probably a "traditional" value, which is forgiving 
for sloppy developers not cleaning up their garbage after a request. I 
have commented out the line in the repository such that users don't draw 
wrong conclusions.


-g

Re: [naviserver-devel] A selection of messages appearing in our error logs

[Brian F. <bri...@ai...>]

Hi Gustaf

thank you so much for such a comprehensive reply. I feel like I just increased my IQ by a few percentage! A couple of follow up questions, if I may:

1. "sockAccept accepted 2 connections" - this was very useful to know, I will start keeping an eye out for values > 10. It would be very useful to be able to configure the threshold at which it appears in the log, as I mentioned we see a huge amount of these (over 2000 in the last few hours on our test server, almost all of them for 2 connections).
Also regarding configuring a second driver thread - just to be clear are you referring to this mail you sent back in 2016 https://sourceforge.net/p/naviserver/mailman/message/35502664/  i.e. enable reuseport and set driverthreads = 2? It's unclear to me if this has benefit when running Naviserver in a Docker container - definitely beyond my level of comprehension. See https://github.com/moby/moby/issues/7536#issuecomment-1039443599 for discussion of SO_REUSEPORT and Docker.

2. "ns_cache create entry collision cache util_memoize key '__util_memoize_installed_p" - again very helpful. We actually have been partitioning our caches, and try not to add anything new to util-memoize cache. In fact, this particular message about  '__util_memoize_installed_p' is the only cache collision notice we are seeing. Your comment helped me to track down to issue to the "util_memoize_initialized_p" and "apm_package_installed_p" procs, and I can see the new improved versions in later OpenACS versions, so I will try those out.

3. "exiting: exceeded max connections per thread" - actually we used to have "connsperthread" set to 10000, but I reduced to 1000 when we moved to Linux, as I noticed that was the value in latest OpenACS config, even though the default is still 10000 https://bitbucket.org/naviserver/naviserver/src/main/openacs-config.tcl  Is 1000 the recommended value for OpenACS?

thanks again
Brian
________________________________
From: Gustaf Neumann <ne...@wu...>
Sent: Wednesday 8 November 2023 9:00 am
To: nav...@li... <nav...@li...>
Subject: Re: [naviserver-devel] A selection of messages appearing in our error logs


Hi Brian,


In general, we try to follow with the logging severities the guideline [1]. "Notice" is informational, "warning" is something unexpected/unwanted (hitting some limits, got some data that looks like an attack, ...), "error" is something which should be looked at by an engineer.


    [1] https://openacs.org/xowiki/logging-conventions


below, I comment inline.



On 07.11.23 18:28, Brian Fenton wrote:
Hi

with the goal of improving signal-to-noise in our error logs, and also to increase my understanding of Naviserver internals, I have a few questions about various different messages we are seeing regularly in our logs.


  1.  [06/Nov/2023:14:58:08][77320.7f7b9a7fd640][-driver:nsssl:0-] Notice: ... sockAccept accepted 2 connections
this is by far the most common entry in our logs - we see a LOT of these. Are they merely informative or indicative of some tuning action we could take? I don't see any way to disable them. I would love to remove these.


This indicates some stress situation of the server, where multiple requests come in at the same time. The value of 2 is not of a concern, if you see values like 10 or 20, you should consider a change in your configuration (e.g., configuring a second driver thread). It would be possible to define a configuration value for the network driver to set the threshold when the notice is written to the log file.


  1.

  2.
[06/Nov/2023:14:57:28][77320.7f7b63fff640][-conn:openacs:default:6:6292-] Notice: ns_cache create entry collision cache util_memoize key '__util_memoize_installed_p', no timeout
We also see a quite a few of these, always for  '__util_memoize_installed_p', and always with 'no timeout'. From looking at cache.c I see  that this means that Naviserver failed to create the cache entry. I checked the 'util_memoize' cache utilization in case the cache was full, but it's quite low. What other cause could there be for this?

This means that concurrent operations are executed to obtain the same cache value, and that the second request (here conn:openacs:default:6:6292) has to wait until the locking process has finished. The problem can become serious, when the locking process "hangs", i.e. taking a very long time to finish. This means, that more and more of caching requests for the same entry will pile up, potentially until a restart, since no timeout was specified. The warning was introduced to give the developer a hint, why suddenly the server is hanging (before the message was introduced, the cause was very hard to determine).


This message tells as well something else:
* It looks as if the installation is based on a very old version of openacs-core (including in particular acs-tcl) installed. At least since 9 years, the code has changed to perform this test just at startup. Upgrade to recent versions will improve the performance and increase security.


* The message tells me as well that the application has a stress on the util-memoize cache. Old versions of OpenACS used this "util-memoize" cache as a kitchen-sink cache for everything. I've seen instances having 300K + entries in this cache. The size itself is not the problem, but there were/are many operations in OpenACS that obtain all keys from the cache. It means that a lock is created, a list with 300K+ entries is created and handed to the interpreter before the cache is released. In the meantime, *all* caching requests for this cache have to be halted, ... causing in high load situations degraded performance and limited concurrency. I would not be surprised, if you see high lock times on this cache.


There are caches aside of the util-memoize cache, where cache collisions might happen. See e.g. [2] for more background. Cache partitioning is in this situation the instrument for scaling.


   [2] https://openacs.org/xowiki/cache-size-tuning


  1.

  2.  These 5 appear usually as a group:
[07/Nov/2023:13:46:07][38.7f352cff9640][-conn:openacs:default:3:4088-] Notice: exiting: exceeded max connections per thread
[07/Nov/2023:13:46:07][38.7f3525b84640][-driver:nsssl:0-] Notice: NsEnsureRunningConnectionThreads wantCreate 1 waiting 1 idle 0 current 2
[07/Nov/2023:13:46:07][38.7f352d7fa640][-conn:openacs:default:5:0-] Notice: start update interpreter openacs to epoch 1, concurrent 1
[07/Nov/2023:13:46:08][38.7f352d7fa640][-conn:openacs:default:5:0-] Notice: update interpreter openacs to epoch 1 done, trace none, time 0.373006 secs concurrent 1
[07/Nov/2023:13:46:08][38.7f352d7fa640][-conn:openacs:default:5:0-] Notice: thread initialized (0.388878 secs)
We see these pretty regularly, over 50 per day on  a  local development server.  "maxconnections" is set to the  default value, which is 100, I believe.  "maxthreads" and "minthreads" are also set to default values. What could  be causing these so regularly on a system with 1 logged-in user?


The message tells that a connection thread has reached its end of live, as defined by the "connsperthread" parameter. When this happens, the thread is stopped, and depending on the configuration parameters, a new thread is created maybe immediately or on demand. The reason for stopping connection threads from time to time is to clean up some interpreter specific data, which was accumulated by some applications. With the modern versions of OpenACS and with well-behaved packages, this is not an issue, the value can be increased substantially.


The relevant counter does not depend on the number of different users, but on the number of requests handled by a particular connection thread.


  1.

  2.
 These 3 are related to each other,  I presume:
[07/Nov/2023:15:36:24][38.7f3536b86640][-conn:openacs:default:28:28082-] Notice: nsdb: closing old handle in pool 'pool1'
[07/Nov/2023:15:45:39][38.7f3536b86640][-conn:openacs:default:31:31287-] Notice: nsdb: closing idle handle in pool 'pool1'
[07/Nov/2023:15:45:39][38.7f3536b86640][-conn:openacs:default:31:31287-] Notice: dbdrv: opening database 'nsoracle:192.168.1.1/ORACLE'
These are quite frequent too. Am I correct in thinking that we can tune these with pool "maxidle" and "maxopen" params mentioned here https://naviserver.sourceforge.io/n/nsdb/files/ns_db.html ? Is there any reason not to turn off the automatic closing feature?

There might be some reasons for keeping the number of db connections low (e.g. many servers use the same database with a limited number of client connections, etc.), since at least 20 years there is no technical reason for the common db drivers not to turn off the automatic closing feature. Reconnecting to the database can be costly, when the db is running on a different host and/or the connection is via HTTPS.


hope this helps a little


all the best

-g

Re: [naviserver-devel] test ns_config-7.4.1, Stack around the variable 'filter' was corrupted

[Gustaf N. <ne...@wu...>]

On 07.11.23 22:56, Andrew Piskorski wrote:
> On the current NaviServer head, that problem has gone away!  The
> ns_config-7.4.1 test now runs fine for me on Windows.

You see: time heals all wounds! -g

Re: [naviserver-devel] A selection of messages appearing in our error logs

[Gustaf N. <ne...@wu...>]

Hi Brian,


In general, we try to follow with the logging severities the guideline 
[1]. "Notice" is informational, "warning" is something 
unexpected/unwanted (hitting some limits, got some data that looks like 
an attack, ...), "error" is something which should be looked at by an 
engineer.


     [1] https://openacs.org/xowiki/logging-conventions


below, I comment inline.



On 07.11.23 18:28, Brian Fenton wrote:
> Hi
>
> with the goal of improving signal-to-noise in our error logs, and also 
> to increase my understanding of Naviserver internals, I have a few 
> questions about various different messages we are seeing regularly in 
> our logs.
>
>  1. [06/Nov/2023:14:58:08][77320.7f7b9a7fd640][-driver:nsssl:0-]
>     Notice: ... sockAccept accepted 2 connections
>     this is by far the most common entry in our logs - we see a LOT of
>     these. Are they merely informative or indicative of some tuning
>     action we could take? I don't see any way to disable them. I would
>     love to remove these.
>

This indicates some stress situation of the server, where multiple 
requests come in at the same time. The value of 2 is not of a concern, 
if you see values like 10 or 20, you should consider a change in your 
configuration (e.g., configuring a second driver thread). It would be 
possible to define a configuration value for the network driver to set 
the threshold when the notice is written to the log file.


> 1.
>
> 2.
>     [06/Nov/2023:14:57:28][77320.7f7b63fff640][-conn:openacs:default:6:6292-]
>     Notice: ns_cache create entry collision cache util_memoize key
>     '__util_memoize_installed_p', no timeout
>     We also see a quite a few of these, always for
>     '__util_memoize_installed_p', and always with 'no timeout'. From
>     looking at cache.c I see  that this means that Naviserver failed
>     to create the cache entry. I checked the 'util_memoize' cache
>     utilization in case the cache was full, but it's quite low. What
>     other cause could there be for this?
>
This means that concurrent operations are executed to obtain the same 
cache value, and that the second request (here 
conn:openacs:default:6:6292) has to wait until the locking process has 
finished. The problem can become serious, when the locking process 
"hangs", i.e. taking a very long time to finish. This means, that more 
and more of caching requests for the same entry will pile up, 
potentially until a restart, since no timeout was specified. The warning 
was introduced to give the developer a hint, why suddenly the server is 
hanging (before the message was introduced, the cause was very hard to 
determine).


This message tells as well something else:
* It looks as if the installation is based on a very old version of 
openacs-core (including in particular acs-tcl) installed. At least since 
9 years, the code has changed to perform this test just at startup. 
Upgrade to recent versions will improve the performance and increase 
security.


* The message tells me as well that the application has a stress on the 
util-memoize cache. Old versions of OpenACS used this "util-memoize" 
cache as a kitchen-sink cache for everything. I've seen instances having 
300K + entries in this cache. The size itself is not the problem, but 
there were/are many operations in OpenACS that obtain all keys from the 
cache. It means that a lock is created, a list with 300K+ entries is 
created and handed to the interpreter before the cache is released. In 
the meantime, *all* caching requests for this cache have to be halted, 
... causing in high load situations degraded performance and limited 
concurrency. I would not be surprised, if you see high lock times on 
this cache.


There are caches aside of the util-memoize cache, where cache collisions 
might happen. See e.g. [2] for more background. Cache partitioning is in 
this situation the instrument for scaling.


    [2] https://openacs.org/xowiki/cache-size-tuning


> 1.
>  2. These 5 appear usually as a group:
>     [07/Nov/2023:13:46:07][38.7f352cff9640][-conn:openacs:default:3:4088-]
>     Notice: exiting: exceeded max connections per thread
>     [07/Nov/2023:13:46:07][38.7f3525b84640][-driver:nsssl:0-] Notice:
>     NsEnsureRunningConnectionThreads wantCreate 1 waiting 1 idle 0
>     current 2
>     [07/Nov/2023:13:46:07][38.7f352d7fa640][-conn:openacs:default:5:0-]
>     Notice: start update interpreter openacs to epoch 1, concurrent 1
>     [07/Nov/2023:13:46:08][38.7f352d7fa640][-conn:openacs:default:5:0-]
>     Notice: update interpreter openacs to epoch 1 done, trace none,
>     time 0.373006 secs concurrent 1
>     [07/Nov/2023:13:46:08][38.7f352d7fa640][-conn:openacs:default:5:0-]
>     Notice: thread initialized (0.388878 secs)
>     We see these pretty regularly, over 50 per day on  a  local
>     development server.  "maxconnections" is set to the default value,
>     which is 100, I believe.  "maxthreads" and "minthreads" are also
>     set to default values. What could  be causing these so regularly
>     on a system with 1 logged-in user?
>

The message tells that a connection thread has reached its end of live, 
as defined by the "connsperthread" parameter. When this happens, the 
thread is stopped, and depending on the configuration parameters, a new 
thread is created maybe immediately or on demand. The reason for 
stopping connection threads from time to time is to clean up some 
interpreter specific data, which was accumulated by some applications. 
With the modern versions of OpenACS and with well-behaved packages, this 
is not an issue, the value can be increased substantially.


The relevant counter does not depend on the number of different users, 
but on the number of requests handled by a particular connection thread.


> 1.
>
> 2.
>      These 3 are related to each other,  I presume:
>     [07/Nov/2023:15:36:24][38.7f3536b86640][-conn:openacs:default:28:28082-]
>     Notice: nsdb: closing old handle in pool 'pool1'
>     [07/Nov/2023:15:45:39][38.7f3536b86640][-conn:openacs:default:31:31287-]
>     Notice: nsdb: closing idle handle in pool 'pool1'
>     [07/Nov/2023:15:45:39][38.7f3536b86640][-conn:openacs:default:31:31287-]
>     Notice: dbdrv: opening database 'nsoracle:192.168.1.1/ORACLE'
>     These are quite frequent too. Am I correct in thinking that we can
>     tune these with pool "maxidle" and "maxopen" params mentioned here
>     https://naviserver.sourceforge.io/n/nsdb/files/ns_db.html ? Is
>     there any reason not to turn off the automatic closing feature?
>
There might be some reasons for keeping the number of db connections low 
(e.g. many servers use the same database with a limited number of client 
connections, etc.), since at least 20 years there is no technical reason 
for the common db drivers not to turn off the automatic closing feature. 
Reconnecting to the database can be costly, when the db is running on a 
different host and/or the connection is via HTTPS.


hope this helps a little


all the best

-g

[naviserver-devel] A selection of messages appearing in our error logs

[Brian F. <bri...@ai...>]

Hi

with the goal of improving signal-to-noise in our error logs, and also to increase my understanding of Naviserver internals, I have a few questions about various different messages we are seeing regularly in our logs.


  1.  [06/Nov/2023:14:58:08][77320.7f7b9a7fd640][-driver:nsssl:0-] Notice: ... sockAccept accepted 2 connections
this is by far the most common entry in our logs - we see a LOT of these. Are they merely informative or indicative of some tuning action we could take? I don't see any way to disable them. I would love to remove these.

  2.
[06/Nov/2023:14:57:28][77320.7f7b63fff640][-conn:openacs:default:6:6292-] Notice: ns_cache create entry collision cache util_memoize key '__util_memoize_installed_p', no timeout
We also see a quite a few of these, always for  '__util_memoize_installed_p', and always with 'no timeout'. From looking at cache.c I see  that this means that Naviserver failed to create the cache entry. I checked the 'util_memoize' cache utilization in case the cache was full, but it's quite low. What other cause could there be for this?

  3.  These 5 appear usually as a group:
[07/Nov/2023:13:46:07][38.7f352cff9640][-conn:openacs:default:3:4088-] Notice: exiting: exceeded max connections per thread
[07/Nov/2023:13:46:07][38.7f3525b84640][-driver:nsssl:0-] Notice: NsEnsureRunningConnectionThreads wantCreate 1 waiting 1 idle 0 current 2
[07/Nov/2023:13:46:07][38.7f352d7fa640][-conn:openacs:default:5:0-] Notice: start update interpreter openacs to epoch 1, concurrent 1
[07/Nov/2023:13:46:08][38.7f352d7fa640][-conn:openacs:default:5:0-] Notice: update interpreter openacs to epoch 1 done, trace none, time 0.373006 secs concurrent 1
[07/Nov/2023:13:46:08][38.7f352d7fa640][-conn:openacs:default:5:0-] Notice: thread initialized (0.388878 secs)
We see these pretty regularly, over 50 per day on  a  local development server.  "maxconnections" is set to the  default value, which is 100, I believe.  "maxthreads" and "minthreads" are also set to default values. What could  be causing these so regularly on a system with 1 logged-in user?

  4.
 These 3 are related to each other,  I presume:
[07/Nov/2023:15:36:24][38.7f3536b86640][-conn:openacs:default:28:28082-] Notice: nsdb: closing old handle in pool 'pool1'
[07/Nov/2023:15:45:39][38.7f3536b86640][-conn:openacs:default:31:31287-] Notice: nsdb: closing idle handle in pool 'pool1'
[07/Nov/2023:15:45:39][38.7f3536b86640][-conn:openacs:default:31:31287-] Notice: dbdrv: opening database 'nsoracle:192.168.1.1/ORACLE'
These are quite frequent too. Am I correct in thinking that we can tune these with pool "maxidle" and "maxopen" params mentioned here https://naviserver.sourceforge.io/n/nsdb/files/ns_db.html ? Is there any reason not to turn off the automatic closing feature?

many thanks in advance for any tips
Brian


  1.

Re: [naviserver-devel] test ns_config-7.4.1, Stack around the variable 'filter' was corrupted

[Andrew P. <at...@pi...>]

On Wed, Mar 15, 2023 at 03:57:40PM -0400, Andrew Piskorski wrote:
> On Tue, Mar 14, 2023 at 04:44:21PM -0400, Andrew Piskorski wrote:
> 
> >   Debug Error!
> >   Program:  C:\web\ns-fork-pub\naviserver\nsd\libnsd.dll
> >   Run-Time Check Failure #2 - Stack around the variable 'filter' was corrupted.
> > 
> > I never see the usual "all.tcl" test summary output.  Maybe nsd is
> > hitting the above "Debug Error!" before getting there?
> 
> On Windows, running the "ns_config.test" tests triggers that one.  All
> the tests through ns_config-7.4.0, pass, then it stops with no further
> output.  Yep, the next test, ns_config-7.4.1, is sufficient to trigger
> the problem all by itself.

On the current NaviServer head, that problem has gone away!  The
ns_config-7.4.1 test now runs fine for me on Windows.

-- 
Andrew Piskorski <at...@pi...>

Re: [naviserver-devel] NaviServer 4.99.29 available

[Gustaf N. <ne...@wu...>]

Hi Brian,


The proper solution is not to skip the error message, but to skip the 
full connection output operations in error situations. Since this 
involves many commands (from ns_return*, ns_write, ns_cookie* ...) this 
is a larger change. I will look into this in the next days. Should be 
doable with moderate effort.


all the best

-g

PS: bitbucket recovery still takes time, i am now at >15 email messages 
with support from support of atlassian and/or bitbucket.


On 06.11.23 15:11, Brian Fenton wrote:
> Hi Gustaf
>
> my apologies, I hadn't realised that silencing the log would lead to 
> different behaviour. If this is a bigger job than expected, please 
> feel free to revert back to previous version. It's a nice-to-have 
> feature for us.
>
> The attached script reproduces the issue when the parameter is set to 
> false.
>
> thanks
> Brian
>
> ------------------------------------------------------------------------
> *From:* Gustaf Neumann <ne...@wu...>
> *Sent:* Monday 6 November 2023 1:59 pm
> *To:* nav...@li... 
> <nav...@li...>
> *Subject:* Re: [naviserver-devel] NaviServer 4.99.29 available
>
> Hi Brian,
>
>
> as stated several times, the right action is to fix your script (as 
> you did) rather than "silencing" NaviServer. I am not surprised, that 
> attempts to write on detached connections can lead to error conditions 
> on several occasions (generating errors avoids this).
>
>
> But since we offer this silencing parameter, i do agree, the crashing 
> is harsh. If you could send a short script triggering the problem it 
> would help to work on such cases.
>
>
> all the best
>
> -g
>
>
> On 06.11.23 14:25, Brian Fenton wrote:
>> Hi Gustaf
>>
>> I just built and ran some tests on the "rejectalreadyclosedconn" 
>> parameter to see how it handles code that triggers the "connection 
>> socket is detached" error.
>>
>>  If I set "rejectalreadyclosedconn"to false, and browse to a page 
>> that triggers the "connection socket is detached" error, Naviserver 
>> crashes with the following error message:
>>
>> [06/Nov/2023:13:13:21][39.7f3489fb9640][-conn:openacs:default:1:30-] 
>> Warning: NsWriterQueue: called without sockPtr size 414 bufs 1 flags 
>> 1030431 stream 000000 chan (nil) fd -1
>> [06/Nov/2023:13:13:21][39.7f3489fb9640][-conn:openacs:default:1:30-] 
>> Fatal: received fatal signal 11
>>
>> If I then fix the code that was triggering the "connection socket is 
>> detached" error, by adding the missing "return" after the offending 
>> "ad_returnredirect", everything works fine.
>>
>> Let me know if you need more info to help reproduce this.
>> thanks,
>> Brian
>>
>> ------------------------------------------------------------------------
>> *From:* Gustaf Neumann <ne...@wu...> <mailto:ne...@wu...>
>> *Sent:* Thursday 2 November 2023 2:30 pm
>> *To:* Navidevel <nav...@li...> 
>> <mailto:nav...@li...>
>> *Subject:* [naviserver-devel] NaviServer 4.99.29 available
>> Dear all,
>>
>> I am glad to announce that the release of NaviServer 4.99.29 is 
>> available at SourceForge [1].  This release is a pure bug-fix and 
>> maintenance release, which fixes a potentiall serious memory leak 
>> when working with PostgreSQL and large text contents. Furthermore, 
>> the release contains a small enhancement as requested by Brian not 
>> very long ago on this list.
>>
>> See below for a summary of the changes.
>>
>> So far, these changes are only available on sourceforge, since i have 
>> lost write access to the repository at bitbucket. The people on 
>> Atlassian seem to have changed some account types, and - on to of 
>> this - they annouced via the Blog post on September 27, 2023, that 
>> the billing model changed (where they also refer to the 
>> "unified-user-management"). It took me a while to figure out, what 
>> happened. The blog post states:
>>
>>     /From October 31st, 2023, Bitbucket Cloud will begin counting all
>>     workspace members as a billable user. ....
>>
>>     Free plans: If you're on a free plan and your billable user count
>>     is higher than 5 as per the new definition of billable user, all
>>     repositories in your workspace will become read-only until you
>>     remove some users or upgrade your workspace to a paid plan./
>>
>> It seems that the users of the "naviserver" group are now counted as 
>> "billable users", and it contains 19 users. According to support, we 
>> have to reduce this number to 5, otherwise nobody will be able to 
>> commit anything.
>>
>> Due to the ability with PRs, i think the reduction will be possible 
>> without too much loss in functionality. If nobody objects, i will go 
>> back in history and reduce the number of commit-member based on the 
>> most recent direct commits. I hope, that non of the "old members" 
>> will be offended by this. One other option would be to upgrade to a 
>> paid plain - but i am not sure, who is gonna pay for this.
>>
>> All the best!
>>
>> -gustaf neumann
>>
>> [1] 
>> https://sourceforge.net/projects/naviserver/files/naviserver/4.99.29/ 
>> <https://sourceforge.net/projects/naviserver/files/naviserver/4.99.29/>
>> [2] https://bitbucket.org/blog/billing-model-change 
>> <https://bitbucket.org/blog/billing-model-change>
>>
>> =======================================
>> NaviServer 4.99.29, released 2023-11-01
>> =======================================
>>
>>   37 files changed, 261 insertions(+), 132 deletions(-)
>>
>> New Features:
>> -------------
>>   - Eased configuration of simple setups
>>     * don't require to specify a "defaultserver" when a single
>>       server is in use.
>>
>>     * reduce warnings for per-server network drivers. This is a not
>>       recommended but possible configuration, global network drivers
>>       should be used.
>>
>>   - The configuration option "rejectalreadyclosedconn", which warns
>>     about attempts to send data to the web client at times, when the
>>     connection is not available anymore, is now applied on closed and
>>     detached connections. Before it was only applied on closed
>>     connections, causing potentially many warnings for legacy
>>     applications.
>>
>> Bug Fixes:
>> ----------
>>
>>    - Fixed a potential memory leak introduced two releases ago (in
>>      4.99.27).
>>
>>    - Fixed a potential compilation problem with glibc 2.38 or newer
>>      (released 31 Jul 2023)
>>
>>    - Fixed reloading of certificates for mass virtual hosting
>>
>> Code Maintenance:
>> -----------------
>>
>>    - fixed typos
>>    - fixed enum/int conversion flagged by gcc13
>>
>>
>> Modules:
>> --------
>> The following list contains the most important changes:
>>
>>   - module nsdbpg:
>>     fixed memory leak (see above).

Re: [naviserver-devel] NaviServer 4.99.29 available

[Brian F. <bri...@ai...>]

Hi Gustaf

my apologies, I hadn't realised that silencing the log would lead to different behaviour. If this is a bigger job than expected, please feel free to revert back to previous version. It's a nice-to-have feature for us.

The attached script reproduces the issue when the parameter is set to false.

thanks
Brian

________________________________
From: Gustaf Neumann <ne...@wu...>
Sent: Monday 6 November 2023 1:59 pm
To: nav...@li... <nav...@li...>
Subject: Re: [naviserver-devel] NaviServer 4.99.29 available


Hi Brian,


as stated several times, the right action is to fix your script (as you did) rather than "silencing" NaviServer. I am not surprised, that attempts to write on detached connections can lead to error conditions on several occasions (generating errors avoids this).


But since we offer this silencing parameter, i do agree, the crashing is harsh. If you could send a short script triggering the problem it would help to work on such cases.


all the best

-g


On 06.11.23 14:25, Brian Fenton wrote:
Hi Gustaf

I just built and ran some tests on the "rejectalreadyclosedconn" parameter to see how it handles code that triggers the "connection socket is detached" error.

 If I set "rejectalreadyclosedconn" to false, and browse to a page that triggers the "connection socket is detached" error, Naviserver crashes with the following error message:

[06/Nov/2023:13:13:21][39.7f3489fb9640][-conn:openacs:default:1:30-] Warning: NsWriterQueue: called without sockPtr size 414 bufs 1 flags 1030431 stream 000000 chan (nil) fd -1
[06/Nov/2023:13:13:21][39.7f3489fb9640][-conn:openacs:default:1:30-] Fatal: received fatal signal 11

If I then fix the code that was triggering the "connection socket is detached" error, by adding the missing "return" after the offending "ad_returnredirect", everything works fine.

Let me know if you need more info to help reproduce this.
thanks,
Brian

________________________________
From: Gustaf Neumann <ne...@wu...><mailto:ne...@wu...>
Sent: Thursday 2 November 2023 2:30 pm
To: Navidevel <nav...@li...><mailto:nav...@li...>
Subject: [naviserver-devel] NaviServer 4.99.29 available

Dear all,

I am glad to announce that the release of NaviServer 4.99.29 is available at SourceForge [1].  This release is a pure bug-fix and maintenance release, which fixes a potentiall serious memory leak when working with PostgreSQL and large text contents. Furthermore, the release contains a small enhancement as requested by Brian not very long ago on this list.

See below for a summary of the changes.

So far, these changes are only available on sourceforge, since i have lost write access to the repository at bitbucket. The people on Atlassian seem to have changed some account types, and - on to of this - they annouced via the Blog post on September 27, 2023, that the billing model changed (where they also refer to the "unified-user-management"). It took me a while to figure out, what happened. The blog post states:

>From October 31st, 2023, Bitbucket Cloud will begin counting all workspace members as a billable user. ....

Free plans: If you're on a free plan and your billable user count is higher than 5 as per the new definition of billable user, all repositories in your workspace will become read-only until you remove some users or upgrade your workspace to a paid plan.

It seems that the users of the "naviserver" group are now counted as "billable users", and it contains 19 users. According to support, we have to reduce this number to 5, otherwise nobody will be able to commit anything.

Due to the ability with PRs, i think the reduction will be possible without too much loss in functionality. If nobody objects, i will go back in history and reduce the number of commit-member based on the most recent direct commits. I hope, that non of the "old members" will be offended by this. One other option would be to upgrade to a paid plain - but i am not sure, who is gonna pay for this.

All the best!

-gustaf neumann

[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.29/
[2] https://bitbucket.org/blog/billing-model-change


=======================================
NaviServer 4.99.29, released 2023-11-01
=======================================

 37 files changed, 261 insertions(+), 132 deletions(-)

New Features:
-------------
 - Eased configuration of simple setups
   * don't require to specify a "defaultserver" when a single
     server is in use.

   * reduce warnings for per-server network drivers. This is a not
     recommended but possible configuration, global network drivers
     should be used.

 - The configuration option "rejectalreadyclosedconn", which warns
   about attempts to send data to the web client at times, when the
   connection is not available anymore, is now applied on closed and
   detached connections. Before it was only applied on closed
   connections, causing potentially many warnings for legacy
   applications.

Bug Fixes:
----------

  - Fixed a potential memory leak introduced two releases ago (in
    4.99.27).

  - Fixed a potential compilation problem with glibc 2.38 or newer
    (released 31 Jul 2023)

  - Fixed reloading of certificates for mass virtual hosting

Code Maintenance:
-----------------

  - fixed typos
  - fixed enum/int conversion flagged by gcc13


Modules:
--------
The following list contains the most important changes:

 - module nsdbpg:
   fixed memory leak (see above).

Re: [naviserver-devel] NaviServer 4.99.29 available

[Gustaf N. <ne...@wu...>]

Hi Brian,


as stated several times, the right action is to fix your script (as you 
did) rather than "silencing" NaviServer. I am not surprised, that 
attempts to write on detached connections can lead to error conditions 
on several occasions (generating errors avoids this).


But since we offer this silencing parameter, i do agree, the crashing is 
harsh. If you could send a short script triggering the problem it would 
help to work on such cases.


all the best

-g


On 06.11.23 14:25, Brian Fenton wrote:
> Hi Gustaf
>
> I just built and ran some tests on the "rejectalreadyclosedconn" 
> parameter to see how it handles code that triggers the "connection 
> socket is detached" error.
>
>  If I set "rejectalreadyclosedconn"to false, and browse to a page that 
> triggers the "connection socket is detached" error, Naviserver crashes 
> with the following error message:
>
> [06/Nov/2023:13:13:21][39.7f3489fb9640][-conn:openacs:default:1:30-] 
> Warning: NsWriterQueue: called without sockPtr size 414 bufs 1 flags 
> 1030431 stream 000000 chan (nil) fd -1
> [06/Nov/2023:13:13:21][39.7f3489fb9640][-conn:openacs:default:1:30-] 
> Fatal: received fatal signal 11
>
> If I then fix the code that was triggering the "connection socket is 
> detached" error, by adding the missing "return" after the offending 
> "ad_returnredirect", everything works fine.
>
> Let me know if you need more info to help reproduce this.
> thanks,
> Brian
>
> ------------------------------------------------------------------------
> *From:* Gustaf Neumann <ne...@wu...>
> *Sent:* Thursday 2 November 2023 2:30 pm
> *To:* Navidevel <nav...@li...>
> *Subject:* [naviserver-devel] NaviServer 4.99.29 available
> Dear all,
>
> I am glad to announce that the release of NaviServer 4.99.29 is 
> available at SourceForge [1].  This release is a pure bug-fix and 
> maintenance release, which fixes a potentiall serious memory leak when 
> working with PostgreSQL and large text contents. Furthermore, the 
> release contains a small enhancement as requested by Brian not very 
> long ago on this list.
>
> See below for a summary of the changes.
>
> So far, these changes are only available on sourceforge, since i have 
> lost write access to the repository at bitbucket. The people on 
> Atlassian seem to have changed some account types, and - on to of this 
> - they annouced via the Blog post on September 27, 2023, that the 
> billing model changed (where they also refer to the 
> "unified-user-management"). It took me a while to figure out, what 
> happened. The blog post states:
>
>     /From October 31st, 2023, Bitbucket Cloud will begin counting all
>     workspace members as a billable user. ....
>
>     Free plans: If you're on a free plan and your billable user count
>     is higher than 5 as per the new definition of billable user, all
>     repositories in your workspace will become read-only until you
>     remove some users or upgrade your workspace to a paid plan./
>
> It seems that the users of the "naviserver" group are now counted as 
> "billable users", and it contains 19 users. According to support, we 
> have to reduce this number to 5, otherwise nobody will be able to 
> commit anything.
>
> Due to the ability with PRs, i think the reduction will be possible 
> without too much loss in functionality. If nobody objects, i will go 
> back in history and reduce the number of commit-member based on the 
> most recent direct commits. I hope, that non of the "old members" will 
> be offended by this. One other option would be to upgrade to a paid 
> plain - but i am not sure, who is gonna pay for this.
>
> All the best!
>
> -gustaf neumann
>
> [1] 
> https://sourceforge.net/projects/naviserver/files/naviserver/4.99.29/ 
> <https://sourceforge.net/projects/naviserver/files/naviserver/4.99.29/>
> [2] https://bitbucket.org/blog/billing-model-change 
> <https://bitbucket.org/blog/billing-model-change>
>
> =======================================
> NaviServer 4.99.29, released 2023-11-01
> =======================================
>
>   37 files changed, 261 insertions(+), 132 deletions(-)
>
> New Features:
> -------------
>   - Eased configuration of simple setups
>     * don't require to specify a "defaultserver" when a single
>       server is in use.
>
>     * reduce warnings for per-server network drivers. This is a not
>       recommended but possible configuration, global network drivers
>       should be used.
>
>   - The configuration option "rejectalreadyclosedconn", which warns
>     about attempts to send data to the web client at times, when the
>     connection is not available anymore, is now applied on closed and
>     detached connections. Before it was only applied on closed
>     connections, causing potentially many warnings for legacy
>     applications.
>
> Bug Fixes:
> ----------
>
>    - Fixed a potential memory leak introduced two releases ago (in
>      4.99.27).
>
>    - Fixed a potential compilation problem with glibc 2.38 or newer
>      (released 31 Jul 2023)
>
>    - Fixed reloading of certificates for mass virtual hosting
>
> Code Maintenance:
> -----------------
>
>    - fixed typos
>    - fixed enum/int conversion flagged by gcc13
>
>
> Modules:
> --------
> The following list contains the most important changes:
>
>   - module nsdbpg:
>     fixed memory leak (see above).

Re: [naviserver-devel] NaviServer 4.99.29 available

[Brian F. <bri...@ai...>]

Hi Gustaf

I just built and ran some tests on the "rejectalreadyclosedconn" parameter to see how it handles code that triggers the "connection socket is detached" error.

 If I set "rejectalreadyclosedconn" to false, and browse to a page that triggers the "connection socket is detached" error, Naviserver crashes with the following error message:

[06/Nov/2023:13:13:21][39.7f3489fb9640][-conn:openacs:default:1:30-] Warning: NsWriterQueue: called without sockPtr size 414 bufs 1 flags 1030431 stream 000000 chan (nil) fd -1
[06/Nov/2023:13:13:21][39.7f3489fb9640][-conn:openacs:default:1:30-] Fatal: received fatal signal 11

If I then fix the code that was triggering the "connection socket is detached" error, by adding the missing "return" after the offending "ad_returnredirect", everything works fine.

Let me know if you need more info to help reproduce this.
thanks,
Brian

________________________________
From: Gustaf Neumann <ne...@wu...>
Sent: Thursday 2 November 2023 2:30 pm
To: Navidevel <nav...@li...>
Subject: [naviserver-devel] NaviServer 4.99.29 available

Dear all,

I am glad to announce that the release of NaviServer 4.99.29 is available at SourceForge [1].  This release is a pure bug-fix and maintenance release, which fixes a potentiall serious memory leak when working with PostgreSQL and large text contents. Furthermore, the release contains a small enhancement as requested by Brian not very long ago on this list.

See below for a summary of the changes.

So far, these changes are only available on sourceforge, since i have lost write access to the repository at bitbucket. The people on Atlassian seem to have changed some account types, and - on to of this - they annouced via the Blog post on September 27, 2023, that the billing model changed (where they also refer to the "unified-user-management"). It took me a while to figure out, what happened. The blog post states:

>From October 31st, 2023, Bitbucket Cloud will begin counting all workspace members as a billable user. ....

Free plans: If you're on a free plan and your billable user count is higher than 5 as per the new definition of billable user, all repositories in your workspace will become read-only until you remove some users or upgrade your workspace to a paid plan.

It seems that the users of the "naviserver" group are now counted as "billable users", and it contains 19 users. According to support, we have to reduce this number to 5, otherwise nobody will be able to commit anything.

Due to the ability with PRs, i think the reduction will be possible without too much loss in functionality. If nobody objects, i will go back in history and reduce the number of commit-member based on the most recent direct commits. I hope, that non of the "old members" will be offended by this. One other option would be to upgrade to a paid plain - but i am not sure, who is gonna pay for this.

All the best!

-gustaf neumann

[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.29/
[2] https://bitbucket.org/blog/billing-model-change


=======================================
NaviServer 4.99.29, released 2023-11-01
=======================================

 37 files changed, 261 insertions(+), 132 deletions(-)

New Features:
-------------
 - Eased configuration of simple setups
   * don't require to specify a "defaultserver" when a single
     server is in use.

   * reduce warnings for per-server network drivers. This is a not
     recommended but possible configuration, global network drivers
     should be used.

 - The configuration option "rejectalreadyclosedconn", which warns
   about attempts to send data to the web client at times, when the
   connection is not available anymore, is now applied on closed and
   detached connections. Before it was only applied on closed
   connections, causing potentially many warnings for legacy
   applications.

Bug Fixes:
----------

  - Fixed a potential memory leak introduced two releases ago (in
    4.99.27).

  - Fixed a potential compilation problem with glibc 2.38 or newer
    (released 31 Jul 2023)

  - Fixed reloading of certificates for mass virtual hosting

Code Maintenance:
-----------------

  - fixed typos
  - fixed enum/int conversion flagged by gcc13


Modules:
--------
The following list contains the most important changes:

 - module nsdbpg:
   fixed memory leak (see above).

Re: [naviserver-devel] NaviServer 4.99.29 available

[Andrew P. <at...@pi...>]

On Thu, Nov 02, 2023 at 03:30:43PM +0100, Gustaf Neumann wrote:

> So far, these changes are only available on sourceforge, since i have 
> lost write access to the repository at bitbucket.

Wow.  Yes, please remove whatever group members are necessary to
restore write access, and let us know when the public NaviServer Git
repository is again up to date.  (As you say, we can always send you
patches via forks in our personal repositories on Bitbucket, if
necessary.)

-- 
Andrew Piskorski <at...@pi...>

Re: [naviserver-devel] NaviServer 4.99.29 available

[Brian F. <bri...@ai...>]

Dear Gustaf

thank you for including my requested enhancement -  a pleasant surprise!

I have a few more thoughts and questions about reducing the volume of log entries - I will put that together and send it on in the near future.

thanks again
Brian

________________________________
From: Gustaf Neumann <ne...@wu...>
Sent: Thursday 2 November 2023 2:30 pm
To: Navidevel <nav...@li...>
Subject: [naviserver-devel] NaviServer 4.99.29 available

Dear all,

I am glad to announce that the release of NaviServer 4.99.29 is available at SourceForge [1].  This release is a pure bug-fix and maintenance release, which fixes a potentiall serious memory leak when working with PostgreSQL and large text contents. Furthermore, the release contains a small enhancement as requested by Brian not very long ago on this list.

See below for a summary of the changes.

So far, these changes are only available on sourceforge, since i have lost write access to the repository at bitbucket. The people on Atlassian seem to have changed some account types, and - on to of this - they annouced via the Blog post on September 27, 2023, that the billing model changed (where they also refer to the "unified-user-management"). It took me a while to figure out, what happened. The blog post states:

>From October 31st, 2023, Bitbucket Cloud will begin counting all workspace members as a billable user. ....

Free plans: If you're on a free plan and your billable user count is higher than 5 as per the new definition of billable user, all repositories in your workspace will become read-only until you remove some users or upgrade your workspace to a paid plan.

It seems that the users of the "naviserver" group are now counted as "billable users", and it contains 19 users. According to support, we have to reduce this number to 5, otherwise nobody will be able to commit anything.

Due to the ability with PRs, i think the reduction will be possible without too much loss in functionality. If nobody objects, i will go back in history and reduce the number of commit-member based on the most recent direct commits. I hope, that non of the "old members" will be offended by this. One other option would be to upgrade to a paid plain - but i am not sure, who is gonna pay for this.

All the best!

-gustaf neumann


[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.29/

[2] https://bitbucket.org/blog/billing-model-change


=======================================
NaviServer 4.99.29, released 2023-11-01
=======================================

 37 files changed, 261 insertions(+), 132 deletions(-)

New Features:
-------------
 - Eased configuration of simple setups
   * don't require to specify a "defaultserver" when a single
     server is in use.

   * reduce warnings for per-server network drivers. This is a not
     recommended but possible configuration, global network drivers
     should be used.

 - The configuration option "rejectalreadyclosedconn", which warns
   about attempts to send data to the web client at times, when the
   connection is not available anymore, is now applied on closed and
   detached connections. Before it was only applied on closed
   connections, causing potentially many warnings for legacy
   applications.

Bug Fixes:
----------

  - Fixed a potential memory leak introduced two releases ago (in
    4.99.27).

  - Fixed a potential compilation problem with glibc 2.38 or newer
    (released 31 Jul 2023)

  - Fixed reloading of certificates for mass virtual hosting

Code Maintenance:
-----------------

  - fixed typos
  - fixed enum/int conversion flagged by gcc13


Modules:
--------
The following list contains the most important changes:

 - module nsdbpg:
   fixed memory leak (see above).

[naviserver-devel] NaviServer 4.99.29 available

[Gustaf N. <ne...@wu...>]

Dear all,

I am glad to announce that the release of NaviServer 4.99.29 is 
available at SourceForge [1].  This release is a pure bug-fix and 
maintenance release, which fixes a potentiall serious memory leak when 
working with PostgreSQL and large text contents. Furthermore, the 
release contains a small enhancement as requested by Brian not very long 
ago on this list.

See below for a summary of the changes.

So far, these changes are only available on sourceforge, since i have 
lost write access to the repository at bitbucket. The people on 
Atlassian seem to have changed some account types, and - on to of this - 
they annouced via the Blog post on September 27, 2023, that the billing 
model changed (where they also refer to the "unified-user-management"). 
It took me a while to figure out, what happened. The blog post states:

    /From October 31st, 2023, Bitbucket Cloud will begin counting all
    workspace members as a billable user. ....

    Free plans: If you're on a free plan and your billable user count is
    higher than 5 as per the new definition of billable user, all
    repositories in your workspace will become read-only until you
    remove some users or upgrade your workspace to a paid plan./

It seems that the users of the "naviserver" group are now counted as 
"billable users", and it contains 19 users. According to support, we 
have to reduce this number to 5, otherwise nobody will be able to commit 
anything.

Due to the ability with PRs, i think the reduction will be possible 
without too much loss in functionality. If nobody objects, i will go 
back in history and reduce the number of commit-member based on the most 
recent direct commits. I hope, that non of the "old members" will be 
offended by this. One other option would be to upgrade to a paid plain - 
but i am not sure, who is gonna pay for this.

All the best!

-gustaf neumann

[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.29/
[2] https://bitbucket.org/blog/billing-model-change

=======================================
NaviServer 4.99.29, released 2023-11-01
=======================================

  37 files changed, 261 insertions(+), 132 deletions(-)

New Features:
-------------
  - Eased configuration of simple setups
    * don't require to specify a "defaultserver" when a single
      server is in use.

    * reduce warnings for per-server network drivers. This is a not
      recommended but possible configuration, global network drivers
      should be used.

  - The configuration option "rejectalreadyclosedconn", which warns
    about attempts to send data to the web client at times, when the
    connection is not available anymore, is now applied on closed and
    detached connections. Before it was only applied on closed
    connections, causing potentially many warnings for legacy
    applications.

Bug Fixes:
----------

   - Fixed a potential memory leak introduced two releases ago (in
     4.99.27).

   - Fixed a potential compilation problem with glibc 2.38 or newer
     (released 31 Jul 2023)

   - Fixed reloading of certificates for mass virtual hosting

Code Maintenance:
-----------------

   - fixed typos
   - fixed enum/int conversion flagged by gcc13


Modules:
--------
The following list contains the most important changes:

  - module nsdbpg:
    fixed memory leak (see above).

[naviserver-devel] Proposal: config file boolean parameter "showserverheader" to show/hide "Server:" response header

[Brian F. <bri...@ai...>]

Hi all

Recently during a client security audit, the "Server: NaviServer/4.99.28" response header was flagged as an issue. The client has asked us to remove the header, if possible.
The RFC suggests that the "Server: " header is optional, so I believe this should be OK to remove. https://www.rfc-editor.org/rfc/rfc7231#section-7.4.2

We would like to propose a new config file boolean parameter "showserverheader" with default true.

Ns_ConnConstructHeaders in return.c could then check this parameter before outputting the "Server: " header e.g. something like this:

    if (Ns_ConfigBool(path, "showserverheader", NS_TRUE) == NS_TRUE) {
      Ns_DStringVarAppend(dsPtr, "Server: ", Ns_InfoServerName(), "/", Ns_InfoServerVersion(), "\r\n");    
    }

Thoughts? Alternatives?

thanks
Brian

Re: [naviserver-devel] "connection already closed" vs "connection socket is detached" and the "rejectalreadyclosedconn" parameter

[Brian F. <bri...@ai...>]

Hi Gustaf

thanks for the explanation and the commit. I got confused because the discussion on https://openacs.org/forums/message-view?message_id=5659253 was about "connection socket is detached", when as you say that distinction came later.

This kind of parameter is handy with clients running older versions of our application where it's not feasible to immediately upgrade them. With the parameter, we could theoretically reduce the noise in the error logs, while the client awaits an upgrade.

thanks as always
Brian
________________________________
From: Gustaf Neumann <ne...@wu...>
Sent: Saturday 30 September 2023 11:08 am
To: nav...@li... <nav...@li...>
Subject: Re: [naviserver-devel] "connection already closed" vs "connection socket is detached" and the "rejectalreadyclosedconn" parameter

Hi Brian,

The parameter "rejectalreadyclosedconn" does what it is supposed to do
(controls error messages, when someone tries to write to a connection,
which was already closed). The parameter was introduced at a time,
before the distinction between closed and detached connections was
introduced in the code. So, I read your request to extend the meaning of
the parameter to cover detached connections as well.

 > Have I misunderstood something or is this not possible?

There is no hard reason rendering this as impossible. One could either
introduce (a) another parameter "rejectalreadydetachedconn" or (b) make
"rejectalreadyclosedconn" slightly misleading from the name to cover as
well the detached cases.

Since the parameter is just for legacy applications, which did not care
about the semantics, when someone tries to write to a connection which
is not available anymore (leading to potentially complex debugging
questions), the updated code [1] follows the (b) approach. I see the
potential misuse of the parameter to encourage lazy coding practices,
but i do understand as well code-maintainers who see large amounts of
errors in their log files on busy sites, not having the resources right
now to address these...

-gn

[1]
https://bitbucket.org/naviserver/naviserver/commits/725cf9475fe78f495918ed4549bf434cf2daaf96





_______________________________________________
naviserver-devel mailing list
nav...@li...
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] "connection already closed" vs "connection socket is detached" and the "rejectalreadyclosedconn" parameter

[Gustaf N. <ne...@wu...>]

On 29.09.23 12:56, Brian Fenton wrote:
> Hi
>
> this discussion here 
> https://openacs.org/forums/message-view?message_id=5659253 suggests 
> that the "rejectalreadyclosedconn" parameter will prevent the 
> "connection socket is detached" message from showing in the logs. 
> However, this doesn't appear to work (tried in 4.99.28). Looking at 
> "NsConnRequire" in conn.c, it looks like 
> "reject_already_closed_connection" is only used in the "connection 
> already closed" case. 
> https://bitbucket.org/naviserver/naviserver/src/b21b6d75a8f2ff41cd00dd40cb4c143d9a8a55ce/nsd/conn.c#lines-2748
>
> It would be good to be able to turn off the "connection socket is 
> detached" (understanding of course that we should in the meantime 
> resolve the source of those messages). Have I misunderstood something 
> or is this not possible?
>
> thanks
> Brian
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

-- 
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"

Re: [naviserver-devel] "connection already closed" vs "connection socket is detached" and the "rejectalreadyclosedconn" parameter

[Gustaf N. <ne...@wu...>]

Hi Brian,

The parameter "rejectalreadyclosedconn" does what it is supposed to do 
(controls error messages, when someone tries to write to a connection, 
which was already closed). The parameter was introduced at a time, 
before the distinction between closed and detached connections was 
introduced in the code. So, I read your request to extend the meaning of 
the parameter to cover detached connections as well.

 > Have I misunderstood something or is this not possible?

There is no hard reason rendering this as impossible. One could either 
introduce (a) another parameter "rejectalreadydetachedconn" or (b) make 
"rejectalreadyclosedconn" slightly misleading from the name to cover as 
well the detached cases.

Since the parameter is just for legacy applications, which did not care 
about the semantics, when someone tries to write to a connection which 
is not available anymore (leading to potentially complex debugging 
questions), the updated code [1] follows the (b) approach. I see the 
potential misuse of the parameter to encourage lazy coding practices, 
but i do understand as well code-maintainers who see large amounts of 
errors in their log files on busy sites, not having the resources right 
now to address these...

-gn

[1] 
https://bitbucket.org/naviserver/naviserver/commits/725cf9475fe78f495918ed4549bf434cf2daaf96