Menu
▾
▴
naviserver-users — Discussion of installation and usage of NaviServer
You can subscribe to this list here.
| 2025 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
(2) |
Aug
(9) |
Sep
(11) |
Oct
|
Nov
(6) |
Dec
|
|---|
|
From: Decent E. <dec...@gm...> - 2025-11-10 04:44:38
|
FYI: it is not a “hack”, this is the standard way, how ALL HTTP methods in NaviServer are implemented. sometimes, the request handler is resolved by a tcl scripts, sometimes it is delivering a file, etc. I see. I thought GET POST HEAD were all handled with internal code and this just happened to be possible. Well, it’s a nice design then, that you’ve made the common methods implemented this way, so as you say, new methods can be easily added by users without needing to look at the C code. -john |
|
From: Gustaf N. (sslmail) <ne...@wu...> - 2025-11-08 00:34:31
|
> On 07.11.2025, at 16:42, Decent Espresso <dec...@gm...> wrote: > > Nice hack! FYI: it is not a “hack”, this is the standard way, how ALL HTTP methods in NaviServer are implemented. sometimes, the request handler is resolved by a tcl scripts, sometimes it is delivering a file, etc. -g |
|
From: Decent E. <dec...@gm...> - 2025-11-07 11:10:18
|
I wanted to report back that I needed to add "Authorization" to
Access-Control-Allow-Headers in order for HTTP AUTH to be permitted.
And now we have a CORS compliant API we've tested is indeed working.
# Simple CORS-style preflight handler
ns_register_proc OPTIONS /support/api/* {
ns_set put [ns_conn outputheaders] Access-Control-Allow-Origin "*"
ns_set put [ns_conn outputheaders] Access-Control-Allow-Methods "GET,
POST, OPTIONS"
ns_set put [ns_conn outputheaders] Access-Control-Allow-Headers
"Content-Type, Authorization, X-Requested-With"
ns_return 204 text/plain ""
}
On Fri, Nov 7, 2025 at 4:58 PM Gustaf Neumann (sslmail) <ne...@wu...>
wrote:
> The following should work:
>
> # Simple CORS-style preflight handler
> ns_register_proc OPTIONS /api/* {
> ns_set put [ns_conn outputheaders] Access-Control-Allow-Origin "*"
> ns_set put [ns_conn outputheaders] Access-Control-Allow-Methods "GET,
> POST, OPTIONS"
> ns_set put [ns_conn outputheaders] Access-Control-Allow-Headers
> "Content-Type, X-Requested-With"
> ns_return 204 text/plain ""
> }
>
>
> -g
>
> On 07.11.2025, at 09:31, Decent Espresso <dec...@gm...> wrote:
>
> I’m being asked why naviserver doesn’t seem to support the OPTIONS method,
> which CORS can occasionally use.
>
> Perhaps it does, and I haven’t configured it?
>
> curl -X OPTIONS http://decentespresso.com
> <!DOCTYPE html>
> <html lang='en'>
> <head>
> <meta charset='UTF-8'>
> <meta name='viewport' content='width=device-width, initial-scale=1.0'>
> <title>Method Not Allowed</title>
> </head>
> <body>
> <h2>Method Not Allowed</h2>
> The requested method is not allowed on this server.
> <p style='text-align: right; font-size: small; font-style: italic;'>
> NaviServer/5.0.1
> </p>
>
> </body>
>
>
> _______________________________________________
> Naviserver-users mailing list
> Nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-users
>
>
>
|
|
From: Decent E. <dec...@gm...> - 2025-11-07 09:42:18
|
Nice hack! curl -I -X OPTIONS "http://localhost:8000/api/sn_batch_details?sn=9486” HTTP/1.1 204 No Content Server: NaviServer/5.0.0a Date: Fri, 07 Nov 2025 09:40:35 GMT Access-Control-Allow-Origin: * Access-Control-Allow-Methods: GET, POST, OPTIONS Access-Control-Allow-Headers: Content-Type, X-Requested-With X-Gustav-Hack: yes Content-Type: text/plain; charset=utf-8 Content-Length: 0 Connection: keep-alive |
|
From: Gustaf N. (sslmail) <ne...@wu...> - 2025-11-07 08:59:00
|
The following should work:
# Simple CORS-style preflight handler
ns_register_proc OPTIONS /api/* {
ns_set put [ns_conn outputheaders] Access-Control-Allow-Origin "*"
ns_set put [ns_conn outputheaders] Access-Control-Allow-Methods "GET, POST, OPTIONS"
ns_set put [ns_conn outputheaders] Access-Control-Allow-Headers "Content-Type, X-Requested-With"
ns_return 204 text/plain ""
}
-g
> On 07.11.2025, at 09:31, Decent Espresso <dec...@gm...> wrote:
>
> I’m being asked why naviserver doesn’t seem to support the OPTIONS method, which CORS can occasionally use.
>
> Perhaps it does, and I haven’t configured it?
>
> curl -X OPTIONS http://decentespresso.com <http://decentespresso.com/>
> <!DOCTYPE html>
> <html lang='en'>
> <head>
> <meta charset='UTF-8'>
> <meta name='viewport' content='width=device-width, initial-scale=1.0'>
> <title>Method Not Allowed</title>
> </head>
> <body>
> <h2>Method Not Allowed</h2>
> The requested method is not allowed on this server.
> <p style='text-align: right; font-size: small; font-style: italic;'>
> NaviServer/5.0.1
> </p>
>
> </body>
>
>
> _______________________________________________
> Naviserver-users mailing list
> Nav...@li... <mailto:Nav...@li...>
> https://lists.sourceforge.net/lists/listinfo/naviserver-users
|
|
From: Decent E. <dec...@gm...> - 2025-11-07 02:31:52
|
I’m being asked why naviserver doesn’t seem to support the OPTIONS method, which CORS can occasionally use. Perhaps it does, and I haven’t configured it? curl -X OPTIONS http://decentespresso.com <!DOCTYPE html> <html lang='en'> <head> <meta charset='UTF-8'> <meta name='viewport' content='width=device-width, initial-scale=1.0'> <title>Method Not Allowed</title> </head> <body> <h2>Method Not Allowed</h2> The requested method is not allowed on this server. <p style='text-align: right; font-size: small; font-style: italic;'> NaviServer/5.0.1 </p> </body> |
|
From: Gustaf N. (s. <ne...@wu...> - 2025-09-24 07:05:04
|
Dear all, We are pleased to announce the availability of two new bug-fix releases of NaviServer: NaviServer 4.99.32 and NaviServer 5.0.1. Both releases contain backported fixes only. No new features have been added; all ongoing development continues on the main branch. These releases are intended for users who prefer to stay with the 4.99 or 5.0 lines for stability while benefiting from selected improvements and fixes. Contributors: - Gustaf Neumann - Hector Romojaro - Nicky Johnstone - Oleg Oleinick Source Code & Tags - GitHub: https://github.com/naviserver-project/naviserver - Tags: naviserver-5.0.1, naviserver-4.99.32 Tarball Downloads - https://sourceforge.net/projects/naviserver/files/naviserver/4.99.32/ - https://sourceforge.net/projects/naviserver/files/naviserver/5.0.1/ Docker Images for 4.99.32 and 5.0.1 (alpine | trixie, linux/amd64 | linux/arm64) - https://hub.docker.com/repository/docker/gustafn/naviserver ------------- NaviServer 5.0.1 (released 2025-09-23) 33 files changed, 463 insertions(+), 129 deletions(-) Bug Fixes Prevented tail truncation of disk-spooled file uploads under certain conditions Fixed potentially harmful negative timeout calculation in driver thread Corrected return type in TLS context initialization error path (TCL_ERROR instead of NS_ERROR) connchan: fixed potential NULL dereferences with debug mode Strengthened UTF-8 validation in Ns_Valid_UTF8() with stricter checks, better error reporting Multiple Windows/MSVC build fixes (stack corruption warnings, _mktemp_s, ns_mkdtemp, Ns_SockConnectUnix, Makefile improvements, EXEEXT, MINGW handling) Build system fixes: corrected tabs in Makefiles, cross-compilation failures, updated MSVC files Test suite improvements: skip lmap tests on Tcl 8.5, add UTF-8 regression tests, adapt ns_strcoll tests for macOS Sequoia Documentation and Configuration Clarified defaults and meaning of schedsperthread in sample configs and docs Minor corrections in comments and error messages for platform-specific builds Notes This is a bug-fix only release for the 5.0 branch. It includes backports from the head branch to improve robustness, portability (especially Windows/MSVC), and test coverage, without introducing new features. NaviServer 4.99.32 (released 2025-09-23) 7 files changed, 80 insertions(+), 17 deletions(-) Bug Fixes Fixed multipart/form-data parsing in ns_parseformfile to avoid truncated or misaligned parts Adjusted OpenACS sample config (nslog.checkforproxy) to use $reverseproxymode (was $proxy_mode) Corrected TLS context setup to return TCL_ERROR instead of NS_ERROR when certificate is missing Made driver poll timeout handling more robust (safe defaults, rounding, assert against negatives) Documentation Updates Refreshed README and manpages to reflect actual version numbers Clarified ns_return headers argument behavior (accepts setid, defaults to ns_conn outputheaders) Corrected ns_set examples (ns_set create instead of ns_set new) Updated ns_issmallint description to reflect unsigned 16-bit semantics Tests Added multipart upload tests (small and larger files, spool handling) Made ns_http.test portable across platforms ($devnull, $contentfile) Relaxed ns_strcoll expectations for macOS Sequoia collation changes Notes This is a bug-fix release for the 4.99 line with no new features. It includes select backports from 5.0 that improve robustness, documentation, and test portability. For Tcl 9 compatibility, please use NaviServer 5.*. |
|
From: Gustaf N. (sslmail) <ne...@wu...> - 2025-09-22 11:38:54
|
> On 22.09.2025, at 12:17, Decent Espresso <dec...@gm...> wrote:
>
> I’m wondering: is there a (new?) server setting that is the max time an ADP can run?
no
> Problem:
> - I’ve got a web page that takes a long time to run (an hour)
> - it keeps the browser up to date with progress via ns_adp_puts; ns_adp_flush
> - On the release version of naviserver 5, it is now timing out and not finishing after a few minutes.
> - On my older version of naviserver 5, the script completes. On older naviserver 4.9x it also was fine.
What makes you thinking, that ADP is causing this? Do you see a message in the log file when this happens?
To trace the timeouts, you might try
ns_logctl severity Debug(timeout) on
(easiest via nssats -> Logging -> Severities)
> I tried making these limits huge, but that did not help:
>
> ns_section "ns/limit/default"
>
> # Conn threads running for limit.
> ns_param maxrun 10000
>
> # Conn threads waiting for limit.
> ns_param maxwait 10000
>
> # Total seconds to wait for resources.
> ns_param timeout 6000
These are non-existing configuration parameters in NaviServer (since ever). From where did you get the information that setting these configuration parameters might help?
Go to nsstats/Configuration/Parameters to see all the configuration parameters that were loaded, the ones which were set but not used, parameters sets to the default values or different values, etc.
All the best
-g
|
|
From: Decent E. <dec...@gm...> - 2025-09-22 10:17:54
|
I’m wondering: is there a (new?) server setting that is the max time an ADP can run? Problem: - I’ve got a web page that takes a long time to run (an hour) - it keeps the browser up to date with progress via ns_adp_puts; ns_adp_flush - On the release version of naviserver 5, it is now timing out and not finishing after a few minutes. - On my older version of naviserver 5, the script completes. On older naviserver 4.9x it also was fine. I tried making these limits huge, but that did not help: ns_section "ns/limit/default" # Conn threads running for limit. ns_param maxrun 10000 # Conn threads waiting for limit. ns_param maxwait 10000 # Total seconds to wait for resources. ns_param timeout 6000 |
|
From: Gustaf N. (sslmail) <ne...@wu...> - 2025-09-19 15:46:11
|
Yes, the semantics of “schedsperthread” are very similar to “jobsperthread”: when the value is 0 (default), the thread is not automatically restarted. i have updated the documentation and added these parameters to the keyword index page. On my wishlist, i have a dict based “database” to contain all configuration parameters with default values and help strings, which can be used to generate also various parts of the documentation automatically. This would also increase maintainability. Currently, i am quite busy with other things. We have no search on the sourceforge pages, there are just static pages. Search engines are the best option. all the best -g > On 17.09.2025, at 14:19, Decent Espresso <dec...@gm...> wrote: > > The default for schedsperthread and jobsperthread appears to be “0”, but I’m unclear if for schedsperthread this means “never” or “after 1 job runs” or “not relevant". > > https://naviserver.sourceforge.io/n/naviserver/files/ns_schedule.html > > If zero for schedsperthread means the same as it does for jobsperthread, then perhaps it means "no influence on cleaning up a thread” ? > > Also, would be possible to have config parameters in the Keyword index? > https://naviserver.sourceforge.io/n/index.html > > I did not see a ‘search’ function on the docs. Is google search the preferred way to find things? > > And finally <smile> perhaps both these two parameters should be mentioned on the tuning page? > https://naviserver.sourceforge.io/n/manual/files/admin-tuning.html |
|
From: Decent E. <dec...@gm...> - 2025-09-19 15:38:23
|
Caching entire content pages is generally not recommended, especially when pages are personalized, since cache reuse is typically poor. Instead, I suggest setting a maxentry value to prevent a single large page from evicting the entire cache. It is also good practice to monitor the cache statistics in nsstats to evaluate how well your caching parameters perform — paying attention to metrics such as cache reuse and mutex contention. I’m using the caching on mysql-heavy pages, such as “what products are currently on sale, and in stock?” with a 10 minute cache refresh. Example: https://decentespresso.com/sale - some of these queries run in the 1 to 2 seconds range, which is slow enough so that crawlers hitting my site hard will take it down. The 10 minute cache occasionally causes problems if people order an item faster than every ten minutes, in which case we can accidentally oversell an item. I’ve thought about having a more general mysql cache in place, as SQL UPDATE commands that invalidate SQL SELECTS are quite rare. Possibly, the right approach would be to rearchitect the mysql schema so that queries always run quickly, so that I don’t need to add my own caching. -john |
|
From: Gustaf N. (sslmail) <ne...@wu...> - 2025-09-19 15:16:10
|
i think, i’ve missed the following posting: > The only thing to figure out for us, is the “untrusted” https domains we host (we have a *.decentespresso.com <http://decentespresso.com/> ssl cert) and that looks like a config change, as the warnings indicate. based on the provided host header field, NaviServer decides to which virtual server configuration (and with this, which blueprint etc.) the requests should be routed. One has to define the “defaultserver” for undecided cases. Checkout [1] for the documentation. Note that the host header field is subject to various attacks, since it is also used for redirects, the server might be used as a gateway. If you are cautious, you can consider to define a defaultserver as a honeypot and use just trusted headerfields for your real site. HTTP/3 will be a little be better in this respect. > > And I believe that ns_cache is now deprecated, so I should look at replacing that. I make heavy use of this: > > proc content_cache {cachename codetorun} { > if {[cookie_get "decent_admin"] != "" || [decentliveserver] == 0} { > return [eval $codetorun] > } > > return [ns_cache_eval tempfile $cachename $codetorun] > } ns_cache is not deprecated. If you have a source claiming otherwise, it should be corrected. In fact, ns_cache provides a unique transaction feature that can be linked to database transactions. This ensures consistency between cached values and the database state. By contrast, traditional caches retain intermediate results even when a complex transaction fails, leaving the cache out of sync with the database. Such inconsistencies can cause elusive “Heisenbugs” that are extremely difficult to diagnose. Caching entire content pages is generally not recommended, especially when pages are personalized, since cache reuse is typically poor. Instead, I suggest setting a maxentry value to prevent a single large page from evicting the entire cache. It is also good practice to monitor the cache statistics in nsstats to evaluate how well your caching parameters perform — paying attention to metrics such as cache reuse and mutex contention. > on pages that use big mysql queries to render. > > Anything else big in navi5 that I should jump on? it depepends, from where you jump. [1] https://naviserver.sourceforge.io/5/manual/files/admin-config.html#subsection7 > > -john > > _______________________________________________ > Naviserver-users mailing list > Nav...@li... <mailto:Nav...@li...> > https://lists.sourceforge.net/lists/listinfo/naviserver-users |
|
From: Decent E. <dec...@gm...> - 2025-09-17 12:19:15
|
The default for schedsperthread and jobsperthread appears to be “0”, but I’m unclear if for schedsperthread this means “never” or “after 1 job runs” or “not relevant". https://naviserver.sourceforge.io/n/naviserver/files/ns_schedule.html If zero for schedsperthread means the same as it does for jobsperthread, then perhaps it means "no influence on cleaning up a thread” ? Also, would be possible to have config parameters in the Keyword index? https://naviserver.sourceforge.io/n/index.html I did not see a ‘search’ function on the docs. Is google search the preferred way to find things? And finally <smile> perhaps both these two parameters should be mentioned on the tuning page? https://naviserver.sourceforge.io/n/manual/files/admin-tuning.html |
|
From: Gustaf N. (sslmail) <ne...@wu...> - 2025-09-10 16:49:22
|
> On 08.09.2025, at 17:40, Decent Espresso <dec...@gm...> wrote: > > Gustaf, what do you think of my theory that scheduled threads could cause bloat, because the thread never exits, thus never reclaiming the peak memory they use? long story short. - scheduled threads should be restarted from time to time when used intensively (parameter schedsperthread) - scheduled threads should be cleaned when they occupy resources also between restarts (see OpenACS function below, you might consider temporary objects, ns_sets, dom nodes, etc). you can use a function like the one below to check, whether some resources are leaked. Call it from time to time in journalism scheduled procs, and see some potential growing. we do not experience a linear grow on servers, running a month without restart, using scheduled procedures intensively (but also ns_jobs, etc.) -g [1] https://openacs.org/api-doc/proc-view?proc=ad_run_scheduled_proc&source_p=1 proc ::xo::stats {{msg ""}} { dict set stats xotcl [llength [::xotcl::Object info instances -closure]] dict set stats nx [llength [::nx::Object info instances -closure]] dict set stats tmpObjs [llength [info commands ::nsf::__#*]] dict set stats tdom [llength [list {*}[info commands domNode0*] {*}[info commands domDoc0x*]]] dict set stats nssets [expr {[acs::icanuse "ns_set stats"] ? [list [ns_set stats]] : [llength [ns_set list]]}] ns_log notice "xo::stats $msg: $stats" return $stats } > > Years ago I had that problem with my own C++ app with threads that never exited. Most modern OSes don’t reclaim freed mallocs from a thread until the thread exits, so as to avoid needing locks to re-allocate memory. > > If it’s a reasonable theory, then some way to have scheduled tasks be run as short run threads, makes sense to do. > > -john > > > > On September 8, 2025 at 17:32:54, Gustaf Neumann (sslmail) (ne...@wu... <mailto:ne...@wu...>) wrote: > >> Hi Jon, >> >> there are really many reasons why one can observe a memory growth over time. >> Some of these are “natural” (reaching asymptotically some value), some of these >> are bugs - from application bugs to bugs in the underlying software. >> >> Some time ago, i did a comparison in terms of performance and memory consumption >> of various malloc implementations, using a typical startup of AOLserver/Naviserver >> (20-30 mio alloc/free operations). The memory footprint of the same malloc operation >> varies significantly, both in terms of RSS and VSize >> >> https://next-scripting.org/2.4.0/doc/misc/thread-mallocs/index1 >> >> Nevertheless, i am aware of a memory leak i introduced in a non-released version of nsf >> >> What version on nsf/xotcl are you using? >> % set $::nsf::patchLeve >> >> Are you using SYSTEM_MALLOC? >> % ns_info buildinfo >> >> If you are using SYSTEM_MALLOC + tcmalloc, you can get nice memory statistics >> via the nsstats module. >> >> all the best >> -g >> >>> On 08.09.2025, at 11:49, Decent Espresso <dec...@gm... <mailto:dec...@gm...>> wrote: >>> >>> I’m still having nsd memory grow over time, even though I’m in v5. >>> >>> We asked Claude to give its opinion, which is reported as: >>> >>>> Thread Analysis identified: >>>> Leaking threads are scheduler idle threads (-sched:idle3-, -sched:idle4-) >>>> These threads execute background scheduled procedures >>>> Memory accumulation occurs at C-level, invisible to Tcl inspection >>>> >>>> Based on scheduled procedure analysis, the most probable causes: >>>> >>>> qbo_daily_import - Running every 5 minutes >>>> fetch_undelivered_shipment_tracking / fetch_delivered_shipment_tracking >>>> easyship_update_all - Hourly updates >>>> >>>> Memory Pool Analysis revealed: >>>> >>>> Thread 7f192effd700: 22+ million allocated 8KB blocks (~184GB worth) >>>> Thread 7f194dffb700: 11+ million allocated 16KB blocks (~183GB worth) >>>> These are C-level memory allocations within NaviServer's memory management system >>> >>> I’m thinking: >>> >>> since each of these is run as a thread, such as: >>> ns_schedule_proc -thread 600 qbo_refresh >>> >>> it’s likely that the thread is never exiting, and thus its malloced memory is never reclaimed. >>> >>> I’m thinking of adding log entries that give the thread id of the proc being run (ns_thread id), so I can figure out which of my tcl threads is consuming all that memory. >>> >>> It’s unclear in the docs how a new ns_thread exits and gets cleaned up. I didn’t see a “ns_thread terminate” for example. >>> >>> -john >>> >>> _______________________________________________ >>> Naviserver-users mailing list >>> Nav...@li... <mailto:Nav...@li...> >>> https://lists.sourceforge.net/lists/listinfo/naviserver-users |
|
From: Decent E. <dec...@gm...> - 2025-09-08 15:40:58
|
Gustaf, what do you think of my theory that scheduled threads could cause bloat, because the thread never exits, thus never reclaiming the peak memory they use? Years ago I had that problem with my own C++ app with threads that never exited. Most modern OSes don’t reclaim freed mallocs from a thread until the thread exits, so as to avoid needing locks to re-allocate memory. If it’s a reasonable theory, then some way to have scheduled tasks be run as short run threads, makes sense to do. -john On September 8, 2025 at 17:32:54, Gustaf Neumann (sslmail) (ne...@wu...) wrote: Hi Jon, there are really many reasons why one can observe a memory growth over time. Some of these are “natural” (reaching asymptotically some value), some of these are bugs - from application bugs to bugs in the underlying software. Some time ago, i did a comparison in terms of performance and memory consumption of various malloc implementations, using a typical startup of AOLserver/Naviserver (20-30 mio alloc/free operations). The memory footprint of the same malloc operation varies significantly, both in terms of RSS and VSize https://next-scripting.org/2.4.0/doc/misc/thread-mallocs/index1 Nevertheless, i am aware of a memory leak i introduced in a non-released version of nsf What version on nsf/xotcl are you using? % set $::nsf::patchLeve Are you using SYSTEM_MALLOC? % ns_info buildinfo If you are using SYSTEM_MALLOC + tcmalloc, you can get nice memory statistics via the nsstats module. all the best -g On 08.09.2025, at 11:49, Decent Espresso <dec...@gm...> wrote: I’m still having nsd memory grow over time, even though I’m in v5. We asked Claude to give its opinion, which is reported as: Thread Analysis identified: Leaking threads are scheduler idle threads (-sched:idle3-, -sched:idle4-) These threads execute background scheduled procedures Memory accumulation occurs at C-level, invisible to Tcl inspection Based on scheduled procedure analysis, the most probable causes: qbo_daily_import - Running every 5 minutes fetch_undelivered_shipment_tracking / fetch_delivered_shipment_tracking easyship_update_all - Hourly updates Memory Pool Analysis revealed: Thread 7f192effd700: 22+ million allocated 8KB blocks (~184GB worth) Thread 7f194dffb700: 11+ million allocated 16KB blocks (~183GB worth) These are C-level memory allocations within NaviServer's memory management system I’m thinking: since each of these is run as a thread, such as: ns_schedule_proc -thread 600 qbo_refresh it’s likely that the thread is never exiting, and thus its malloced memory is never reclaimed. I’m thinking of adding log entries that give the thread id of the proc being run (ns_thread id), so I can figure out which of my tcl threads is consuming all that memory. It’s unclear in the docs how a new ns_thread exits and gets cleaned up. I didn’t see a “ns_thread terminate” for example. -john _______________________________________________ Naviserver-users mailing list Nav...@li... https://lists.sourceforge.net/lists/listinfo/naviserver-users |
|
From: Gustaf N. (sslmail) <ne...@wu...> - 2025-09-08 15:33:06
|
Hi Jon,
there are really many reasons why one can observe a memory growth over time.
Some of these are “natural” (reaching asymptotically some value), some of these
are bugs - from application bugs to bugs in the underlying software.
Some time ago, i did a comparison in terms of performance and memory consumption
of various malloc implementations, using a typical startup of AOLserver/Naviserver
(20-30 mio alloc/free operations). The memory footprint of the same malloc operation
varies significantly, both in terms of RSS and VSize
https://next-scripting.org/2.4.0/doc/misc/thread-mallocs/index1
Nevertheless, i am aware of a memory leak i introduced in a non-released version of nsf
What version on nsf/xotcl are you using?
% set $::nsf::patchLeve
Are you using SYSTEM_MALLOC?
% ns_info buildinfo
If you are using SYSTEM_MALLOC + tcmalloc, you can get nice memory statistics
via the nsstats module.
all the best
-g
> On 08.09.2025, at 11:49, Decent Espresso <dec...@gm...> wrote:
>
> I’m still having nsd memory grow over time, even though I’m in v5.
>
> We asked Claude to give its opinion, which is reported as:
>
>> Thread Analysis identified:
>> Leaking threads are scheduler idle threads (-sched:idle3-, -sched:idle4-)
>> These threads execute background scheduled procedures
>> Memory accumulation occurs at C-level, invisible to Tcl inspection
>>
>> Based on scheduled procedure analysis, the most probable causes:
>>
>> qbo_daily_import - Running every 5 minutes
>> fetch_undelivered_shipment_tracking / fetch_delivered_shipment_tracking
>> easyship_update_all - Hourly updates
>>
>> Memory Pool Analysis revealed:
>>
>> Thread 7f192effd700: 22+ million allocated 8KB blocks (~184GB worth)
>> Thread 7f194dffb700: 11+ million allocated 16KB blocks (~183GB worth)
>> These are C-level memory allocations within NaviServer's memory management system
>
> I’m thinking:
>
> since each of these is run as a thread, such as:
> ns_schedule_proc -thread 600 qbo_refresh
>
> it’s likely that the thread is never exiting, and thus its malloced memory is never reclaimed.
>
> I’m thinking of adding log entries that give the thread id of the proc being run (ns_thread id), so I can figure out which of my tcl threads is consuming all that memory.
>
> It’s unclear in the docs how a new ns_thread exits and gets cleaned up. I didn’t see a “ns_thread terminate” for example.
>
> -john
>
> _______________________________________________
> Naviserver-users mailing list
> Nav...@li... <mailto:Nav...@li...>
> https://lists.sourceforge.net/lists/listinfo/naviserver-users
|
|
From: Decent E. <dec...@gm...> - 2025-09-08 09:49:30
|
I’m still having nsd memory grow over time, even though I’m in v5. We asked Claude to give its opinion, which is reported as: Thread Analysis identified: Leaking threads are scheduler idle threads (-sched:idle3-, -sched:idle4-) These threads execute background scheduled procedures Memory accumulation occurs at C-level, invisible to Tcl inspection Based on scheduled procedure analysis, the most probable causes: qbo_daily_import - Running every 5 minutes fetch_undelivered_shipment_tracking / fetch_delivered_shipment_tracking easyship_update_all - Hourly updates Memory Pool Analysis revealed: Thread 7f192effd700: 22+ million allocated 8KB blocks (~184GB worth) Thread 7f194dffb700: 11+ million allocated 16KB blocks (~183GB worth) These are C-level memory allocations within NaviServer's memory management system I’m thinking: since each of these is run as a thread, such as: ns_schedule_proc -thread 600 qbo_refresh it’s likely that the thread is never exiting, and thus its malloced memory is never reclaimed. I’m thinking of adding log entries that give the thread id of the proc being run (ns_thread id), so I can figure out which of my tcl threads is consuming all that memory. It’s unclear in the docs how a new ns_thread exits and gets cleaned up. I didn’t see a “ns_thread terminate” for example. -john |
|
From: Decent E. <dec...@gm...> - 2025-08-21 17:08:34
|
Today we moved our production server https://decentespresso.com/ from navi4.99 (built 6 years ago) to release navi5. Wow, it’s quite a bit faster. I’m also curious to see if: 1 - the slowly growing memory footprint of nsd 4.99 is no longer happening 2 - if nsd no longer occasionally crashes Due to much increased load on the server due to AI company spiders, we made the move. We also moved all media file serving to a separate nginx process, just to remove a possible cause, and to reduce the possible causes. Every once in a while with nsd 4.99, we’d see a huge spike in cpu utilization, and couldn’t see anything in the access logs that was a likely cause. I do realize that #1 could be caused by my own code, or poor tracking of ns_cache. The only thing to figure out for us, is the “untrusted” https domains we host (we have a *.decentespresso.com ssl cert) and that looks like a config change, as the warnings indicate. And I believe that ns_cache is now deprecated, so I should look at replacing that. I make heavy use of this: proc content_cache {cachename codetorun} { if {[cookie_get "decent_admin"] != "" || [decentliveserver] == 0} { return [eval $codetorun] } return [ns_cache_eval tempfile $cachename $codetorun] } on pages that use big mysql queries to render. Anything else big in navi5 that I should jump on? -john |
|
From: Decent E. <dec...@gm...> - 2025-08-15 11:41:04
|
There is now a redirect to sourceforge, so the link you got should also work. I get different links from Google search for “NaviServer Manual” (see below)… probably due to the personalized search… I often test with https://www.startpage.com/ because they are an anonymized version of google’s index. Thus, no personalized search. “naviserver manual” gives hit #2 as the old manual, which I confirm now auto-redirects perfectly. -john |
|
From: Gustaf N. (sslmail) <ne...@wu...> - 2025-08-15 10:38:29
|
> On 14.08.2025, at 16:45, Decent Espresso <dec...@gm...> wrote: > >> Hi john. >> From where do you have the “manual” link? > I google search for "naviserver manual” and the first hit is to https://naviserver.sourceforge.io/ > > See attached picture. > There is now a redirect to sourceforge, so the link you got should also work. I get different links from Google search for “NaviServer Manual” … probably due to the personalized search… all the best -g |
|
From: Decent E. <dec...@gm...> - 2025-08-14 14:58:32
|
For many years, I’ve patched nsd/request.c so that http requests that do not have an extension specified, auto-add .adp. If an extension is specified (ie, jpg) the add nothing. And the .adp appended to the url is never shown to the site visitor. This gives me very clean URLs like: http://mydomain.com/user/edit which actually resolve to: http://mydomain.com/user/edit.adp Those are nicely human readable. Most every server-side parsing http servers show the filename extension, and I find it a nice advantage of using nsd that I don’t have to. I’m wondering what cleaner way there is to do this, in a high performance way? I could use ns_register_proc for every page, but then I don’t get the use of ADP pages: ns_register_proc GET /battery/* direct_battery 0 I could register a 404 handler, redirect to the .adp page, and then use javascript to rewrite the URL to remove .adp, but that’s pretty ugly engineering. -john Here’s my original C code for nsd/request.c: ==== /* john buckman added 4/14/06 */ /* check if should add default filename extension of .adp */ /* only if no / on end of url which indicates a directory */ char * dotpos; if (ds2.string[ds2.length - 1] != '/') { /* if not . in the entire url, or if there is a dot before the final / (indicating a . in a directory name, which is ok, then add the default filename extension */ dotpos = strrchr(ds2.string, '.'); if ((dotpos == NULL) || (strchr(dotpos, '/') != NULL)) { Ns_DStringAppend(&ds2, ".adp"); /* Ns_Log(Notice, "added default extension to get '%s'", ds2.string); */ } } /* end john buckman added */ request->url = ns_strdup(ds2.string); === |
|
From: Decent E. <dec...@gm...> - 2025-08-14 14:52:23
|
The suggestion from the AI is not completely off, we have a parameter “extraheaders”: https://naviserver.sourceforge.io/5.0/nssock/files/nssock.html It is used e.g. in https://github.com/naviserver-project/naviserver/blob/main/nsd-config.tcl#L61 This is great. I couldn’t find the 5.0 manual yesterday, so I didn’t know about it. I used to hack the C source code to do this, in older nsd. -john |
|
From: Gustaf N. (sslmail) <ne...@wu...> - 2025-08-14 10:22:28
|
The suggestion from the AI is not completely off, we have a parameter “extraheaders”: https://naviserver.sourceforge.io/5.0/nssock/files/nssock.html It is used e.g. in https://github.com/naviserver-project/naviserver/blob/main/nsd-config.tcl#L61 If you want to set it on just for certain requests, a filter is not so bad. -g > On 13.08.2025, at 17:36, Decent Espresso <dec...@gm...> wrote: > > I’m looking for the most efficient way of adding the CORS headers that are needed to serve CSS/json files from a separate server. > > This works: > > — > ns_register_filter preauth GET /css/* add_css_headers > > proc add_css_headers {preauth} { > ns_set put [ns_conn outputheaders] "Access-Control-Allow-Origin" "*" > ns_set put [ns_conn outputheaders] "Access-Control-Allow-Methods" "GET, POST, OPTIONS" > ns_set put [ns_conn outputheaders] "Access-Control-Allow-Headers" "Content-Type, Authorization" > return "filter_ok" > } > — > > > asking an AI for alternatives gives me hallucinations, such as adding this to nsd.conf > > ns_section "ns/server/${server}/module/nsheaders" > ns_param defaultheaders "Access-Control-Allow-Origin: *" > _______________________________________________ > Naviserver-users mailing list > Nav...@li... <mailto:Nav...@li...> > https://lists.sourceforge.net/lists/listinfo/naviserver-users |
|
From: Gustaf N. (sslmail) <ne...@wu...> - 2025-08-14 10:14:39
|
Hi john. From where do you have the “manual” link? The manual is for 5.0 https://naviserver.sourceforge.io/5.0/manual/toc.html and the reference + manual is https://naviserver.sourceforge.io/5.0/toc.html with cross links to 4.99 The version agnostic link has the “n” (for historic versions). Certainly, we can add a bunch or more links, but up to my knowledge, the link you provided is not mentioned on official pages. If there is one, it is probably easier to fix this. -g > On 13.08.2025, at 17:23, Decent Espresso <dec...@gm...> wrote: > > I’m getting http errors on the manual at > https://naviserver.sourceforge.io/manual/toc.html > > -john > > An error has been encountered in accessing this page. > > 1. Server: naviserver.sourceforge.io <http://naviserver.sourceforge.io/> > 2. URL path: /manual/toc.html > 3. Error notes: NONE > 4. Error type: 404 > 5. Request method: GET > 6. Request query string: NONE > 7. Time: 2025-08-13 14:57:41 UTC (1755097061 <tel:1755097061>) > > Reporting this problem: The problem you have encountered is with a project web site hosted by SourceForge.net <http://sourceforge.net/>. This issue should be reported to the SourceForge.net <http://sourceforge.net/>-hosted project (not to SourceForge.net <http://sourceforge.net/>). > > If this is a severe or recurring/persistent problem, please do one of the following, and provide the error text (numbered 1 through 7, above): > > Contact the project via their designated support resources <https://sourceforge.net/support/prweb-lookup.php?host=naviserver.sourceforge.io&support=1>. > Contact the project administrators of this project via email (see the upper right-hand corner of the Project Summary page <https://sourceforge.net/support/prweb-lookup.php?host=naviserver.sourceforge.io> for their usernames) at use...@us... <http://users.sourceforge.net/> > If you are a maintainer of this web content, please refer to the Site Documentation regarding web services <https://sourceforge.net/p/forge/documentation/Project%20Web%20Services/> for further assistance. > > NOTE: As of 2008-10-23 directory index display has been disabled by default. This option may be re-enabled by the project by placing a file with the name ".htaccess" with this line: > > > > _______________________________________________ > Naviserver-users mailing list > Nav...@li... <mailto:Nav...@li...> > https://lists.sourceforge.net/lists/listinfo/naviserver-users |
|
From: Decent E. <dec...@gm...> - 2025-08-13 15:36:51
|
I’m looking for the most efficient way of adding the CORS headers that are
needed to serve CSS/json files from a separate server.
This works:
—
ns_register_filter preauth GET /css/* add_css_headers
proc add_css_headers {preauth} {
ns_set put [ns_conn outputheaders] "Access-Control-Allow-Origin" "*"
ns_set put [ns_conn outputheaders] "Access-Control-Allow-Methods" "GET,
POST, OPTIONS"
ns_set put [ns_conn outputheaders] "Access-Control-Allow-Headers"
"Content-Type, Authorization"
return "filter_ok"
}
—
asking an AI for alternatives gives me hallucinations, such as adding this
to nsd.conf
ns_section "ns/server/${server}/module/nsheaders"
ns_param defaultheaders "Access-Control-Allow-Origin: *"
|
