Re: [Naviserver-users] memory bloat

[Decent E. <dec...@gm...>]

 Gustaf, what do you think of my theory that scheduled threads could cause
bloat, because the thread never exits, thus never reclaiming the peak
memory they use?

Years ago I had that problem with my own C++ app with threads that never
exited. Most modern OSes don’t reclaim freed mallocs from a thread until
the thread exits, so as to avoid needing locks to re-allocate memory.

If it’s a reasonable theory, then some way to have scheduled tasks be run
as short run threads, makes sense to do.

-john



On September 8, 2025 at 17:32:54, Gustaf Neumann (sslmail) (ne...@wu...)
wrote:

Hi Jon,

there are really many reasons why one can observe a memory growth over time.
Some of these are “natural” (reaching asymptotically some value), some of
these
are bugs - from application bugs to bugs in the underlying software.

Some time ago, i did a comparison in terms of performance and memory
consumption
of various malloc implementations, using a typical startup of
AOLserver/Naviserver
(20-30 mio alloc/free operations). The memory footprint of the same malloc
operation
varies significantly, both in terms of RSS and VSize

  https://next-scripting.org/2.4.0/doc/misc/thread-mallocs/index1

Nevertheless, i am aware of a memory leak i introduced in a non-released
version of nsf

What version on nsf/xotcl are you using?
% set $::nsf::patchLeve

Are you using SYSTEM_MALLOC?
% ns_info buildinfo

If you are using SYSTEM_MALLOC + tcmalloc, you can get nice memory
statistics
via the nsstats module.

all the best
-g

On 08.09.2025, at 11:49, Decent Espresso <dec...@gm...> wrote:

I’m still having nsd memory grow over time, even though I’m in v5.

We asked Claude to give its opinion, which is reported as:

Thread Analysis identified:
Leaking threads are scheduler idle threads (-sched:idle3-, -sched:idle4-)
These threads execute background scheduled procedures
Memory accumulation occurs at C-level, invisible to Tcl inspection

Based on scheduled procedure analysis, the most probable causes:

qbo_daily_import -  Running every 5 minutes
fetch_undelivered_shipment_tracking / fetch_delivered_shipment_tracking
easyship_update_all - Hourly updates

Memory Pool Analysis revealed:

Thread 7f192effd700: 22+ million allocated 8KB blocks (~184GB worth)
Thread 7f194dffb700: 11+ million allocated 16KB blocks (~183GB worth)
These are C-level memory allocations within NaviServer's memory management
system


I’m thinking:

since each of these is run as a thread, such as:
ns_schedule_proc -thread 600 qbo_refresh

it’s likely that the thread is never exiting, and thus its malloced memory
is never reclaimed.

I’m thinking of adding log entries that give the thread id of the proc
being run (ns_thread id), so I can figure out which of my tcl threads is
consuming all that memory.

It’s unclear in the docs how a new ns_thread exits and gets cleaned up.  I
didn’t see a “ns_thread terminate” for example.

-john

_______________________________________________
Naviserver-users mailing list
Nav...@li...
https://lists.sourceforge.net/lists/listinfo/naviserver-users