Re: [Naviserver-users] memory bloat

[Gustaf N. (sslmail) <ne...@wu...>]

> On 08.09.2025, at 17:40, Decent Espresso <dec...@gm...> wrote:
> 
> Gustaf, what do you think of my theory that scheduled threads could cause bloat, because the thread never exits, thus never reclaiming the peak memory they use?

long story short. 
- scheduled threads should be restarted from time to time when used intensively (parameter schedsperthread)
- scheduled threads should be cleaned when they occupy resources also between restarts (see OpenACS function below, you might consider temporary objects, ns_sets, dom nodes, etc).

you can use a function like the one below to check, whether some resources are leaked. Call it from time to time in journalism scheduled procs, and see some potential growing.

we do not experience a linear grow on servers, running a month without restart, using scheduled procedures intensively (but also ns_jobs, etc.)

-g

[1] https://openacs.org/api-doc/proc-view?proc=ad_run_scheduled_proc&source_p=1

  proc ::xo::stats {{msg ""}} {
    dict set stats xotcl   [llength [::xotcl::Object info instances -closure]]
    dict set stats nx      [llength [::nx::Object info instances  -closure]]
    dict set stats tmpObjs [llength [info commands ::nsf::__#*]]
    dict set stats tdom    [llength [list {*}[info commands domNode0*] {*}[info commands domDoc0x*]]]
    dict set stats nssets  [expr {[acs::icanuse "ns_set stats"] ? [list [ns_set stats]] : [llength [ns_set list]]}]
    ns_log notice "xo::stats $msg: $stats"
    return $stats
  }


> 
> Years ago I had that problem with my own C++ app with threads that never exited. Most modern OSes don’t reclaim freed mallocs from a thread until the thread exits, so as to avoid needing locks to re-allocate memory.
> 
> If it’s a reasonable theory, then some way to have scheduled tasks be run as short run threads, makes sense to do.
> 
> -john
> 
> 
> 
> On September 8, 2025 at 17:32:54, Gustaf Neumann (sslmail) (ne...@wu... <mailto:ne...@wu...>) wrote:
> 
>> Hi Jon,
>> 
>> there are really many reasons why one can observe a memory growth over time.
>> Some of these are “natural” (reaching asymptotically some value), some of these
>> are bugs - from application bugs to bugs in the underlying software.
>> 
>> Some time ago, i did a comparison in terms of performance and memory consumption
>> of various malloc implementations, using a typical startup of AOLserver/Naviserver
>> (20-30 mio alloc/free operations). The memory footprint of the same malloc operation
>> varies significantly, both in terms of RSS and VSize 
>>       
>>   https://next-scripting.org/2.4.0/doc/misc/thread-mallocs/index1
>> 
>> Nevertheless, i am aware of a memory leak i introduced in a non-released version of nsf
>> 
>> What version on nsf/xotcl are you using?
>> % set $::nsf::patchLeve
>> 
>> Are you using SYSTEM_MALLOC?
>> % ns_info buildinfo
>> 
>> If you are using SYSTEM_MALLOC + tcmalloc, you can get nice memory statistics
>> via the nsstats module.
>> 
>> all the best
>> -g
>> 
>>> On 08.09.2025, at 11:49, Decent Espresso <dec...@gm... <mailto:dec...@gm...>> wrote:
>>> 
>>> I’m still having nsd memory grow over time, even though I’m in v5.
>>> 
>>> We asked Claude to give its opinion, which is reported as:
>>> 
>>>> Thread Analysis identified:
>>>> Leaking threads are scheduler idle threads (-sched:idle3-, -sched:idle4-)
>>>> These threads execute background scheduled procedures
>>>> Memory accumulation occurs at C-level, invisible to Tcl inspection
>>>> 
>>>> Based on scheduled procedure analysis, the most probable causes:
>>>> 
>>>> qbo_daily_import -  Running every 5 minutes
>>>> fetch_undelivered_shipment_tracking / fetch_delivered_shipment_tracking 
>>>> easyship_update_all - Hourly updates
>>>> 
>>>> Memory Pool Analysis revealed:
>>>> 
>>>> Thread 7f192effd700: 22+ million allocated 8KB blocks (~184GB worth)
>>>> Thread 7f194dffb700: 11+ million allocated 16KB blocks (~183GB worth)
>>>> These are C-level memory allocations within NaviServer's memory management system
>>> 
>>> I’m thinking:
>>> 
>>> since each of these is run as a thread, such as:
>>> ns_schedule_proc -thread 600 qbo_refresh
>>> 
>>> it’s likely that the thread is never exiting, and thus its malloced memory is never reclaimed.
>>> 
>>> I’m thinking of adding log entries that give the thread id of the proc being run (ns_thread id), so I can figure out which of my tcl threads is consuming all that memory.
>>> 
>>> It’s unclear in the docs how a new ns_thread exits and gets cleaned up.  I didn’t see a “ns_thread terminate” for example.
>>> 
>>> -john
>>> 
>>> _______________________________________________
>>> Naviserver-users mailing list
>>> Nav...@li... <mailto:Nav...@li...>
>>> https://lists.sourceforge.net/lists/listinfo/naviserver-users