|
From: Decent E. <dec...@gm...> - 2025-09-08 15:40:58
|
Gustaf, what do you think of my theory that scheduled threads could cause bloat, because the thread never exits, thus never reclaiming the peak memory they use? Years ago I had that problem with my own C++ app with threads that never exited. Most modern OSes don’t reclaim freed mallocs from a thread until the thread exits, so as to avoid needing locks to re-allocate memory. If it’s a reasonable theory, then some way to have scheduled tasks be run as short run threads, makes sense to do. -john On September 8, 2025 at 17:32:54, Gustaf Neumann (sslmail) (ne...@wu...) wrote: Hi Jon, there are really many reasons why one can observe a memory growth over time. Some of these are “natural” (reaching asymptotically some value), some of these are bugs - from application bugs to bugs in the underlying software. Some time ago, i did a comparison in terms of performance and memory consumption of various malloc implementations, using a typical startup of AOLserver/Naviserver (20-30 mio alloc/free operations). The memory footprint of the same malloc operation varies significantly, both in terms of RSS and VSize https://next-scripting.org/2.4.0/doc/misc/thread-mallocs/index1 Nevertheless, i am aware of a memory leak i introduced in a non-released version of nsf What version on nsf/xotcl are you using? % set $::nsf::patchLeve Are you using SYSTEM_MALLOC? % ns_info buildinfo If you are using SYSTEM_MALLOC + tcmalloc, you can get nice memory statistics via the nsstats module. all the best -g On 08.09.2025, at 11:49, Decent Espresso <dec...@gm...> wrote: I’m still having nsd memory grow over time, even though I’m in v5. We asked Claude to give its opinion, which is reported as: Thread Analysis identified: Leaking threads are scheduler idle threads (-sched:idle3-, -sched:idle4-) These threads execute background scheduled procedures Memory accumulation occurs at C-level, invisible to Tcl inspection Based on scheduled procedure analysis, the most probable causes: qbo_daily_import - Running every 5 minutes fetch_undelivered_shipment_tracking / fetch_delivered_shipment_tracking easyship_update_all - Hourly updates Memory Pool Analysis revealed: Thread 7f192effd700: 22+ million allocated 8KB blocks (~184GB worth) Thread 7f194dffb700: 11+ million allocated 16KB blocks (~183GB worth) These are C-level memory allocations within NaviServer's memory management system I’m thinking: since each of these is run as a thread, such as: ns_schedule_proc -thread 600 qbo_refresh it’s likely that the thread is never exiting, and thus its malloced memory is never reclaimed. I’m thinking of adding log entries that give the thread id of the proc being run (ns_thread id), so I can figure out which of my tcl threads is consuming all that memory. It’s unclear in the docs how a new ns_thread exits and gets cleaned up. I didn’t see a “ns_thread terminate” for example. -john _______________________________________________ Naviserver-users mailing list Nav...@li... https://lists.sourceforge.net/lists/listinfo/naviserver-users |