Thank you for the tips, Nikodemus
To recap, I have an SBCL 1.0.42 process that is allocating about 2 million generic and unsigned-64 vectors of length
100, for about 2GB of mystery allocation. I have no idea what they are.
Following your tips, I looked through room.lisp for more undocumented functions that could be of use in probing the memory usage.
I managed to lobotomize my SBCL process so much trying to wipe the root object responsible for all my uncollectable vectors
(like deleting the info database) that I can't define any crawlers that use SB-VM:MAP-ALLOCATED-OBJECTS.
I also found that I have to be pretty careful using SB-VM:MAP-ALLOCATED-OBJECTS because it suspends GC and then
consing can crash SBCL.
Using (SB-VM::INSTANCE-USAGE :DYNAMIC), it seems that I have about 500K hash tables and 500K spinlocks and 500K
MYSQL-DATABASE objects. That's a number sort of close to the 2.5M unexplained vectors that also exist.
I never make spinlocks, just mutexes. Spinlocks are apparently deprecated after 1.0.42, so this might all be some quirk
or misuse of an outdated SBCL. CLSQL just uses SB-THREAD:MAKE-MUTEX
It's evident that something weird happened with threading and open databases, and I don't know where the uncollectable root is.
I'd be inclined to guess that a bunch of threads died, and didn't release some spinlocks,
which are someone pinning down some objects against GC. Is this a silly hypothesis?
For future reference, if this happens again, is the following a complete enumeration of roots where a mass of un-GC'able objects might reside:
1. conservative roots in the stack
2. user toplevel variables including *,**,etc, and functions.
3. another active thread (none, if no other threads)
4. objects referenced in a closure
5. sb-c::*info-environment* - can point to functions that are in deleted packages.
6. NOT global vars in PCL/CLOS because CLOS doesn't store pointers to instances
7. ??? Something weird with terminated threads not releasing locks …. the mechanics of which I don't get.
I realize that these are vague questions, so I can't expect precise answers, but if any blatant misconceptions jump out,
please let me know.
On Oct 30, 2012, at 4:49 AM, Nikodemus Siivola wrote:
> On 30 October 2012 09:37, JTK <jetmonk@...> wrote:
>> So I deleted package after package, each time running (PROGN (GC :FULL T) (ROOM)). I'm left with nothing but
>> ASDF, the SB-, and GRAY-STREAM packages, and the dummy package I'm working in; CL-USER is gone. Yet the mystery
>> objects are still there, consuming 2.9 gigabytes of memory. I would have thought that at some point their top level
>> references would be gone along with the package, and they'd be deleted.
> Deleting a package is no sort of guarantee that eg. functions
> accessible through symbols in that package can be collected, even if
> there are no other user-visible references to them.
> (It would be nice, yes.)
>> Some detritus left after foreign function calls? Internal references to CLOS objects I don't know about? Just a leak in the GC?
> My first guess would be CLOS instance vectors. Each CLOS object looks
> roughly like this:
> where instance-vector is a SIMPLE-VECTOR that holds the slot-contents
> of that object.
> My second guess would be hash-tables (which also use SIMPLE-VECTORs
> internally) or SBCL's info database, which holds those pesky
> references to global functions and variables I alluded to earlier.
> Method dispatch caches also uses SIMPLE_VECTORs internally.
> It /is/ possibly to find out what they are. See eg.
> SB-VM:MAP-ALLOCATED-OBJECTS. Both SB-VM (internal, unsupported) and
> SB-INTROSPECT (supported) hold some tools for figuring out what refers
> to what.
> -- Nikodemus