I experienced an OOME with my app so I opened up the heap dump in MAT. Amongst other things I noticed the SoftCache running at much higher memory consumption than expected, with thousands of SoftCache$Entry instances. I would expect all soft references that pass the normal heuristics (no strong references to the referent) to be GC'd so I was surprised to see these still there.
Looking inside the SoftCache$Entrys I noticed that the referent, a BPage, is also the _ser attribute. This must stop the referent being garbage collected. Is this expected behaviour? If so, how does the SoftCache ever have its entries GC'd?
It's all the more confusing because when I look at the code I can only see one path to create an Entry, and that sets the _ser as null! That is, via SoftCache.get:250 -> SoftCache.put(key, value, false, null).
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Not sure what version you're using but first question is: are you calling
recman.flush() and/or recman.commit() often enough? Any object still
referenced by outstanding (non-committed/persisted) work will still have
hard reference(s) to it.
I experienced an OOME with my app so I opened up the heap dump in MAT.
Amongst other things I noticed the SoftCache running at much higher memory
consumption than expected, with thousands of SoftCache$Entry instances. I
would expect all soft references that pass the normal heuristics (no strong
references to the referent) to be GC'd so I was surprised to see these
still there.
Looking inside the SoftCache$Entrys I noticed that the referent, a BPage,
is also the _ser attribute. This must stop the referent being garbage
collected. Is this expected behaviour? If so, how does the SoftCache ever
have its entries GC'd?
It's all the more confusing because when I look at the code I can only see
one path to create an Entry, and that sets the _ser as null! That is, via
SoftCache.get:250 -> SoftCache.put(key, value, false, null).
With addressing your concerns about the Bpage reference, soft references make for a much less reliable memory management scheme. I would only use a weak reference cache. The capacity of the hard reference cache fronting for the weak reference cache then sets the retention capacity for the objects.
Bryan
I experienced an OOME with my app so I opened up the heap dump in MAT. Amongst other things I noticed the SoftCache running at much higher memory consumption than expected, with thousands of SoftCache$Entry instances. I would expect all soft references that pass the normal heuristics (no strong references to the referent) to be GC'd so I was surprised to see these still there.
Looking inside the SoftCache$Entrys I noticed that the referent, a BPage, is also the _ser attribute. This must stop the referent being garbage collected. Is this expected behaviour? If so, how does the SoftCache ever have its entries GC'd?
It's all the more confusing because when I look at the code I can only see one path to create an Entry, and that sets the _ser as null! That is, via SoftCache.get:250 -> SoftCache.put(key, value, false, null).
Can you explain a little more what you mean by "The capacity of the hard reference cache fronting for the weak reference cache then sets the retention capacity for the objects"? I notice the Entry inner class for WeakCache doesn't store the values in a hard reference.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
It has been several years since I was involved in that cache code, and that was with JDBM before there was a code / project fork. I assume that we are talking about the original JDBM code line.
The concept of a weak reference cache is that hard references, e.g., in a fixed size ring buffer, pin some number of objects. Once those objects are evicted from the ring buffer they are either ignored (if dirty) or evicted to the disk. The weak reference cache acts as a canonicalizing mapping so you always get the right view of the object. This generally involves having a per connection cache if you support multiple connections.
If the record evicted from the ring buffer is clean, then it remains weakly reachable and can still be retrieved from the cache. If any hard reference exists to that record, then it will be pinned into the cache. The ring buffers is a handy supply of hard references to recently used records.
Weak references that are swept by gc are reported to a reference queue object. The weak reference has already been cleared, but he entry for the record key still needs to be removed from the cache since it is now associated with a weak reference that does not point anywhere.
I do not remember the JDBM implementation In detail. The bigdata project on source forge has several implementations of this pattern that I wrote. Infinispan also has some caches with this pattern - we collaborated on lock amortization approaches.
Can you explain a little more what you mean by "The capacity of the hard reference cache fronting for the weak reference cache then sets the retention capacity for the objects"? I notice the Entry inner class for WeakCache doesn't store the values in a hard reference.
Thanks for trying to remember Bryan, I appreciate it's an old code base. FWIW the version I'm using has no diffs to the version in SVN in sf.net.
I tried running with the WeakReference cache and this seems to at least garbage collect, so that's some progress. I've got another (actually more problematic) memory issue with JDBM so I'll make myself unpopular by opening another thread.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I experienced an OOME with my app so I opened up the heap dump in MAT. Amongst other things I noticed the SoftCache running at much higher memory consumption than expected, with thousands of SoftCache$Entry instances. I would expect all soft references that pass the normal heuristics (no strong references to the referent) to be GC'd so I was surprised to see these still there.
Looking inside the SoftCache$Entrys I noticed that the referent, a BPage, is also the _ser attribute. This must stop the referent being garbage collected. Is this expected behaviour? If so, how does the SoftCache ever have its entries GC'd?
It's all the more confusing because when I look at the code I can only see one path to create an Entry, and that sets the _ser as null! That is, via SoftCache.get:250 -> SoftCache.put(key, value, false, null).
Hi Dan,
Not sure what version you're using but first question is: are you calling
recman.flush() and/or recman.commit() often enough? Any object still
referenced by outstanding (non-committed/persisted) work will still have
hard reference(s) to it.
alex
On Wed, Jan 15, 2014 at 4:09 AM, Dan Gravell gravelld2@users.sf.net wrote:
With addressing your concerns about the Bpage reference, soft references make for a much less reliable memory management scheme. I would only use a weak reference cache. The capacity of the hard reference cache fronting for the weak reference cache then sets the retention capacity for the objects.
Bryan
From: Dan Gravell gravelld2@users.sf.net<mailto:gravelld2@users.sf.net>
Reply-To: "[jdbm:discussion]" 12570@discussion.jdbm.p.re.sf.net<mailto:12570@discussion.jdbm.p.re.sf.net>
Date: Wednesday, January 15, 2014 7:09 AM
To: "[jdbm:discussion]" 12570@discussion.jdbm.p.re.sf.net<mailto:12570@discussion.jdbm.p.re.sf.net>
Subject: [jdbm:discussion] SoftCache not garbage collecting
I experienced an OOME with my app so I opened up the heap dump in MAT. Amongst other things I noticed the SoftCache running at much higher memory consumption than expected, with thousands of SoftCache$Entry instances. I would expect all soft references that pass the normal heuristics (no strong references to the referent) to be GC'd so I was surprised to see these still there.
Looking inside the SoftCache$Entrys I noticed that the referent, a BPage, is also the _ser attribute. This must stop the referent being garbage collected. Is this expected behaviour? If so, how does the SoftCache ever have its entries GC'd?
It's all the more confusing because when I look at the code I can only see one path to create an Entry, and that sets the _ser as null! That is, via SoftCache.get:250 -> SoftCache.put(key, value, false, null).
SoftCache not garbage collectinghttps://sourceforge.net/p/jdbm/discussion/12570/thread/7f298952/?limit=25#1b2e
Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/jdbm/discussion/12570/
To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/
As per https://weblogs.java.net/blog/enicholas/archive/2006/05/understanding_w.html I tend to use SoftReferences for caches, but understand this might not be the best way to work with the JDBM cache.
Can you explain a little more what you mean by "The capacity of the hard reference cache fronting for the weak reference cache then sets the retention capacity for the objects"? I notice the Entry inner class for WeakCache doesn't store the values in a hard reference.
It has been several years since I was involved in that cache code, and that was with JDBM before there was a code / project fork. I assume that we are talking about the original JDBM code line.
The concept of a weak reference cache is that hard references, e.g., in a fixed size ring buffer, pin some number of objects. Once those objects are evicted from the ring buffer they are either ignored (if dirty) or evicted to the disk. The weak reference cache acts as a canonicalizing mapping so you always get the right view of the object. This generally involves having a per connection cache if you support multiple connections.
If the record evicted from the ring buffer is clean, then it remains weakly reachable and can still be retrieved from the cache. If any hard reference exists to that record, then it will be pinned into the cache. The ring buffers is a handy supply of hard references to recently used records.
Weak references that are swept by gc are reported to a reference queue object. The weak reference has already been cleared, but he entry for the record key still needs to be removed from the cache since it is now associated with a weak reference that does not point anywhere.
I do not remember the JDBM implementation In detail. The bigdata project on source forge has several implementations of this pattern that I wrote. Infinispan also has some caches with this pattern - we collaborated on lock amortization approaches.
Thanks,
Bryan
On Jan 21, 2014, at 1:42 PM, "Dan Gravell" gravelld2@users.sf.net<mailto:gravelld2@users.sf.net> wrote:
As per https://weblogs.java.net/blog/enicholas/archive/2006/05/understanding_w.html I tend to use SoftReferences for caches, but understand this might not be the best way to work with the JDBM cache.
Can you explain a little more what you mean by "The capacity of the hard reference cache fronting for the weak reference cache then sets the retention capacity for the objects"? I notice the Entry inner class for WeakCache doesn't store the values in a hard reference.
SoftCache not garbage collectinghttps://sourceforge.net/p/jdbm/discussion/12570/thread/7f298952/?limit=25#1b2e/8ae7/5fec
Sent from sourceforge.nethttp://sourceforge.net because you indicated interest in https://sourceforge.net/p/jdbm/discussion/12570/
To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/
Thanks for trying to remember Bryan, I appreciate it's an old code base. FWIW the version I'm using has no diffs to the version in SVN in sf.net.
I tried running with the WeakReference cache and this seems to at least garbage collect, so that's some progress. I've got another (actually more problematic) memory issue with JDBM so I'll make myself unpopular by opening another thread.
I'm committing on each update. An SVN diff shows no changes to svn://svn.code.sf.net/p/jdbm/svn/trunk