-
Brian - I've got a question for you towards the bottom of this post - please be sure to read!
No, they shouldn't be the same size (although I am surprised that the difference is as much as you indicate). The byte arrays that actually consume the pages in the file don't get constructed until you commit. So if you have an object that changes 1000 times, it only gets serialized once, and...
2009-12-28 16:04:37 UTC by trumpetinc
-
Merry Christmas! So I realized how foolish I have been - I am serializing a collection of strings as a single value, and that collection grows with every insert. This is the source of my OOM. So I fixed that, and now even Compression doesn't cause OOM. I feel like an idiot... Thanks for all the help guys!
In order to fix the use of a collection as a value though I had to waste storage...
2009-12-24 19:18:20 UTC by frutchey
-
Memory leaks are generally the result of member variables of objects - you should start there. Serializers are usually best implemented as stateless, immutable objects, so you might want to check your implementation and make sure that is the case.
My recommendation is to create a small application that just runs your serializer in a tight loop, and monitor with a profiler. In other words:
2009-12-20 15:09:53 UTC by trumpetinc
-
It would seem the cause of the ever growing bytearray is my serializer. If all I do is serialize a single string for the key and value, even if those strings produce the same number of bytes as the average object my serializer handles, the bytearray seems to stay the same size. My serializer which has longs, single bytes, and sets of strings seems to be the problem - any ideas?.
2009-12-19 18:58:27 UTC by frutchey
-
Here's a test:
package jdbm.btree;
import java.io.File;
import java.util.Properties;
import jdbm.RecordManager;
import jdbm.RecordManagerFactory;
import jdbm.RecordManagerOptions;
import jdbm.helper.StringComparator;
import jdbm.helper.StringSerializer;
import jdbm.helper.compression.LeadingValueCompressionProvider;
/**.
2009-12-18 21:48:06 UTC by trumpetinc
-
this is different from my test results. I show that after 7 minutes or so, the heap usage goes flat and stays that way (granted, I didn't let it run for hours, but I did let it run for another 30 minutes, and saw no heap growth at all). I'll put together a simplified test that demonstrates this without having to have 3 or 4 source files.
2009-12-18 20:36:58 UTC by trumpetinc
-
I should probably also mention that I am setting Xmx to 1024M. I can avoid OOM by increasing memory limit enough to support # of inserts I am doing. Like 20M needs Xmx6000M.
2009-12-18 19:51:30 UTC by frutchey
-
Guys, I haven't been able to test the Windows-specific mods, but I am still seeing the byte array grow until OOM even when I do not use compression. I just ran another test where I removed all properties but the disabling of transactions, and still OOM. I will try the windows-specific settings next, then dig into the heap dump on a run. Seems like the byte array is much smaller (only 78Mb at...
2009-12-18 19:49:44 UTC by frutchey
-
Hello,
I'm doing some performance test with JDBM. I'm using the following config:
props.setProperty(RecordManagerOptions.DISABLE_TRANSACTIONS, "TRUE");
props.setProperty(RecordManagerOptions.DISABLE_TRANSACTIONS_PERFORMSYNCONCLOSE, "TRUE");
props.setProperty(RecordManagerOptions.NO_CACHE, "TRUE");
Is there something more i can configure to tune performance?
Thanks,
Olivier.
2009-12-18 00:01:58 UTC by olivier_smadja
-
I do see the Java heap growing without bound, but the OS memory also gets clobbered. The issue is almost certainly related to holding the Deflater object for too long.
But generic record level compression is really not desirable except for special circumstances (and in those cases, a custom serializer can do the trick). Rather than going in and trying to change the mechanism for caching...
2009-12-15 15:53:30 UTC by trumpetinc
