I've taken a fairly large break from development, so I haven't had time to continue our conversation for quite a while.
I do have an interesting issue for you though. Here is the stack trace:
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Unknown Source)
at java.nio.DirectByteBuffer.<init>(Unknown Source)
at java.nio.ByteBuffer.allocateDirect(Unknown Source)
at net.fortuna.mstor.data.MboxFile.getMessagePositions(Unknown Source)
at net.fortuna.mstor.data.MboxFile.getMessageCount(Unknown Source)
at net.fortuna.mstor.MStorFolder.getMessageCount(Unknown Source)
at net.ivanovic.synchingthunder.SynchingThread.syncFolders(SynchingThread.java:118)
at net.ivanovic.synchingthunder.SynchingThread.run(SynchingThread.java :71)
at foxtrot.AbstractWorkerThread$2.run(AbstractWorkerThread.java:49)
at java.security.AccessController.doPrivileged(Native Method)
at foxtrot.AbstractWorkerThread.runTask(AbstractWorkerThread.java :45)
at foxtrot.Worker.post(Worker.java:108)
at net.ivanovic.synchingthunder.SynchingThunderUI$7.run(SynchingThunderUI.java:668)
at foxtrot.AbstractWorkerThread$2.run(AbstractWorkerThread.java:49)
at java.security.AccessController.doPrivileged(Native Method)
at foxtrot.AbstractWorkerThread.runTask(AbstractWorkerThread.java:45)
at foxtrot.Worker.post(Worker.java:108)
at net.ivanovic.synchingthunder.SynchingThunderUI.syncFolders (SynchingThunderUI.java:656)
at net.ivanovic.synchingthunder.SynchingThunderUI.syncTargetToSource(SynchingThunderUI.java:598)
at net.ivanovic.synchingthunder.SynchingThunderUI.access$4(SynchingThunderUI.java :584)
at net.ivanovic.synchingthunder.SynchingThunderUI$6.run(SynchingThunderUI.java:613)
at foxtrot.AbstractWorkerThread$2.run(AbstractWorkerThread.java:49)
at java.security.AccessController.doPrivileged (Native Method)
at foxtrot.AbstractWorkerThread.runTask(AbstractWorkerThread.java:45)
at foxtrot.workers.DefaultWorkerThread.run(DefaultWorkerThread.java:153)
at java.lang.Thread.run(Unknown Source)
basically calling getMessageCount on a large folder (by large, I mean 100MB+) seems to cause the problem. I'm using mstor 0.9.7 - and while I haven't had any time to try and debug the problem in detail, I can tell you that this occured in a JVM which was allocated at least 256Mb of RAM.
Any ideas that you may have on how to reduce the overall memory usage would be excellent...
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
This is a bit strange to me, as the maximum size that is allocated for a buffer is 1024k. Anyway, I did notice some apache code that catches an OutOfMemoryError when calling ByteBuffer.allocateDirect() and calls ByteBuffer.allocate() instead. As an experiment I've added a flag to disable direct buffers, which is set via the following system property:
mstor.mbox.directBuffer=false
This is now in CVS so you can try it out on your 100Mb file (I haven't got one that big so I can't really test it).
regards,
ben
ps. Its always good to take a break from time to time. This is one advantage of Open Source (you don't have an obligation to keep up a certain pace), although once your software gets (relatively) popular the demand for it can be a bit of a driving force (in a good way usually).
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Testing large mailboxes is high on the agenda, as its definitely the best way to identify the memory issues (which I think have priority at the moment).
I haven't done much with mstor myself lately, but hopefully soon will get a chance to improve on it.
b.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi Ben,
I've taken a fairly large break from development, so I haven't had time to continue our conversation for quite a while.
I do have an interesting issue for you though. Here is the stack trace:
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Unknown Source)
at java.nio.DirectByteBuffer.<init>(Unknown Source)
at java.nio.ByteBuffer.allocateDirect(Unknown Source)
at net.fortuna.mstor.data.MboxFile.getMessagePositions(Unknown Source)
at net.fortuna.mstor.data.MboxFile.getMessageCount(Unknown Source)
at net.fortuna.mstor.MStorFolder.getMessageCount(Unknown Source)
at net.ivanovic.synchingthunder.SynchingThread.syncFolders(SynchingThread.java:118)
at net.ivanovic.synchingthunder.SynchingThread.run(SynchingThread.java :71)
at foxtrot.AbstractWorkerThread$2.run(AbstractWorkerThread.java:49)
at java.security.AccessController.doPrivileged(Native Method)
at foxtrot.AbstractWorkerThread.runTask(AbstractWorkerThread.java :45)
at foxtrot.Worker.post(Worker.java:108)
at net.ivanovic.synchingthunder.SynchingThunderUI$7.run(SynchingThunderUI.java:668)
at foxtrot.AbstractWorkerThread$2.run(AbstractWorkerThread.java:49)
at java.security.AccessController.doPrivileged(Native Method)
at foxtrot.AbstractWorkerThread.runTask(AbstractWorkerThread.java:45)
at foxtrot.Worker.post(Worker.java:108)
at net.ivanovic.synchingthunder.SynchingThunderUI.syncFolders (SynchingThunderUI.java:656)
at net.ivanovic.synchingthunder.SynchingThunderUI.syncTargetToSource(SynchingThunderUI.java:598)
at net.ivanovic.synchingthunder.SynchingThunderUI.access$4(SynchingThunderUI.java :584)
at net.ivanovic.synchingthunder.SynchingThunderUI$6.run(SynchingThunderUI.java:613)
at foxtrot.AbstractWorkerThread$2.run(AbstractWorkerThread.java:49)
at java.security.AccessController.doPrivileged (Native Method)
at foxtrot.AbstractWorkerThread.runTask(AbstractWorkerThread.java:45)
at foxtrot.workers.DefaultWorkerThread.run(DefaultWorkerThread.java:153)
at java.lang.Thread.run(Unknown Source)
basically calling getMessageCount on a large folder (by large, I mean 100MB+) seems to cause the problem. I'm using mstor 0.9.7 - and while I haven't had any time to try and debug the problem in detail, I can tell you that this occured in a JVM which was allocated at least 256Mb of RAM.
Any ideas that you may have on how to reduce the overall memory usage would be excellent...
Hi Rusty,
This is a bit strange to me, as the maximum size that is allocated for a buffer is 1024k. Anyway, I did notice some apache code that catches an OutOfMemoryError when calling ByteBuffer.allocateDirect() and calls ByteBuffer.allocate() instead. As an experiment I've added a flag to disable direct buffers, which is set via the following system property:
mstor.mbox.directBuffer=false
This is now in CVS so you can try it out on your 100Mb file (I haven't got one that big so I can't really test it).
regards,
ben
ps. Its always good to take a break from time to time. This is one advantage of Open Source (you don't have an obligation to keep up a certain pace), although once your software gets (relatively) popular the demand for it can be a bit of a driving force (in a good way usually).
I've renamed this flag. You should now specify the following to disable direct buffers:
mstor.mbox.useDirectBuffers=false
regards,
ben
Hi Ben,
Thanks for the quick update. I'll give it a go when I get some time this week.
Do you have any plans for testing your code on large mailboxes in the future?
cheers,
Rusty
p.s. I agree on the time advantage of Open Source, and luckily my project is not that popular, so I can take it easy :-)
Rusty,
Testing large mailboxes is high on the agenda, as its definitely the best way to identify the memory issues (which I think have priority at the moment).
I haven't done much with mstor myself lately, but hopefully soon will get a chance to improve on it.
b.
cheers Ben, I'll let you know how it goes.