running out of memory on a large insert?

2005-07-14
2013-06-03
  • Bryan Thompson
    Bryan Thompson
    2005-07-14

    Hello,

    I am trying to figure out why jdbm runs out of memory on a large insert operation.  I thought that there was an option (jdbm.disableTransactions) to turn off the in-memory transaction buffer, but that does not seem to solve the problem.  I have not looked into this deeply (yet), but I was wondering if other folks had already explored this problem in their applications and what tradeoffs are involved as the size of the transaction grows very large.

    Thanks,

    -bryan

     
    • Alex Boisvert
      Alex Boisvert
      2005-07-15

      Have you made sure that your application is not "hoarding" a large object graph?

      Beyond the LRU cache and the transaction buffers, JDBM does not retain any object in memory.

      Maybe it's time to launch your memory profiler to see what objects stay in memory and how they are being referenced...

      alex

       
    • Bryan Thompson
      Bryan Thompson
      2005-07-16

      Alex,

      We're definately not hoarding.  All references are recids.  The application  itself is layered.  We have a different persistence layer that we also run over which does not have these problems.  I ran the profiler earlier, but I have not delved into the data yet to see where all the memory is going.  I was hoping that there was an easy answer with respect to transaction buffers.

      One thing that I do not understand is the tradeoff when the in-memory transaction buffers are disabled.  Could you expand on that?

      Thanks,

      -bryan

       
      • Bryan Thompson
        Bryan Thompson
        2005-07-17

        Alex,

        I spoke too soon.  There is a graph of listeners that are coded as pointers and not recids....oops!

        I was thinking in terms of the application, which is the same over jdbm and the other store.  However the intermediate layer for jdbm is different -- and that is where we are hoarding.  My bad.

        Thanks,

        -bryan

         
        • Alex Boisvert
          Alex Boisvert
          2005-07-17

          Good news;  I'm glad you resolved your problem.

          alex