Work at SourceForge, help us to make it a better place! We have an immediate need for a Support Technician in our San Francisco or Denver office.

Close

Memory Problem with JDBM's H-Tree

Chen Yang
2005-10-08
2013-06-03
  • Chen Yang
    Chen Yang
    2005-10-08

    Hi all,

    Please help me out with this memory problem with JDBM's H-Tree, thanks in advance.

    I am using JDBM to develop my search system.  While I load the data into JDBM, I put them into HTree.  After I add certain information, I commit and then close RecordManager, and then I reload RecordManager and HTree and add new data again.

    Then I got a OutOfMemory Exception from JVM.  I suppose this Exception is due to huge H-Tree after certain amount of data.  How can I take care of H-Tree so that the H-Tree wouldn't be too huge in my memory?
    Should I use two gdbm to take care of this?  Or should I use two H-Tree and check them one at a time?

    Thank you very much for your help.

    -CY

     
    • Alex Boisvert
      Alex Boisvert
      2005-10-10

      Chen,

      There are no know memory-related problems with HTree.

      Most likely, your application itself keeps direct references to the objects you place in the HTree.  This may cause memory retention and lead to OutOfMemory exceptions.

      When using JDBM, your application should normally access objects via the RecordManager/HTree/BTree interfaces.   Except for caching and short-term operations, you application should not maintain references to objects stored in JDBM.

      I recommend using a memory profilter to track down this issue.

      Hope this helps,
      alex

       
    • Kevin Day
      Kevin Day
      2005-10-10

      Chen-

      What Alex says is true - there are no known memory leaks in HTree, and HTree does not load it's entire contents into memory (that's the point of a file based database, after all).

      If you are not maintaining references to the objects in the HTree (please confirm this), then there may be an interaction between how you are using jdbm and the underlying transaction manager that could cause your problem.

      Are you certain that you are not holding on to references to the objects returned by the HTree?

      Do you have transactions disabled or enabled? 

      How many entries are you adding to the HTree before you commit changes?

      Thanks,

      - K

       
    • Chen Yang
      Chen Yang
      2005-10-10

      Thanks a lot.  I'm using Memory profiler now and see if I can find flaw in my program.

      This is basic what my program does:  It parses around 10G textual content into jdbm, and does basic searching.  When I test my program in a small scale content like 1MB, the result turns out fine; but when I tried the full scale content, it became very slow and finally got this OutOfMemoryException.  I also tried -Xms512m -Xmx512m but this didn't help much either.

      My program does commit() every 10,000 adding to HTree. The transaction was enabled all the time.

       
      • Kevin Day
        Kevin Day
        2005-10-10

        Just for testing, try calling commit() every 500 insertions.  Depending on the size of your values, 10,000 insertions may be a lot to ask the transaction manager to hold in memory at the same time.

        Let us know what happens with that test, and we'll proceed from there.

        - K