java.lang.Error: in use list not empty at com

2010-05-07
2013-06-03
  • Frank Nestel

    Frank Nestel - 2010-05-07

    Hello,

    we have recently used JDBM to move a growing internal HashSet from a long running application (a special kind of a web spider, running 12h daily) to disc in form of a JDBM database. We are using jdbm 1.0. This has worked very smooth and performant without any errors for approximately 50 days now. Until tonight when our job was stopped by below error (error and stderr output down below in this message).

    Now the question: Is there anything we can do against this on the caller side or by settings or whatever. Are there any later versions of JDBM (a quick google suggested, that there are branches of JDBM on SF.net, Codehaus and maybe Apache). Are there later versions of JDBM. Hope for relieving below problem or any other other problem?

    Any help highly appreciated,
    Thank you,
    Frank

    java.lang.Error: in use list not empty at commit time (2)
            at jdbm.recman.RecordFile.commit(RecordFile.java:569)
            at jdbm.recman.RecordFile.release(RecordFile.java:519)
            at jdbm.recman.RecordFile.release(RecordFile.java:503)
            at jdbm.recman.LogicalRowIdManager.fetch(LogicalRowIdManager.java:172)
            at jdbm.recman.BaseRecordManager.fetch(BaseRecordManager.java:593)
            at jdbm.recman.CacheRecordManager.fetch(CacheRecordManager.java:313)
            at jdbm.btree.BTree._fetch(BTree.java:863)
            at jdbm.btree.BPage.loadBPage(BPage.java:918)
            at jdbm.btree.BPage.childBPage(BPage.java:907)
            at jdbm.btree.BPage.find(BPage.java:312)
            at jdbm.btree.BPage.find(BPage.java:313)
            at jdbm.btree.BPage.find(BPage.java:313)
            at jdbm.btree.BPage.find(BPage.java:313)
            at jdbm.btree.BPage.find(BPage.java:313)
            at jdbm.btree.BTree.find(BTree.java:441)

    stderr:
    elem 0: BlockIO(0,true,null)
    elem 1: BlockIO(213780,false,null)

    both repeated until process stalled (not stopped at all).

     
  • Kevin Day

    Kevin Day - 2010-05-07

    Interesting bug.  From what I can see, it looks like you are running with transactions disabled.  As near as I can tell, the problem is in this block of code (my code may be different from yours):

        /**
         *  Releases a block.
         *
         *  @param block The block to release.
         * @throws IOException
         */
        void release(BlockIo block) throws IOException {
            long key = block.getBlockId();
            inUse.remove(key);
            if (block.isDirty()) {
                // System.out.println( "Dirty: " + key + block );
                dirty.put(key, block);
                if (transactionsDisabled && dirty.size() > maxDirtySize)
                commit();
            } else {
                if (!transactionsDisabled && block.isInTransaction()) {
                    inTxn.put(key, block);
                } else {
                    free.add(block);
                }
            }
        }

    namely, the auto-commit is triggering when a certain number of dirty blocks are seen.  This normally happens when user blocks are released - but in the case of your problem, the block that's triggering this is in the logical row table, which is something that can change in unexpected ways.

    The reason this hasn't come up is that most people run with transactions enabled - if that is an option for you, it will almost certainly prevent the problem without any code change required.

    A real fix, I think, involves adjusting the following line of code:

    if (transactionsDisabled && dirty.size() > maxDirtySize)

    to be:

    if (transactionsDisabled && dirty.size() > maxDirtySize && inUse.isEmpty())

    I'll wait to see if anyone else has opinions on this, then commit that change.

     
  • Alex Boisvert

    Alex Boisvert - 2010-05-07

    Yes, interesting bug.   Kevin's proposed fix looks right.

    Alternatively, calling RecordManager.commit() periodically in your application (even if you're not actually using transactions) will release buffers early and should avoid this condition.

    alex

     
  • Kevin Day

    Kevin Day - 2010-05-07

    and now the question of the day - is jdbm.cvs.sourceforge.net still the active repo?  I want to make sure I commit the change to the right place.

     
  • Alex Boisvert

    Alex Boisvert - 2010-05-07

    Yes, jdbm.cvs.sourceforge.net is still the reference repository.

     
  • Kevin Day

    Kevin Day - 2010-05-08

    It looks like the repo is actually now at jdbm.svn.sourceforge.net - anyway, I have released this bug fix to head.  nestefan, you will need to build this from HEAD to get the fix (it'll probably be while before an official next release occurs)

     
  • Frank Nestel

    Frank Nestel - 2010-05-08

    Thank you all for the quick and helpful respose!

    I'll try to obtain the SVN HEAD and build it (I am somewhat trapped behind a firewall that does not allow me many protocolls, so subversion via HTTP suits me best). I hope, there is anything worrying new (say breakable) in HEAD.

    In case the answers to questions are still of interest.
    * We replaced a (RAM) HashSet by JDBM and it seemed like "no transactions" is faster. In the current setting JDBM is really fine for us, and we felt (did not take the time to test), that transaction might slow down?!
    * We use autocommit, which should, as I understood do occasional commits from time to time automatically.

    I will take a few days to transfer the patch into production. So feedback might not come soon.

    Thanx once more
    Frank (nestefan)

     

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks