Menu
▾
▴
Re: [Jdbm-developer] Resending to both of you as this msg exceeds mailing list msg limit
|
From: Kevin D. <ke...@tr...> - 2008-02-18 21:36:16
|
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META content="MSHTML 6.00.2900.3059" name=GENERATOR></HEAD>
<BODY leftMargin=1 topMargin=1 rightMargin=1><FONT face=Arial size=2>
<DIV>Glad to hear it. FYI - the best way to delete a BTree is to use it's .delete() method (I wrote that about a year ago when I had the need to efficiently clear out a BTree). If you are truly trying to remove the entire contents of the BTree, just delete it and re-create it. Much faster than iterating keys and deleting that way.</DIV>
<DIV> </DIV>
<DIV>- K</DIV>
<DIV> </DIV></FONT>
<DIV style="FONT-SIZE: x-small; FONT-FAMILY: Tahoma">
<DIV>----------------------- <B>Original Message</B> -----------------------</DIV>
<DIV> </DIV>
<DIV><B>From:</B> "NAVEEN UPADHYAY" <A href="mailto:nav...@gm..."><FONT color=#0000ff><nav...@gm...></FONT></A></DIV>
<DIV><B>To:</B> "Kevin Day" <A href="mailto:ke...@tr..."><FONT color=#0000ff><ke...@tr...>,</FONT></A> "Alex Boisvert" <A href="mailto:boi...@in..."><FONT color=#0000ff><boi...@in...></FONT></A></DIV>
<DIV><B>Cc:</B> "JDBM Developer listserv" <A href="mailto:jdb...@li..."><FONT color=#0000ff><jdb...@li...></FONT></A></DIV>
<DIV><B>Date:</B> Sat, 16 Feb 2008 06:58:44 +0530</DIV>
<DIV><B>Subject: <U>Re: [Jdbm-developer] Resending to both of you as this msg exceeds mailing list msg limit</U></B></DIV>
<DIV> </DIV></DIV>One more thing that i would like to add is the changes are really working as desired in the actual application that i had problems with.<BR><BR>Thanks for your support.<BR><BR><BR>Regards,<BR>Naveen<BR><BR><BR>
<DIV class=gmail_quote>On Feb 14, 2008 11:13 PM, Kevin Day <<A href="mailto:ke...@tr...">ke...@tr...</A>> wrote:<BR>
<BLOCKQUOTE class=gmail_quote style="PADDING-LEFT: 1ex; MARGIN: 0pt 0pt 0pt 0.8ex; BORDER-LEFT: rgb(204,204,204) 1px solid">
<DIV><FONT face=Arial size=2>
<DIV>Alex-</DIV>
<DIV> </DIV>
<DIV>All tests in the Test.java test suite pass for me, and I have confirmed that the adjustement you make addresses Naveen's situation. I would still like to get Naveen to shoot me a test case so we can get test coverage on this, but I have made Alex's change and commited it to CVS HEAD.</DIV>
<DIV> </DIV>
<DIV>- K</DIV>
<DIV> </DIV></FONT>
<DIV>
<DIV></DIV>
<DIV class=Wj3C7c>
<DIV style="FONT-SIZE: x-small; FONT-FAMILY: Tahoma">
<DIV>----------------------- <B>Original Message</B> -----------------------</DIV>
<DIV> </DIV>
<DIV><B>From:</B> "Alex Boisvert" <A href="mailto:boi...@in..." target=_blank><FONT color=#0000ff><boi...@in...></FONT></A></DIV>
<DIV><B>To:</B> "Kevin Day" <A href="mailto:ke...@tr..." target=_blank><FONT color=#0000ff><ke...@tr...></FONT></A></DIV>
<DIV><B>Cc:</B> "JDBM Developer listserv" <A href="mailto:jdb...@li..." target=_blank><FONT color=#0000ff><jdb...@li...></FONT></A></DIV>
<DIV><B>Date:</B> Thu, 14 Feb 2008 09:32:08 -0800</DIV>
<DIV><B>Subject: <U>Re: [Jdbm-developer] Resending to both of you as this msg exceeds mailing list msg limit</U></B></DIV>
<DIV> </DIV></DIV>I took a quick look at Naveen's test and the BTree code and found that we didn't correctly release the record for the root page of the BTree:<BR><BR>Index: src/main/jdbm/btree/BTree.java<BR>===================================================================<BR>RCS file: /cvsroot/jdbm/jdbm/src/main/jdbm/btree/BTree.java,v<BR>retrieving revision 1.17<BR>diff -c -r1.17 BTree.java<BR>*** src/main/jdbm/btree/BTree.java 31 May 2006 22:25:31 -0000 1.17<BR>--- src/main/jdbm/btree/BTree.java 14 Feb 2008 17:07:29 -0000<BR>***************<BR>*** 396,402 ****<BR> _height -= 1;<BR> dirty = true;<BR><BR>! // TODO: check contract for BPages to
be removed from recman.<BR> if ( _height == 0 ) {<BR> _root = 0;<BR> } else {<BR>--- 396,402 ----<BR> _height -= 1;<BR> dirty = true;<BR><BR>! _recman.delete(_root);<BR> if ( _height == 0 ) {<BR> _root = 0;<BR> } else {<BR><BR>So the
leak appears to be specific to the case where the BTree is emptied and the root page replaced. With this change, I don't see any leak with his test case anymore.<BR><BR>Unfortunately, the test cases don't all pass on my machine right now so I hesitate to commit the change until I can be assured that I'm not breaking anything else. (The failing test don't seem to be released to the code I've changed, though)<BR><BR><SPAN style="FONT-FAMILY: courier new,monospace"> [junit] Tests FAILED</SPAN><BR style="FONT-FAMILY: courier new,monospace"><SPAN style="FONT-FAMILY: courier new,monospace">The following tests failed:</SPAN><BR style="FONT-FAMILY: courier new,monospace"><SPAN style="FONT-FAMILY: courier new,monospace">jdbm.helper.TestWeakCache</SPAN><BR style="FONT-FAMILY: courier new,monospace"><SPAN style="FONT-FAMILY: courier new,monospace">jdbm.recman.TestBufferedRecordInstallManager</SPAN><BR style="FONT-FAMILY: courier new,monospace"><BR>Kevin,
are all the tests working for you? I can provide details about the failures if needed.<BR><BR>alex<BR><BR><BR></DIV></DIV>
<DIV>
<DIV>
<DIV></DIV>
<DIV class=Wj3C7c><SPAN class=gmail_quote>On 2/14/08, <B class=gmail_sendername>Kevin Day</B> <<A href="mailto:ke...@tr..." target=_blank>ke...@tr...</A>> wrote:</SPAN> </DIV></DIV>
<BLOCKQUOTE class=gmail_quote style="PADDING-LEFT: 1ex; MARGIN: 0pt 0pt 0pt 0.8ex; BORDER-LEFT: rgb(204,204,204) 1px solid">
<DIV>
<DIV>
<DIV></DIV>
<DIV class=Wj3C7c><FONT face=Arial size=2></FONT>
<DIV><FONT face=Arial size=2>Naveen-</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>This is pretty basic jdbm stuff - but your code certainly doesn't work with the latest version of jdbm (what you are using looks really, really old...). Can you please make sure you are running jdbm 1.0 or higher? <A href="http://jdbm.sourceforge.net/" target=_blank>http://jdbm.sourceforge.net/</A></FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>All that said, what you really need to provide is a unit test case that can run and show a failure. If you are not familiar with writing JUnit tests, the JUnit cookbook can get you started: <A href="http://junit.sourceforge.net/doc/cookbook/cookbook.htm" target=_blank>http://junit.sourceforge.net/doc/cookbook/cookbook.htm</A></FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>Most likely you are going to want the unit test to check the size of the .db file before and after your operations and show that it continues to grow, regardless of the number of insert/remove cycles you perform.</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>Once I adjusted your code so it would compile and work with the jdbm 1.0 libraries, I manually watched the size of the db file, and I can confirm that it does appear to be ever-increasing - but in order for this to be useful for development purposes, you've got to create a test case. The test case will show a failure (presumably), then we can all have a discussion about *why* the test case is failing.</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>So the first step is a unit test that results in a failure that demonstrates the problem. Without that, the testing can't be automated.</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>If you haven't created test cases before, take a stab at it, and send it to just me (no need to bug Alex with it). I'll do a code review and make suggestions, then once we have a solid unit test, you can submit it.</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>For your reference, her is your code adjusted to work with jdbm 1.0 (I have added comments starting with '<FONT color=#3f7f5f size=2>NAVEEN - </FONT>' in places that you'll have to adjust to convert this into a unit test.</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>package jdbm.btree;</FONT></DIV>
<DIV><FONT face=Arial size=2>import java.io.PrintStream;<BR>import java.util.Properties;</FONT></DIV>
<DIV><FONT face=Arial size=2>import jdbm.RecordManager;<BR>import jdbm.RecordManagerFactory;<BR>import jdbm.helper.StringComparator;<BR>import jdbm.helper.Tuple;<BR>import jdbm.helper.TupleBrowser;</FONT></DIV>
<DIV><FONT face=Arial size=2>public class DBSizeTest {</FONT></DIV>
<DIV><FONT face=Arial size=2> static String DATABASE = "testpeople";</FONT></DIV>
<DIV><FONT face=Arial size=2> static String BTREE_NAME = "FamousPeople";</FONT></DIV>
<DIV><FONT face=Arial size=2> static String[] people =<BR> { "Greenspan, Alan",<BR> "Williams-Byrd, Julie",<BR> "Picasso, Pablo",<BR> "Stallman, Richard",<BR> "Fort, Paul",<BR> "Søndergaard, Ole",<BR> "Schwarzenegger, Arnold",<BR> "Dulkinys, Susanna"<BR> };</FONT></DIV>
<DIV><FONT face=Arial size=2> static String[] occupations =<BR> { "Federal Reserve Board Chairman",<BR> "Engineer",<BR> "Painter",<BR> "Programmer",<BR> "Poet",<BR> "Typographer",<BR> "Actor",<BR> "Designer"<BR> };</FONT></DIV>
<DIV><FONT face=Arial size=2> static String PREFIX = "S";</FONT></DIV>
<DIV><FONT face=Arial size=2><BR> /**<BR> * Example main entrypoint.<BR> */<BR> <BR> // NAVEEN - need to change this to be a JUnit compatible method signature<BR> public static void main( String[] args ) {<BR> RecordManager recman;<BR> BTree tree;<BR> long recid;<BR> Tuple tuple = new Tuple();<BR> TupleBrowser browser;</FONT></DIV>
<DIV><FONT face=Arial size=2> try {<BR> // open database and setup an object cache<BR> Properties props = new Properties();<BR> recman = RecordManagerFactory.createRecordManager( DATABASE );</FONT></DIV></DIV></DIV>
<DIV><FONT face=Arial size=2>
<DIV>
<DIV></DIV>
<DIV class=Wj3C7c> // try to reload an existing B+Tree<BR> recid = recman.getNamedObject( BTREE_NAME );<BR> if ( recid != 0 ) {<BR> tree = BTree.load( recman, recid );<BR> System.out.println( "Reloaded existing BTree with " + tree.size()<BR> + " famous people." );<BR> } else {<BR></DIV></DIV> &
amp;n bsp; &n bsp; // create a new B+Tree data structure and use a StringComparator
<DIV class=Ih2E3d><BR> // to order the records based on people's name.<BR> tree = BTree.createInstance(recman, new StringComparator());<BR> recman.setNamedObject( BTREE_NAME, tree.getRecid() );<BR> System.out.println( "Created a new empty BTree" );<BR> <BR> <BR> &nbs p; }<BR> &nb
sp; <BR> // NAVEEN - need to get the size of the DB file here<BR> for (int i = 0; i < 500; i++){<BR> <BR> // insert people with their respective occupation<BR> for ( int i=0; i<people.length; i++ ) {<BR> System.out.println( "Insert: " + people[i] );<BR></DIV> tree.insert( peopl e[ i ], occupations[ i ], true );< BR>
}
<DIV class=Ih2E3d><BR> <BR> for ( int i=0; i<people.length; i++ ) {<BR> System.out.println( "Remove: " + people[i] );<BR> tree.remove( people[ i ]);<BR> }<BR> <BR> // make the data persistent in the database<BR></DIV> &nb sp; r ecman.commit();
<DIV class=Ih2E3d><BR> }<BR> <BR> // NAVEEN - need to get the size of the db file here and compare it to the original size using an asserTrue() call</DIV></FONT></DIV>
<DIV class=Ih2E3d>
<DIV><FONT face=Arial size=2> } catch ( Exception except ) {<BR> except.printStackTrace();<BR> }<BR> }</FONT></DIV>
<DIV><FONT face=Arial size=2><BR>}<BR></FONT></DIV><SPAN>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>- K</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV></SPAN></DIV>
<DIV><SPAN>
<DIV class=Ih2E3d>
<DIV style="FONT-SIZE: x-small; FONT-FAMILY: Tahoma">
<DIV>----------------------- <B>Original Message</B> -----------------------</DIV>
<DIV> </DIV>
<DIV><B>From:</B> "NAVEEN UPADHYAY" <A href="mailto:nav...@gm..." target=_blank><FONT color=#0000ff><nav...@gm...></FONT></A></DIV>
<DIV><B>To:</B> "Alex Boisvert" <A href="mailto:boi...@in..." target=_blank><FONT color=#0000ff><boi...@in...>,</FONT></A> "Kevin Day" <A href="mailto:ke...@tr..." target=_blank><FONT color=#0000ff><ke...@tr...></FONT></A></DIV>
<DIV><B>Cc:</B> </DIV>
<DIV><B>Date:</B> Thu, 14 Feb 2008 17:31:04 +0530</DIV>
<DIV><B>Subject: <U>Resending to both of you as this msg exceeds mailing list msg limit</U></B></DIV>
<DIV> </DIV></DIV>Hi Alex/Kevin,<BR><BR> Here is the sample program that illustrates the problem.<BR>All that i am doing is adding and removing data from the same Btree.<BR>Ideally such a case should keep the size of the .db file within a threshold.<BR>But as you can see (while running this program) this is NOT the case.<BR>The file size is ever increasing.<BR><BR>I am wondering whether its a problem with JDBM or something wrong at my end.<BR>In case any more input is required i will be online for the next 10 hours .<BR><A href="mailto:sag...@ya..." target=_blank>sag...@ya...</A><BR><BR>Please look into this.<BR><BR>Thanks in advance,<BR><FONT color=#888888>Naveen<BR></FONT>PS- there is a specific reason why my application has to remove data after every poll cycle.<BR></DIV>If you want to know my application logic i can provide the same .However the given sample program(attached) i s itself sufficient!
<DIV class=Ih2E3d><BR>PSS- The file can be put in<B> /src/examples</B> folder of<B> jdbm-0.12.jar</B> </DIV></SPAN></DIV></DIV>
<DIV class=Ih2E3d><BR>-------------------------------------------------------------------------<BR>This SF.net email is sponsored by: Microsoft<BR>Defy all challenges. Microsoft(R) Visual Studio 2008.<BR><A href="http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/" target=_blank>http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/</A><BR>_______________________________________________<BR>Jdbm-developer mailing list<BR><A href="mailto:Jdb...@li..." target=_blank>Jdb...@li...</A><BR><A href="https://lists.sourceforge.net/lists/listinfo/jdbm-developer" target=_blank>https://lists.sourceforge.net/lists/listinfo/jdbm-developer</A><BR><BR></DIV></BLOCKQUOTE></DIV><BR></DIV><BR>-------------------------------------------------------------------------<BR>This SF.net email is sponsored by: Microsoft<BR>Defy all challenges. Microsoft(R) Visual Studio 2008.<BR><A href="http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/" target=
_blank>http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/</A><BR>_______________________________________________<BR>Jdbm-developer mailing list<BR><A href="mailto:Jdb...@li...">Jdb...@li...</A><BR><A href="https://lists.sourceforge.net/lists/listinfo/jdbm-developer" target=_blank>https://lists.sourceforge.net/lists/listinfo/jdbm-developer</A><BR><BR></BLOCKQUOTE></DIV><BR></BODY></HTML>
|