Naveen,

Can you reduce your application down to a test-case that would exemplify this behavior?

alex


On 1/31/08, NAVEEN UPADHYAY <naveen.invincible@gmail.com> wrote:
Hi Alex and Kevin,

All said and done.
I did put purging capabilities in my code such that if file size is greater than >threshold ...purge a few records from BTree and commit!

Now let me put some facts before i ask my final question.

I have got basically two database tables tab$1 and tab$2.
there are hardly 50 rows in them with all text data.

Now i constantly poll these two tables and write into two BTrees (by the same name)

Everytime i write i assume there could be changes to the values in various rows .therefore i say btree.insert(key,value,true) .
each time
.
Now first of all if the data is the same the .db file should not change its size(as i will be just overwrting the same info)...Right? If so i should NOT need purging capabilities for such small data..!

Wrong ..as i see it just growing and growing..from a 2 MB file it has grown on to 45 MB in the last one week..

 All of that just to store records just storing people names and there ids ...(My entire databse MySql size is smaller now by good 15 MB than the .db file)


The cause just evades me... :-(

I am using jdbm0.12.jar .Dont know if that jar had such issues.

I really dont know whether i should be exploring other ways to get the persistance thing working!

Best Regards,
Naveen


On Jan 29, 2008 9:59 AM, NAVEEN UPADHYAY <naveen.invincible@gmail.com> wrote:
Yes Kevin,you are right but the very reason why i used timestamp was to make the records can be retrieved in the order in they were persisted..Now if i put back the recid into as the key .. i'll be no better than before(without timestamp) as the order might not be guaranteed
-Naveen!


On Jan 28, 2008 10:00 PM, Kevin Day <kevin@trumpetinc.com> wrote:
FYI - my comment on having the same timestamp does not preclude your use of a timestamp - you just have to build your index key so it is always unique.
 
Creating a key that consists of the time stamp, followed by the recid (or some other unique identifying information of you data - such as the key in your original btree) will achieve this.
 
- K
 
----------------------- Original Message -----------------------
  
From: "NAVEEN UPADHYAY" <naveen.invincible@gmail.com>
Cc: 
Date: Mon, 28 Jan 2008 19:29:52 +0530
Subject: Re: [Jdbm-developer] Need help on .db file optimisation | JDBM
  

Hi all,
       In my code i am using the jdbm-0.12.jar and i am cleaning up records from btree using a pseudocode like
if (_btree.size() >= threshold){//NaveenBEG
                    //Going for bulk purging!Please wait.....
                    int count =0;
                    Tuple tuple;
                    TupleBrowser browser;
                    browser = _btree.browse();
            &n bsp;       tuple = new Tuple();

                    while(count <threshold/2 &&browser.getNext(tuple)){
                        _btree.remove(tuple.getKey());
                            count++ ;
                    }

and there is indeed a recman commit to follow...Still i am not seeing any decrease in the .db file size.
Is there still a problem with my code or something wrong with the jdbm-0.12.jar.

Naveen
Alex i didnt use the timestamp method as u suggested ,since as rightly p ointed out by Kevin lot many transaction were hitting the same timestamp :(

On Jan 25, 2008 2:44 AM, NAVEEN UPADHYAY <naveen.invincible@gmail.com> wrote:
Kevin,
Regarding your point, if I do BTree.remove, then I don't need not bother separately for the record manager..
 
Naveen 

On Jan 24, 2008 11:52 PM, NAVEEN UPADHYAY <naveen.invincible@gmail.com> wrote:
Just to confirm to Alex's comment..
We need another Btree(one more)..??this time its value being the key to the first one...



On Jan 24, 2008 3:37 AM, Kevin Day < kevin@trumpetinc.com> wrote:
Just make sure you delete the records from the BTree when you delete them from the record manager :-)
 
For these situations, we actually store the key in the btree as:
 
[timestamp][recid]
 
and the value as a null value.  that way we don't have to worry about key collision if two transactions hit at the exact same time stamp.
 
I have a class around here somehwere that allows you to easily implement a multi-keyed index like the above using longs or strings.  If it's of interest, let me know and I'll send it to you directly.
 
- K
 
----------------------- Original Message -----------------------
  
From: "Alex Boisvert" <boisvert@intalio.com>
To: "NAVEEN UPADHYAY" <naveen.invincible@gmail.com>
Date: Wed, 23 Jan 2008 05:32:38 -0800
Subject: Re: [Jdbm-developer] Need help on .db file optimisation | JDBM
  
Naveen,

You can delete/remove whatever you are putting into the database.  It's up to your application code to do that.  

For example, if you are recording events, simply tag the events with a timestamp, create an index (btree) on the timestamp and delete old events when you reach a given threshold.  You can collect statistics on a sample set of records to determine your average space requirement and then extrapolate to obtain your upper limit based on your file system limitations (2G), minus some conservative margin.

regards,
alex



On 1/22/08, NAVEEN UPADHYAY <naveen.invincible@gmail.com> wrote:
Thanks Alex,Kevin for your responses ;^)
 The proposed solutions seem to be quite good.
Only thing i have is my main concern is to purge old data with these .db files and NOT their corruption beyond 2gb.
My only aim is to keep the size of the file always within a limit.
For my driver only the recent data is of use..
So all that i am looking for is a way to purge old data

Will throwing IOExceptions work?
Please suggest some solutions...
Thanks in advance,
Naveen

On Jan 23, 2008 5:02 AM, Alex Boisvert < boisvert@intalio.com> wrote:
On 1/22/08, Kevin Day <kevin@trumpetinc.com> wrote:
Another option would be to have an OOSProvider that could be configured with an instrumentable version for testing (this would be more akin to dependency injection)...  It feels weird to add code to the runtime implementation of a class that forces it to fail...
 
Any preference? (although I love Brian's idea of using a thumb drive - or maybe even a RAM disk - it's just hard to enforce that stuff in a test suite :-)  )

I understand the weird feeling and I don't have a strong preference.  I think the OOSProvider solution is going to involve more code to achieve the same goal.  If the abstraction isn't needed in the first place, I don't think it's worth the added complexity, meaning adding 4-5 more classes for this test alone.

alex


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Jdbm-developer mailing list
Jdbm-developer@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jdbm-developer



-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

_______________________________________________
Jdbm-developer mailing list
Jdbm-developer@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jdbm-developer



-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Jdbm-developer mailing list
Jdbm-developer@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jdbm-developer




-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

_______________________________________________
Jdbm-developer mailing list
Jdbm-developer@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jdbm-developer





-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Jdbm-developer mailing list
Jdbm-developer@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jdbm-developer