From: NAVEEN U. <nav...@gm...> - 2008-01-31 14:00:33
|
Hi Alex and Kevin, All said and done. I did put purging capabilities in my code such that if file size is greater than >threshold ...purge a few records from BTree and commit! Now let me put some facts before i ask my final question. I have got basically two database tables tab$1 and tab$2. there are *hardly* 50 rows in them with all text data. Now i constantly poll these two tables and write into two BTrees (by the same name) Everytime i write i assume there could be changes to the values in various rows .therefore i say* btree.insert(key,value,true) . each time*. Now first of all if the data is the same the .db file should not change its size(as i will be just overwrting the same info)...Right? If so i should NOT need purging capabilities for such small data..! Wrong ..as i see it just growing and growing..from a 2 MB file it has grown on to 45 MB in the last one week.. All of that just to store records just storing people names and there ids ...(My entire databse MySql size is smaller now by good 15 MB than the .db file) The cause just evades me... :-( I am using *jdbm0.12.jar .Dont know if that jar had such issues. *I really dont know whether i should be exploring other ways to get the persistance thing working! Best Regards, Naveen On Jan 29, 2008 9:59 AM, NAVEEN UPADHYAY <nav...@gm...> wrote: > Yes Kevin,you are right but the very reason why i used timestamp was to > make the records can be retrieved in the order in they were persisted..Now > if i put back the recid into as the key .. i'll be no better than > before(without timestamp) as the order might not be guaranteed > -Naveen! > > > On Jan 28, 2008 10:00 PM, Kevin Day <ke...@tr...> wrote: > > > FYI - my comment on having the same timestamp does not preclude your > > use of a timestamp - you just have to build your index key so it is always > > unique. > > > > Creating a key that consists of the time stamp, followed by the recid > > (or some other unique identifying information of you data - such as the key > > in your original btree) will achieve this. > > > > - K > > > > ----------------------- *Original Message* ----------------------- > > > > *From:* "NAVEEN UPADHYAY" <nav...@gm...><nav...@gm...> > > *To:* jdb...@li... > > *Cc:* > > *Date:* Mon, 28 Jan 2008 19:29:52 +0530 > > *Subject: Re: [Jdbm-developer] Need help on .db file optimisation | JDBM > > * > > > > > > Hi all, > > In my code i am using the jdbm-0.12.jar and i am cleaning up > > records from btree using a pseudocode like > > if (_btree.size() >= threshold){//NaveenBEG > > //Going for bulk purging!Please wait..... > > int count =0; > > Tuple tuple; > > TupleBrowser browser; > > browser = _btree.browse(); > > &n bsp; tuple = new Tuple(); > > while(count <threshold/2 &&browser.getNext(tuple)){ > > _btree.remove(tuple.getKey()); > > count++ ; > > } > > > > and there is indeed a recman commit to follow...Still i am not seeing > > any decrease in the .db file size. > > Is there still a problem with my code or something wrong with the * > > jdbm-0.12.jar. > > > > Naveen > > * > > Alex i didnt use the timestamp method as u suggested ,since as rightly p > > ointed out by Kevin lot many transaction were hitting the same timestamp :( > > > > On Jan 25, 2008 2:44 AM, NAVEEN UPADHYAY <nav...@gm...> > > wrote: > > > > > Kevin, > > > Regarding your point, if I do BTree.remove, then I don't need not > > > bother separately for the record manager.. > > > > > > Naveen > > > > > > On Jan 24, 2008 11:52 PM, NAVEEN UPADHYAY < > > > nav...@gm...> wrote: > > > > > > > Just to confirm to Alex's comment.. > > > > We need *another *Btree(*one more)*..??this time its value being the > > > > key to the first one... > > > > > > > > > > > > > > > > On Jan 24, 2008 3:37 AM, Kevin Day < ke...@tr...> wrote: > > > > > > > > > Just make sure you delete the records from the BTree when you > > > > > delete them from the record manager :-) > > > > > > > > > > For these situations, we actually store the key in the btree as: > > > > > > > > > > [timestamp][recid] > > > > > > > > > > and the value as a null value. that way we don't have to worry > > > > > about key collision if two transactions hit at the exact same time stamp. > > > > > > > > > > I have a class around here somehwere that allows you to easily > > > > > implement a multi-keyed index like the above using longs or strings. If > > > > > it's of interest, let me know and I'll send it to you directly. > > > > > > > > > > - K > > > > > > > > > > ----------------------- *Original Message*----------------------- > > > > > > > > > > *From:* "Alex Boisvert" <boi...@in...><boi...@in...> > > > > > *To:* "NAVEEN UPADHYAY" <nav...@gm...><nav...@gm...> > > > > > *Cc:* jdb...@li...,<jdb...@li...>Kevin Day > > > > > <ke...@tr...> <ke...@tr...> > > > > > *Date:* Wed, 23 Jan 2008 05:32:38 -0800 > > > > > *Subject: Re: [Jdbm-developer] Need help on .db file optimisation > > > > > | JDBM* > > > > > > > > > > Naveen, > > > > > > > > > > You can delete/remove whatever you are putting into the database. > > > > > It's up to your application code to do that. > > > > > > > > > > For example, if you are recording events, simply tag the events > > > > > with a timestamp, create an index (btree) on the timestamp and delete old > > > > > events when you reach a given threshold. You can collect statistics on a > > > > > sample set of records to determine your average space requirement and then > > > > > extrapolate to obtain your upper limit based on your file system limitations > > > > > (2G), minus some conservative margin. > > > > > > > > > > regards, > > > > > alex > > > > > > > > > > > > > > > > > > > > On 1/22/08, NAVEEN UPADHYAY <nav...@gm...> wrote: > > > > > > > > > > > > Thanks Alex,Kevin for your responses ;^) > > > > > > The proposed solutions seem to be quite good. > > > > > > Only thing i have is my main concern is to purge old data with > > > > > > these .db files and* NOT *their corruption beyond 2gb. > > > > > > My only aim is to keep the size of the file always within a > > > > > > limit. > > > > > > For my driver only the recent data is of use.. > > > > > > So all that i am looking for is a way to purge old data > > > > > > > > > > > > Will throwing IOExceptions work? > > > > > > Please suggest some solutions... > > > > > > Thanks in advance, > > > > > > Naveen > > > > > > > > > > > > On Jan 23, 2008 5:02 AM, Alex Boisvert < boi...@in...> > > > > > > wrote: > > > > > > > > > > > > > On 1/22/08, Kevin Day <ke...@tr...> wrote: > > > > > > > > > > > > > > > Another option would be to have an OOSProvider that could > > > > > > > > be configured with an instrumentable version for testing (this would be more > > > > > > > > akin to dependency injection)... It feels weird to add code to the runtime > > > > > > > > implementation of a class that forces it to fail... > > > > > > > > > > > > > > > > Any preference? (although I love Brian's idea of using a > > > > > > > > thumb drive - or maybe even a RAM disk - it's just hard to enforce that > > > > > > > > stuff in a test suite :-) ) > > > > > > > > > > > > > > > > > > > > > > I understand the weird feeling and I don't have a strong > > > > > > > preference. I think the OOSProvider solution is going to involve more code > > > > > > > to achieve the same goal. If the abstraction isn't needed in the first > > > > > > > place, I don't think it's worth the added complexity, meaning adding 4-5 > > > > > > > more classes for this test alone. > > > > > > > > > > > > > > alex > > > > > > > > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------------- > > > > > > > This SF.net email is sponsored by: Microsoft > > > > > > > Defy all challenges. Microsoft(R) Visual Studio 2008. > > > > > > > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > > > > > > > _______________________________________________ > > > > > > > Jdbm-developer mailing list > > > > > > > Jdb...@li... > > > > > > > <Jdb...@li...urcefor%0A+ge.net> > > > > > > > https://lists.sourceforge.net/lists/listinfo/jdbm-developer > > > > > > > <https://lists.sourceforge.net/lists/%0A+listinfo/jdbm-developer> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------------- > > > > > This SF.net email is sponsored by: Microsoft > > > > > Defy all challenges. Microsoft(R) Visual Studio 2008. > > > > > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > > > > > > > > > > _______________________________________________ > > > > > Jdbm-developer mailing list > > > > > Jdb...@li... > > > > > https://lists.sourceforge.net/lists/listinfo/jdbm-developer > > > > > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------------- > > > > > This SF.net email is sponsored by: Microsoft > > > > > Defy all challenges. Microsoft(R) Visual Studio 2008. > > > > > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > > > > > _______________________________________________ > > > > > Jdbm-developer mailing list > > > > > Jdb...@li... > > > > > https://lists.sourceforge.net/lists/listinfo/jdbm-developer > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------------- > > This SF.net email is sponsored by: Microsoft > > Defy all challenges. Microsoft(R) Visual Studio 2008. > > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > > > > _______________________________________________ > > Jdbm-developer mailing list > > Jdb...@li... > > https://lists.sourceforge.net/lists/listinfo/jdbm-developer > > > > > > > |