From: Alex B. <boi...@in...> - 2008-01-31 16:13:46
|
Naveen, Can you reduce your application down to a test-case that would exemplify this behavior? alex On 1/31/08, NAVEEN UPADHYAY <nav...@gm...> wrote: > > Hi Alex and Kevin, > > All said and done. > I did put purging capabilities in my code such that if file size is > greater than >threshold ...purge a few records from BTree and commit! > > Now let me put some facts before i ask my final question. > > I have got basically two database tables tab$1 and tab$2. > there are *hardly* 50 rows in them with all text data. > > Now i constantly poll these two tables and write into two BTrees (by the > same name) > > Everytime i write i assume there could be changes to the values in various > rows .therefore i say* btree.insert(key,value,true) . > each time*. > Now first of all if the data is the same the .db file should not change > its size(as i will be just overwrting the same info)...Right? If so i should > NOT need purging capabilities for such small data..! > > Wrong ..as i see it just growing and growing..from a 2 MB file it has > grown on to 45 MB in the last one week.. > > All of that just to store records just storing people names and there ids > ...(My entire databse MySql size is smaller now by good 15 MB than the .db > file) > > > The cause just evades me... :-( > > I am using *jdbm0.12.jar .Dont know if that jar had such issues. > > *I really dont know whether i should be exploring other ways to get the > persistance thing working! > > Best Regards, > Naveen > > > On Jan 29, 2008 9:59 AM, NAVEEN UPADHYAY <nav...@gm...> > wrote: > > > Yes Kevin,you are right but the very reason why i used timestamp was to > > make the records can be retrieved in the order in they were persisted..Now > > if i put back the recid into as the key .. i'll be no better than > > before(without timestamp) as the order might not be guaranteed > > -Naveen! > > > > > > On Jan 28, 2008 10:00 PM, Kevin Day <ke...@tr...> wrote: > > > > > FYI - my comment on having the same timestamp does not preclude your > > > use of a timestamp - you just have to build your index key so it is always > > > unique. > > > > > > Creating a key that consists of the time stamp, followed by the recid > > > (or some other unique identifying information of you data - such as the key > > > in your original btree) will achieve this. > > > > > > - K > > > > > > ----------------------- *Original Message* ----------------------- > > > > > > *From:* "NAVEEN UPADHYAY" <nav...@gm...><nav...@gm...> > > > *To:* jdb...@li... > > > *Cc:* > > > *Date:* Mon, 28 Jan 2008 19:29:52 +0530 > > > *Subject: Re: [Jdbm-developer] Need help on .db file optimisation | > > > JDBM* > > > > > > > > > Hi all, > > > In my code i am using the jdbm-0.12.jar and i am cleaning up > > > records from btree using a pseudocode like > > > if (_btree.size() >= threshold){//NaveenBEG > > > //Going for bulk purging!Please wait..... > > > int count =0; > > > Tuple tuple; > > > TupleBrowser browser; > > > browser = _btree.browse(); > > > &n bsp; tuple = new Tuple(); > > > while(count <threshold/2 > > > &&browser.getNext(tuple)){ > > > _btree.remove(tuple.getKey()); > > > count++ ; > > > } > > > > > > and there is indeed a recman commit to follow...Still i am not seeing > > > any decrease in the .db file size. > > > Is there still a problem with my code or something wrong with the * > > > jdbm-0.12.jar. > > > > > > Naveen > > > * > > > Alex i didnt use the timestamp method as u suggested ,since as rightly > > > p ointed out by Kevin lot many transaction were hitting the same timestamp > > > :( > > > > > > On Jan 25, 2008 2:44 AM, NAVEEN UPADHYAY <nav...@gm...> > > > wrote: > > > > > > > Kevin, > > > > Regarding your point, if I do BTree.remove, then I don't need not > > > > bother separately for the record manager.. > > > > > > > > Naveen > > > > > > > > On Jan 24, 2008 11:52 PM, NAVEEN UPADHYAY < > > > > nav...@gm...> wrote: > > > > > > > > > Just to confirm to Alex's comment.. > > > > > We need *another *Btree(*one more)*..??this time its value being > > > > > the key to the first one... > > > > > > > > > > > > > > > > > > > > On Jan 24, 2008 3:37 AM, Kevin Day < ke...@tr...> wrote: > > > > > > > > > > > Just make sure you delete the records from the BTree when you > > > > > > delete them from the record manager :-) > > > > > > > > > > > > For these situations, we actually store the key in the btree as: > > > > > > > > > > > > [timestamp][recid] > > > > > > > > > > > > and the value as a null value. that way we don't have to worry > > > > > > about key collision if two transactions hit at the exact same time stamp. > > > > > > > > > > > > I have a class around here somehwere that allows you to easily > > > > > > implement a multi-keyed index like the above using longs or strings. If > > > > > > it's of interest, let me know and I'll send it to you directly. > > > > > > > > > > > > - K > > > > > > > > > > > > ----------------------- *Original Message*----------------------- > > > > > > > > > > > > *From:* "Alex Boisvert" <boi...@in...><boi...@in...> > > > > > > *To:* "NAVEEN UPADHYAY" <nav...@gm...><nav...@gm...> > > > > > > *Cc:* jdb...@li...,<jdb...@li...>Kevin Day > > > > > > <ke...@tr...> <ke...@tr...> > > > > > > *Date:* Wed, 23 Jan 2008 05:32:38 -0800 > > > > > > *Subject: Re: [Jdbm-developer] Need help on .db file > > > > > > optimisation | JDBM* > > > > > > > > > > > > Naveen, > > > > > > > > > > > > You can delete/remove whatever you are putting into the > > > > > > database. It's up to your application code to do that. > > > > > > > > > > > > For example, if you are recording events, simply tag the events > > > > > > with a timestamp, create an index (btree) on the timestamp and delete old > > > > > > events when you reach a given threshold. You can collect statistics on a > > > > > > sample set of records to determine your average space requirement and then > > > > > > extrapolate to obtain your upper limit based on your file system limitations > > > > > > (2G), minus some conservative margin. > > > > > > > > > > > > regards, > > > > > > alex > > > > > > > > > > > > > > > > > > > > > > > > On 1/22/08, NAVEEN UPADHYAY <nav...@gm...> wrote: > > > > > > > > > > > > > > > > > > > > Thanks Alex,Kevin for your responses ;^) > > > > > > > The proposed solutions seem to be quite good. > > > > > > > Only thing i have is my main concern is to purge old data with > > > > > > > these .db files and* NOT *their corruption beyond 2gb. > > > > > > > My only aim is to keep the size of the file always within a > > > > > > > limit. > > > > > > > For my driver only the recent data is of use.. > > > > > > > So all that i am looking for is a way to purge old data > > > > > > > > > > > > > > Will throwing IOExceptions work? > > > > > > > Please suggest some solutions... > > > > > > > Thanks in advance, > > > > > > > Naveen > > > > > > > > > > > > > > On Jan 23, 2008 5:02 AM, Alex Boisvert < boi...@in...> > > > > > > > wrote: > > > > > > > > > > > > > > > On 1/22/08, Kevin Day <ke...@tr...> wrote: > > > > > > > > > > > > > > > > > Another option would be to have an OOSProvider that could > > > > > > > > > be configured with an instrumentable version for testing (this would be more > > > > > > > > > akin to dependency injection)... It feels weird to add code to the runtime > > > > > > > > > implementation of a class that forces it to fail... > > > > > > > > > > > > > > > > > > Any preference? (although I love Brian's idea of using a > > > > > > > > > thumb drive - or maybe even a RAM disk - it's just hard to enforce that > > > > > > > > > stuff in a test suite :-) ) > > > > > > > > > > > > > > > > > > > > > > > > > I understand the weird feeling and I don't have a strong > > > > > > > > preference. I think the OOSProvider solution is going to involve more code > > > > > > > > to achieve the same goal. If the abstraction isn't needed in the first > > > > > > > > place, I don't think it's worth the added complexity, meaning adding 4-5 > > > > > > > > more classes for this test alone. > > > > > > > > > > > > > > > > alex > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------------- > > > > > > > > This SF.net email is sponsored by: Microsoft > > > > > > > > Defy all challenges. Microsoft(R) Visual Studio 2008. > > > > > > > > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > > > > > > > > _______________________________________________ > > > > > > > > Jdbm-developer mailing list > > > > > > > > Jdb...@li... > > > > > > > > <Jdb...@li...urcefor%0A+ge.net> > > > > > > > > https://lists.sourceforge.net/lists/listinfo/jdbm-developer > > > > > > > > <https://lists.sourceforge.net/lists/%0A+listinfo/jdbm-developer> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------------- > > > > > > This SF.net email is sponsored by: Microsoft > > > > > > Defy all challenges. Microsoft(R) Visual Studio 2008. > > > > > > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > > > > > > > > > > > > _______________________________________________ > > > > > > Jdbm-developer mailing list > > > > > > Jdb...@li... > > > > > > https://lists.sourceforge.net/lists/listinfo/jdbm-developer > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------------- > > > > > > This SF.net email is sponsored by: Microsoft > > > > > > Defy all challenges. Microsoft(R) Visual Studio 2008. > > > > > > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > > > > > > _______________________________________________ > > > > > > Jdbm-developer mailing list > > > > > > Jdb...@li... > > > > > > https://lists.sourceforge.net/lists/listinfo/jdbm-developer > > > > > > > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------------- > > > This SF.net email is sponsored by: Microsoft > > > Defy all challenges. Microsoft(R) Visual Studio 2008. > > > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > > > > > > _______________________________________________ > > > Jdbm-developer mailing list > > > Jdb...@li... > > > https://lists.sourceforge.net/lists/listinfo/jdbm-developer > > > > > > > > > > > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > Jdbm-developer mailing list > Jdb...@li... > https://lists.sourceforge.net/lists/listinfo/jdbm-developer > > |