Kevin,


I’m sorry for the continuing barge of email. Can you elaborate at all on the option for a reduced durability mode? It sounds like it may be a good fit for my current task but I don’t see anything in RecordManagerOptions.

 

Eric

 


From: jdbm-general-admin@lists.sourceforge.net [mailto:jdbm-general-admin@lists.sourceforge.net] On Behalf Of Kevin Day
Sent: Wednesday, January 18, 2006 9:53 AM
To: JDBM General listserv
Subject: re[6]: [Jdbm-general] Transaction memory size

 

Eric-

 

At minimum, I would recommend grabbing the 1.0 release.  In terms of changes that have been made, your best bet is to look at the commit logs - but the differences between the .13 and 1.0 release were very minor (mostly updates of comments and that sort of thing - a few bug fixes).

 

The auto-commit interval option is only available in the current development build (HEAD), which is undergoing some significant development right now...  As of today, it appears to be relatively stable (I'm not aware of any glaring problems with the code base, all unit tests pass, etc...), but it is a development build...

 

If you are comfortable running a development build, then take a crack at HEAD - otherwise, you'll have to periodically call commit()...

 

Other major features in the current development build:

 

A revamped serialization system that optionally allows association of serializers with records

BTree key compression

Option for running in a reduced durability mode (this makes the system faster, at the cost of potentially losing comitted transactions during a crash.  The database will not get corrupted during a crash, though).

 

There may be others, but those are the three that come immediately to mind...

 

If you do work with HEAD and run into anything, be sure to post!

 

- K

 

 

 

>
Im not sure exactly but it looks like the version Im using is from 2003.  
 
I didnt realize people JDBM was being actively developed again. Do you recommend the CVS version? It looks like if I just move to 1.0 I wont get the DISABLE_TRANSACTIONS_AUTOCOMMITINTERVAL option.
 
Is there a high level overview of what has changed over the last few years anywhere? It doesnt look like the CHANGES or README file has been updated in a while.  

Eric
 


From: jdbm-general-admin@lists.sourceforge.net [mailto:jdbm-general-admin@lists.sourceforge.net] On Behalf Of Kevin Day
Sent: Tuesday, January 17, 2006 4:46 PM
To: JDBM General listserv
Subject: re[4]: [Jdbm-general] Transaction memory size
 

Eric-

 

What code are you running?  The code was updated on 8/22/05 to provide  DISABLE_TRANSACTIONS_AUTOCOMMITINTERVAL option - this option is enabled by default, so you should be able to use it out of the box to get the behavior you are askingabout.

 

The very latest version of code in CVS has the default for this value set at 10,000 pages, which is ~80 MB - you've got to write a *lot* of data before the auto-commit hits.  You can adjust the value downwards by setting the jdbm.disableTransactions.autoCommitInterval property to a lower number.

 

- K

 

  

 
>
Kevin,
 
Ive been experimenting running with the transactions shut off. Even with transactions shut off it seems like the db file never gets written to unless I call commit on the RecordManager. Is this the expected behavior?
 
Your description gave me the impression that the cached updates would be periodically written to the database file by jdbm and that there was no reason to call commit on the RecordManager directly.
 
 
Eric
 


From: jdbm-general-admin@lists.sourceforge.net [mailto:jdbm-general-admin@lists.sourceforge.net] On Behalf Of Kevin Day
Sent: Monday, January 16, 2006 4:23 PM
To: JDBM General listserv
Subject: re[2]: [Jdbm-general] Transaction memory size
 

Eric-

 

No - when transactions are disabled, they are cached and commited to disk periodically (behind the scenes, we manage the size of the uncomitted page load and write to disk as necessary).  When Tx are off, writes go to only one file (thedb file), instead of writing to the log, then to the db file.

 

Because writes to the log are actually synced to the phyiscal disk they are usually considerably slower than non-transactional writes.

 

Running without transactions should be several times faster than running with them (at the risk of corrupting the database file in the event of a crash, of course).

 

- K

 

  

 
>
Kevin,
 
Thanks for such a quick response. The main reason I was trying to avoid committing was because committing seems to be relatively slow. If I disable transactions will things be written to disk immediately? Will this writing be faster thena commit?

Eric
 


From: Kevin Day [mailto:kevin@trumpetinc.com]
Sent: Monday, January 16, 2006 4:02 PM
To: Rosenberg, Eric
Subject: re: [Jdbm-general] Transaction memory size
 

Eric-

 

This is a limitation in the current jdbm transaction manager (there is a group of us working on an alternate technique that will support arbitrarily large transactions, but that's a long way off).

 

The recommended approach at this point is to either:

 

A.  Commit your transactions periodically

 

or

 

B.  Run with transcations disabled for large inserts (understand that this will open you up to potential file corruption, so you only want to run without transactions when it is OK to lose the entire database - i.e. during initial data population).

 

 

Cheers,

 

- Kevin

 

  

 
>If I add / update a bunch of objects in JDBM during a transaction is
everything kept in memory until I do a commit?

I am trying to add an arbitrary number of objects to JDBM and am
wondering if it is safe to add them all from a single transaction or if
I need to commit periodically to ensure that I don't run out of memory.

Thanks,
Eric


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_idv37&alloc_id865&op└ick
_______________________________________________
Jdbm-general mailing list
Jdbm-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jdbm-general

<
<
<
<

 

------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 _______________________________________________ Jdbm-general mailing list Jdbm-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/jdbm-general