From: Rosenberg, E. <eri...@ng...> - 2006-01-17 20:30:57
|
Kevin, =20 I've been experimenting running with the transactions shut off. Even = with transactions shut off it seems like the db file never gets written = to unless I call commit on the RecordManager. Is this the expected = behavior? =20 Your description gave me the impression that the cached updates would be = periodically written to the database file by jdbm and that there was no = reason to call commit on the RecordManager directly. =20 =20 Eric =20 ________________________________ From: jdb...@li... = [mailto:jdb...@li...] On Behalf Of Kevin Day Sent: Monday, January 16, 2006 4:23 PM To: JDBM General listserv Subject: re[2]: [Jdbm-general] Transaction memory size =20 Eric- =20 No - when transactions are disabled, they are cached and commited to = disk periodically (behind the scenes, we manage the size of the = uncomitted page load and write to disk as necessary). When Tx are off, = writes go to only one file (the db file), instead of writing to the log, = then to the db file. =20 Because writes to the log are actually synced to the phyiscal disk they = are usually considerably slower than non-transactional writes. =20 Running without transactions should be several times faster than running = with them (at the risk of corrupting the database file in the event of a = crash, of course). =20 - K =20 =20 =20 >=20 Kevin,=20 =20 Thanks for such a quick response. The main reason I was trying to avoid = committing was because committing seems to be relatively slow. If I = disable transactions will things be written to disk immediately? Will = this writing be faster thena commit?=20 Eric=20 =20 From: Kevin Day [mailto:ke...@tr...] = <mailto:ke...@tr...> =20 Sent: Monday, January 16, 2006 4:02 PM To: Rosenberg, Eric Subject: re: [Jdbm-general] Transaction memory size=20 =20 Eric-=20 =20 This is a limitation in the current jdbm transaction manager (there is a = group of us working on an alternate technique that will support = arbitrarily large transactions, but that's a long way off).=20 =20 The recommended approach at this point is to either:=20 =20 A. Commit your transactions periodically=20 =20 or=20 =20 B. Run with transcations disabled for large inserts (understand that = this will open you up to potential file corruption, so you only want to = run without transactions when it is OK to lose the entire database - = i.e. during initial data population).=20 =20 =20 Cheers,=20 =20 - Kevin=20 =20 =20 =20 > If I add / update a bunch of objects in JDBM during a transaction is everything kept in memory until I do a commit? I am trying to add an arbitrary number of objects to JDBM and am wondering if it is safe to add them all from a single transaction or if I need to commit periodically to ensure that I don't run out of memory. Thanks, Eric ------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Do you grep through log = files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! http://ads.osdn.com/?ad_idv37&alloc_id865&op=C0ick _______________________________________________ Jdbm-general mailing list Jdb...@li... https://lists.sourceforge.net/lists/listinfo/jdbm-general < < =20 ------------------------------------------------------- This SF.net = email is sponsored by: Splunk Inc. Do you grep through log files for = problems? Stop! Download the new AJAX search engine that makes searching = your log files as easy as surfing the web. DOWNLOAD SPLUNK! = http://ads.osdn.com/?ad_id=3D7637&alloc_id=3D16865&op=3Dclick = _______________________________________________ Jdbm-general mailing = list Jdb...@li... = https://lists.sourceforge.net/lists/listinfo/jdbm-general |
From: Rosenberg, E. <eri...@ng...> - 2006-01-18 14:29:39
|
I'm not sure exactly but it looks like the version I'm using is from = 2003.=20 =20 I didn't realize people JDBM was being actively developed again. Do you = recommend the CVS version? It looks like if I just move to 1.0 I won't = get the DISABLE_TRANSACTIONS_AUTOCOMMITINTERVAL option. =20 Is there a high level overview of what has changed over the last few = years anywhere? It doesn't look like the CHANGES or README file has been = updated in a while.=20 Eric =20 ________________________________ From: jdb...@li... = [mailto:jdb...@li...] On Behalf Of Kevin Day Sent: Tuesday, January 17, 2006 4:46 PM To: JDBM General listserv Subject: re[4]: [Jdbm-general] Transaction memory size =20 Eric- =20 What code are you running? The code was updated on 8/22/05 to provide = DISABLE_TRANSACTIONS_AUTOCOMMITINTERVAL option - this option is enabled = by default, so you should be able to use it out of the box to get the = behavior you are asking about. =20 The very latest version of code in CVS has the default for this value = set at 10,000 pages, which is ~80 MB - you've got to write a *lot* of = data before the auto-commit hits. You can adjust the value downwards by = setting the jdbm.disableTransactions.autoCommitInterval property to a = lower number. =20 - K =20 =20 =20 >=20 Kevin,=20 =20 Ive been experimenting running with the transactions shut off. Even with = transactions shut off it seems like the db file never gets written to = unless I call commit on the RecordManager. Is this the expected = behavior?=20 =20 Your description gave me the impression that the cached updates would be = periodically written to the database file by jdbm and that there was no = reason to call commit on the RecordManager directly.=20 =20 =20 Eric=20 =20 From: jdb...@li... = [mailto:jdb...@li...] = <mailto:jdb...@li...> On Behalf Of Kevin = Day Sent: Monday, January 16, 2006 4:23 PM To: JDBM General listserv Subject: re[2]: [Jdbm-general] Transaction memory size=20 =20 Eric-=20 =20 No - when transactions are disabled, they are cached and commited to = disk periodically (behind the scenes, we manage the size of the = uncomitted page load and write to disk as necessary). When Tx are off, = writes go to only one file (the db file), instead of writing to the log, = then to the db file.=20 =20 Because writes to the log are actually synced to the phyiscal disk they = are usually considerably slower than non-transactional writes.=20 =20 Running without transactions should be several times faster than running = with them (at the risk of corrupting the database file in the event of a = crash, of course).=20 =20 - K=20 =20 =20 =20 >=20 Kevin,=20 =20 Thanks for such a quick response. The main reason I was trying to avoid = committing was because committing seems to be relatively slow. If I = disable transactions will things be written to disk immediately? Will = this writing be faster thena commit?=20 Eric=20 =20 From: Kevin Day [mailto:ke...@tr...] = <mailto:ke...@tr...> =20 Sent: Monday, January 16, 2006 4:02 PM To: Rosenberg, Eric Subject: re: [Jdbm-general] Transaction memory size=20 =20 Eric-=20 =20 This is a limitation in the current jdbm transaction manager (there is a = group of us working on an alternate technique that will support = arbitrarily large transactions, but that's a long way off).=20 =20 The recommended approach at this point is to either:=20 =20 A. Commit your transactions periodically=20 =20 or=20 =20 B. Run with transcations disabled for large inserts (understand that = this will open you up to potential file corruption, so you only want to = run without transactions when it is OK to lose the entire database - = i.e. during initial data population).=20 =20 =20 Cheers,=20 =20 - Kevin=20 =20 =20 =20 >If I add / update a bunch of objects in JDBM during a transaction is everything kept in memory until I do a commit? I am trying to add an arbitrary number of objects to JDBM and am wondering if it is safe to add them all from a single transaction or if I need to commit periodically to ensure that I don't run out of memory. Thanks, Eric ------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Do you grep through log = files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! http://ads.osdn.com/?ad_idv37&alloc_id865&op=C0ick _______________________________________________ Jdbm-general mailing list Jdb...@li... https://lists.sourceforge.net/lists/listinfo/jdbm-general < < < =20 ------------------------------------------------------- This SF.net = email is sponsored by: Splunk Inc. Do you grep through log files for = problems? Stop! Download the new AJAX search engine that makes searching = your log files as easy as surfing the web. DOWNLOAD SPLUNK! = http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=3D103432&bid=3D230486&dat=3D= 121642 _______________________________________________ Jdbm-general = mailing list Jdb...@li... = https://lists.sourceforge.net/lists/listinfo/jdbm-general |
From: Kevin D. <ke...@tr...> - 2006-01-18 14:50:03
|
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"> <HTML><HEAD> <STYLE type=text/css> P, UL, OL, DL, DIR, MENU, PRE { margin: 0 auto;}</STYLE> <META content="MSHTML 6.00.2900.2802" name=GENERATOR></HEAD> <BODY leftMargin=1 topMargin=1 rightMargin=1><FONT face=Tahoma> <DIV><FONT face=Arial size=2>Eric-</FONT></DIV> <DIV><FONT face=Arial size=2></FONT> </DIV> <DIV><FONT face=Arial size=2>At minimum, I would recommend grabbing the 1.0 release. In terms of changes that have been made, your best bet is to look at the commit logs - but the differences between the .13 and 1.0 release were very minor (mostly updates of comments and that sort of thing - a few bug fixes).</FONT></DIV> <DIV><FONT face=Arial size=2></FONT> </DIV> <DIV><FONT face=Arial size=2>The auto-commit interval option is only available in the current development build (HEAD), which is undergoing some significant development right now... As of today, it appears to be relatively stable (I'm not aware of any glaring problems with the code base, all unit tests pass, etc...), but it is a development build...</FONT></DIV> <DIV><FONT face=Arial size=2></FONT> </DIV> <DIV><FONT face=Arial size=2>If you are comfortable running a development build, then take a crack at HEAD - otherwise, you'll have to periodically call commit()...</FONT></DIV> <DIV><FONT face=Arial size=2></FONT> </DIV> <DIV><FONT face=Arial size=2>Other major features in the current development build:</FONT></DIV> <DIV><FONT face=Arial size=2></FONT> </DIV> <DIV><FONT face=Arial size=2>A revamped serialization system that optionally allows association of serializers with records</FONT></DIV> <DIV><FONT face=Arial size=2>BTree key compression</FONT></DIV> <DIV><FONT face=Arial size=2>Option for running in a reduced durability mode (this makes the system faster, at the cost of potentially losing comitted transactions during a crash. The database will not get corrupted during a crash, though).</FONT></DIV> <DIV><FONT face=Arial size=2></FONT> </DIV> <DIV><FONT face=Arial size=2>There may be others, but those are the three that come immediately to mind...</FONT></DIV> <DIV><FONT face=Arial size=2></FONT> </DIV> <DIV><FONT face=Arial size=2>If you do work with HEAD and run into anything, be sure to post!</FONT></DIV> <DIV><FONT face=Arial size=2></FONT> </DIV> <DIV><FONT face=Arial size=2>- K</FONT></DIV> <DIV><FONT face=Arial size=2></FONT></DIV> <DIV><FONT face=Arial size=2></FONT> </DIV> <DIV><FONT face=Arial size=2> </FONT> <TABLE> <TBODY> <TR> <TD width=1 bgColor=blue><FONT face=Arial size=2></FONT></TD> <TD><FONT face=Arial size=2><FONT color=green>> <BR>Im not sure exactly but it looks like the version Im using is from 2003. <BR> <BR>I didnt realize people JDBM was being actively developed again. Do you recommend the CVS version? It looks like if I just move to 1.0 I wont get the DISABLE_TRANSACTIONS_AUTOCOMMITINTERVAL option. <BR> <BR>Is there a high level overview of what has changed over the last few years anywhere? It doesnt look like the CHANGES or README file has been updated in a while. <BR><BR>Eric <BR> <BR><BR><BR>From: <A href="mailto:jdb...@li..."><FONT color=#0000ff>jdb...@li...</FONT></A> <A href="mailto:jdb...@li..."><FONT color=#0000ff>[mailto:jdb...@li...]</FONT></A> On Behalf Of Kevin Day<BR>Sent: Tuesday, January 17, 2006 4:46 PM<BR>To: JDBM General listserv<BR>Subject: re[4]: [Jdbm-general] Transaction memory size <BR> <BR><BR>Eric- <BR><BR> <BR><BR>What code are you running? The code was updated on 8/22/05 to provide DISABLE_TRANSACTIONS_AUTOCOMMITINTERVAL option - this option is enabled by default, so you should be able to use it out of the box to get the behavior you are askingabout. <BR><BR> <BR><BR>The very latest version of code in CVS has the default for this value set at 10,000 pages, which is ~80 MB - you've got to write a *lot* of data before the auto-commit hits. You can adjust the value downwards by setting the jdbm.disableTransactions.autoCommitInterval property to a lower number. <BR><BR> <BR><BR>- K <BR><BR> <BR><BR> <BR><BR> <BR>> <BR>Kevin, <BR> <BR>Ive been experimenting running with the transactions shut off. Even with transactions shut off it seems like the db file never gets written to unless I call commit on the RecordManager. Is this the expected behavior? <BR> <BR>Your description gave me the impression that the cached updates would be periodically written to the database file by jdbm and that there was no reason to call commit on the RecordManager directly. <BR> <BR> <BR>Eric <BR> <BR><BR><BR>From: <A href="mailto:jdb...@li..."><FONT color=#0000ff>jdb...@li...</FONT></A> <A href="mailto:jdb...@li..."><FONT color=#0000ff>[mailto:jdb...@li...]</FONT></A> On Behalf Of Kevin Day<BR>Sent: Monday, January 16, 2006 4:23 PM<BR>To: JDBM General listserv<BR>Subject: re[2]: [Jdbm-general] Transaction memory size <BR> <BR><BR>Eric- <BR><BR> <BR><BR>No - when transactions are disabled, they are cached and commited to disk periodically (behind the scenes, we manage the size of the uncomitted page load and write to disk as necessary). When Tx are off, writes go to only one file (thedb file), instead of writing to the log, then to the db file. <BR><BR> <BR><BR>Because writes to the log are actually synced to the phyiscal disk they are usually considerably slower than non-transactional writes. <BR><BR> <BR><BR>Running without transactions should be several times faster than running with them (at the risk of corrupting the database file in the event of a crash, of course). <BR><BR> <BR><BR>- K <BR><BR> <BR><BR> <BR><BR> <BR>><BR>Kevin, <BR> <BR>Thanks for such a quick response. The main reason I was trying to avoid committing was because committing seems to be relatively slow. If I disable transactions will things be written to disk immediately? Will this writing be faster thena commit? <BR><BR>Eric <BR> <BR><BR><BR>From: Kevin Day <A href="mailto:ke...@tr..."><FONT color=#0000ff>[mailto:ke...@tr...]</FONT></A> <BR>Sent: Monday, January 16, 2006 4:02 PM<BR>To: Rosenberg, Eric<BR>Subject: re: [Jdbm-general] Transaction memory size <BR> <BR><BR>Eric- <BR><BR> <BR><BR>This is a limitation in the current jdbm transaction manager (there is a group of us working on an alternate technique that will support arbitrarily large transactions, but that's a long way off). <BR><BR> <BR><BR>The recommended approach at this point is to either: <BR><BR> <BR><BR>A. Commit your transactions periodically <BR><BR> <BR><BR>or <BR><BR> <BR><BR>B. Run with transcations disabled for large inserts (understand that this will open you up to potential file corruption, so you only want to run without transactions when it is OK to lose the entire database - i.e. during initial data population). <BR><BR> <BR><BR> <BR><BR>Cheers, <BR><BR> <BR><BR>- Kevin <BR><BR> <BR><BR> <BR><BR> <BR>>If I add / update a bunch of objects in JDBM during a transaction is<BR>everything kept in memory until I do a commit?<BR><BR>I am trying to add an arbitrary number of objects to JDBM and am<BR>wondering if it is safe to add them all from a single transaction or if<BR>I need to commit periodically to ensure that I don't run out of memory.<BR><BR>Thanks,<BR>Eric<BR><BR><BR>-------------------------------------------------------<BR>This SF.net email is sponsored by: Splunk Inc. Do you grep through log files<BR>for problems? Stop! Download the new AJAX search engine that makes<BR>searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!<BR><A href="http://ads.osdn.com/?ad_idv37&alloc_id865&opÀick"><FONT color=#0000ff>http://ads.osdn.com/?ad_idv37&alloc_id865&opÀick</FONT></A><BR>_______________________________________________<BR>Jdbm-general mailing list<BR><A href="mailto:Jdb...@li..."><FONT color=#0000ff>Jdb...@li...</FONT></A><BR><A href="https://lists.sourceforge.net/lists/listinfo/jdbm-general"><FONT color=#0000ff>https://lists.sourceforge.net/lists/listinfo/jdbm-general</FONT></A><BR><BR><<BR><<BR><<BR><<BR></FONT></FONT></TD></TR></TBODY></TABLE></DIV></FONT></BODY></HTML> |
From: Rosenberg, E. <eri...@ng...> - 2006-01-18 15:51:22
|
Kevin, I'm sorry for the continuing barge of email. Can you elaborate at all on = the option for a reduced durability mode? It sounds like it may be a = good fit for my current task but I don't see anything in = RecordManagerOptions. =20 Eric =20 ________________________________ From: jdb...@li... = [mailto:jdb...@li...] On Behalf Of Kevin Day Sent: Wednesday, January 18, 2006 9:53 AM To: JDBM General listserv Subject: re[6]: [Jdbm-general] Transaction memory size =20 Eric- =20 At minimum, I would recommend grabbing the 1.0 release. In terms of = changes that have been made, your best bet is to look at the commit logs = - but the differences between the .13 and 1.0 release were very minor = (mostly updates of comments and that sort of thing - a few bug fixes). =20 The auto-commit interval option is only available in the current = development build (HEAD), which is undergoing some significant = development right now... As of today, it appears to be relatively = stable (I'm not aware of any glaring problems with the code base, all = unit tests pass, etc...), but it is a development build... =20 If you are comfortable running a development build, then take a crack at = HEAD - otherwise, you'll have to periodically call commit()... =20 Other major features in the current development build: =20 A revamped serialization system that optionally allows association of = serializers with records BTree key compression Option for running in a reduced durability mode (this makes the system = faster, at the cost of potentially losing comitted transactions during a = crash. The database will not get corrupted during a crash, though). =20 There may be others, but those are the three that come immediately to = mind... =20 If you do work with HEAD and run into anything, be sure to post! =20 - K =20 =20 =20 >=20 Im not sure exactly but it looks like the version Im using is from 2003. = =20 =20 I didnt realize people JDBM was being actively developed again. Do you = recommend the CVS version? It looks like if I just move to 1.0 I wont = get the DISABLE_TRANSACTIONS_AUTOCOMMITINTERVAL option.=20 =20 Is there a high level overview of what has changed over the last few = years anywhere? It doesnt look like the CHANGES or README file has been = updated in a while. =20 Eric=20 =20 From: jdb...@li... = [mailto:jdb...@li...] = <mailto:jdb...@li...> On Behalf Of Kevin = Day Sent: Tuesday, January 17, 2006 4:46 PM To: JDBM General listserv Subject: re[4]: [Jdbm-general] Transaction memory size=20 =20 Eric-=20 =20 What code are you running? The code was updated on 8/22/05 to provide = DISABLE_TRANSACTIONS_AUTOCOMMITINTERVAL option - this option is enabled = by default, so you should be able to use it out of the box to get the = behavior you are askingabout.=20 =20 The very latest version of code in CVS has the default for this value = set at 10,000 pages, which is ~80 MB - you've got to write a *lot* of = data before the auto-commit hits. You can adjust the value downwards by = setting the jdbm.disableTransactions.autoCommitInterval property to a = lower number.=20 =20 - K=20 =20 =20 =20 >=20 Kevin,=20 =20 Ive been experimenting running with the transactions shut off. Even with = transactions shut off it seems like the db file never gets written to = unless I call commit on the RecordManager. Is this the expected = behavior?=20 =20 Your description gave me the impression that the cached updates would be = periodically written to the database file by jdbm and that there was no = reason to call commit on the RecordManager directly.=20 =20 =20 Eric=20 =20 From: jdb...@li... = [mailto:jdb...@li...] = <mailto:jdb...@li...> On Behalf Of Kevin = Day Sent: Monday, January 16, 2006 4:23 PM To: JDBM General listserv Subject: re[2]: [Jdbm-general] Transaction memory size=20 =20 Eric-=20 =20 No - when transactions are disabled, they are cached and commited to = disk periodically (behind the scenes, we manage the size of the = uncomitted page load and write to disk as necessary). When Tx are off, = writes go to only one file (thedb file), instead of writing to the log, = then to the db file.=20 =20 Because writes to the log are actually synced to the phyiscal disk they = are usually considerably slower than non-transactional writes.=20 =20 Running without transactions should be several times faster than running = with them (at the risk of corrupting the database file in the event of a = crash, of course).=20 =20 - K=20 =20 =20 =20 > Kevin,=20 =20 Thanks for such a quick response. The main reason I was trying to avoid = committing was because committing seems to be relatively slow. If I = disable transactions will things be written to disk immediately? Will = this writing be faster thena commit?=20 Eric=20 =20 From: Kevin Day [mailto:ke...@tr...] = <mailto:ke...@tr...> =20 Sent: Monday, January 16, 2006 4:02 PM To: Rosenberg, Eric Subject: re: [Jdbm-general] Transaction memory size=20 =20 Eric-=20 =20 This is a limitation in the current jdbm transaction manager (there is a = group of us working on an alternate technique that will support = arbitrarily large transactions, but that's a long way off).=20 =20 The recommended approach at this point is to either:=20 =20 A. Commit your transactions periodically=20 =20 or=20 =20 B. Run with transcations disabled for large inserts (understand that = this will open you up to potential file corruption, so you only want to = run without transactions when it is OK to lose the entire database - = i.e. during initial data population).=20 =20 =20 Cheers,=20 =20 - Kevin=20 =20 =20 =20 >If I add / update a bunch of objects in JDBM during a transaction is everything kept in memory until I do a commit? I am trying to add an arbitrary number of objects to JDBM and am wondering if it is safe to add them all from a single transaction or if I need to commit periodically to ensure that I don't run out of memory. Thanks, Eric ------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Do you grep through log = files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! http://ads.osdn.com/?ad_idv37&alloc_id865&op=C0ick _______________________________________________ Jdbm-general mailing list Jdb...@li... https://lists.sourceforge.net/lists/listinfo/jdbm-general < < < < =20 ------------------------------------------------------- This SF.net = email is sponsored by: Splunk Inc. Do you grep through log files for = problems? Stop! Download the new AJAX search engine that makes searching = your log files as easy as surfing the web. DOWNLOAD SPLUNK! = http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=3D103432&bid=3D230486&dat=3D= 121642 _______________________________________________ Jdbm-general = mailing list Jdb...@li... = https://lists.sourceforge.net/lists/listinfo/jdbm-general |
From: Kevin D. <ke...@tr...> - 2006-01-18 16:01:20
|
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"> <HTML><HEAD> <STYLE type=text/css> P, UL, OL, DL, DIR, MENU, PRE { margin: 0 auto;}</STYLE> <META content="MSHTML 6.00.2900.2802" name=GENERATOR></HEAD> <BODY leftMargin=1 topMargin=1 rightMargin=1><FONT face=Tahoma> <DIV><FONT face=Arial size=2>hmpppphhhh. I thought this was part of the code base, but apparently not...</FONT></DIV> <DIV><FONT face=Arial size=2></FONT> </DIV> <DIV><FONT face=Arial size=2>You can take a look at the patches on the sourceforge web site and apply that to your code:</FONT></DIV> <DIV><FONT face=Arial size=2></FONT> </DIV> <DIV><A href="http://sourceforge.net/tracker/index.php?func=detail&aid=1063039&group_id=4155&atid=304155">http://sourceforge.net/tracker/index.php?func=detail&aid=1063039&group_id=4155&atid=304155</A></DIV> <DIV><FONT face=Arial size=2></FONT> </DIV> <DIV><FONT face=Arial size=2></FONT> </DIV> <DIV><FONT face=Arial size=2>I haven't applied this patch in awhile, so it may take some manual tweaking to get it to apply - but it's a relatively straightforward change to the system...</FONT></DIV> <DIV><FONT face=Arial size=2></FONT> </DIV> <DIV><FONT face=Arial size=2>Sorry about the confusion on that one...</FONT></DIV> <DIV><FONT face=Arial size=2></FONT> </DIV> <DIV><FONT face=Arial size=2>- K</FONT></DIV> <DIV><FONT face=Arial size=2></FONT> </DIV> <DIV><FONT face=Arial size=2></FONT> </DIV> <DIV><FONT face=Arial size=2></FONT></DIV> <DIV><FONT face=Arial size=2></FONT> </DIV> <DIV><FONT face=Arial size=2> </FONT> <TABLE> <TBODY> <TR> <TD width=1 bgColor=blue><FONT face=Arial size=2></FONT></TD> <TD><FONT face=Arial size=2><FONT color=red>> <BR>Kevin, <BR><BR>Im sorry for the continuing barge of email. Can you elaborate at all on the option for a reduced durability mode? It sounds like it may be a good fit for my current task but I dont see anything in RecordManagerOptions. <BR> <BR>Eric <BR> <BR><BR><BR>From: <A href="mailto:jdb...@li..."><FONT color=#0000ff>jdb...@li...</FONT></A> <A href="mailto:jdb...@li..."><FONT color=#0000ff>[mailto:jdb...@li...]</FONT></A> On Behalf Of Kevin Day<BR>Sent: Wednesday, January 18, 2006 9:53 AM<BR>To: JDBM General listserv<BR>Subject: re[6]: [Jdbm-general] Transaction memory size <BR> <BR><BR>Eric- <BR><BR> <BR><BR>At minimum, I would recommend grabbing the 1.0 release. In terms of changes that have been made, your best bet is to look at the commit logs - but the differences between the .13 and 1.0 release were very minor (mostly updates of comments and that sort of thing - a few bug fixes). <BR><BR> <BR><BR>The auto-commit interval option is only available in the current development build (HEAD), which is undergoing some significant development right now... As of today, it appears to be relatively stable (I'm not aware of any glaring problems with the code base, all unit tests pass, etc...), but it is a development build... <BR><BR> <BR><BR>If you are comfortable running a development build, then take a crack at HEAD - otherwise, you'll have to periodically call commit()... <BR><BR> <BR><BR>Other major features in the current development build: <BR><BR> <BR><BR>A revamped serialization system that optionally allows association of serializers with records <BR><BR>BTree key compression <BR><BR>Option for running in a reduced durability mode (this makes the system faster, at the cost of potentially losing comitted transactions during a crash. The database will not get corrupted during a crash, though). <BR><BR> <BR><BR>There may be others, but those are the three that come immediately to mind... <BR><BR> <BR><BR>If you do work with HEAD and run into anything, be sure to post! <BR><BR> <BR><BR>- K <BR><BR> <BR><BR> <BR><BR> <BR>> <BR>Im not sure exactly but it looks like the version Im using is from 2003. <BR> <BR>I didnt realize people JDBM was being actively developed again. Do you recommend the CVS version? It looks like if I just move to 1.0 I wont get the DISABLE_TRANSACTIONS_AUTOCOMMITINTERVAL option. <BR> <BR>Is there a high level overview of what has changed over the last few years anywhere? It doesnt look like the CHANGES or README file has been updated in a while. <BR><BR>Eric <BR> <BR><BR><BR>From: <A href="mailto:jdb...@li..."><FONT color=#0000ff>jdb...@li...</FONT></A> <A href="mailto:jdb...@li..."><FONT color=#0000ff>[mailto:jdb...@li...]</FONT></A> On Behalf Of Kevin Day<BR>Sent: Tuesday, January 17, 2006 4:46 PM<BR>To: JDBM General listserv<BR>Subject: re[4]: [Jdbm-general] Transaction memory size <BR> <BR><BR>Eric- <BR><BR> <BR><BR>What code are you running? The code was updated on 8/22/05 to provide DISABLE_TRANSACTIONS_AUTOCOMMITINTERVAL option - this option is enabled by default, so you should be able to use it out of the box to get the behavior you are askingabout. <BR><BR> <BR><BR>The very latest version of code in CVS has the default for this value set at 10,000 pages, which is ~80 MB - you've got to write a *lot* of data before the auto-commit hits. You can adjust the value downwards by setting the jdbm.disableTransactions.autoCommitInterval property to a lower number. <BR><BR> <BR><BR>- K <BR><BR> <BR><BR> <BR><BR> <BR>><BR>Kevin, <BR> <BR>Ive been experimenting running with the transactions shut off. Even with transactions shut off it seems like the db file never gets written to unless I call commit on the RecordManager. Is this the expected behavior? <BR> <BR>Your description gave me the impression that the cached updates would be periodically written to the database file by jdbm and that there was no reason to call commit on the RecordManager directly. <BR> <BR> <BR>Eric <BR> <BR><BR><BR>From: <A href="mailto:jdb...@li..."><FONT color=#0000ff>jdb...@li...</FONT></A> <A href="mailto:jdb...@li..."><FONT color=#0000ff>[mailto:jdb...@li...]</FONT></A> On Behalf Of Kevin Day<BR>Sent: Monday, January 16, 2006 4:23 PM<BR>To: JDBM General listserv<BR>Subject: re[2]: [Jdbm-general] Transaction memory size <BR> <BR><BR>Eric- <BR><BR> <BR><BR>No - when transactions are disabled, they are cached and commited to disk periodically (behind the scenes, we manage the size of the uncomitted page load and write to disk as necessary). When Tx are off, writes go to only one file (thedb file), instead of writing to the log, then to the db file. <BR><BR> <BR><BR>Because writes to the log are actually synced to the phyiscal disk they are usually considerably slower than non-transactional writes. <BR><BR> <BR><BR>Running without transactions should be several times faster than running with them (at the risk of corrupting the database file in the event of a crash, of course). <BR><BR> <BR><BR>- K <BR><BR> <BR><BR> <BR><BR> <BR>><BR>Kevin, <BR> <BR>Thanks for such a quick response. The main reason I was trying to avoid committing was because committing seems to be relatively slow. If I disable transactions will things be written to disk immediately? Will this writing be faster thena commit? <BR><BR>Eric <BR> <BR><BR><BR>From: Kevin Day <A href="mailto:ke...@tr..."><FONT color=#0000ff>[mailto:ke...@tr...]</FONT></A> <BR>Sent: Monday, January 16, 2006 4:02 PM<BR>To: Rosenberg, Eric<BR>Subject: re: [Jdbm-general] Transaction memory size <BR> <BR><BR>Eric- <BR><BR> <BR><BR>This is a limitation in the current jdbm transaction manager (there is a group of us working on an alternate technique that will support arbitrarily large transactions, but that's a long way off). <BR><BR> <BR><BR>The recommended approach at this point is to either: <BR><BR> <BR><BR>A. Commit your transactions periodically <BR><BR> <BR><BR>or <BR><BR> <BR><BR>B. Run with transcations disabled for large inserts (understand that this will open you up to potential file corruption, so you only want to run without transactions when it is OK to lose the entire database - i.e. during initial data population). <BR><BR> <BR><BR> <BR><BR>Cheers, <BR><BR> <BR><BR>- Kevin <BR><BR> <BR><BR> <BR><BR> <BR>>If I add / update a bunch of objects in JDBM during a transaction is<BR>everything kept in memory until I do a commit?<BR><BR>I am trying to add an arbitrary number of objects to JDBM and am<BR>wondering if it is safe to add them all from a single transaction or if<BR>I need to commit periodically to ensure that I don't run out of memory.<BR><BR>Thanks,<BR>Eric<BR><BR><BR>-------------------------------------------------------<BR>This SF.net email is sponsored by: Splunk Inc. Do you grep through log files<BR>for problems? Stop! Download the new AJAX search engine that makes<BR>searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!<BR><A href="http://ads.osdn.com/?ad_idv37&alloc_id865&opÀick"><FONT color=#0000ff>http://ads.osdn.com/?ad_idv37&alloc_id865&opÀick</FONT></A><BR>_______________________________________________<BR>Jdbm-general mailing list<BR><A href="mailto:Jdb...@li..."><FONT color=#0000ff>Jdb...@li...</FONT></A><BR><A href="https://lists.sourceforge.net/lists/listinfo/jdbm-general"><FONT color=#0000ff>https://lists.sourceforge.net/lists/listinfo/jdbm-general</FONT></A><BR><BR><<BR><<BR><<BR><<BR><<BR></FONT></FONT></TD></TR></TBODY></TABLE></DIV></FONT></BODY></HTML> |
From: Kevin D. <ke...@tr...> - 2006-01-17 21:42:26
|
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"> <HTML><HEAD> <STYLE type=text/css> P, UL, OL, DL, DIR, MENU, PRE { margin: 0 auto;}</STYLE> <META content="MSHTML 6.00.2900.2802" name=GENERATOR></HEAD> <BODY leftMargin=1 topMargin=1 rightMargin=1><FONT face=Tahoma> <DIV><FONT face=Arial size=2>Eric-</FONT></DIV> <DIV><FONT face=Arial size=2></FONT> </DIV> <DIV><FONT face=Arial size=2></FONT></DIV> <DIV><FONT face=Arial size=2>What code are you running? The code was updated on 8/22/05 to provide <FONT size=2>DISABLE_TRANSACTIONS_AUTOCOMMITINTERVAL option - this option is enabled by default, so you should be able to use it out of the box to get the behavior you are asking about.</FONT></FONT></DIV> <DIV><FONT face=Arial size=2></FONT> </DIV> <DIV><FONT face=Arial size=2>The very latest version of code in CVS has the default for this value set at 10,000 pages, which is ~80 MB - you've got to write a *lot* of data before the auto-commit hits. You can adjust the value downwards by setting the jdbm.disableTransactions.autoCommitInterval property to a lower number.</FONT></DIV> <DIV><FONT face=Arial size=2></FONT> </DIV> <DIV><FONT face=Arial size=2>- K</FONT></DIV> <DIV><FONT face=Arial size=2></FONT> </DIV> <DIV><FONT face=Arial size=2> </FONT> <TABLE> <TBODY> <TR> <TD width=1 bgColor=blue><FONT face=Arial size=2></FONT></TD> <TD><FONT face=Arial size=2><FONT color=blue>> <BR>Kevin, <BR> <BR>Ive been experimenting running with the transactions shut off. Even with transactions shut off it seems like the db file never gets written to unless I call commit on the RecordManager. Is this the expected behavior? <BR> <BR>Your description gave me the impression that the cached updates would be periodically written to the database file by jdbm and that there was no reason to call commit on the RecordManager directly. <BR> <BR> <BR>Eric <BR> <BR><BR><BR>From: <A href="mailto:jdb...@li..."><FONT color=#0000ff>jdb...@li...</FONT></A> <A href="mailto:jdb...@li..."><FONT color=#0000ff>[mailto:jdb...@li...]</FONT></A> On Behalf Of Kevin Day<BR>Sent: Monday, January 16, 2006 4:23 PM<BR>To: JDBM General listserv<BR>Subject: re[2]: [Jdbm-general] Transaction memory size <BR> <BR><BR>Eric- <BR><BR> <BR><BR>No - when transactions are disabled, they are cached and commited to disk periodically (behind the scenes, we manage the size of the uncomitted page load and write to disk as necessary). When Tx are off, writes go to only one file (the db file), instead of writing to the log, then to the db file. <BR><BR> <BR><BR>Because writes to the log are actually synced to the phyiscal disk they are usually considerably slower than non-transactional writes. <BR><BR> <BR><BR>Running without transactions should be several times faster than running with them (at the risk of corrupting the database file in the event of a crash, of course). <BR><BR> <BR><BR>- K <BR><BR> <BR><BR> <BR><BR> <BR>> <BR>Kevin, <BR> <BR>Thanks for such a quick response. The main reason I was trying to avoid committing was because committing seems to be relatively slow. If I disable transactions will things be written to disk immediately? Will this writing be faster thena commit? <BR><BR>Eric <BR> <BR><BR><BR>From: Kevin Day <A href="mailto:ke...@tr..."><FONT color=#0000ff>[mailto:ke...@tr...]</FONT></A> <BR>Sent: Monday, January 16, 2006 4:02 PM<BR>To: Rosenberg, Eric<BR>Subject: re: [Jdbm-general] Transaction memory size <BR> <BR><BR>Eric- <BR><BR> <BR><BR>This is a limitation in the current jdbm transaction manager (there is a group of us working on an alternate technique that will support arbitrarily large transactions, but that's a long way off). <BR><BR> <BR><BR>The recommended approach at this point is to either: <BR><BR> <BR><BR>A. Commit your transactions periodically <BR><BR> <BR><BR>or <BR><BR> <BR><BR>B. Run with transcations disabled for large inserts (understand that this will open you up to potential file corruption, so you only want to run without transactions when it is OK to lose the entire database - i.e. during initial data population). <BR><BR> <BR><BR> <BR><BR>Cheers, <BR><BR> <BR><BR>- Kevin <BR><BR> <BR><BR> <BR><BR> <BR>>If I add / update a bunch of objects in JDBM during a transaction is<BR>everything kept in memory until I do a commit?<BR><BR>I am trying to add an arbitrary number of objects to JDBM and am<BR>wondering if it is safe to add them all from a single transaction or if<BR>I need to commit periodically to ensure that I don't run out of memory.<BR><BR>Thanks,<BR>Eric<BR><BR><BR>-------------------------------------------------------<BR>This SF.net email is sponsored by: Splunk Inc. Do you grep through log files<BR>for problems? Stop! Download the new AJAX search engine that makes<BR>searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!<BR><A href="http://ads.osdn.com/?ad_idv37&alloc_id865&opÀick"><FONT color=#0000ff>http://ads.osdn.com/?ad_idv37&alloc_id865&opÀick</FONT></A><BR>_______________________________________________<BR>Jdbm-general mailing list<BR><A href="mailto:Jdb...@li..."><FONT color=#0000ff>Jdb...@li...</FONT></A><BR><A href="https://lists.sourceforge.net/lists/listinfo/jdbm-general"><FONT color=#0000ff>https://lists.sourceforge.net/lists/listinfo/jdbm-general</FONT></A><BR><BR><<BR><<BR><<BR></FONT></FONT></TD></TR></TBODY></TABLE></DIV></FONT></BODY></HTML> |