RE: [Jdbm-developer] lazy insert (commit)

[Thompson, B. B. <BRY...@sa...>]

Ok.  I've committed the changes.  The test suites (including the new
TestLazyInsert) all run.  The lazy insert feature is on by default.
The execution should be backward compatible.  The commit is rather
small since there was not much code that was directly affected.  I
have a large set of changes related to the extensible serialization
that I have not yet committed, so I introduced the lazy insert changes
into my "clean" version of jdbm and validated them there.  However I
would certainly appreciate 3rd party validation that this commit does
not break anything.

I increased the exposure of the logical manager in BaseRecordManager
since I needed to tunnel through to get a new logical row id without
allocating a physical row id at the same time.  I also increased the
exposure of the CacheEntry class in CacheRecordManager, but restricted
the expose of its data.

I have not yet run a profiler over the result, but that 15% gain is
pretty much the maximium that I had hoped to gain by dealing with the
free physical row id stuff.  Hopefully that hotspot is eliminated by
this commit.

Cheers,

-bryan

-----Original Message-----
From: Kevin Day
To: Thompson, Bryan B.; 'jdb...@li...'
Cc: 'tho...@sa...'; Bebee, Bradley R.; Personick, Michael R.;
'bp...@is...'
Sent: 10/26/2005 12:01 PM
Subject: re: [Jdbm-developer] lazy insert

Bryan-

I'm confused by this.  The current implementation of jdbm already defers
serialization...  Objects aren't serialized or written to a record until
they are evicted from cache....
 
hmmmm - ah- I think I see.  You are refering only to insert operations
(the current jdbm does what you describe on updates...  but maybe not on
inserts).
 
So a logical record id is being assigned during the insert, but not the
physical id?  I guess I'm not entirely clear on how the
logical->physical id mapping is getting created if not during the insert
call...  I'm looking forward to seeing the code.
 
 
Thanks,
 
- K


 > All,

I just implemented (not committed yet) an option to defer the
serialization of the object and the allocation of the physical row
during an insert operation until the object is either evicted from the
cache or a commit is performed.  This was faily simple to do and I am
going to make it available as a runtime option.  The net savings was 15%
on our test application :-)

This option occurred to me when examining jdbm under a profiler again.
As you know, there is a definate hotspot in the reuse of "free" physical
rows.  It occurred to me that the thing driving the physical row reuse
was that the size of the record was changing (growing) from its original
insert through a series of updates leading to the commit.  By deferring
the allocation of the physical row, I just mark the object as dirty in
the cache during the insert (rather than clean) and handle the case in
update on BaseRecordManager when the physical row id is zero.

This issue is characteristic of our application which evolves the state
of the objects in cache before they are ready to be laid down on the
disk.
Unlike Alex (who reports being disk bound), we are actually CPU bound as
nearly all operations touch cache rather than going to disk.  It is only
on very long transactions that we need to spin data onto the log before
the commit.

Along those lines, I have added some counters into RecordFile.  Among
other things, these counters can report the #of blocks in memory (and
the size in MB of those blocks).  The current "maxDirtySize" is very
low and I think that we should boost it to 10,000 or so (80M).  Of
course,
what we really need to do is tear out the RecordFile/TransactionManager
and replace it with a DBCache version so that we have better I/O through
put for long transactions and support for concurrent transactions.

Cheers,

-bryan


-------------------------------------------------------
This SF.Net email is sponsored by the JBoss Inc.
Get Certified Today * Register for a JBoss Training Course
Free Certification Exam for All Training Attendees Through End of 2005
Visit  <http://www.jboss.com/services/certification>
http://www.jboss.com/services/certification for more information
_______________________________________________
Jdbm-developer mailing list
 <mailto:Jdb...@li...>
Jdb...@li...
 <https://lists.sourceforge.net/lists/listinfo/jdbm-developer>
https://lists.sourceforge.net/lists/listinfo/jdbm-developer

<