[Hibernate-devel] deep lazy collections

[Doug C. <de...@fl...>]

I have been thinking about "deep lazy collections." For example, a map
where only individual entries are read from the database as needed,
and only added or modified entries are written. [This is probably only
really useful for maps, and maybe sets.]

Deep lazy collections would be very useful for large collections that
are accessed sparsely. An example might be a map keyed by date where
several years worth of data are stored but only a few days worth used
at any one time.

Of course you could create such a map by using a DATE field as the
primary key (id) for some class and using custom queries. But there is
a certain beauty in using Hibernate collections and Java accesses
directly.

I believe that deep lazy collections (lazy="deep") could be
implemented without *too* much effort. I would appreciate critiques on
my proposal...

1. Deep Lazy collection classes would extend the Lazy classes in
cirrus.hibernate.collections.*. As a fallback, if the user does
something inconsistent with deep laziness, e.g., asking for a
collection of elements via Map.values() for example, the Deep Lazy
class would simply revert to Lazy behavior.

2. The methods get(), put(), remove(), size(), and a few others would
be optimized to perform minimal database activity.

3. Deep Lazy collection classes would maintain a number of local maps
to record additions, removals, and modifications. These maps, along
with a cache of read objects, are consulted before going to the
database. See logic and invariants below.

4. CollectionPersister would be extended to support sql for select by
map-key, delete by map-key, and update by map-key, as well as count(*).

5. To maintain some backward compatibility with the current
RelationalDatabaseSession and its use of dorecreate and doremove,
PersistentCollection could add methods for
     isDeepLazy() => false (if true, don't delete all, instead ask for:)
  deleteEntries()
  insertEntries() => entries() for old collection classes
  updateEntries() => null for old collection classes

There are surely some things I haven't considered or don't fully
appreciate. Some areas that concern me:

1. The notion of ce.initialized must be refined. There will be
collections that have a persister, but have not been read entirely.
I.e., ce.initialized, but elements() not valid; see: flush,
updateReachableCollection, searchForDirtyCollections,
prepareCollectionForUpdate.

2. prepareCollectionForUpdate must be smarter, e.g., deciding when to
totally recreate a collection versus when to update only; this may be
a separate dirty case verus a copied or moved case, I'm not sure.

3. How does this affect copying and reusing (sub-)collections?

Again, comments are welcome. I am willing to code the
cirrus.hibernate.collections.* part of the implementation (an outline
of that portion is below), and maybe the changes in
CollectionPersister if we can agree on the interface to
RelationalDatabaseSession.

e

-=-

map with deep laziness

dbase - the elements in the db
cache - the elements looked up (unchanged from dbase version)
repls - the modified elements
addns - the newly created elements
delts - the newly deleted elements

invariants:
 cache is just a copy of what's in dbase
 an entity is only in one of: delts or addns or repls
 if in delts, then     in dbase
 if in addns, then not in dbase
 if in repls, then     in dbase

logic...
     each of these operations searches the maps in order, and
     if found in
     -------
put - delts: remove from delts, create a repl
      addns: modify the addn
      repls: modify the repl
      dbase: create a repl
      else:  create a addn

rem - delts: nothing to do
      addns: remove from addns
      repls: remove from repls, add to delts
      dbase: create a delt
      else:  nothing to do
      
get - delts: not found
      addns: ok
      repls: ok
      dbase: ok
      else:  not found
 
size := dbase.sqlCount() + addns.size() - delts.size()
isEmpty := count == 0
containsKey := get != null
get := get
remove := rem
put := put
putAll := put in loop
the rest := revert to old mode
  i.e., read(); apply addns, repls, delts; wipe addns, repls, delts, cache.
  maybe containsElement := get(byElement?) != null
  
-=-