xorm-devel

[Xorm-devel] Distributed Cache Management

[Harry E. <ha...@tr...>]

I have a simple design for distributed cache management.  It has a 
couple of holes, but doesn't seem like a bad first pass at it.  I would 
like to get feedback from the group to see what others think.

CacheManager methods (don't worry about local vs. remote right now)
-lock(Object o, transactionId) : Lock
Checks version of object, and attempts to create a lock object for it.  
Version is either some kind of change counter, or some time kind of time 
stamp of last update time in the datastore.
--o
The xorm managed object to lock
--transactionId
a relatively unique id for the transaction obtaining the lock.  Used to 
allow the same transaction to acquire a lock on the same object more 
than once.
--returns
If the object is already locked, returns the special LOCK_FAILED Lock 
object. (An overridden version of the method might allow the caller to 
specify waiting until the lock is obtained)
If a lock is granted, returns a Lock object with status LOCK_SUCCESS, 
and a valid lock expiration time.
If a lock is granted, but the object is stale, returns a Lock object 
with LOCK_REFRESH, and a valid lock expiration time.  This means that 
the caller has obtained a lock on the object, but needs to refresh the 
object with the data store.


-commit(Object o, transactionId) : Lock
Verifies that this object is has a lock held by the transactionId.  If 
so, it will update the version in the Cache Manager to match the version 
held by the object.  This method should be called AFTER a change to the 
datastore, but before committing the changes to the datastore.  This 
method also depends on the version of the object being updated based on 
the changes to the datastore, even though it is not yet committed.  
Regardless of outcome, this method will result in releasing the lock on 
this object in the CacheManager.
--o
The xorm managed object that a lock was previously obtained for
--transactionId
the transactionId that was used to obtain the lock for o
--returns
If there is a lock for this object, and the transactionId matches, and 
the lock has not expired, returns COMMIT_SUCCESS
Otherwise if the lock does not exist, is not for this transactionId, or 
has expired, returns COMMIT_FAILURE.  The proper action on a 
COMMIT_FAILURE might be a retry, or a rollback.

-check(Object o) : boolean
Checks whether the object is the most current version of the object seen 
by the CacheManager.
--o
The object to check
--returns
Returns true if this version of the object is the most recent seen by 
the CacheManager
otherwise false.

-refreshLock(Object o, transactionId) : LOCK
Checks for a valid lock on the object, and if all is good, extends the 
expiration of the Lock.  Might not be needed (could just be rolled into 
lock method).
--return values as per lock() method.


The concept behind all this is that the CacheManager object does not 
talk to the db, but simply deals with objects as it sees them.  Given a 
compliant local implementation of XORM, all objects would be seen 
whenever a transaction was involved, so there shouldn't be a race 
condition issue.  A simple sequence diagram follows:

XORM                    CacheManager                DataStore
----                    ------------                ---------
  |                           |                          |
 | |---startTrans()--         |                          |
 | |                |         |                          |
 | |<----------------         |                          |
 | |                          |                          |
 | |---lock(o, tId)--------->| |                         |
 | |<----------Lock object---| |                         |
 | |                          |                          |
 | |---refreshObject(o) (if stale)--------------------->| |
 | |<---------------------------------refreshedObject---| |
 | |                          |                          |
 | |---make changes--         |                          |
 | |                |         |                          |
 | |<----------------         |                          |
 | |---commitTrans() (begin)- |                          |
 | |                        | |                          |
 | |<------------------------ |                          |
 | |                          |                          |
 | |---makeUpdates (don't commit)---------------------->| |
 | |<-----------------------------------------success---| |
 | |                          |                         | |
 | |---commit(o, tid)------->| |                        | |
 | |<--------------success---| |                        | |
 | |                          |                         | |
 | |---commitTransaction------------------------------->| |
 | |<-----------------------------------------success---| |
 | |                          |                          |
 | |---commitTrans() (end)-   |                          |
 | |                      |   |                          |
 | |<----------------------   |                          |
  |                           |                          |

I think the whole scheme might work, if you assume that no one operates 
outside of it.  I don't know if that is a reasonable caveat or not.  
Obviously, it is possible for the distributed cache manager to provide 
methods for locking or committing more than one object at the same 
time.  Also, it is assumes that the distributed cache manager would most 
likely be "remote" from the XORM instance using it, so the "local" 
methods I showed should be assumed to be local facades to a remote 
distributed cache manager.  I tis probably obvious (but worth stating)  
that all XORM instances dealing with a particular datastore would use 
the same CacheManager, possibly with CacheManager replication for 
failover.  Persistent connections, and good communications design should 
make it relatively low overhead data wise.

The Lock object as it is internally held by the CacheManager might look 
something like this:
Lock
----
class : Class
id : objectPrimaryKey
version : Version (int, timestamp, etc)
txnId : TransactionId (int, long, etc)
expires: Time (long, Date, etc)

For objects that have been seen, but are not currently locked, txnId and 
expires would be null.  For objects that are locked, all values would be 
non null.  Locks can be removed lazily, by checking a lock when needed, 
and if it is expired, nulling txnId and expires.

This might be way off base, but it seemed like a good first pass.  I am 
not trying to reinvent the wheel here, but it seems like something like 
this might solve most of the "coordinated" or "distributed" cache 
management issues we see with xorm right now.

Thoughts / comments / questions?

Harry

[Xorm-devel] Distributed Cache Management

[Harry E. <ha...@gm...>]

Optimistic Distributed Cache Management

This is a proposal for a simple distributed cache invalidation
strategy for xorm.  It might be implementable at only the cache
manager layer, or might require integration at a higher layer.  It
does not do coordinated lock management, but would do reasonable cache
invalidation in the cases where the same enitity was not modified in
multiple instances in rapid succession.  This is seen as the first
phase in a more pessimistic strategy that would do actual lock
management.

It uses the multicast transmission facilities in Java to set up a peer
to peer local network on which cache invalidation messages are sent to
other servers using the same datastore.  Detected events are broadcast
to other servers, allowing them to either hollow the referenced object
or remove them completely from cache.  Below are the major pieces of
the implementation as I see them.

I am looking at using JRMS (Java Reliable Multicast Service) jar
(http://www.experimentalstuff.com/Technologies/JRMS/index.html) to
reasonably guarantee the receipt of packets to each server
(specifically the LRMP variety, with late-join catchup disabled).=20
Basic desription of major areas below, followed by questions.

I would really really like to find a way to do this at the cache
manager level, as it would be a simple properties change to enable it,
by using a version of the LRU cache that implemented the additional
functionality, but I am not sure this is possible.  Feedback
appreciated.

Harry

Detection:
"Broadcast" means notification sent out over multicast
Objects that are deleted must be broadcast to be removed
Objects that have simple attributes or one way references updated must
be broadcast to be hollowed or removed
Changes to collections must be broadcast:
  ObjParent has a collection that contains objects, including ObjChild
  ObjChild has a direct reference to ObjParent
  ObjChild has its reference to ObjParent removed (or reassigned to
ObjNewParent)
  ObjChild must be broadcast to be hollowed or removed
  ObjParent must either be broadcast to be hollowed or removed or must
be broadcast have its collection set to unresolved
  ObjNewParent must either be broadcast to be hollowed or removed or
must be broadcast to have its collection set to unresolved

Transmission:
Initial transmission is seen as non locking multicast packets
All cache instances are both transmitters and receivers of packets
(there is no central server)
When a cache instance detects a broadcast event:
  It formulates a broadcast packet, containing the table name and
primary key of the Object
  (Should this be class instead of table?)
  It transmits the packet on the multicast address for the group
  Each cache instance (except the sender) receives the packet
  Each cache instance purges (or makes hollow?) the referenced object
  (Does this need to distinguish between hollow vs removal for change
vs delete, respectively?)
  (How are collections handled?)

Packet format:
Packets are datagrams, pssoibly of fixed length:
  Format: (Action ( Remove vs hollow?) + ) Primary key + Hash of
(Class | table name?)
  (Can we use fixed length for packet?  ie can we reliably hash class
or table name?)
  Packet is fixed length to minimize message size.  Variable length is
possible, but seems bad.
  If using a hash, each cache manager calculates hash value upon entry
of (class | table) into cache space.  Unknown hashes are ignored (as
they aren't in cache).

Questions:
How do we detect the appropriate broadcast events at the cache manager leve=
l?
Can multiple VMs on the same host bind to the same multicast group
address?  If not, do we need some form of local relay? (Tested to be
okay on windows)
Do we use table name, class name, or something else for entity representati=
on?
Can we use a hash of the entity representation (class or table name)
to allow for fixed length packets?
Which hash formula do we use?
Is there too much multicast traffic on a network of 60 machines? (we
can use separate addresses for separate clusters)
Are networks (or switches) commonly configured to support local
multicast packets?

[Xorm-devel] Re: Distributed Cache Management

[Harry E. <ha...@gm...>]

Having spoken with Doug, I am looking at using the JGroups (or JavaGroups,
gotta love that Sun trademark enforcement) as the mechanism for transportin=
g
the notifications.

Thoughts:
1. Modify LRUCache to optionally notify a listener when a Row is added,
updated, or removed.
2. Subclass LRUCache to use (most likely) a JGroups NotificationBus for
cache events.
3. Setup JGroups to manage this notification, and provide properties the
subclass can read to allow for enhanced settings (UDP versus TCP
notifications, etc).
4. I walked through some xorm code, and it doesn't appear that relationship
proxy information is ever directly conveyed to the cache level, which means
that there is no way, at that level, to invalidate the Collection case I
listed previously. I could really use some insight into an approach for thi=
s
area (*cough* Wes *cough*) as this seems like a kinda important part. I
might have this wrong, but it appears xorm manages all collection level
information (12M and M2M) at the ObjectProxy level, and not at all in the
Row DataCache level. Is that correct?
5. I am unclear whether InterfaceInvocationHandlers need to have their Row
cleared in this scheme, or if they generally pass out of scope frequently
enough that the Row is the main thing to focus on. This has to do with the
concern that they hold a direct hard reference to a Row object, so throwing
that out of cache isn't going to accomplish much if they are still pointing
to it.

I am really interested in the JGroups DistributedLockManager class, but it
would have to be combined / enhanced with cache management to use it, and i=
t
seems like there should be an option, regardless, to only do cache
management without the overhead of distributed locks. The JBossCache stuff
also might be more appropriate in the long term, as it uses JGroups stuff,
but seems like it can handle both locks and caching. I will have to do more
research.

Input very much appreciated.

Harry

On 10/4/05, Harry Evans <ha...@gm...> wrote:
>
> Optimistic Distributed Cache Management
>
> This is a proposal for a simple distributed cache invalidation
> strategy for xorm. It might be implementable at only the cache
> manager layer, or might require integration at a higher layer. It
> does not do coordinated lock management, but would do reasonable cache
> invalidation in the cases where the same enitity was not modified in
> multiple instances in rapid succession. This is seen as the first
> phase in a more pessimistic strategy that would do actual lock
> management.
>
> It uses the multicast transmission facilities in Java to set up a peer
> to peer local network on which cache invalidation messages are sent to
> other servers using the same datastore. Detected events are broadcast
> to other servers, allowing them to either hollow the referenced object
> or remove them completely from cache. Below are the major pieces of
> the implementation as I see them.
>
> I am looking at using JRMS (Java Reliable Multicast Service) jar
> (http://www.experimentalstuff.com/Technologies/JRMS/index.html) to
> reasonably guarantee the receipt of packets to each server
> (specifically the LRMP variety, with late-join catchup disabled).
> Basic desription of major areas below, followed by questions.
>
> I would really really like to find a way to do this at the cache
> manager level, as it would be a simple properties change to enable it,
> by using a version of the LRU cache that implemented the additional
> functionality, but I am not sure this is possible. Feedback
> appreciated.
>
> Harry
>
> Detection:
> "Broadcast" means notification sent out over multicast
> Objects that are deleted must be broadcast to be removed
> Objects that have simple attributes or one way references updated must
> be broadcast to be hollowed or removed
> Changes to collections must be broadcast:
> ObjParent has a collection that contains objects, including ObjChild
> ObjChild has a direct reference to ObjParent
> ObjChild has its reference to ObjParent removed (or reassigned to
> ObjNewParent)
> ObjChild must be broadcast to be hollowed or removed
> ObjParent must either be broadcast to be hollowed or removed or must
> be broadcast have its collection set to unresolved
> ObjNewParent must either be broadcast to be hollowed or removed or
> must be broadcast to have its collection set to unresolved
>
> Transmission:
> Initial transmission is seen as non locking multicast packets
> All cache instances are both transmitters and receivers of packets
> (there is no central server)
> When a cache instance detects a broadcast event:
> It formulates a broadcast packet, containing the table name and
> primary key of the Object
> (Should this be class instead of table?)
> It transmits the packet on the multicast address for the group
> Each cache instance (except the sender) receives the packet
> Each cache instance purges (or makes hollow?) the referenced object
> (Does this need to distinguish between hollow vs removal for change
> vs delete, respectively?)
> (How are collections handled?)
>
> Packet format:
> Packets are datagrams, pssoibly of fixed length:
> Format: (Action ( Remove vs hollow?) + ) Primary key + Hash of
> (Class | table name?)
> (Can we use fixed length for packet? ie can we reliably hash class
> or table name?)
> Packet is fixed length to minimize message size. Variable length is
> possible, but seems bad.
> If using a hash, each cache manager calculates hash value upon entry
> of (class | table) into cache space. Unknown hashes are ignored (as
> they aren't in cache).
>
> Questions:
> How do we detect the appropriate broadcast events at the cache manager
> level?
> Can multiple VMs on the same host bind to the same multicast group
> address? If not, do we need some form of local relay? (Tested to be
> okay on windows)
> Do we use table name, class name, or something else for entity
> representation?
> Can we use a hash of the entity representation (class or table name)
> to allow for fixed length packets?
> Which hash formula do we use?
> Is there too much multicast traffic on a network of 60 machines? (we
> can use separate addresses for separate clusters)
> Are networks (or switches) commonly configured to support local
> multicast packets?
>

Re: [Xorm-devel] Re: Distributed Cache Management

[Wes B. <we...@ca...>]

Changes to collections are always accomplished by changes to the data
model.  That being said, I forget if XORM is keeping many-to-many table
Rows in cache currently -- it would need to for this scheme to work,
unless you want to change the whole thing and cache the
persistence-capable objects directly (I don't think this is a good idea).

You might look at JCache.sourceforge.net which uses JavaGroups.  This is
the approach that some of the commercial JDO vendors have taken --
integrate a third party distributed cache management solution.  The
JCache API (it's a JSR) is basically a Map, and the implementation
supposedly takes care of all the notification and sync behind the
scenes.  But that might be harder to integrate with the existing
LRUCache, etc.

Wes

Harry Evans wrote:

> Having spoken with Doug, I am looking at using the JGroups (or 
> JavaGroups, gotta love that Sun trademark enforcement) as the 
> mechanism for transporting the notifications.
>
> Thoughts:
> 1.  Modify LRUCache to optionally notify a listener when a Row is 
> added, updated, or removed.
> 2.  Subclass LRUCache to use (most likely) a JGroups NotificationBus 
> for cache events.
> 3.  Setup JGroups to manage this notification, and provide properties 
> the subclass can read to allow for enhanced settings (UDP versus TCP 
> notifications, etc).
> 4.  I walked through some xorm code, and it doesn't appear that 
> relationship proxy information is ever directly conveyed to the cache 
> level, which means that there is no way, at that level, to invalidate 
> the Collection case I listed previously.  I could really use some 
> insight into an approach for this area (*cough* Wes *cough*) as this 
> seems like a kinda important part.  I might have this wrong, but it 
> appears xorm manages all collection level information (12M and M2M) at 
> the ObjectProxy level, and not at all in the Row DataCache level.  Is 
> that correct?
> 5.  I am unclear whether InterfaceInvocationHandlers need to have 
> their Row cleared in this scheme, or if they generally pass out of 
> scope frequently enough that the Row is the main thing to focus on.  
> This has to do with the concern that they hold a direct hard reference 
> to a Row object, so throwing that out of cache isn't going to 
> accomplish much if they are still pointing to it.
>
>  I am really interested in the JGroups DistributedLockManager class, 
> but it would have to be combined / enhanced with cache management to 
> use it, and it seems like there should be an option, regardless, to 
> only do cache management without the overhead of distributed locks.  
> The JBossCache stuff also might be more appropriate in the long term, 
> as it uses JGroups stuff, but seems like it can handle both locks and 
> caching.  I will have to do more research.
>
> Input very much appreciated.
>
> Harry
>
> On 10/4/05, *Harry Evans* <ha...@gm... <mailto:ha...@gm...>> 
> wrote:
>
>     Optimistic Distributed Cache Management
>
>     This is a proposal for a simple distributed cache invalidation
>     strategy for xorm.  It might be implementable at only the cache
>     manager layer, or might require integration at a higher layer.  It
>     does not do coordinated lock management, but would do reasonable cache
>     invalidation in the cases where the same enitity was not modified in
>     multiple instances in rapid succession.  This is seen as the first
>     phase in a more pessimistic strategy that would do actual lock
>     management.
>
>     It uses the multicast transmission facilities in Java to set up a peer
>     to peer local network on which cache invalidation messages are sent to
>     other servers using the same datastore.  Detected events are
>     broadcast
>     to other servers, allowing them to either hollow the referenced object
>     or remove them completely from cache.  Below are the major pieces of
>     the implementation as I see them.
>
>     I am looking at using JRMS (Java Reliable Multicast Service) jar
>     (http://www.experimentalstuff.com/Technologies/JRMS/index.html) to
>     reasonably guarantee the receipt of packets to each server
>     (specifically the LRMP variety, with late-join catchup disabled).
>     Basic desription of major areas below, followed by questions.
>
>     I would really really like to find a way to do this at the cache
>     manager level, as it would be a simple properties change to enable it,
>     by using a version of the LRU cache that implemented the additional
>     functionality, but I am not sure this is possible.  Feedback
>     appreciated.
>
>     Harry
>
>     Detection:
>     "Broadcast" means notification sent out over multicast
>     Objects that are deleted must be broadcast to be removed
>     Objects that have simple attributes or one way references updated must
>     be broadcast to be hollowed or removed
>     Changes to collections must be broadcast:
>       ObjParent has a collection that contains objects, including
>     ObjChild
>       ObjChild has a direct reference to ObjParent
>       ObjChild has its reference to ObjParent removed (or reassigned to
>     ObjNewParent)
>       ObjChild must be broadcast to be hollowed or removed
>       ObjParent must either be broadcast to be hollowed or removed or
>     must
>     be broadcast have its collection set to unresolved
>       ObjNewParent must either be broadcast to be hollowed or removed or
>     must be broadcast to have its collection set to unresolved
>
>     Transmission:
>     Initial transmission is seen as non locking multicast packets
>     All cache instances are both transmitters and receivers of packets
>     (there is no central server)
>     When a cache instance detects a broadcast event:
>       It formulates a broadcast packet, containing the table name and
>     primary key of the Object
>       (Should this be class instead of table?)
>       It transmits the packet on the multicast address for the group
>       Each cache instance (except the sender) receives the packet
>       Each cache instance purges (or makes hollow?) the referenced object
>       (Does this need to distinguish between hollow vs removal for change
>     vs delete, respectively?)
>       (How are collections handled?)
>
>     Packet format:
>     Packets are datagrams, pssoibly of fixed length:
>       Format: (Action ( Remove vs hollow?) + ) Primary key + Hash of
>     (Class | table name?)
>       (Can we use fixed length for packet?  ie can we reliably hash class
>     or table name?)
>       Packet is fixed length to minimize message size.  Variable length is
>     possible, but seems bad.
>       If using a hash, each cache manager calculates hash value upon entry
>     of (class | table) into cache space.  Unknown hashes are ignored (as
>     they aren't in cache).
>
>     Questions:
>     How do we detect the appropriate broadcast events at the cache
>     manager level?
>     Can multiple VMs on the same host bind to the same multicast group
>     address?  If not, do we need some form of local relay? (Tested to be
>     okay on windows)
>     Do we use table name, class name, or something else for entity
>     representation?
>     Can we use a hash of the entity representation (class or table name)
>     to allow for fixed length packets?
>     Which hash formula do we use?
>     Is there too much multicast traffic on a network of 60 machines? (we
>     can use separate addresses for separate clusters)
>     Are networks (or switches) commonly configured to support local
>     multicast packets?
>
>

Re: [Xorm-devel] Re: Distributed Cache Management

[Harry E. <ha...@gm...>]

NO! to caching persistence capable objects directly.  That is too big
of a change for me to even contemplate right now.

I will take a look at JCache (it seems like an awful lot of folks are
doing cool things with JavaGroups.  Where have I been?).  I think a
map based api would most likely be pretty easy to integrate into LRU
or some derivative, given that it is implemented using a map.

On the Collections front, when I scanned through the code, I think I
found all the places in InterfaceInvocationHandler, InterfaceManager,
and TransactionImpl where xorm was interacting with the DataCache.  I
saw where row structures were being put in, updated and taken out of
the cache that backed the InterfaceInvocationHandlers (TransactionImpl
lines 247-282), but the same areas of code also update Collection
Relationship proxies, and don't seem to make any cache calls
(TransactionImpl lines 225-240).

Even if it did, they probably wouldn't be in cache, given the fact
that even the DataCache interface specifies that anything without a
primary key that you call add() with is, by contract, thrown away.=20
However, it might be worth making the calls for Collection mods, even
if they don't end up getting stored, to give the cache layer a chance
to deal with propagating the changes.  I just need to get a little bit
better understanding of where this info is being held onto in (if at
all), if it isn't the DataCache layer. (Maybe collections are always
thrown away with the InterfaceInvocationHandler, and then requeried?)

Harry

On 10/5/05, Wes Biggs <we...@ca...> wrote:
> Changes to collections are always accomplished by changes to the data
> model.  That being said, I forget if XORM is keeping many-to-many table
> Rows in cache currently -- it would need to for this scheme to work,
> unless you want to change the whole thing and cache the
> persistence-capable objects directly (I don't think this is a good idea).
>
> You might look at JCache.sourceforge.net which uses JavaGroups.  This is
> the approach that some of the commercial JDO vendors have taken --
> integrate a third party distributed cache management solution.  The
> JCache API (it's a JSR) is basically a Map, and the implementation
> supposedly takes care of all the notification and sync behind the
> scenes.  But that might be harder to integrate with the existing
> LRUCache, etc.
>
> Wes
>
> Harry Evans wrote:
>
> > Having spoken with Doug, I am looking at using the JGroups (or
> > JavaGroups, gotta love that Sun trademark enforcement) as the
> > mechanism for transporting the notifications.
> >
> > Thoughts:
> > 1.  Modify LRUCache to optionally notify a listener when a Row is
> > added, updated, or removed.
> > 2.  Subclass LRUCache to use (most likely) a JGroups NotificationBus
> > for cache events.
> > 3.  Setup JGroups to manage this notification, and provide properties
> > the subclass can read to allow for enhanced settings (UDP versus TCP
> > notifications, etc).
> > 4.  I walked through some xorm code, and it doesn't appear that
> > relationship proxy information is ever directly conveyed to the cache
> > level, which means that there is no way, at that level, to invalidate
> > the Collection case I listed previously.  I could really use some
> > insight into an approach for this area (*cough* Wes *cough*) as this
> > seems like a kinda important part.  I might have this wrong, but it
> > appears xorm manages all collection level information (12M and M2M) at
> > the ObjectProxy level, and not at all in the Row DataCache level.  Is
> > that correct?
> > 5.  I am unclear whether InterfaceInvocationHandlers need to have
> > their Row cleared in this scheme, or if they generally pass out of
> > scope frequently enough that the Row is the main thing to focus on.
> > This has to do with the concern that they hold a direct hard reference
> > to a Row object, so throwing that out of cache isn't going to
> > accomplish much if they are still pointing to it.
> >
> >  I am really interested in the JGroups DistributedLockManager class,
> > but it would have to be combined / enhanced with cache management to
> > use it, and it seems like there should be an option, regardless, to
> > only do cache management without the overhead of distributed locks.
> > The JBossCache stuff also might be more appropriate in the long term,
> > as it uses JGroups stuff, but seems like it can handle both locks and
> > caching.  I will have to do more research.
> >
> > Input very much appreciated.
> >
> > Harry
> >
> > On 10/4/05, *Harry Evans* <ha...@gm... <mailto:ha...@gm...>>
> > wrote:
> >
> >     Optimistic Distributed Cache Management
> >
> >     This is a proposal for a simple distributed cache invalidation
> >     strategy for xorm.  It might be implementable at only the cache
> >     manager layer, or might require integration at a higher layer.  It
> >     does not do coordinated lock management, but would do reasonable ca=
che
> >     invalidation in the cases where the same enitity was not modified i=
n
> >     multiple instances in rapid succession.  This is seen as the first
> >     phase in a more pessimistic strategy that would do actual lock
> >     management.
> >
> >     It uses the multicast transmission facilities in Java to set up a p=
eer
> >     to peer local network on which cache invalidation messages are sent=
 to
> >     other servers using the same datastore.  Detected events are
> >     broadcast
> >     to other servers, allowing them to either hollow the referenced obj=
ect
> >     or remove them completely from cache.  Below are the major pieces o=
f
> >     the implementation as I see them.
> >
> >     I am looking at using JRMS (Java Reliable Multicast Service) jar
> >     (http://www.experimentalstuff.com/Technologies/JRMS/index.html) to
> >     reasonably guarantee the receipt of packets to each server
> >     (specifically the LRMP variety, with late-join catchup disabled).
> >     Basic desription of major areas below, followed by questions.
> >
> >     I would really really like to find a way to do this at the cache
> >     manager level, as it would be a simple properties change to enable =
it,
> >     by using a version of the LRU cache that implemented the additional
> >     functionality, but I am not sure this is possible.  Feedback
> >     appreciated.
> >
> >     Harry
> >
> >     Detection:
> >     "Broadcast" means notification sent out over multicast
> >     Objects that are deleted must be broadcast to be removed
> >     Objects that have simple attributes or one way references updated m=
ust
> >     be broadcast to be hollowed or removed
> >     Changes to collections must be broadcast:
> >       ObjParent has a collection that contains objects, including
> >     ObjChild
> >       ObjChild has a direct reference to ObjParent
> >       ObjChild has its reference to ObjParent removed (or reassigned to
> >     ObjNewParent)
> >       ObjChild must be broadcast to be hollowed or removed
> >       ObjParent must either be broadcast to be hollowed or removed or
> >     must
> >     be broadcast have its collection set to unresolved
> >       ObjNewParent must either be broadcast to be hollowed or removed o=
r
> >     must be broadcast to have its collection set to unresolved
> >
> >     Transmission:
> >     Initial transmission is seen as non locking multicast packets
> >     All cache instances are both transmitters and receivers of packets
> >     (there is no central server)
> >     When a cache instance detects a broadcast event:
> >       It formulates a broadcast packet, containing the table name and
> >     primary key of the Object
> >       (Should this be class instead of table?)
> >       It transmits the packet on the multicast address for the group
> >       Each cache instance (except the sender) receives the packet
> >       Each cache instance purges (or makes hollow?) the referenced obje=
ct
> >       (Does this need to distinguish between hollow vs removal for chan=
ge
> >     vs delete, respectively?)
> >       (How are collections handled?)
> >
> >     Packet format:
> >     Packets are datagrams, pssoibly of fixed length:
> >       Format: (Action ( Remove vs hollow?) + ) Primary key + Hash of
> >     (Class | table name?)
> >       (Can we use fixed length for packet?  ie can we reliably hash cla=
ss
> >     or table name?)
> >       Packet is fixed length to minimize message size.  Variable length=
 is
> >     possible, but seems bad.
> >       If using a hash, each cache manager calculates hash value upon en=
try
> >     of (class | table) into cache space.  Unknown hashes are ignored (a=
s
> >     they aren't in cache).
> >
> >     Questions:
> >     How do we detect the appropriate broadcast events at the cache
> >     manager level?
> >     Can multiple VMs on the same host bind to the same multicast group
> >     address?  If not, do we need some form of local relay? (Tested to b=
e
> >     okay on windows)
> >     Do we use table name, class name, or something else for entity
> >     representation?
> >     Can we use a hash of the entity representation (class or table name=
)
> >     to allow for fixed length packets?
> >     Which hash formula do we use?
> >     Is there too much multicast traffic on a network of 60 machines? (w=
e
> >     can use separate addresses for separate clusters)
> >     Are networks (or switches) commonly configured to support local
> >     multicast packets?
> >
> >
>
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by:
> Power Architecture Resource Center: Free content, downloads, discussions,
> and more. http://solutions.newsforge.com/ibmarch.tmpl
> _______________________________________________
> Xorm-devel mailing list
> Xor...@li...
> https://lists.sourceforge.net/lists/listinfo/xorm-devel
>

Re: [Xorm-devel] Re: Distributed Cache Management

[Wes B. <we...@ca...>]

Yes, collection relationships are always thrown away currently.  That's 
because XORM is not smart about bidirectional relationships.  So to add 
distributed caching on top of currently functionality, they don't need 
to be propagated.

Wes

Harry Evans wrote:

>NO! to caching persistence capable objects directly.  That is too big
>of a change for me to even contemplate right now.
>
>I will take a look at JCache (it seems like an awful lot of folks are
>doing cool things with JavaGroups.  Where have I been?).  I think a
>map based api would most likely be pretty easy to integrate into LRU
>or some derivative, given that it is implemented using a map.
>
>On the Collections front, when I scanned through the code, I think I
>found all the places in InterfaceInvocationHandler, InterfaceManager,
>and TransactionImpl where xorm was interacting with the DataCache.  I
>saw where row structures were being put in, updated and taken out of
>the cache that backed the InterfaceInvocationHandlers (TransactionImpl
>lines 247-282), but the same areas of code also update Collection
>Relationship proxies, and don't seem to make any cache calls
>(TransactionImpl lines 225-240).
>
>Even if it did, they probably wouldn't be in cache, given the fact
>that even the DataCache interface specifies that anything without a
>primary key that you call add() with is, by contract, thrown away. 
>However, it might be worth making the calls for Collection mods, even
>if they don't end up getting stored, to give the cache layer a chance
>to deal with propagating the changes.  I just need to get a little bit
>better understanding of where this info is being held onto in (if at
>all), if it isn't the DataCache layer. (Maybe collections are always
>thrown away with the InterfaceInvocationHandler, and then requeried?)
>
>Harry
>
>On 10/5/05, Wes Biggs <we...@ca...> wrote:
>  
>
>>Changes to collections are always accomplished by changes to the data
>>model.  That being said, I forget if XORM is keeping many-to-many table
>>Rows in cache currently -- it would need to for this scheme to work,
>>unless you want to change the whole thing and cache the
>>persistence-capable objects directly (I don't think this is a good idea).
>>
>>You might look at JCache.sourceforge.net which uses JavaGroups.  This is
>>the approach that some of the commercial JDO vendors have taken --
>>integrate a third party distributed cache management solution.  The
>>JCache API (it's a JSR) is basically a Map, and the implementation
>>supposedly takes care of all the notification and sync behind the
>>scenes.  But that might be harder to integrate with the existing
>>LRUCache, etc.
>>
>>Wes
>>
>>Harry Evans wrote:
>>
>>    
>>
>>>Having spoken with Doug, I am looking at using the JGroups (or
>>>JavaGroups, gotta love that Sun trademark enforcement) as the
>>>mechanism for transporting the notifications.
>>>
>>>Thoughts:
>>>1.  Modify LRUCache to optionally notify a listener when a Row is
>>>added, updated, or removed.
>>>2.  Subclass LRUCache to use (most likely) a JGroups NotificationBus
>>>for cache events.
>>>3.  Setup JGroups to manage this notification, and provide properties
>>>the subclass can read to allow for enhanced settings (UDP versus TCP
>>>notifications, etc).
>>>4.  I walked through some xorm code, and it doesn't appear that
>>>relationship proxy information is ever directly conveyed to the cache
>>>level, which means that there is no way, at that level, to invalidate
>>>the Collection case I listed previously.  I could really use some
>>>insight into an approach for this area (*cough* Wes *cough*) as this
>>>seems like a kinda important part.  I might have this wrong, but it
>>>appears xorm manages all collection level information (12M and M2M) at
>>>the ObjectProxy level, and not at all in the Row DataCache level.  Is
>>>that correct?
>>>5.  I am unclear whether InterfaceInvocationHandlers need to have
>>>their Row cleared in this scheme, or if they generally pass out of
>>>scope frequently enough that the Row is the main thing to focus on.
>>>This has to do with the concern that they hold a direct hard reference
>>>to a Row object, so throwing that out of cache isn't going to
>>>accomplish much if they are still pointing to it.
>>>
>>> I am really interested in the JGroups DistributedLockManager class,
>>>but it would have to be combined / enhanced with cache management to
>>>use it, and it seems like there should be an option, regardless, to
>>>only do cache management without the overhead of distributed locks.
>>>The JBossCache stuff also might be more appropriate in the long term,
>>>as it uses JGroups stuff, but seems like it can handle both locks and
>>>caching.  I will have to do more research.
>>>
>>>Input very much appreciated.
>>>
>>>Harry
>>>
>>>On 10/4/05, *Harry Evans* <ha...@gm... <mailto:ha...@gm...>>
>>>wrote:
>>>
>>>    Optimistic Distributed Cache Management
>>>
>>>    This is a proposal for a simple distributed cache invalidation
>>>    strategy for xorm.  It might be implementable at only the cache
>>>    manager layer, or might require integration at a higher layer.  It
>>>    does not do coordinated lock management, but would do reasonable cache
>>>    invalidation in the cases where the same enitity was not modified in
>>>    multiple instances in rapid succession.  This is seen as the first
>>>    phase in a more pessimistic strategy that would do actual lock
>>>    management.
>>>
>>>    It uses the multicast transmission facilities in Java to set up a peer
>>>    to peer local network on which cache invalidation messages are sent to
>>>    other servers using the same datastore.  Detected events are
>>>    broadcast
>>>    to other servers, allowing them to either hollow the referenced object
>>>    or remove them completely from cache.  Below are the major pieces of
>>>    the implementation as I see them.
>>>
>>>    I am looking at using JRMS (Java Reliable Multicast Service) jar
>>>    (http://www.experimentalstuff.com/Technologies/JRMS/index.html) to
>>>    reasonably guarantee the receipt of packets to each server
>>>    (specifically the LRMP variety, with late-join catchup disabled).
>>>    Basic desription of major areas below, followed by questions.
>>>
>>>    I would really really like to find a way to do this at the cache
>>>    manager level, as it would be a simple properties change to enable it,
>>>    by using a version of the LRU cache that implemented the additional
>>>    functionality, but I am not sure this is possible.  Feedback
>>>    appreciated.
>>>
>>>    Harry
>>>
>>>    Detection:
>>>    "Broadcast" means notification sent out over multicast
>>>    Objects that are deleted must be broadcast to be removed
>>>    Objects that have simple attributes or one way references updated must
>>>    be broadcast to be hollowed or removed
>>>    Changes to collections must be broadcast:
>>>      ObjParent has a collection that contains objects, including
>>>    ObjChild
>>>      ObjChild has a direct reference to ObjParent
>>>      ObjChild has its reference to ObjParent removed (or reassigned to
>>>    ObjNewParent)
>>>      ObjChild must be broadcast to be hollowed or removed
>>>      ObjParent must either be broadcast to be hollowed or removed or
>>>    must
>>>    be broadcast have its collection set to unresolved
>>>      ObjNewParent must either be broadcast to be hollowed or removed or
>>>    must be broadcast to have its collection set to unresolved
>>>
>>>    Transmission:
>>>    Initial transmission is seen as non locking multicast packets
>>>    All cache instances are both transmitters and receivers of packets
>>>    (there is no central server)
>>>    When a cache instance detects a broadcast event:
>>>      It formulates a broadcast packet, containing the table name and
>>>    primary key of the Object
>>>      (Should this be class instead of table?)
>>>      It transmits the packet on the multicast address for the group
>>>      Each cache instance (except the sender) receives the packet
>>>      Each cache instance purges (or makes hollow?) the referenced object
>>>      (Does this need to distinguish between hollow vs removal for change
>>>    vs delete, respectively?)
>>>      (How are collections handled?)
>>>
>>>    Packet format:
>>>    Packets are datagrams, pssoibly of fixed length:
>>>      Format: (Action ( Remove vs hollow?) + ) Primary key + Hash of
>>>    (Class | table name?)
>>>      (Can we use fixed length for packet?  ie can we reliably hash class
>>>    or table name?)
>>>      Packet is fixed length to minimize message size.  Variable length is
>>>    possible, but seems bad.
>>>      If using a hash, each cache manager calculates hash value upon entry
>>>    of (class | table) into cache space.  Unknown hashes are ignored (as
>>>    they aren't in cache).
>>>
>>>    Questions:
>>>    How do we detect the appropriate broadcast events at the cache
>>>    manager level?
>>>    Can multiple VMs on the same host bind to the same multicast group
>>>    address?  If not, do we need some form of local relay? (Tested to be
>>>    okay on windows)
>>>    Do we use table name, class name, or something else for entity
>>>    representation?
>>>    Can we use a hash of the entity representation (class or table name)
>>>    to allow for fixed length packets?
>>>    Which hash formula do we use?
>>>    Is there too much multicast traffic on a network of 60 machines? (we
>>>    can use separate addresses for separate clusters)
>>>    Are networks (or switches) commonly configured to support local
>>>    multicast packets?
>>>      
>>>

Re: [Xorm-devel] Re: Distributed Cache Management

[Harry E. <ha...@gm...>]

I took a look at jCache.  It seems to do object coordination in the
cache.  In other words, it actually moves objects around from cache
instance to cache instance, based on demand.  The version I was
looking at doing for Xorm would be coordinated eviction only.  The
differences;
1.  JCache gives a distributed hashtable view of the cache.  If one
vm's cache doesn't have a particular key, but another one does, it
will copy the object over, and give it to the requester.  JCache also
gives you a strategy to load an object from a datasource, if it the
requested key is not currently cached (think Xorm lookups by primary
key for non cached objects).
2.  The approach I was taking was eviction based.  If an object is
changed in VM 1, it would disappear from VM 2' cache, such that a
request from VM 2 would cause a miss, and it would go back to the DB
directly.

The JCache version gives you 1 big cache, with objects being copied
around as needed.  The approach I was taking doesn't pass around
objects, only local cache eviction notices.

The only big worry I have is the amount of data being shipped around
by JCache for all those miss / copy commands.  The equivalent in my
approach is all the eviction notices being shipped around.

I am open to either approach.  I can see a JCache like approach having
the benefit that it is only steps away from also being able to hold
locks on the same objects.  However, I doubt the current
implementation would be malleable enough to do that with on its own.=20
My version doesn't get you any closer to locking, but "might" be more
network friendly.

I could really use some input on the approach to take, as I can see
the value in each, and not enough discriminating factors to choose
either one.

Harry

On 10/5/05, Wes Biggs <we...@ca...> wrote:
> Yes, collection relationships are always thrown away currently.  That's
> because XORM is not smart about bidirectional relationships.  So to add
> distributed caching on top of currently functionality, they don't need
> to be propagated.
>
> Wes
>
> Harry Evans wrote:
>
> >NO! to caching persistence capable objects directly.  That is too big
> >of a change for me to even contemplate right now.
> >
> >I will take a look at JCache (it seems like an awful lot of folks are
> >doing cool things with JavaGroups.  Where have I been?).  I think a
> >map based api would most likely be pretty easy to integrate into LRU
> >or some derivative, given that it is implemented using a map.
> >
> >On the Collections front, when I scanned through the code, I think I
> >found all the places in InterfaceInvocationHandler, InterfaceManager,
> >and TransactionImpl where xorm was interacting with the DataCache.  I
> >saw where row structures were being put in, updated and taken out of
> >the cache that backed the InterfaceInvocationHandlers (TransactionImpl
> >lines 247-282), but the same areas of code also update Collection
> >Relationship proxies, and don't seem to make any cache calls
> >(TransactionImpl lines 225-240).
> >
> >Even if it did, they probably wouldn't be in cache, given the fact
> >that even the DataCache interface specifies that anything without a
> >primary key that you call add() with is, by contract, thrown away.
> >However, it might be worth making the calls for Collection mods, even
> >if they don't end up getting stored, to give the cache layer a chance
> >to deal with propagating the changes.  I just need to get a little bit
> >better understanding of where this info is being held onto in (if at
> >all), if it isn't the DataCache layer. (Maybe collections are always
> >thrown away with the InterfaceInvocationHandler, and then requeried?)
> >
> >Harry
> >
> >On 10/5/05, Wes Biggs <we...@ca...> wrote:
> >
> >
> >>Changes to collections are always accomplished by changes to the data
> >>model.  That being said, I forget if XORM is keeping many-to-many table
> >>Rows in cache currently -- it would need to for this scheme to work,
> >>unless you want to change the whole thing and cache the
> >>persistence-capable objects directly (I don't think this is a good idea=
).
> >>
> >>You might look at JCache.sourceforge.net which uses JavaGroups.  This i=
s
> >>the approach that some of the commercial JDO vendors have taken --
> >>integrate a third party distributed cache management solution.  The
> >>JCache API (it's a JSR) is basically a Map, and the implementation
> >>supposedly takes care of all the notification and sync behind the
> >>scenes.  But that might be harder to integrate with the existing
> >>LRUCache, etc.
> >>
> >>Wes
> >>
> >>Harry Evans wrote:
> >>
> >>
> >>
> >>>Having spoken with Doug, I am looking at using the JGroups (or
> >>>JavaGroups, gotta love that Sun trademark enforcement) as the
> >>>mechanism for transporting the notifications.
> >>>
> >>>Thoughts:
> >>>1.  Modify LRUCache to optionally notify a listener when a Row is
> >>>added, updated, or removed.
> >>>2.  Subclass LRUCache to use (most likely) a JGroups NotificationBus
> >>>for cache events.
> >>>3.  Setup JGroups to manage this notification, and provide properties
> >>>the subclass can read to allow for enhanced settings (UDP versus TCP
> >>>notifications, etc).
> >>>4.  I walked through some xorm code, and it doesn't appear that
> >>>relationship proxy information is ever directly conveyed to the cache
> >>>level, which means that there is no way, at that level, to invalidate
> >>>the Collection case I listed previously.  I could really use some
> >>>insight into an approach for this area (*cough* Wes *cough*) as this
> >>>seems like a kinda important part.  I might have this wrong, but it
> >>>appears xorm manages all collection level information (12M and M2M) at
> >>>the ObjectProxy level, and not at all in the Row DataCache level.  Is
> >>>that correct?
> >>>5.  I am unclear whether InterfaceInvocationHandlers need to have
> >>>their Row cleared in this scheme, or if they generally pass out of
> >>>scope frequently enough that the Row is the main thing to focus on.
> >>>This has to do with the concern that they hold a direct hard reference
> >>>to a Row object, so throwing that out of cache isn't going to
> >>>accomplish much if they are still pointing to it.
> >>>
> >>> I am really interested in the JGroups DistributedLockManager class,
> >>>but it would have to be combined / enhanced with cache management to
> >>>use it, and it seems like there should be an option, regardless, to
> >>>only do cache management without the overhead of distributed locks.
> >>>The JBossCache stuff also might be more appropriate in the long term,
> >>>as it uses JGroups stuff, but seems like it can handle both locks and
> >>>caching.  I will have to do more research.
> >>>
> >>>Input very much appreciated.
> >>>
> >>>Harry
> >>>
> >>>On 10/4/05, *Harry Evans* <ha...@gm... <mailto:ha...@gm...>>
> >>>wrote:
> >>>
> >>>    Optimistic Distributed Cache Management
> >>>
> >>>    This is a proposal for a simple distributed cache invalidation
> >>>    strategy for xorm.  It might be implementable at only the cache
> >>>    manager layer, or might require integration at a higher layer.  It
> >>>    does not do coordinated lock management, but would do reasonable c=
ache
> >>>    invalidation in the cases where the same enitity was not modified =
in
> >>>    multiple instances in rapid succession.  This is seen as the first
> >>>    phase in a more pessimistic strategy that would do actual lock
> >>>    management.
> >>>
> >>>    It uses the multicast transmission facilities in Java to set up a =
peer
> >>>    to peer local network on which cache invalidation messages are sen=
t to
> >>>    other servers using the same datastore.  Detected events are
> >>>    broadcast
> >>>    to other servers, allowing them to either hollow the referenced ob=
ject
> >>>    or remove them completely from cache.  Below are the major pieces =
of
> >>>    the implementation as I see them.
> >>>
> >>>    I am looking at using JRMS (Java Reliable Multicast Service) jar
> >>>    (http://www.experimentalstuff.com/Technologies/JRMS/index.html) to
> >>>    reasonably guarantee the receipt of packets to each server
> >>>    (specifically the LRMP variety, with late-join catchup disabled).
> >>>    Basic desription of major areas below, followed by questions.
> >>>
> >>>    I would really really like to find a way to do this at the cache
> >>>    manager level, as it would be a simple properties change to enable=
 it,
> >>>    by using a version of the LRU cache that implemented the additiona=
l
> >>>    functionality, but I am not sure this is possible.  Feedback
> >>>    appreciated.
> >>>
> >>>    Harry
> >>>
> >>>    Detection:
> >>>    "Broadcast" means notification sent out over multicast
> >>>    Objects that are deleted must be broadcast to be removed
> >>>    Objects that have simple attributes or one way references updated =
must
> >>>    be broadcast to be hollowed or removed
> >>>    Changes to collections must be broadcast:
> >>>      ObjParent has a collection that contains objects, including
> >>>    ObjChild
> >>>      ObjChild has a direct reference to ObjParent
> >>>      ObjChild has its reference to ObjParent removed (or reassigned t=
o
> >>>    ObjNewParent)
> >>>      ObjChild must be broadcast to be hollowed or removed
> >>>      ObjParent must either be broadcast to be hollowed or removed or
> >>>    must
> >>>    be broadcast have its collection set to unresolved
> >>>      ObjNewParent must either be broadcast to be hollowed or removed =
or
> >>>    must be broadcast to have its collection set to unresolved
> >>>
> >>>    Transmission:
> >>>    Initial transmission is seen as non locking multicast packets
> >>>    All cache instances are both transmitters and receivers of packets
> >>>    (there is no central server)
> >>>    When a cache instance detects a broadcast event:
> >>>      It formulates a broadcast packet, containing the table name and
> >>>    primary key of the Object
> >>>      (Should this be class instead of table?)
> >>>      It transmits the packet on the multicast address for the group
> >>>      Each cache instance (except the sender) receives the packet
> >>>      Each cache instance purges (or makes hollow?) the referenced obj=
ect
> >>>      (Does this need to distinguish between hollow vs removal for cha=
nge
> >>>    vs delete, respectively?)
> >>>      (How are collections handled?)
> >>>
> >>>    Packet format:
> >>>    Packets are datagrams, pssoibly of fixed length:
> >>>      Format: (Action ( Remove vs hollow?) + ) Primary key + Hash of
> >>>    (Class | table name?)
> >>>      (Can we use fixed length for packet?  ie can we reliably hash cl=
ass
> >>>    or table name?)
> >>>      Packet is fixed length to minimize message size.  Variable lengt=
h is
> >>>    possible, but seems bad.
> >>>      If using a hash, each cache manager calculates hash value upon e=
ntry
> >>>    of (class | table) into cache space.  Unknown hashes are ignored (=
as
> >>>    they aren't in cache).
> >>>
> >>>    Questions:
> >>>    How do we detect the appropriate broadcast events at the cache
> >>>    manager level?
> >>>    Can multiple VMs on the same host bind to the same multicast group
> >>>    address?  If not, do we need some form of local relay? (Tested to =
be
> >>>    okay on windows)
> >>>    Do we use table name, class name, or something else for entity
> >>>    representation?
> >>>    Can we use a hash of the entity representation (class or table nam=
e)
> >>>    to allow for fixed length packets?
> >>>    Which hash formula do we use?
> >>>    Is there too much multicast traffic on a network of 60 machines? (=
we
> >>>    can use separate addresses for separate clusters)
> >>>    Are networks (or switches) commonly configured to support local
> >>>    multicast packets?
> >>>
> >>>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by:
> Power Architecture Resource Center: Free content, downloads, discussions,
> and more. http://solutions.newsforge.com/ibmarch.tmpl
> _______________________________________________
> Xorm-devel mailing list
> Xor...@li...
> https://lists.sourceforge.net/lists/listinfo/xorm-devel
>

Re: [Xorm-devel] Re: Distributed Cache Management

[Wes B. <we...@ca...>]

Harry Evans wrote:

>The JCache version gives you 1 big cache, with objects being copied
>around as needed.
>
That's sort of annoying, especially if you have applications that use 
partially (but not fully) overlapping sets of tables.  I suppose it's an 
implementation detail on the cache layer, but I agree it doesn't sound 
that flexible for all kinds of distributed JDO systems.

> 
>I could really use some input on the approach to take, as I can see
>the value in each, and not enough discriminating factors to choose
>either one.
>  
>
Thanks for doing the analysis.  I think that an eviction-notice-only 
approach would be a good lightweight starting point.

XORM's all about doing things differently, anyway.

Wes

Re: [Xorm-devel] Distributed Cache Management

[Wes B. <we...@ca...>]

This looks really good, thanks for the writeup.

Two quick comments:
1) This isn't cache management, it's lock management.. I'm guessing the
existing CacheManager implementation would be the component that
interacts with this, though.

2) We might want to see if the javax.transaction.xa interfaces map well
to this in terms of a standardized API.  I'll look at the Javadoc and
see if I can find a fit.

Cheers,

Wes

Harry Evans wrote:

> I have a simple design for distributed cache management.  It has a 
> couple of holes, but doesn't seem like a bad first pass at it.  I 
> would like to get feedback from the group to see what others think.
>
> CacheManager methods (don't worry about local vs. remote right now)
> -lock(Object o, transactionId) : Lock
> Checks version of object, and attempts to create a lock object for 
> it.  Version is either some kind of change counter, or some time kind 
> of time stamp of last update time in the datastore.
> --o
> The xorm managed object to lock
> --transactionId
> a relatively unique id for the transaction obtaining the lock.  Used 
> to allow the same transaction to acquire a lock on the same object 
> more than once.
> --returns
> If the object is already locked, returns the special LOCK_FAILED Lock 
> object. (An overridden version of the method might allow the caller to 
> specify waiting until the lock is obtained)
> If a lock is granted, returns a Lock object with status LOCK_SUCCESS, 
> and a valid lock expiration time.
> If a lock is granted, but the object is stale, returns a Lock object 
> with LOCK_REFRESH, and a valid lock expiration time.  This means that 
> the caller has obtained a lock on the object, but needs to refresh the 
> object with the data store.
>
>
> -commit(Object o, transactionId) : Lock
> Verifies that this object is has a lock held by the transactionId.  If 
> so, it will update the version in the Cache Manager to match the 
> version held by the object.  This method should be called AFTER a 
> change to the datastore, but before committing the changes to the 
> datastore.  This method also depends on the version of the object 
> being updated based on the changes to the datastore, even though it is 
> not yet committed.  Regardless of outcome, this method will result in 
> releasing the lock on this object in the CacheManager.
> --o
> The xorm managed object that a lock was previously obtained for
> --transactionId
> the transactionId that was used to obtain the lock for o
> --returns
> If there is a lock for this object, and the transactionId matches, and 
> the lock has not expired, returns COMMIT_SUCCESS
> Otherwise if the lock does not exist, is not for this transactionId, 
> or has expired, returns COMMIT_FAILURE.  The proper action on a 
> COMMIT_FAILURE might be a retry, or a rollback.
>
> -check(Object o) : boolean
> Checks whether the object is the most current version of the object 
> seen by the CacheManager.
> --o
> The object to check
> --returns
> Returns true if this version of the object is the most recent seen by 
> the CacheManager
> otherwise false.
>
> -refreshLock(Object o, transactionId) : LOCK
> Checks for a valid lock on the object, and if all is good, extends the 
> expiration of the Lock.  Might not be needed (could just be rolled 
> into lock method).
> --return values as per lock() method.
>
>
> The concept behind all this is that the CacheManager object does not 
> talk to the db, but simply deals with objects as it sees them.  Given 
> a compliant local implementation of XORM, all objects would be seen 
> whenever a transaction was involved, so there shouldn't be a race 
> condition issue.  A simple sequence diagram follows:
>
> XORM                    CacheManager                DataStore
> ----                    ------------                ---------
>  |                           |                          |
> | |---startTrans()--         |                          |
> | |                |         |                          |
> | |<----------------         |                          |
> | |                          |                          |
> | |---lock(o, tId)--------->| |                         |
> | |<----------Lock object---| |                         |
> | |                          |                          |
> | |---refreshObject(o) (if stale)--------------------->| |
> | |<---------------------------------refreshedObject---| |
> | |                          |                          |
> | |---make changes--         |                          |
> | |                |         |                          |
> | |<----------------         |                          |
> | |---commitTrans() (begin)- |                          |
> | |                        | |                          |
> | |<------------------------ |                          |
> | |                          |                          |
> | |---makeUpdates (don't commit)---------------------->| |
> | |<-----------------------------------------success---| |
> | |                          |                         | |
> | |---commit(o, tid)------->| |                        | |
> | |<--------------success---| |                        | |
> | |                          |                         | |
> | |---commitTransaction------------------------------->| |
> | |<-----------------------------------------success---| |
> | |                          |                          |
> | |---commitTrans() (end)-   |                          |
> | |                      |   |                          |
> | |<----------------------   |                          |
>  |                           |                          |
>
> I think the whole scheme might work, if you assume that no one 
> operates outside of it.  I don't know if that is a reasonable caveat 
> or not.  Obviously, it is possible for the distributed cache manager 
> to provide methods for locking or committing more than one object at 
> the same time.  Also, it is assumes that the distributed cache manager 
> would most likely be "remote" from the XORM instance using it, so the 
> "local" methods I showed should be assumed to be local facades to a 
> remote distributed cache manager.  I tis probably obvious (but worth 
> stating)  that all XORM instances dealing with a particular datastore 
> would use the same CacheManager, possibly with CacheManager 
> replication for failover.  Persistent connections, and good 
> communications design should make it relatively low overhead data wise.
>
> The Lock object as it is internally held by the CacheManager might 
> look something like this:
> Lock
> ----
> class : Class
> id : objectPrimaryKey
> version : Version (int, timestamp, etc)
> txnId : TransactionId (int, long, etc)
> expires: Time (long, Date, etc)
>
> For objects that have been seen, but are not currently locked, txnId 
> and expires would be null.  For objects that are locked, all values 
> would be non null.  Locks can be removed lazily, by checking a lock 
> when needed, and if it is expired, nulling txnId and expires.
>
> This might be way off base, but it seemed like a good first pass.  I 
> am not trying to reinvent the wheel here, but it seems like something 
> like this might solve most of the "coordinated" or "distributed" cache 
> management issues we see with xorm right now.
>
> Thoughts / comments / questions?
>
> Harry 

Re: [Xorm-devel] Distributed Cache Management

[Wes B. <we...@ca...>]

Harry Evans wrote:

> For objects that have been seen, but are not currently locked, txnId 
> and expires would be null.  For objects that are locked, all values 
> would be non null.  Locks can be removed lazily, by checking a lock 
> when needed, and if it is expired, nulling txnId and expires.

It's OK for the lock manager to not keep strong references to every 
object it has seen, right?

Also, should the lock manager be informed of newly persisted objects 
(i.e. creates) or is it only concerned with updates?  I think it's the 
latter but wanted to make sure I was understanding the whole scheme.

Some good news for caching in general -- JDO 2.0 will provide a 
PersistenceManagerFactory.getDataStoreCache() method that can be used to 
pin or evict objects, extents, or all cached data.

I think the next thing to document (again, in PDF would be best) is the 
full interaction diagram between a user-level transaction, XORM's 
DataCache (LRUCache) and the LockManager.

Wes