[Xorm-devel] Distributed Cache Management

[Harry E. <ha...@gm...>]

Optimistic Distributed Cache Management

This is a proposal for a simple distributed cache invalidation
strategy for xorm.  It might be implementable at only the cache
manager layer, or might require integration at a higher layer.  It
does not do coordinated lock management, but would do reasonable cache
invalidation in the cases where the same enitity was not modified in
multiple instances in rapid succession.  This is seen as the first
phase in a more pessimistic strategy that would do actual lock
management.

It uses the multicast transmission facilities in Java to set up a peer
to peer local network on which cache invalidation messages are sent to
other servers using the same datastore.  Detected events are broadcast
to other servers, allowing them to either hollow the referenced object
or remove them completely from cache.  Below are the major pieces of
the implementation as I see them.

I am looking at using JRMS (Java Reliable Multicast Service) jar
(http://www.experimentalstuff.com/Technologies/JRMS/index.html) to
reasonably guarantee the receipt of packets to each server
(specifically the LRMP variety, with late-join catchup disabled).=20
Basic desription of major areas below, followed by questions.

I would really really like to find a way to do this at the cache
manager level, as it would be a simple properties change to enable it,
by using a version of the LRU cache that implemented the additional
functionality, but I am not sure this is possible.  Feedback
appreciated.

Harry

Detection:
"Broadcast" means notification sent out over multicast
Objects that are deleted must be broadcast to be removed
Objects that have simple attributes or one way references updated must
be broadcast to be hollowed or removed
Changes to collections must be broadcast:
  ObjParent has a collection that contains objects, including ObjChild
  ObjChild has a direct reference to ObjParent
  ObjChild has its reference to ObjParent removed (or reassigned to
ObjNewParent)
  ObjChild must be broadcast to be hollowed or removed
  ObjParent must either be broadcast to be hollowed or removed or must
be broadcast have its collection set to unresolved
  ObjNewParent must either be broadcast to be hollowed or removed or
must be broadcast to have its collection set to unresolved

Transmission:
Initial transmission is seen as non locking multicast packets
All cache instances are both transmitters and receivers of packets
(there is no central server)
When a cache instance detects a broadcast event:
  It formulates a broadcast packet, containing the table name and
primary key of the Object
  (Should this be class instead of table?)
  It transmits the packet on the multicast address for the group
  Each cache instance (except the sender) receives the packet
  Each cache instance purges (or makes hollow?) the referenced object
  (Does this need to distinguish between hollow vs removal for change
vs delete, respectively?)
  (How are collections handled?)

Packet format:
Packets are datagrams, pssoibly of fixed length:
  Format: (Action ( Remove vs hollow?) + ) Primary key + Hash of
(Class | table name?)
  (Can we use fixed length for packet?  ie can we reliably hash class
or table name?)
  Packet is fixed length to minimize message size.  Variable length is
possible, but seems bad.
  If using a hash, each cache manager calculates hash value upon entry
of (class | table) into cache space.  Unknown hashes are ignored (as
they aren't in cache).

Questions:
How do we detect the appropriate broadcast events at the cache manager leve=
l?
Can multiple VMs on the same host bind to the same multicast group
address?  If not, do we need some form of local relay? (Tested to be
okay on windows)
Do we use table name, class name, or something else for entity representati=
on?
Can we use a hash of the entity representation (class or table name)
to allow for fixed length packets?
Which hash formula do we use?
Is there too much multicast traffic on a network of 60 machines? (we
can use separate addresses for separate clusters)
Are networks (or switches) commonly configured to support local
multicast packets?