[ZooLib-dev] ZAssetRep_Std_XX assertions diagnosis

[Andrew G. <ag...@em...>]

We've had reports of assertions tripping in ZAssetRep_Std_XX when used
under load.

First some background on ZRef and ZRefCountedWithFinalization.

ZRef is a template, and thus must be parameterized by some type. An
instance holds a pointer to an object of that type. Currently the type
is always a subclass of either ZRefCounted or
ZRefCountedWithFinalization. However, any class that has
implementations of the static methods sIncRefCount, sDecRefCount, 
sCheckAccessRef, sCheckAccess and sCheckAccessEnabled can be used as
the type parameter to a ZRef. [Ultimately, or sooner, I plan to allow
for ZRef-ing of arbitrary types, eg CTabHandle, through the use of
traits templates.]

When a ZRef<X> is constructed, copy-constructed, destroyed or assigned
the refcounts on the object or objects involved are manipulated
appropriately. So the key methods for our purposes are sIncReCount and
sDecRefCount.

For ZRefCounted-derived classes, sIncReCount and sDecRefCount call
ZThread_SafeInc and ZThread_SafeDecAndTest respectively. Those methods
are themselves defined in terms of ZAtomic_XX on platforms/runtime
environments where preemption may occur, and expand to simple ++ and
-- operators otherwise (classic MacOS basically). If sDecRefCount's
call of ZThread_SafeDecAndTest returns true this indicates that the
refcount has hit zero and so the pointed-to object is deleted.

For ZRefCountedWithFinalization-derived classes, sIncReCount and
sDecRefCount are a little trickier because they have to do slightly
more than just atomically increment/decrement a count. On
non-preemptive platforms they can just use regular arithmetic
operations. On preemptive platforms they acquire a static mutex.

For sIncRefCount if the resulting refcount is 1 (one) then the
pointed-to object's virtual Initialize method is called. The increment
is protected by the static mutex, but the call to Initialize is not.

For sDecRefCount if the resulting refcount is zero then it is changed
back to one (hence the need for the mutex, there's no equivalent
atomic-op) and the object's virtual Finalize method is called -- again
the call is not protected by the mutex, just the funky
subtract-one-if-not-already-one-and-return-true-if-it-was-one
operation.

You'll note that the result is that the object now has a refcount of
one but we *know* that there are no extant ZRef's to it. There may
however be pointer(s) to the object, and in fact this is the crux.

If some other entity keeps pointers to objects that are
ZRefCountedWithFinalization descendants we have a way to cleanly find
out when the last ZRef to each of those objects goes away and can take
an action other than simple deletion when that occurs. We use this
technique in ZBTreeNodeSpace to cache nodes.

Because the now non-ZRefed object has a refcount of one, any other
thread that might somehow acquire a ZRef to it (in the example I'm
using by calling ZDBTreeNodeSpace to get a node) will take the
refcount from one to two -- it thus will not call the object's
initialize method. And that's fine, because we haven't actually
finished finalizing it.

If the fetch of such an object (potentially taking the refcount from
one to two) does so under the protection of some shared mutex, and if
the Finalize method of an object acquires the same shared mutex, then
checking GetRefCount's result against one tells us if some other
thread ducked in front of us and effectively revived the object. So
the finalization action should acquire this shared mutex, and only if
the refcount is one should the finalization action(s) occur (in our
example move it off the in-use nodes list and on to the recently-used
nodes list).

Hopefully that all makes sense.


What's happening with ZAssetRep_Std_Data_XX and ZAssetRep_Std_Union?

Well there are several situations, but they all boil down to the fact
that the asset reps are all created when the ZAssetTree is first
opened -- the parse of the asset tree's table of contents builds a
tree of ZAssetReps, and that tree is what is walked to locate an asset
given some path. We don't actually load the data for the assets -- the
idea is that we only keep in memory data that's actually in use.

You'll remember that ZAssetTree itself is derived from
ZRefCountedWithFinalization, but that you as the user do not need to
keep a ZRef to it. Simply having a ZAsset (containing a
ZRef<ZAssetRep>) will keep the asset tree loaded. When the last asset
from the tree goes out of scope the tree itself is unloaded
(un-mmaping it, or releasing it's file or streamer depending on its
implementation).

So a ZAssetRep with a greater-than-zero refcount must have a ZRef to
its asset tree, but a ZAssetRep with a zero refcount must not. This is
where the Initialize/Finalize stuff comes in. When a ZAssetRep's
refcount goes from zero to one it assigns the pointer to its asset
tree that it was constructed with to its field fRef_AssetTree (of type
ZRef<ZAssetTree>). When a ZAssetRep's refcount goes from one to zero
it un-assigns its fRef_AssetTree field, thus taking the refcount on
the asset tree down by one.

Additionally some ZAssetRep derivatives have additional data that they
need to keep loaded whilst they're in use, but should unload when
they're no longer being referenced. A ZAssetRep that's had its GetData
method called may have loaded its data from an underlying file or
stream -- so long as the rep is in use that data should stick around,
so that callers of GetData can rely on the data staying in the same
place. (ZAssetReps that come from trees that are memory-mapped of
course don't need to do this). A ZAssetRep that is a union when asked
for a child must walk the list of asset reps that it references and
build an overlay asset rep. It's relatively costly to do this
resolution, so it does it the first time it needs to and then holds on
to the result.

As you can see, asset reps may need to take several steps when they're
Initialized or Finalized (when their refcount goes from zero to one,
or from one to zero). And those steps need to be atomic with respect
to another potential Initialization/Finalization on the same object
that may be happening in another thread, or atomic with respect to the
fetch of data or resolution of unioned assets. That atomicity is
provided by the ZMutex returned by the asset's asset tree's GetMutex
method.

The ZRefCountedWithFinalization sIncRefCount and sDecRefCount methods
provide the basic mechanism for ensuring that a single object can
transfer protection of its lifetime from the refcount value to some
arbitrary protection mechanism, without requiring that that mechanism
be a mutex accessible to ZRef itself. There does need to be a
mechanism visible to the refcounted objects themselves, and that
mechanism must be shared by all refcounted objects that participate in
the same scheme of overall ownership The key here is that anytime a
pointer to such an object is converted to a ZRef to that object there
is the potential that it's being taken from a refcount of zero to one
and *that* occurrence must be protected by the same mechanism that the
protects finalization of the object.

And that's what's *not* happening in ZAssetRep_Std currently. The
shared mutex is acquired in Finalize, which is okay, but it is also
being acquired in Initialize, which is not correct. The mutex needs to
be acquired by the code that is doing the pointer-->ZRef assignment
before the assigment actually happens. I'll be checking in new code
that does this shortly -- hopefuly in time for use on Monday.

A
-- 
Andrew Green                           mailto:ag...@em...
Electric Magic Co.           Vox/Fax: +1 (408) 907 2101