Re: [Wonder-disc] Deadlock using EOObjectStore.invalidateObjectsWithGlobalIDs(NSArray)

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 454-5900

> 	On application load, I get deadlocks on  
> "_invalidateObjectsWithGlobalIDs(editingContext, globalIDs)" line.  
> Unfortunately, I couldn't get the stack trace of the thread: A  
> "kill -3 <pid>" is not enought, I must perform a "kill -9 <pid>" to  
> stop the application.
>
> 	So, here is my question: is there a safer way to invalidate  
> globalIDs rather than using "editingContext.parentObjectStore 
> ().invalidateObjectsWithGlobalIDs(globalIDs);" with properly locked  
> context and object store? Any hints or ideas to avoid these deadlocks?
>

I did try ec.rootObjectStore.invalidate... with correct locking and  
also got deadlocks. I was able to get some Thread dumps. In almost  
all cases, the SharedEC was involved. Sample at the bottom of the  
message.

Even when the sharedEC was disabled, I got IllegalStateExceptions. I  
posted a message concerning that a few days ago regarding that:

> Somehow it appears impossible to me to go the "invalidate global  
> IDs" route because I'm getting IllegalStateExceptions during  
> saveChanges (see bottom of post). I made some reproduction code  
> which reproduces the problem quite reliably (also see bottom of post).
>
> I tried tons of alternatives for the locking (ec.lockObjectectStore 
> () instead ec.rootObjectStore().lock(), getting down to the  
> database context and locking it as well, ...) but the problem  
> remains. Also tried EODatabase.forgetSnapshotForGlobalID with the  
> same result.
>
> Before I finally give up on the subject, could someone with a  
> deeper understanding of the locking stuff (Chuck?) take a look at  
> it. I think it should work this way. If I'm not missing something,  
> I currently see no way in getting any kind of change notification  
> framework work reliably with the "invalidate global IDs" approach.  
> And the "snapshot" approach is already known for causing deadlocks.
>
> Locally I'm testing on Intel/Tiger/WO 5.3.1, but have seen the same  
> Exceptions on PPC/Panther Server/WO 5.2.3.
>
> Any help would be greatly appreciated.
>
> Timo
>
> PS: As Apple is going out of the tools business for WO and wants to  
> concentrate more on the framework, I *really* hope they take a deep  
> look into EOF concurrency and cross-VM cache synchronization. I  
> guess that will remain a wish but would make life soooooo much  
> easier...
>
> PPS:
>
> The meat of the code is:
>
> a) in some thread read and write data concurrently (with correct  
> locking)
>
> b) simulate incoming change notifications in another tread
>
> EOEditingContext ec = new EOEditingContext();
> try {
> 	ec.lock();
> 	ec.rootObjectStore().lock();
> 	ec.rootObjectStore().invalidateAllObjects(); // just for testing,  
> real code would only invalidate certain GIDs
> } finally {
> 	ec.rootObjectStore().unlock();
> 	ec.unlock();
> 	ec.dispose();
> }
>
> Result is e.g.:
>
> java.lang.IllegalStateException:
> rowDiffsForAttributes:
> snapshot in com.webobjects.eoaccess.EODatabaseOperation {
> _dbSnapshot = {};
> _entity = "TestEntity";
> _newRow = {id = 45; testValue = "Thread-2/2200"; lastUpdate =  
> 2006-09-13 15:12:40 Etc/GMT; };
> _object = "{values = {lastUpdate = 2006-09-13 15:12:40 Etc/GMT;  
> testValue = "Thread-2/2200"; };
> this = "<timo.testet.TestEntity 5b8e6c _EOIntegralKeyGlobalID 
> [TestEntity (java.lang.Integer)45]>"; }";
> _globalID = _EOIntegralKeyGlobalID[TestEntity (java.lang.Integer)45];
> _databaseOperator = "EODatabaseUpdateOperator";
> }
> does not contain value for attribute named id with snapshot key: id
>
>

Sample of deadlock with sharedEC involved while invalidating objects  
on rootObjectStore:

> "ActiveMQ Session Task" daemon prio=7 tid=0x005d7bb0 nid=0x30ad8e00  
> in Object.wait() [f181e000..f1820b20]
> 	at java.lang.Object.wait(Native Method)
> 	- waiting on <0x999cb18> (a  
> com.webobjects.foundation.NSMultiReaderLock$ConditionLock)
> 	at java.lang.Object.wait(Object.java:429)
> 	at com.webobjects.foundation.NSMultiReaderLock$ConditionLock.await 
> (NSMultiReaderLock.java:505)
> 	- locked <0x999cb18> (a com.webobjects.foundation.NSMultiReaderLock 
> $ConditionLock)
> 	at com.webobjects.foundation.NSMultiReaderLock._lockForWriting 
> (NSMultiReaderLock.java:204)
> 	at com.webobjects.foundation.NSMultiReaderLock.lockForWriting 
> (NSMultiReaderLock.java:165)
> 	at com.webobjects.eocontrol.EOSharedEditingContext.lock 
> (EOSharedEditingContext.java:700)
> 	at com.webobjects.eocontrol.EOEditingContext.lockObjectStore 
> (EOEditingContext.java:4673)
> 	at com.webobjects.eocontrol.EOEditingContext.refaultObject 
> (EOEditingContext.java:3980)
> 	at er.extensions.ERXEC.refaultObject(ERXEC.java:753)
> 	at  
> com.webobjects.eocontrol.EOEditingContext._refaultObjectWithGlobalID 
> (EOEditingContext.java:3232)
> 	at  
> com.webobjects.eocontrol.EOEditingContext._newChangesFromInvalidatingO 
> bjectsWithGlobalIDs(EOEditingContext.java:3461)
> 	at  
> com.webobjects.eocontrol.EOEditingContext._processObjectStoreChanges 
> (EOEditingContext.java:3497)
> 	at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke 
> (DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:324)
> 	at com.webobjects.foundation.NSSelector.invoke(NSSelector.java:354)
> 	at com.webobjects.foundation.NSSelector._safeInvokeSelector 
> (NSSelector.java:108)
> 	at  
> com.webobjects.eocontrol.EOEditingContext._processNotificationQueue 
> (EOEditingContext.java:4750)
> 	at com.webobjects.eocontrol.EOEditingContext.lock 
> (EOEditingContext.java:4627)
> 	at er.extensions.ERXEC.lock(ERXEC.java:259)
> 	at com.webobjects.eocontrol.EOEditingContext.tryLock 
> (EOEditingContext.java:4639)
> 	at  
> com.webobjects.eocontrol.EOEditingContext._sendOrEnqueueNotification 
> (EOEditingContext.java:4714)
> 	at com.webobjects.eocontrol.EOEditingContext._objectsChangedInStore 
> (EOEditingContext.java:3537)
> 	at er.extensions.ERXEC._objectsChangedInStore(ERXEC.java:919)
> 	at sun.reflect.GeneratedMethodAccessor88.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke 
> (DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:324)
> 	at com.webobjects.foundation.NSSelector._safeInvokeMethod 
> (NSSelector.java:120)
> 	at com.webobjects.foundation.NSNotificationCenter 
> $_Entry.invokeMethod(NSNotificationCenter.java:598)
> 	at com.webobjects.foundation.NSNotificationCenter.postNotification 
> (NSNotificationCenter.java:542)
> 	at com.webobjects.foundation.NSNotificationCenter.postNotification 
> (NSNotificationCenter.java:572)
> 	at  
> com.webobjects.eocontrol.EOObjectStoreCoordinator._objectsChangedInSub 
> Store(EOObjectStoreCoordinator.java:744)
> 	at sun.reflect.GeneratedMethodAccessor94.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke 
> (DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:324)
> 	at com.webobjects.foundation.NSSelector._safeInvokeMethod 
> (NSSelector.java:120)
> 	at com.webobjects.foundation.NSNotificationCenter 
> $_Entry.invokeMethod(NSNotificationCenter.java:598)
> 	at com.webobjects.foundation.NSNotificationCenter.postNotification 
> (NSNotificationCenter.java:542)
> 	at com.webobjects.foundation.NSNotificationCenter.postNotification 
> (NSNotificationCenter.java:572)
> 	at  
> com.webobjects.eoaccess.EODatabaseContext._snapshotsChangedInDatabase( 
> EODatabaseContext.java:3812)
> 	at sun.reflect.GeneratedMethodAccessor93.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke 
> (DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:324)
> 	at com.webobjects.foundation.NSSelector._safeInvokeMethod 
> (NSSelector.java:120)
> 	at com.webobjects.foundation.NSNotificationCenter 
> $_Entry.invokeMethod(NSNotificationCenter.java:598)
> 	at com.webobjects.foundation.NSNotificationCenter.postNotification 
> (NSNotificationCenter.java:542)
> 	at com.webobjects.foundation.NSNotificationCenter.postNotification 
> (NSNotificationCenter.java:572)
> 	at com.webobjects.eoaccess.EODatabase.forgetSnapshotsForGlobalIDs 
> (EODatabase.java:951)
> 	at  
> com.webobjects.eoaccess.EODatabaseContext.forgetSnapshotsForGlobalIDs( 
> EODatabaseContext.java:2397)
> 	at  
> com.webobjects.eoaccess.EODatabaseContext.invalidateObjectsWithGlobalI 
> Ds(EODatabaseContext.java:3781)
> 	at  
> com.webobjects.eocontrol.EOObjectStoreCoordinator.invalidateObjectsWit 
> hGlobalIDs(EOObjectStoreCoordinator.java:718)
> 	at er.changenotification.ERCNSubscriber._processUpdates 
> (ERCNSubscriber.java:166)
> 	at er.changenotification.ERCNSubscriber.onMessage 
> (ERCNSubscriber.java:103)
> 	at org.apache.activemq.ActiveMQMessageConsumer.dispatch 
> (ActiveMQMessageConsumer.java:795)
> 	at org.apache.activemq.ActiveMQSessionExecutor.dispatch 
> (ActiveMQSessionExecutor.java:96)
> 	at org.apache.activemq.ActiveMQSessionExecutor.iterate 
> (ActiveMQSessionExecutor.java:149)
> 	at org.apache.activemq.thread.PooledTaskRunner.runTask 
> (PooledTaskRunner.java:110)
> 	at org.apache.activemq.thread.PooledTaskRunner.access$100 
> (PooledTaskRunner.java:25)
> 	at org.apache.activemq.thread.PooledTaskRunner$1.run 
> (PooledTaskRunner.java:43)
> 	at  
> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor 
> $Worker.runTask(ThreadPoolExecutor.java:650)
> 	at  
> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor 
> $Worker.run(ThreadPoolExecutor.java:675)
> 	at java.lang.Thread.run(Thread.java:552)
>

Timo