[JBoss-dev] minerva / tm problems under load ?

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Hello all,

We have been stress testing jboss this past week and have found stability
problems under load. Perhaps this is simply a configuration problem (I
hope).

The system setup : jboss-2.2.2, DB2 UDB w/ app version of jdbc 2.0 driver,
redhat 6.2 on Intel, sun jdk 1.3.0
The test app : single BMP Entity bean with "Required" transaction attribute

Under low load, the application runs perfectly. As load increases the
exception A (shown at end) occurs about every 5 - 10 minutes.

The root problem is being obscured by the ConcurrentModificationException.
Looking at the minerva code in ObjectPool, there are 4 places where an
iterator is used with insufficient protection from another thread modifying
the underlying HashMap (field name : objects).

I hacked ObjectPool in each case so a ConcurrentModificationException will
restart the iteration. This allows exposing the real exception which is
causing the transaction failure. That exception is shown as exception B.

At this point I do not understand enough yet about minerva and the TxCapsule
to know what causes the null XaRes exception, but I should point out that
this exception occurs less than 1% of the time the business method is
invoked, and only under load. After the first occurance of exception B,
other transaction related exceptions start to appear (XAER_PROTO and
XAER_NOTA). The worst part is that the database connection is forgotten,
neither rolled back nor commited nor ever used again.

Does anyone have a suggestion how to begin debugging this?

Thanks alot for any suggestions,
Wayne

Exception A (stock minerva in jboss-2.2.2)
[Mailbox] CONTAINER EXCEPTION:null
[Mailbox] java.util.ConcurrentModificationException
[Mailbox]       at java.util.HashMap$HashIterator.next(HashMap.java:736)
[Mailbox]       at
java.util.AbstractCollection.addAll(AbstractCollection.java:317)
[Mailbox]       at java.util.HashSet.<init>(HashSet.java:86)
[Mailbox]       at
org.opentools.minerva.pool.ObjectPool.getUsedCount(ObjectPool.java:693)
[Mailbox]       at
org.opentools.minerva.pool.ObjectPool.toString(ObjectPool.java:705)
[Mailbox]       at java.lang.String.valueOf(String.java:1925)
[Mailbox]       at java.lang.StringBuffer.append(StringBuffer.java:373)
[Mailbox]       at
org.opentools.minerva.pool.ObjectPool.releaseObject(ObjectPool.java:680)
[Mailbox]       at
org.opentools.minerva.jdbc.xa.XAConnectionFactory$3.transactionFailed(XAConn
ectionFactory.j
ava:153)
[Mailbox]       at
org.opentools.minerva.jdbc.xa.wrapper.XAConnectionImpl.transactionFailed(XAC
onnectionImpl.j
ava:150)
[Mailbox]       at
org.opentools.minerva.jdbc.xa.wrapper.XAResourceImpl.forget(XAResourceImpl.j
ava:142)
[Mailbox]       at org.jboss.tm.TxCapsule.gotHeuristic(TxCapsule.java:1310)
[Mailbox]       at
org.jboss.tm.TxCapsule.prepareResources(TxCapsule.java:1450)
[Mailbox]       at org.jboss.tm.TxCapsule.commit(TxCapsule.java:347)
[Mailbox]       at
org.jboss.tm.TransactionImpl.commit(TransactionImpl.java:76)
[Mailbox]       at
org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.
java:318)
[Mailbox]       at
org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:99)
[Mailbox]       at
org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:19
0)
[Mailbox]       at
org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:195)
[Mailbox]       at
org.jboss.ejb.EntityContainer.invoke(EntityContainer.java:323)
[Mailbox]       at
org.jboss.ejb.plugins.jrmp.server.JRMPContainerInvoker.invoke(JRMPContainerI
nvoker.java:392
)
[Mailbox]       at java.lang.reflect.Method.invoke(Native Method)
[Mailbox]       at
sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:241)
[Mailbox]       at sun.rmi.transport.Transport$1.run(Transport.java:142)
[Mailbox]       at java.security.AccessController.doPrivileged(Native
Method)
[Mailbox]       at
sun.rmi.transport.Transport.serviceCall(Transport.java:139)
[Mailbox]       at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:443)
[Mailbox]       at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:6
43)
[Mailbox]       at java.lang.Thread.run(Thread.java:484)

Exception B (patched minerva)
[Mailbox] TRANSACTION ROLLBACK EXCEPTION:null; nested exception is:
        javax.ejb.EJBException
[Mailbox] java.lang.RuntimeException: Unable to deregister with
TransactionManager: java.lang.IllegalArgumentE
xception: null xaRes
[Mailbox]       at
org.opentools.minerva.jdbc.xa.XAConnectionFactory$2.closeConnection(XAConnec
tionFactory.jav
a:103)
[Mailbox]       at
org.opentools.minerva.jdbc.xa.XAConnectionFactory$2.connectionClosed(XAConne
ctionFactory.ja
va:82)
[Mailbox]       at
org.opentools.minerva.jdbc.xa.wrapper.XAConnectionImpl.clientConnectionClose
d(XAConnectionI
mpl.java:126)
[Mailbox]       at
org.opentools.minerva.jdbc.xa.wrapper.XAClientConnection.close(XAClientConne
ction.java:250)
[Mailbox]       at eon.infra.dss.BoxBean.loadBoxSummary(BoxBean.java:631)
[Mailbox]       at eon.infra.dss.BoxBean.getOwnerID(BoxBean.java:597)
[Mailbox]       at
eon.infra.dss.MailboxBean.getFilterSet(MailboxBean.java:168)
[Mailbox]       at eon.infra.dss.MailboxBean.deliver(MailboxBean.java:27)
[Mailbox]       at java.lang.reflect.Method.invoke(Native Method)
[Mailbox]       at
org.jboss.ejb.EntityContainer$ContainerInterceptor.invoke(EntityContainer.ja
va:722)
[Mailbox]       at
org.jboss.ejb.plugins.EntitySynchronizationInterceptor.invoke(EntitySynchron
izationIntercep
tor.java:208)
[Mailbox]       at
org.jboss.ejb.plugins.EntityInstanceInterceptor.invoke(EntityInstanceInterce
ptor.java:186)
[Mailbox]       at
org.jboss.ejb.plugins.TxInterceptorCMT.invokeNext(TxInterceptorCMT.java:133)
[Mailbox]       at
org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.
java:263)
[Mailbox]       at
org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:99)
[Mailbox]       at
org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:19
0)
[Mailbox]       at
org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:195)
[Mailbox]       at
org.jboss.ejb.EntityContainer.invoke(EntityContainer.java:323)
[Mailbox]       at
org.jboss.ejb.plugins.jrmp.server.JRMPContainerInvoker.invoke(JRMPContainerI
nvoker.java:392
)
[Mailbox]       at java.lang.reflect.Method.invoke(Native Method)
[Mailbox]       at
sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:241)
[Mailbox]       at sun.rmi.transport.Transport$1.run(Transport.java:142)
[Mailbox]       at java.security.AccessController.doPrivileged(Native
Method)
[Mailbox]       at
sun.rmi.transport.Transport.serviceCall(Transport.java:139)
[Mailbox]       at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:443)
[Mailbox]       at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:6
43)
[Mailbox]       at java.lang.Thread.run(Thread.java:484)