While issueing certificates in stress test, EJBCA generated an exception.
The stack trace shows "Application deadlock detected, resource=org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock@......"
Some part of stack trace is shown on the end of this message.
Is there any relation to the database?
We use MySQL 5.0.41 with all tables as MyISAM.
Does anyone show the same error?
Regards,
Kouichi Sugimoto.
Traceback (innermost last):
File "n_batch.py", line 51, in ?
org.jboss.tm.JBossTransactionRolledbackException: javax.ejb.EJBException: javax.ejb.TransactionRolledbackLocalException: Application deadlock detected, resource=org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock@1860da6, bean=GlobalConfigurationData, id=0, refs=2, tx=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f23f status: ActionStatus.RUNNING >, synched=WorkerThread#2[127.0.0.1:35216], timeout=5000, queue=[], holder=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f194 status: ActionStatus.RUNNING >, waitingResource=org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock@182ba68, bean=CAData, id=-1598425179, refs=2, tx=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f194 status: ActionStatus.RUNNING >, synched=null, timeout=5000, queue=[TXLOCK waitingTx=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f187 status: ActionStatus.RUNNING > id=0 thread=Thread[Thread-12,5,jboss] queued=true], waitingResourceHolder=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f194 status: ActionStatus.RUNNING >; nested exception is:
javax.ejb.EJBException: javax.ejb.EJBException: javax.ejb.TransactionRolledbackLocalException: Application deadlock detected, resource=org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock@1860da6, bean=GlobalConfigurationData, id=0, refs=2, tx=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f23f status: ActionStatus.RUNNING >, synched=WorkerThread#2[127.0.0.1:35216], timeout=5000, queue=[], holder=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f194 status: ActionStatus.RUNNING >, waitingResource=org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock@182ba68, bean=CAData, id=-1598425179, refs=2, tx=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f194 status: ActionStatus.RUNNING >, synched=null, timeout=5000, queue=[TXLOCK waitingTx=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f187 status: ActionStatus.RUNNING > id=0 thread=Thread[Thread-12,5,jboss] queued=true], waitingResourceHolder=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f194 status: ActionStatus.RUNNING >; - nested throwable: (javax.ejb.EJBException: javax.ejb.EJBException: javax.ejb.TransactionRolledbackLocalException: Application deadlock detected, resource=org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock@1860da6, bean=GlobalConfigurationData, id=0, refs=2, tx=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f23f status: ActionStatus.RUNNING >, synched=WorkerThread#2[127.0.0.1:35216], timeout=5000, queue=[], holder=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f194 status: ActionStatus.RUNNING >, waitingResource=org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock@182ba68, bean=CAData, id=-1598425179, refs=2, tx=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f194 status: ActionStatus.RUNNING >, synched=null, timeout=5000, queue=[TXLOCK waitingTx=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f187 status: ActionStatus.RUNNING > id=0 thread=Thread[Thread-12,5,jboss] queued=true], waitingResourceHolder=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f194 status: ActionStatus.RUNNING >)
at org.jboss.ejb.plugins.LogInterceptor.handleException(LogInterceptor.java:280)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:209)
at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)
at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:648)
at org.jboss.ejb.Container.invoke(Container.java:960)
at sun.reflect.GeneratedMethodAccessor166.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
at org.jboss.invocation.unified.server.UnifiedInvoker.invoke(UnifiedInvoker.java:231)
at sun.reflect.GeneratedMethodAccessor189.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:201)
at $Proxy16.invoke(Unknown Source)
at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:734)
at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:560)
at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:383)
at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:165)
Caused by: javax.ejb.EJBException: javax.ejb.EJBException: javax.ejb.TransactionRolledbackLocalException: Application deadlock detected, resource=org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock@1860da6, bean=GlobalConfigurationData, id=0, refs=2, tx=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f23f status: ActionStatus.RUNNING >, synched=WorkerThread#2[127.0.0.1:35216], timeout=5000, queue=[], holder=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f194 status: ActionStatus.RUNNING >, waitingResource=org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock@182ba68, bean=CAData, id=-1598425179, refs=2, tx=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f194 status: ActionStatus.RUNNING >, synched=null, timeout=5000, queue=[TXLOCK waitingTx=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f187 status: ActionStatus.RUNNING > id=0 thread=Thread[Thread-12,5,jboss] queued=true], waitingResourceHolder=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f194 status: ActionStatus.RUNNING >
at org.ejbca.core.ejb.ca.auth.LocalAuthenticationSessionBean.finishUser(LocalAuthenticationSessionBean.java:264)
at sun.reflect.GeneratedMethodAccessor295.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.invocation.Invocation.performCall(Invocation.java:359)
at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:237)
at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:158)
at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:169)
at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)
at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:350)
at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)
at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:168)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)
at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:648)
at org.jboss.ejb.Container.invoke(Container.java:960)
at org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java:430)
at org.jboss.ejb.plugins.local.StatelessSessionProxy.invoke(StatelessSessionProxy.java:103)
at $Proxy187.finishUser(Unknown Source)
at org.ejbca.core.ejb.ca.sign.RSASignSessionBean.finishUser(RSASignSessionBean.java:1247)
at org.ejbca.core.ejb.ca.sign.RSASignSessionBean.createCertificate(RSASignSessionBean.java:1350)
at org.ejbca.core.ejb.ca.sign.RSASignSessionBean.createCertificate(RSASignSessionBean.java:436)
at org.ejbca.core.ejb.ca.sign.RSASignSessionBean.createCertificate(RSASignSessionBean.java:382)
at sun.reflect.GeneratedMethodAccessor267.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.invocation.Invocation.performCall(Invocation.java:359)
at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:237)
at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:158)
at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:169)
at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)
at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:350)
at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)
at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:163)
at org.jboss.remoting.Client.invoke(Client.java:1550)
at org.jboss.remoting.Client.invoke(Client.java:530)
at org.jboss.invocation.unified.interfaces.UnifiedInvokerProxy.invoke(UnifiedInvokerProxy.java:183)
at org.jboss.invocation.InvokerInterceptor.invokeInvoker(InvokerInterceptor.java:365)
at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:197)
at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:61)
at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:70)
at org.jboss.proxy.ejb.StatelessSessionInterceptor.invoke(StatelessSessionInterceptor.java:112)
at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:100)
at $Proxy5.createCertificate(Unknown Source)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java)
at org.python.core.PyMethod.__call__(PyMethod.java)
at org.python.core.PyObject.__call__(PyObject.java)
at org.python.core.PyObject.invoke(PyObject.java)
at org.python.pycode._pyx0.f$0(n_batch.py:51)
at org.python.pycode._pyx0.call_function(n_batch.py)
at org.python.core.PyTableCode.call(PyTableCode.java)
at org.python.core.PyCode.call(PyCode.java)
at org.python.core.Py.runCode(Py.java)
at org.python.core.__builtin__.execfile_flags(__builtin__.java)
at org.python.util.PythonInterpreter.execfile(PythonInterpreter.java)
at org.python.util.jython.main(jython.java)
Thank you for the reply.
We use a jython script that publishes certificates periodically, which we show at the end of this message.
We published around 200000 certificates. Then the ratio of detecting deadlock raised relatively.
It is likely to raise the ratio in relation to the following cases:
1. Certificate publishing while CRL creation
2. Under the condition of index creation (shown as HOWTO-database.txt)
3. Simultaneous certificates publishing
As mentioned in the jira thread,we made get* methond of GlobalConfigurationData bean read-only, and that seemed to work. So far, we have not seen application deadlock errors even
we tried simultaneous access.
Add the following xdoclet tag in GlobalGonfigurationData.java,
Dear sirs,
While issueing certificates in stress test, EJBCA generated an exception.
The stack trace shows "Application deadlock detected, resource=org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock@......"
Some part of stack trace is shown on the end of this message.
Is there any relation to the database?
We use MySQL 5.0.41 with all tables as MyISAM.
Does anyone show the same error?
Regards,
Kouichi Sugimoto.
Traceback (innermost last):
File "n_batch.py", line 51, in ?
org.jboss.tm.JBossTransactionRolledbackException: javax.ejb.EJBException: javax.ejb.TransactionRolledbackLocalException: Application deadlock detected, resource=org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock@1860da6, bean=GlobalConfigurationData, id=0, refs=2, tx=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f23f status: ActionStatus.RUNNING >, synched=WorkerThread#2[127.0.0.1:35216], timeout=5000, queue=[], holder=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f194 status: ActionStatus.RUNNING >, waitingResource=org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock@182ba68, bean=CAData, id=-1598425179, refs=2, tx=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f194 status: ActionStatus.RUNNING >, synched=null, timeout=5000, queue=[TXLOCK waitingTx=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f187 status: ActionStatus.RUNNING > id=0 thread=Thread[Thread-12,5,jboss] queued=true], waitingResourceHolder=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f194 status: ActionStatus.RUNNING >; nested exception is:
javax.ejb.EJBException: javax.ejb.EJBException: javax.ejb.TransactionRolledbackLocalException: Application deadlock detected, resource=org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock@1860da6, bean=GlobalConfigurationData, id=0, refs=2, tx=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f23f status: ActionStatus.RUNNING >, synched=WorkerThread#2[127.0.0.1:35216], timeout=5000, queue=[], holder=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f194 status: ActionStatus.RUNNING >, waitingResource=org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock@182ba68, bean=CAData, id=-1598425179, refs=2, tx=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f194 status: ActionStatus.RUNNING >, synched=null, timeout=5000, queue=[TXLOCK waitingTx=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f187 status: ActionStatus.RUNNING > id=0 thread=Thread[Thread-12,5,jboss] queued=true], waitingResourceHolder=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f194 status: ActionStatus.RUNNING >; - nested throwable: (javax.ejb.EJBException: javax.ejb.EJBException: javax.ejb.TransactionRolledbackLocalException: Application deadlock detected, resource=org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock@1860da6, bean=GlobalConfigurationData, id=0, refs=2, tx=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f23f status: ActionStatus.RUNNING >, synched=WorkerThread#2[127.0.0.1:35216], timeout=5000, queue=[], holder=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f194 status: ActionStatus.RUNNING >, waitingResource=org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock@182ba68, bean=CAData, id=-1598425179, refs=2, tx=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f194 status: ActionStatus.RUNNING >, synched=null, timeout=5000, queue=[TXLOCK waitingTx=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f187 status: ActionStatus.RUNNING > id=0 thread=Thread[Thread-12,5,jboss] queued=true], waitingResourceHolder=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f194 status: ActionStatus.RUNNING >)
at org.jboss.ejb.plugins.LogInterceptor.handleException(LogInterceptor.java:280)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:209)
at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)
at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:648)
at org.jboss.ejb.Container.invoke(Container.java:960)
at sun.reflect.GeneratedMethodAccessor166.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
at org.jboss.invocation.unified.server.UnifiedInvoker.invoke(UnifiedInvoker.java:231)
at sun.reflect.GeneratedMethodAccessor189.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:201)
at $Proxy16.invoke(Unknown Source)
at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:734)
at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:560)
at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:383)
at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:165)
Caused by: javax.ejb.EJBException: javax.ejb.EJBException: javax.ejb.TransactionRolledbackLocalException: Application deadlock detected, resource=org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock@1860da6, bean=GlobalConfigurationData, id=0, refs=2, tx=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f23f status: ActionStatus.RUNNING >, synched=WorkerThread#2[127.0.0.1:35216], timeout=5000, queue=[], holder=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f194 status: ActionStatus.RUNNING >, waitingResource=org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock@182ba68, bean=CAData, id=-1598425179, refs=2, tx=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f194 status: ActionStatus.RUNNING >, synched=null, timeout=5000, queue=[TXLOCK waitingTx=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f187 status: ActionStatus.RUNNING > id=0 thread=Thread[Thread-12,5,jboss] queued=true], waitingResourceHolder=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f194 status: ActionStatus.RUNNING >
at org.ejbca.core.ejb.ca.auth.LocalAuthenticationSessionBean.finishUser(LocalAuthenticationSessionBean.java:264)
at sun.reflect.GeneratedMethodAccessor295.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.invocation.Invocation.performCall(Invocation.java:359)
at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:237)
at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:158)
at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:169)
at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)
at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:350)
at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)
at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:168)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)
at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:648)
at org.jboss.ejb.Container.invoke(Container.java:960)
at org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java:430)
at org.jboss.ejb.plugins.local.StatelessSessionProxy.invoke(StatelessSessionProxy.java:103)
at $Proxy187.finishUser(Unknown Source)
at org.ejbca.core.ejb.ca.sign.RSASignSessionBean.finishUser(RSASignSessionBean.java:1247)
at org.ejbca.core.ejb.ca.sign.RSASignSessionBean.createCertificate(RSASignSessionBean.java:1350)
at org.ejbca.core.ejb.ca.sign.RSASignSessionBean.createCertificate(RSASignSessionBean.java:436)
at org.ejbca.core.ejb.ca.sign.RSASignSessionBean.createCertificate(RSASignSessionBean.java:382)
at sun.reflect.GeneratedMethodAccessor267.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.invocation.Invocation.performCall(Invocation.java:359)
at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:237)
at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:158)
at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:169)
at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)
at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:350)
at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)
at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:163)
at org.jboss.remoting.Client.invoke(Client.java:1550)
at org.jboss.remoting.Client.invoke(Client.java:530)
at org.jboss.invocation.unified.interfaces.UnifiedInvokerProxy.invoke(UnifiedInvokerProxy.java:183)
at org.jboss.invocation.InvokerInterceptor.invokeInvoker(InvokerInterceptor.java:365)
at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:197)
at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:61)
at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:70)
at org.jboss.proxy.ejb.StatelessSessionInterceptor.invoke(StatelessSessionInterceptor.java:112)
at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:100)
at $Proxy5.createCertificate(Unknown Source)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java)
at org.python.core.PyMethod.__call__(PyMethod.java)
at org.python.core.PyObject.__call__(PyObject.java)
at org.python.core.PyObject.invoke(PyObject.java)
at org.python.pycode._pyx0.f$0(n_batch.py:51)
at org.python.pycode._pyx0.call_function(n_batch.py)
at org.python.core.PyTableCode.call(PyTableCode.java)
at org.python.core.PyCode.call(PyCode.java)
at org.python.core.Py.runCode(Py.java)
at org.python.core.__builtin__.execfile_flags(__builtin__.java)
at org.python.util.PythonInterpreter.execfile(PythonInterpreter.java)
at org.python.util.jython.main(jython.java)
org.jboss.tm.JBossTransactionRolledbackException: org.jboss.tm.JBossTransactionRolledbackException: javax.ejb.EJBException: javax.ejb.TransactionRolledbackLocalException: Application deadlock detected, resource=org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock@1860da6, bean=GlobalConfigurationData, id=0, refs=2, tx=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f23f status: ActionStatus.RUNNING >, synched=WorkerThread#2[127.0.0.1:35216], timeout=5000, queue=[], holder=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f194 status: ActionStatus.RUNNING >, waitingResource=org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock@182ba68, bean=CAData, id=-1598425179, refs=2, tx=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f194 status: ActionStatus.RUNNING >, synched=null, timeout=5000, queue=[TXLOCK waitingTx=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f187 status: ActionStatus.RUNNING > id=0 thread=Thread[Thread-12,5,jboss] queued=true], waitingResourceHolder=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f194 status: ActionStatus.RUNNING >; nested exception is:
javax.ejb.EJBException: javax.ejb.EJBException: javax.ejb.TransactionRolledbackLocalException: Application deadlock detected, resource=org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock@1860da6, bean=GlobalConfigurationData, id=0, refs=2, tx=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f23f status: ActionStatus.RUNNING >, synched=WorkerThread#2[127.0.0.1:35216], timeout=5000, queue=[], holder=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f194 status: ActionStatus.RUNNING >, waitingResource=org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock@182ba68, bean=CAData, id=-1598425179, refs=2, tx=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f194 status: ActionStatus.RUNNING >, synched=null, timeout=5000, queue=[TXLOCK waitingTx=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f187 status: ActionStatus.RUNNING > id=0 thread=Thread[Thread-12,5,jboss] queued=true], waitingResourceHolder=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f194 status: ActionStatus.RUNNING >; - nested throwable: (javax.ejb.EJBException: javax.ejb.EJBException: javax.ejb.TransactionRolledbackLocalException: Application deadlock detected, resource=org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock@1860da6, bean=GlobalConfigurationData, id=0, refs=2, tx=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f23f status: ActionStatus.RUNNING >, synched=WorkerThread#2[127.0.0.1:35216], timeout=5000, queue=[], holder=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f194 status: ActionStatus.RUNNING >, waitingResource=org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock@182ba68, bean=CAData, id=-1598425179, refs=2, tx=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f194 status: ActionStatus.RUNNING >, synched=null, timeout=5000, queue=[TXLOCK waitingTx=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f187 status: ActionStatus.RUNNING > id=0 thread=Thread[Thread-12,5,jboss] queued=true], waitingResourceHolder=TransactionImple < ac, BasicAction: 7f000001:87ae:46de1737:11f194 status: ActionStatus.RUNNING >)
We have had reports of deadlock in rare conditions before. Exteremely rare though.
Can you reproduce it?
What kind of stress-test did you perform?
That would help to try to solve the issue.
Regards,
Tomas
Dear Tomas,
Thank you for the reply.
We use a jython script that publishes certificates periodically, which we show at the end of this message.
We published around 200000 certificates. Then the ratio of detecting deadlock raised relatively.
It is likely to raise the ratio in relation to the following cases:
1. Certificate publishing while CRL creation
2. Under the condition of index creation (shown as HOWTO-database.txt)
3. Simultaneous certificates publishing
The scrypt is as follows:
#
# jython batch.py
#
import time
import javax.naming.Context
import org.apache.log4j.Logger
import org.ejbca.core.ejb.InitialContextBuilder
import org.ejbca.core.ejb.ra.IUserAdminSessionHome
import org.ejbca.core.ejb.ca.caadmin.ICAAdminSessionHome
import org.ejbca.core.ejb.ca.sign.ISignSessionHome
import java.security.KeyPairGenerator
administrator = org.ejbca.core.model.log.Admin(4)
jndictx = org.ejbca.core.ejb.InitialContextBuilder.getInstance().getInitialContext()
import javax.rmi.PortableRemoteObject
obj1 = jndictx.lookup("UserAdminSession")
obj2 = jndictx.lookup("CAAdminSession")
obj3 = jndictx.lookup("RSASignSession")
adminhome = javax.rmi.PortableRemoteObject.narrow(obj1,org.ejbca.core.ejb.ra.IUserAdminSessionHome)
caadminhome = javax.rmi.PortableRemoteObject.narrow(obj2,org.ejbca.core.ejb.ca.caadmin.ICAAdminSessionHome)
signhome = javax.rmi.PortableRemoteObject.narrow(obj3,org.ejbca.core.ejb.ca.sign.ISignSessionHome)
admin = adminhome.create()
caadmin = caadminhome.create()
signobj = signhome.create()
caid = caadmin.getCAInfo(administrator,'ThirdCA').getCAId()
rsagen = java.security.KeyPairGenerator.getInstance('RSA')
rsakeypair = rsagen.generateKeyPair()
pubkey = rsakeypair.getPublic()
#log = open('batch_exe.log','w')
#log.write(time.strftime('%Y-%m-%d %H:%M:%S'))
# token type - 1
password='himitsu'
for n in range(60000,100000):
username = 'CA3User%d' % n
subjectdn = 'C=JP,O=SECOM Trust Systems,CN=' + username
altname = 'UPN=' + username + '@secom-sts.co.jp'
admin.addUser(administrator,username,password,subjectdn,altname,None,1,1,1,1,1,0,caid)
# admin.setClearTextPassword(administrator,username,password)
print 'User: ' + username + ' is ready for issuing cert'
cert = signobj.createCertificate(administrator,username,password,pubkey)
#log.write(time.strftime('%Y-%m-%d %H:%M:%S'))
#log.close()
# admin.revokeUser(administrator,username,reason)
# reason - 0 unspecified - 1 keycompromised
Regards,
Kouichi Sugimoto.
Thanks for the script.
Until we have time to dig deep into this, you should read this issue (and the solution):
https://jira.primekey.se/browse/ECA-445
Cheers,
Tomas
As mentioned in the jira thread,we made get* methond of GlobalConfigurationData bean read-only, and that seemed to work. So far, we have not seen application deadlock errors even
we tried simultaneous access.
Add the following xdoclet tag in GlobalGonfigurationData.java,
* @jboss.method-attributes
* pattern = "get*"
* read-only = "true"
"Ant bootstrap" will generate jboss.xml including the read-only method attributes as follows:
<entity>
<ejb-name>GlobalConfigurationData</ejb-name>
<local-jndi-name>GlobalConfigurationDataLocal</local-jndi-name>
<method-attributes>
<method>
<method-name>get*</method-name>
<read-only>true</read-only>
</method>
</method-attributes>
</entity>
Cheers,
-Yuji
Very good. I have tested (that it doesn't break) and committed this fix.
Thank you. Added you to contirbutors list Yuji.
Cheers,
Tomas