[c3p0-users] a lot of threads waiting for a GlobalMaxOnlyStatementCache

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Hi all!

I ran into problem in my application. The threaddump shows that there are
111 threads (almost 30%) are waiting for a monitor
com.mchange.v2.c3p0.stmt.GlobalMaxOnlyStatementCache and 35 threads are
locking this monitor. Below are typical stacktraces of a locking and
waiting thread. My application has a lot of users and db load is very high,
so I guess I need to adjust some settings in hibernate, but I can't figure
out which ones.

Can anyone help? Thanks in advance

locking thread stacktrace:
*Code:*
"http-9090-Processor43" daemon prio=10 tid=0x00007fd7007c9800 nid=0x173f in
Object.wait() [0x00007fd6f8e35000]
   java.lang.Thread.State: BLOCKED (on object monitor)
   at java.lang.Object.wait(Native Method)
   at java.lang.Object.wait(Object.java:502)
   at
com.mchange.v2.c3p0.stmt.GooGooStatementCache.acquireStatement(GooGooStatementCache.java:552)
   at
com.mchange.v2.c3p0.stmt.GooGooStatementCache.checkoutStatement(GooGooStatementCache.java:168)
   - locked <0x0000000683d90348> (a
com.mchange.v2.c3p0.stmt.GlobalMaxOnlyStatementCache)
   at
com.mchange.v2.c3p0.impl.NewPooledConnection.checkoutStatement(NewPooledConnection.java:234)
   - locked <0x000000068abfeeb8> (a
com.mchange.v2.c3p0.impl.NewPooledConnection)
   at
com.mchange.v2.c3p0.impl.NewProxyConnection.prepareStatement(NewProxyConnection.java:199)
   - locked <0x000000079d2b3e98> (a
com.mchange.v2.c3p0.impl.NewProxyConnection)
   at
org.hibernate.jdbc.AbstractBatcher.getPreparedStatement(AbstractBatcher.java:534)
   at
org.hibernate.jdbc.AbstractBatcher.getPreparedStatement(AbstractBatcher.java:452)
   at
org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:161)
   at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1573)
   at org.hibernate.loader.Loader.doQuery(Loader.java:696)
   at
org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259)
   at org.hibernate.loader.Loader.doList(Loader.java:2228)
   at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2125)
   at org.hibernate.loader.Loader.list(Loader.java:2120)
   at
org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:118)
   at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1596)
   at org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:306)
   at
com.gridnine.xtrip.server.storage.SearchQueryHelper.execute(SearchQueryHelper.java:76)
   at
com.gridnine.xtrip.server.storage.EntityStorage.search(EntityStorage.java:82)
   at
com.gridnine.xtrip.server.storage.EntityStorage.getModifications(EntityStorage.java:701)
   at
com.gridnine.xtrip.server.rpc.service.EntityServiceImpl.getModifications(EntityServiceImpl.java:215)
   at sun.reflect.GeneratedMethodAccessor110.invoke(Unknown Source)
   at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:616)
   at
org.jboss.remoting.transporter.TransporterHandler.invoke(TransporterHandler.java:87)
   at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:908)
   at
org.jboss.remoting.transport.servlet.ServletServerInvoker.processRequest(ServletServerInvoker.java:309)
   at
org.jboss.remoting.util.ServletSecurityUtility.processRequest(ServletSecurityUtility.java:53)
   at
org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.processRequest(ServerInvokerServlet.java:128)
   at
com.gridnine.xtrip.server.web.rpc.RPCServlet.processRequest(RPCServlet.java:50)
   at
org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.doPost(ServerInvokerServlet.java:157)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
   at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
   at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
   at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:210)
   at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)
   at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
   at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
   at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
   at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
   at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:870)
   at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
   at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
   at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
   at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:685)
   at java.lang.Thread.run(Thread.java:679)

waiting thread stractrace:
*Code:*
"pool-315-thread-3" prio=10 tid=0x00007fd708a39800 nid=0x2845 waiting for
monitor entry [0x00007fd6e16d5000]
   java.lang.Thread.State: BLOCKED (on object monitor)
   at
com.mchange.v2.c3p0.stmt.GooGooStatementCache.checkoutStatement(GooGooStatementCache.java:158)
   - waiting to lock <0x0000000683d90348> (a
com.mchange.v2.c3p0.stmt.GlobalMaxOnlyStatementCache)
   at
com.mchange.v2.c3p0.impl.NewPooledConnection.checkoutStatement(NewPooledConnection.java:234)
   - locked <0x00000007475304a0> (a
com.mchange.v2.c3p0.impl.NewPooledConnection)
   at
com.mchange.v2.c3p0.impl.NewProxyConnection.prepareStatement(NewProxyConnection.java:199)
   - locked <0x00000007a26a33b0> (a
com.mchange.v2.c3p0.impl.NewProxyConnection)
   at
org.hibernate.jdbc.AbstractBatcher.getPreparedStatement(AbstractBatcher.java:534)
   at
org.hibernate.jdbc.AbstractBatcher.getPreparedStatement(AbstractBatcher.java:452)
   at
org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:161)
   at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1573)
   at org.hibernate.loader.Loader.doQuery(Loader.java:696)
   at
org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259)
   at org.hibernate.loader.Loader.loadEntity(Loader.java:1881)
   at
org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:71)
   at
org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:65)
   at
org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:3072)
   at
org.hibernate.event.def.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:434)
   at
org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:415)
   at
org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:165)
   at
org.hibernate.event.def.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:223)
   at
org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:126)
   at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:905)
   at org.hibernate.impl.SessionImpl.get(SessionImpl.java:842)
   at org.hibernate.impl.SessionImpl.get(SessionImpl.java:835)
   at
com.gridnine.xtrip.server.storage.EntityStorage.findData(EntityStorage.java:799)
   at
com.gridnine.xtrip.server.storage.EntityStorage.save(EntityStorage.java:256)
   at
com.gridnine.xtrip.server.storage.EntityStorage.save(EntityStorage.java:227)
   at
com.gridnine.xtrip.server.model.export.service.ExportDocumentsHelper.processDocuments(ExportDocumentsHelper.java:279)
   at
com.gridnine.xtrip.server.model.export.service.ProcessExportDocumentsService$1.run(ProcessExportDocumentsService.java:146)
   at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
   at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
   at java.util.concurrent.FutureTask.run(FutureTask.java:166)
   at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
   at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
   at java.lang.Thread.run(Thread.java:679)