c3p0:JDBC DataSources/Resource Pools / Bugs / #133 java.sql.SQLException: An SQLException was provoked by the following failure: java.lang.RuntimeException: Internal inconsistency: A (not new) checking-out statement is not in deathmarch.

#133 java.sql.SQLException: An SQLException was provoked by the following failure: java.lang.RuntimeException: Internal inconsistency: A (not new) checking-out statement is not in deathmarch.

Milestone: v1.0 (example)

Status: open

Owner: Steve Waldman

Labels: mysql_replication_connection (1)

Priority: 5

Updated: 2016-03-11

Created: 2015-06-10

Creator: Silvio Fragata Silva

Private: No

Hi,

I've an application in prod that works with C3p0, hibernate and MySQL

I've an error when I use C3p0 and hibernate with MySQL REPLICATION (com.mysql.jdbc.ReplicationDriver instead of com.mysql.jdbc.Driver)

So, my c3p0.property is configured to use max statements per connection
c3p0.maxStatementsPerConnection=30

and my hibernate.cfg.xml is configured to use max connections

    <property name="hibernate.c3p0.acquire_increment">25</property>
    <property name="hibernate.c3p0.max_size">225</property>
    <property name="hibernate.c3p0.max_statements">1000</property>        
    <property name="hibernate.c3p0.min_size">25</property>

I use that jdbc url connection

jdbc:mysql:replication://<master>:<port>,<slave>:<port>/<database>?zeroDateTimeBehavior=convertToNull&replicationEnableJMX=true&replicationConnectionGroup=group&failOverReadOnly=false&autoReconnect=false&profileSQL=true</database></port></slave></port></master>

When I execute a few tests, everything is OK, however, if I execute several tests I've the exceptions below

SQL Error: 0, SQLState: null
An SQLException was provoked by the following failure: java.lang.RuntimeException: Internal inconsistency: A (not new) checking-out statement is not in deathmarch.
HHH000327: Error performing load command : org.hibernate.exception.GenericJDBCException: could not prepare statement
could not prepare statement
org.hibernate.exception.GenericJDBCException: could not prepare statement
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:54)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:126)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:196)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.prepareQueryStatement(StatementPreparerImpl.java:160)
at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.prepareQueryStatement(AbstractLoadPlanBasedLoader.java:257)
at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.executeQueryStatement(AbstractLoadPlanBasedLoader.java:201)
at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.executeLoad(AbstractLoadPlanBasedLoader.java:137)
at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.executeLoad(AbstractLoadPlanBasedLoader.java:102)
at org.hibernate.loader.entity.plan.AbstractLoadPlanBasedEntityLoader.load(AbstractLoadPlanBasedEntityLoader.java:186)
at org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:4126)
at org.hibernate.event.internal.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:503)
at org.hibernate.event.internal.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:468)
at org.hibernate.event.internal.DefaultLoadEventListener.load(DefaultLoadEventListener.java:213)
at org.hibernate.event.internal.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:275)
at org.hibernate.event.internal.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:151)
at org.hibernate.internal.SessionImpl.fireLoad(SessionImpl.java:1106)
at org.hibernate.internal.SessionImpl.access$2000(SessionImpl.java:176)
at org.hibernate.internal.SessionImpl$IdentifierLoadAccessImpl.load(SessionImpl.java:2587)
at org.hibernate.internal.SessionImpl.get(SessionImpl.java:991)
at com.microflex.persistence.session.hibernate.HibernateSession.get(HibernateSession.java:142)
at com.microflex.persistence.transactional.repositories.event.HibernateEventSystemLinkRepository.findByEventId(HibernateEventSystemLinkRepository.java:99)
at com.microflex.persistence.transactional.repositories.event.HibernateInfoPageEventLinkRepository.findByIdentifier(HibernateInfoPageEventLinkRepository.java:60)
at com.microflex.persistence.transactional.repositories.infopage.HibernateInfoPageRepository.findEventInfoPage(HibernateInfoPageRepository.java:202)
at com.microflex.services.integration.ticketing.domain.eventpackage.PackageServiceImpl.getPackagePageInfo(PackageServiceImpl.java:600)
at com.microflex.services.integration.ticketing.domain.eventpackage.PackageServiceImpl.createTicketingPackage(PackageServiceImpl.java:348)
at com.microflex.services.integration.ticketing.domain.eventpackage.PackageServiceImpl.findPackageDetailsWithOnSaleValidation(PackageServiceImpl.java:166)
at com.microflex.services.integration.ticketing.domain.eventpackage.PackageServiceImpl.findPackageDetails(PackageServiceImpl.java:124)
at com.microflex.services.integration.ticketing.domain.eventpackage.PackageServiceImpl.findPackageDetails(PackageServiceImpl.java:1058)
at com.microflex.services.integration.ticketing.domain.eventpackage.PackageServiceImpl.findPackageDetails(PackageServiceImpl.java:114)
at com.microflex.services.integration.ticketing.domain.cart.event.CartEventServiceImpl.getEventDetailsByBlockId(CartEventServiceImpl.java:295)
at com.microflex.services.integration.ticketing.domain.cart.event.CartEventServiceImpl.getEventDetailsByBlockId(CartEventServiceImpl.java:559)
at com.microflex.services.integration.ticketing.cart.CartServiceImpl.getEventDetailsByBlockId(CartServiceImpl.java:356)
at com.microflex.services.integration.ticketing.domain.payment.SaleServiceImpl.processTickets(SaleServiceImpl.java:903)
at com.microflex.services.integration.ticketing.domain.payment.SaleServiceImpl.execute(SaleServiceImpl.java:217)
at com.microflex.services.integration.ticketing.cart.reserve.CartReserveServiceImpl.reserve(CartReserveServiceImpl.java:91)
at com.microflex.services.integration.ticketing.cart.reserve.CartReserveMessageServiceAdapter.processMessageRequest(CartReserveMessageServiceAdapter.java:44)
at com.microflex.services.integration.ticketing.cart.reserve.CartReserveMessageServiceAdapter.processMessageRequest(CartReserveMessageServiceAdapter.java:1)
at com.microflex.services.integration.ticketing.AbstractTicketingMessageServiceAdapter.processRequest(AbstractTicketingMessageServiceAdapter.java:65)
at com.microflex.services.integration.ticketing.messaging.JsonMessageProcessor.processMessageRequest(JsonMessageProcessor.java:194)
at com.microflex.services.integration.ticketing.messaging.JsonMessageProcessor.processMessage(JsonMessageProcessor.java:121)
at com.microflex.services.commons.messaging.AbstractMessageProcessor.process(AbstractMessageProcessor.java:48)
at com.microflex.services.integration.ticketing.TicketingRequestHandlerServlet.doPost(TicketingRequestHandlerServlet.java:103)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:688)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.microflex.idp.client.auth.SimpleAuthenticationFilter.doFilter(SimpleAuthenticationFilter.java:72)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.microflex.services.commons.servlet.RequestLoggerFilter.doFilter(RequestLoggerFilter.java:35)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.microflex.monitoring.performance.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:123)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:516)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1086)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:659)
at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:223)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1558)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1515)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.sql.SQLException: An SQLException was provoked by the following failure: java.lang.RuntimeException: Internal inconsistency: A (not new) checking-out statement is not in deathmarch.
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:118)
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:77)
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:74)
at com.mchange.v2.c3p0.impl.NewPooledConnection.handleThrowable(NewPooledConnection.java:505)
at com.mchange.v2.c3p0.impl.NewProxyConnection.prepareStatement(NewProxyConnection.java:486)
at com.microflex.monitoring.performance.MonitoredConnection.prepareStatement(MonitoredConnection.java:272)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$5.doPrepare(StatementPreparerImpl.java:162)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:186)
... 68 more
Caused by: java.lang.RuntimeException: Internal inconsistency: A (not new) checking-out statement is not in deathmarch.
at com.mchange.v2.c3p0.stmt.GooGooStatementCache$Deathmarch.undeathmarchStatement(GooGooStatementCache.java:706)
at com.mchange.v2.c3p0.stmt.DoubleMaxStatementCache.removeStatementFromDeathmarches(DoubleMaxStatementCache.java:70)
at com.mchange.v2.c3p0.stmt.GooGooStatementCache.checkoutStatement(GooGooStatementCache.java:224)
at com.mchange.v2.c3p0.impl.NewPooledConnection.checkoutStatement(NewPooledConnection.java:321)
at com.mchange.v2.c3p0.impl.NewProxyConnection.prepareStatement(NewProxyConnection.java:455)
... 71 more
exception: could not prepare statement
cause: An SQLException was provoked by the following failure: java.lang.RuntimeException: Internal inconsistency: A (not new) checking-out statement is not in deathmarch.
cause: Internal inconsistency: A (not new) checking-out statement is not in deathmarch.

I've checked the source code and I've found that method
class GooGooStatementCache

protected class Deathmarch
    public void undeathmarchStatement( Object ps )
    {
        assert Thread.holdsLock(GooGooStatementCache.this);

        Long old = (Long) stmtsToLongs.remove( ps );
        if (Debug.DEBUG && old == null)
            throw new RuntimeException("Internal inconsistency: " +
            "A (not new) checking-out statement is not in deathmarch.");
        Object check = longsToStmts.remove( old );
        if (Debug.DEBUG && old == null)
            throw new RuntimeException("Internal inconsistency: " +
            "A (not new) checking-out statement is not in deathmarch.");
    }

I've debuged and I've seen that the ps Object has the value
"com.mysql.jdbc.JDBC4PreparedStatement@731b21a0: EXCEPTION: java.sql.SQLException: No operations allowed after statement closed."

Discussion

Silvio Fragata Silva - 2015-06-10

If I remove the cache statements
c3p0.maxStatementsPerConnection=0 and
<property name="hibernate.c3p0.max_statements">0</property>

I don't have the problem

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Steve Waldman - 2015-06-11

first, what version of c3p0 are you using?

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Silvio Fragata Silva - 2015-06-11

<groupid>com.mchange</groupid>
<artifactid>c3p0</artifactid>
<version>0.9.5</version>

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Thomas Beckmann - 2015-09-03

Post awaiting moderation.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Silvio Fragata Silva - 2016-02-29

Hi, any news

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Taoufiq Boukhcha - 2016-03-11

+1 for this, I have the same issue with version 0.9.5.2

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

java.sql.SQLException: An SQLException was provoked by the following...

Group

Searches

Help

#133 java.sql.SQLException: An SQLException was provoked by the following failure: java.lang.RuntimeException: Internal inconsistency: A (not new) checking-out statement is not in deathmarch.

Discussion