#24 Webserver hanging using c3p0 v0.9.0.4

closed
nobody
None
5
2006-02-15
2005-12-17
Fabian Gonzales
No

My webserver hangs every now and then, and I've never
been able to figure out quite why. These days, every
time it happens, I do a jstack dump, and a lot of
threads show that they are blocked within c3p0. I'm not
very good at reading these stack traces, but it looks
to me like c3p0 is causing the hang.

I'm attaching the jstack output. The first lines got
cut off because my buffer was only (!) set to 9999
lines. But I believe the important information should
be there.

Once I run jstack the webserver starts responding again.

Note that there is definitely something going on
between the webserver and the database, since static
pages load fine, while dynamic ones that access the
database do not.

My setup is Jetty 5.1.4 + Hibernate 2.1.9 + c3p0 0.9.02
+ Mysql 4.1 on Linux.

Discussion

1 2 > >> (Page 1 of 2)
  • Steve Waldman
    Steve Waldman
    2005-12-21

    Logged In: YES
    user_id=175530

    Hi. I don't see the jstack output attached. Also, please atttach your configuration
    dump. Thanks!

     
  • JStack Output

     
    Attachments
  • Logged In: YES
    user_id=727196

    Sorry for the late reply. I had to wait for my server to
    hang again to give you a new output.

    I had to truncate the output because I can only upload files
    <256k, but the threads all pretty much repeat the same
    information.

     
  • Logged In: YES
    user_id=727196

    Here is my c3p0 configuration, as configured in
    hibernate.properties:

    hibernate.c3p0.max_size=90
    hibernate.c3p0.min_size=1
    hibernate.c3p0.timeout=300
    hibernate.c3p0.max_statements=0
    hibernate.c3p0.idle_test_period=300
    hibernate.c3p0.acquire_increment=1

     
  • Steve Waldman
    Steve Waldman
    2006-01-20

    Logged In: YES
    user_id=175530

    Okay. I think the culprit here is some unnecessary synchronization, making all
    acquistions hang when (for some random reason) the acquisition of one
    Connection from the underlying driver freezes. I've undone the unnecessary
    synchronization (as of c3p0-0.9.0.3). Let me know if it solves the problem. And
    thanks for the jstack logs. They were helpful!

     
  • Steve Waldman
    Steve Waldman
    2006-01-20

    • status: open --> closed
     
  • Logged In: YES
    user_id=727196

    I upgraded to c3p0 0.9.0.4 and am still seeing the same
    problem.

    Excerpt from jstack:

    Thread 9133: (state = BLOCKED)
    - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
    -
    com.mchange.v2.resourcepool.BasicResourcePool.awaitAcquire(long)
    @bci=136, line=968 (Interpreted frame)
    -
    com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(long)
    @bci=40, line=208 (Interpreted frame)
    -
    com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection()
    @bci=9, line=260 (Interpreted frame)
    - com.mchange.v2.c3p0.PoolBackedDataSource.getConnection()
    @bci=7, line=94 (Interpreted frame)
    -
    net.sf.hibernate.connection.C3P0ConnectionProvider.getConnection()
    @bci=4, line=33 (Interpreted frame)
    - net.sf.hibernate.impl.BatcherImpl.openConnection()
    @bci=9, line=292 (Interpreted frame)
    - net.sf.hibernate.impl.SessionImpl.connect() @bci=74,
    line=3373 (Interpreted frame)
    - net.sf.hibernate.impl.SessionImpl.connection() @bci=15,
    line=3333 (Interpreted frame)
    -
    net.sf.hibernate.impl.BatcherImpl.prepareStatement(java.lang.String,
    boolean) @bci=12, line=63 (Interpreted frame)
    -
    net.sf.hibernate.persister.EntityPersister.insert(java.lang.Object[],
    boolean[], java.lang.String, java.lang.Object,
    net.sf.hibernate.engine.SessionImplementor) @bci=236,
    line=515 (Interpreted frame)
    -
    net.sf.hibernate.persister.EntityPersister.insert(java.lang.Object[],
    java.lang.Object,
    net.sf.hibernate.engine.SessionImplementor) @bci=43,
    line=426 (Interpreted frame)
    -
    net.sf.hibernate.impl.ScheduledIdentityInsertion.execute()
    @bci=23, line=28 (Interpreted frame)
    -
    net.sf.hibernate.impl.SessionImpl.execute(net.sf.hibernate.impl.SessionImpl$Executable)
    @bci=51, line=2449 (Interpreted frame)
    -
    net.sf.hibernate.impl.SessionImpl.doSave(java.lang.Object,
    net.sf.hibernate.engine.Key,
    net.sf.hibernate.persister.ClassPersister, boolean, boolean,
    net.sf.hibernate.engine.Cascades$CascadingAction,
    java.lang.Object) @bci=341, line=943 (Interpreted frame)
    -
    net.sf.hibernate.impl.SessionImpl.doSave(java.lang.Object,
    java.io.Serializable,
    net.sf.hibernate.persister.ClassPersister, boolean,
    net.sf.hibernate.engine.Cascades$CascadingAction,
    java.lang.Object) @bci=184, line=868 (Interpreted frame)
    -
    net.sf.hibernate.impl.SessionImpl.saveWithGeneratedIdentifier(java.lang.Object,
    net.sf.hibernate.engine.Cascades$CascadingAction,
    java.lang.Object) @bci=88, line=786 (Interpreted frame)
    - net.sf.hibernate.impl.SessionImpl.save(java.lang.Object)
    @bci=69, line=749 (Interpreted frame)
    -
    com.goldengateimages.photobase.PersistenceEngine.save(java.lang.Object)
    @bci=6, line=52 (Interpreted frame)
    -
    com.goldengateimages.photobase.servlet.RequestFilter.saveSearchQuery(javax.servlet.http.HttpServletRequest,
    java.lang.String, java.lang.String) @bci=65, line=152
    (Interpreted frame)
    -
    com.goldengateimages.photobase.servlet.RequestFilter.doFilter(javax.servlet.ServletRequest,
    javax.servlet.ServletResponse, javax.servlet.FilterChain)
    @bci=305, line=79 (Compiled frame)
    -
    org.mortbay.jetty.servlet.WebApplicationHandler$CachedChain.doFilter(javax.servlet.ServletRequest,
    javax.servlet.ServletResponse) @bci=63, line=823
    (Interpreted frame)
    -
    com.goldengateimages.photobase.servlet.HibernateFilter.doFilter(javax.servlet.ServletRequest,
    javax.servlet.ServletResponse, javax.servlet.FilterChain)
    @bci=31, line=78 (Compiled frame)
    -
    org.mortbay.jetty.servlet.WebApplicationHandler$CachedChain.doFilter(javax.servlet.ServletRequest,
    javax.servlet.ServletResponse) @bci=63, line=823 (Compiled
    frame)
    - org.mortbay.http.HttpContext.handle(java.lang.String,
    java.lang.String, org.mortbay.http.HttpRequest,
    org.mortbay.http.HttpResponse) @bci=157, line=1565 (Compiled
    frame)
    -
    org.mortbay.jetty.servlet.WebApplicationContext.handle(java.lang.String,
    java.lang.String, org.mortbay.http.HttpRequest,
    org.mortbay.http.HttpResponse) @bci=14, line=635 (Compiled
    frame)
    -
    org.mortbay.http.HttpContext.handle(org.mortbay.http.HttpRequest,
    org.mortbay.http.HttpResponse) @bci=455, line=1517
    (Interpreted frame)
    -
    org.mortbay.http.HttpServer.service(org.mortbay.http.HttpRequest,
    org.mortbay.http.HttpResponse) @bci=246, line=954 (Compiled
    frame)
    -
    org.mortbay.http.HttpConnection.service(org.mortbay.http.HttpRequest,
    org.mortbay.http.HttpResponse) @bci=24, line=814 (Compiled
    frame)
    - org.mortbay.http.HttpConnection.handle() @bci=17,
    line=831 (Compiled frame)
    -
    org.mortbay.http.SocketListener.handleConnection(java.net.Socket)
    @bci=65, line=244 (Interpreted frame)
    - org.mortbay.util.ThreadedServer.handle(java.lang.Object)
    @bci=19, line=357 (Interpreted frame)
    - org.mortbay.util.ThreadPool$PoolThread.run() @bci=124,
    line=534 (Compiled frame)

    Thread 9132: (state = BLOCKED)
    -
    com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(java.lang.String,
    java.lang.String) @bci=0, line=88 (Interpreted frame)
    -
    com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(java.lang.String,
    java.lang.String) @bci=6, line=109 (Interpreted frame)
    -
    com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1.acquireResource()
    @bci=43, line=89 (Interpreted frame)
    -
    com.mchange.v2.resourcepool.BasicResourcePool.acquireUntil(int)
    @bci=33, line=665 (Interpreted frame)
    -
    com.mchange.v2.resourcepool.BasicResourcePool.access$500(com.mchange.v2.resourcepool.BasicResourcePool,
    int) @bci=2, line=32 (Interpreted frame)
    -
    com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run()
    @bci=33, line=1206 (Interpreted frame)
    -
    com.mchange.v2.async.ThreadPerTaskAsynchronousRunner$TaskThread.run()
    @bci=41, line=255 (Interpreted frame)

    Any ideas?

     
    • status: closed --> open
     
    • summary: Webserver hanging using c3p0 v0.9.0.2 --> Webserver hanging using c3p0 v0.9.0.4
     
  • Complete JStack output from 0.9.0.4 run

     
1 2 > >> (Page 1 of 2)