[EsiGate-users] Connection thread leak

[Auffredou T. <TAu...@vo...>]

Hi,

I'm experimenting issues with the ESI clientPool.
At some point (I can't figure out what are the causes), threads start to grow until reaching the thread pool saturation.
The leaks happens linearly when requesting a healthcheck url, and any other request accelerates the behavior.

The following threaddump extract shows it hangs while acquiring the http client.

"http-bio-10.98.48.138-51125-exec-1" #40 daemon prio=5 os_prio=0 tid=0x00007fb760001800 nid=0x6ff4 waiting on condition [0x00007fb7dc507000]
   java.lang.Thread.State: WAITING (parking)
                at sun.misc.Unsafe.park(Native Method)
                - parking to wait for  <0x00000000eb970880> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
                at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
                at org.apache.http.pool.PoolEntryFuture.await(PoolEntryFuture.java:133)
                at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:282)
                at org.apache.http.pool.AbstractConnPool.access$000(AbstractConnPool.java:64)
                at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:177)
                at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:170)
                at org.apache.http.pool.PoolEntryFuture.get(PoolEntryFuture.java:102)
                at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:244)
                at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:231)
                at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:173)
                at org.esigate.http.ProxyingHttpClientBuilder$1.execute(ProxyingHttpClientBuilder.java:105)
                at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)
                at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)
                at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
                at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
                at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
                at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
                at org.esigate.http.HttpClientRequestExecutor.execute(HttpClientRequestExecutor.java:307)
                at org.esigate.Driver.render(Driver.java:188)
                at org.esigate.esi.IncludeElement.processPage(IncludeElement.java:193)
                at org.esigate.esi.IncludeElement.onTagEnd(IncludeElement.java:81)
                at org.esigate.parser.ParserContextImpl.endElement(ParserContextImpl.java:92)
                at org.esigate.parser.Parser.parse(Parser.java:76)
                at org.esigate.esi.EsiRenderer.render(EsiRenderer.java:122)
                at org.esigate.Driver.performRendering(Driver.java:401)
                at org.esigate.Driver.performRendering(Driver.java:360)
                at org.esigate.Driver.proxy(Driver.java:295)
                at org.esigate.DriverFactory.proxy(DriverFactory.java:403)
                at org.esigate.servlet.ProxyFilter.doFilter(ProxyFilter.java:60)
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
                at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
                at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
                at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
                at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
                at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
                at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
                at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
                at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
                at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
                at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
                at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)
                - locked <0x00000000ec8c1e70> (a org.apache.tomcat.util.net.SocketWrapper)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                at java.lang.Thread.run(Thread.java:745)

Did someone already encountered this issue ?

Thanks,
Thomas