Re: [EsiGate-users] Connection thread leak

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

We found out something !

It seems that we implemented a too complex error management with bad redirections :
If an error occurs on an ESI inclusion request (ex : "resource not found" on application site), our app returns a 302 to rederect users on a “clean 404 error page” (which is on DRUPAL side).
At this stage, EsiGate don’t manage the 302 and responds with a 502 + the thread is blocked.

We are going to improve our application and the error management to not respond a 302 to an ESI request.

Do you know about EsiGate problems when getting a 302 response in the context of an ESI tag request? Do you have ideas to manage these cases.
Thx,
Emilie Poupon Adam
Responsable Delivery Web Mobile
VSC Technologies

De : fx....@gm... [mailto:fx....@gm...] De la part de Francois-Xavier Bonnet
Envoyé : mardi 12 juillet 2016 15:41
À : Poupon Emilie <EP...@VO...>
Cc : Francois-Xavier Bonnet <fra...@ce...>; Auffredou Thomas <TAu...@vo...>; web...@li...
Objet : Re: [EsiGate-users] Connection thread leak

It looks like you might have an issue with your DNS server. Did you keep the full thread dump where you took the abstract in your first mail?

2016-07-12 12:23 GMT+02:00 Poupon Emilie <EP...@vo...<mailto:EP...@vo...>>:

The complete stack trace for the UnknownHostException

2016-07-12 12:19:53,161 ERROR http-bio-10.98.64.220-51125-exec-10 org.esigate.HttpErrorPage - Error retrieving URL
java.net.UnknownHostException: preprod1.tri.vsct.fr<http://preprod1.tri.vsct.fr>: unknown error
        at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
        at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:907)
        at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1302)
        at java.net.InetAddress.getAllByName0(InetAddress.java:1255)
        at java.net.InetAddress.getAllByName(InetAddress.java:1171)
        at java.net.InetAddress.getAllByName(InetAddress.java:1105)
        at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:44)
        at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:101)
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:318)
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:363)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:219)
        at org.esigate.http.ProxyingHttpClientBuilder$1.execute(ProxyingHttpClientBuilder.java:105)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
        at org.esigate.http.HttpClientRequestExecutor.execute(HttpClientRequestExecutor.java:307)
        at org.esigate.Driver.render(Driver.java:188)
        at org.esigate.esi.IncludeElement.processPage(IncludeElement.java:193)
        at org.esigate.esi.IncludeElement.onTagEnd(IncludeElement.java:81)
        at org.esigate.parser.ParserContextImpl.endElement(ParserContextImpl.java:92)
        at org.esigate.parser.Parser.parse(Parser.java:76)
        at org.esigate.esi.EsiRenderer.render(EsiRenderer.java:122)
        at org.esigate.Driver.performRendering(Driver.java:401)
        at org.esigate.Driver.performRendering(Driver.java:360)
        at org.esigate.Driver.proxy(Driver.java:295)
        at org.esigate.DriverFactory.proxy(DriverFactory.java:403)
        at org.esigate.servlet.ProxyFilter.doFilter(ProxyFilter.java:60)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

activate FetchLogging : Ok, we are doing that right now.

Emilie

De : fx....@gm...<mailto:fx....@gm...> [mailto:fx....@gm...<mailto:fx....@gm...>] De la part de Francois-Xavier Bonnet
Envoyé : mardi 12 juillet 2016 11:57

À : Poupon Emilie <EP...@VO...<mailto:EP...@VO...>>
Cc : Francois-Xavier Bonnet <fra...@ce...<mailto:fra...@ce...>>; Auffredou Thomas <TAu...@vo...<mailto:TAu...@vo...>>; web...@li...<mailto:web...@li...>
Objet : Re: [EsiGate-users] Connection thread leak

Could you send the complete stack trace for the UnknownHostException?
It would also be helpful to activate FetchLogging as explained here: http://www.esigate.org/reference.html#FragmentLogging_and_FetchLogging_extensions

2016-07-12 11:46 GMT+02:00 Poupon Emilie <EP...@vo...<mailto:EP...@vo...>>:

1/ When falling this morning (with no user), we had no error in our EsiGate logs.
What we see now is that the Java Application responds with some 302 to the EsiGate (bloc inclusion) : we are searching around these responses if they could explain the instance “death”.

2/ When falling past night (with some users), we had one error message we didn’t explain yet :
org.esigate.HttpErrorPage - Error retrieving URL
java.net.UnknownHostException: www.transilien.com<http://www.transilien.com>: unknown error
            at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
            at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:907)
            at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1302)
            at java.net.InetAddress.getAllByName0(InetAddress.java:1255)
            at java.net.InetAddress.getAllByName(InetAddress.java:1171)
The EsiGate should not access the www.transilien.com<http://www.transilien.com> domain as an external domain, in fact it is the website itself.

Emilie Poupon Adam
Responsable Delivery Web Mobile
VSC Technologies

De : fx....@gm...<mailto:fx....@gm...> [mailto:fx....@gm...<mailto:fx....@gm...>] De la part de Francois-Xavier Bonnet
Envoyé : mardi 12 juillet 2016 10:44

À : Poupon Emilie <EP...@VO...<mailto:EP...@VO...>>
Cc : Francois-Xavier Bonnet <fra...@ce...<mailto:fra...@ce...>>; Auffredou Thomas <TAu...@vo...<mailto:TAu...@vo...>>; web...@li...<mailto:web...@li...>
Objet : Re: [EsiGate-users] Connection thread leak

Do you have some errors in the log files of EsiGate?

2016-07-12 9:33 GMT+02:00 Poupon Emilie <EP...@vo...<mailto:EP...@vo...>>:
Hi,

I did suscribe yesterday (but after sendind my e-mail) !

What is the value of {{socketTimeout}} ?
socketTimeout = 45000

I can see in your configuration that you let the default value for maxConnectionsPerHost which is 20.
Yes, we saw that point and agreed to put an upper value (70) since the number of connections is limited at entry.
We are going to test this today.

When the problem occurs, do you have a lot of users on the application? How many hits/s?
First time we saw the problem we had an several users connected, but yesterday problem occurred on a server not used at all (only life check was running).
We didn't find out how make the problem occurs, we tried with several stress tests and not succeed.

Does the problem fix itself when the load decreases or not?
NO ! We had to reboot the servers, even after several hours, number of open thread didn’t decrease.

Is EsiGate used also for static files (images, js, css) from the drupal backend? If so why did you disable the cache.
EsiGate mix transactional content from a Java application and editorial content from a drupal backend.
We are using a VARNISH cache between the EsiGate and the Drupal backend.

Find here the JVM / Tomcat parameters set on our ESIGate server :
connectionTimeout = 20000
maxThreads = 70
acceptCount = 50
compression no value
compressionMinSize no value
compressableMimeType no value
JDK = 1.8u45 64bits
Tomcat = 7.0.53
JVM params = Xms512m Xmx512m XX:+UseConcMarkSweepGC

Thx,
Emilie Poupon Adam

De : fx....@gm...<mailto:fx....@gm...> [mailto:fx....@gm...<mailto:fx....@gm...>] De la part de Francois-Xavier Bonnet
Envoyé : mardi 12 juillet 2016 08:36
À : Poupon Emilie <EP...@VO...<mailto:EP...@VO...>>
Cc : Francois-Xavier Bonnet <fra...@ce...<mailto:fra...@ce...>>; Auffredou Thomas <TAu...@vo...<mailto:TAu...@vo...>>; web...@li...<mailto:web...@li...>

Objet : Re: [EsiGate-users] Connection thread leak

Hi,

Please subscribe to the mailing list for faster answers.

What is the value of {{socketTimeout}} ?
I can see in your configuration that you let the default value for maxConnectionsPerHost which is 20. When the problem occurs, do you have a lot of users on the application? How many hits/s? Does the problem fix itself when the load decreases or not?
Is EsiGate used also for static files (images, js, css) from the drupal backend? If so why did you disable the cache.

2016-07-11 20:02 GMT+02:00 Poupon Emilie <EP...@vo...<mailto:EP...@vo...>>:
Hi,

I’m working with Thomas, he left few minutes ago (he is coming back on wednesday).
We still have the issue and no idea about the causes.

ESI Gate 5.1 + esigate.properties
Thx,
Emilie Poupon Adam

De : fx....@gm...<mailto:fx....@gm...> [mailto:fx....@gm...<mailto:fx....@gm...>] De la part de Francois-Xavier Bonnet
Envoyé : lundi 11 juillet 2016 18:22
À : Auffredou Thomas <TAu...@vo...<mailto:TAu...@vo...>>
Cc : web...@li...<mailto:web...@li...>; Poupon Emilie <EP...@VO...<mailto:EP...@VO...>>
Objet : Re: [EsiGate-users] Connection thread leak

Hi,

In order to help me understand your issue, could you give more information: Which version of EsiGate are you using? Could you share your esigate.properties configuration file?

Please subscribe to the mailing so that your future messages will be forwarded directly without requiring an administrator to validate them.

2016-07-11 17:36 GMT+02:00 Auffredou Thomas <TAu...@vo...<mailto:TAu...@vo...>>:
Hi,

I’m experimenting issues with the ESI clientPool.
At some point (I can’t figure out what are the causes), threads start to grow until reaching the thread pool saturation.
The leaks happens linearly when requesting a healthcheck url, and any other request accelerates the behavior.

The following threaddump extract shows it hangs while acquiring the http client.

"http-bio-10.98.48.138-51125-exec-1" #40 daemon prio=5 os_prio=0 tid=0x00007fb760001800 nid=0x6ff4 waiting on condition [0x00007fb7dc507000]
   java.lang.Thread.State: WAITING (parking)
                at sun.misc.Unsafe.park(Native Method)
                - parking to wait for  <0x00000000eb970880> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
                at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
                at org.apache.http.pool.PoolEntryFuture.await(PoolEntryFuture.java:133)
                at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:282)
                at org.apache.http.pool.AbstractConnPool.access$000(AbstractConnPool.java:64)
                at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:177)
                at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:170)
                at org.apache.http.pool.PoolEntryFuture.get(PoolEntryFuture.java:102)
                at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:244)
                at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:231)
                at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:173)
                at org.esigate.http.ProxyingHttpClientBuilder$1.execute(ProxyingHttpClientBuilder.java:105)
                at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)
                at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)
                at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
                at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
                at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
                at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
                at org.esigate.http.HttpClientRequestExecutor.execute(HttpClientRequestExecutor.java:307)
                at org.esigate.Driver.render(Driver.java:188)
                at org.esigate.esi.IncludeElement.processPage(IncludeElement.java:193)
                at org.esigate.esi.IncludeElement.onTagEnd(IncludeElement.java:81)
                at org.esigate.parser.ParserContextImpl.endElement(ParserContextImpl.java:92)
                at org.esigate.parser.Parser.parse(Parser.java:76)
                at org.esigate.esi.EsiRenderer.render(EsiRenderer.java:122)
                at org.esigate.Driver.performRendering(Driver.java:401)
                at org.esigate.Driver.performRendering(Driver.java:360)
                at org.esigate.Driver.proxy(Driver.java:295)
                at org.esigate.DriverFactory.proxy(DriverFactory.java:403)
                at org.esigate.servlet.ProxyFilter.doFilter(ProxyFilter.java:60)
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
                at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
                at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
                at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
                at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
                at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
                at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
                at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
                at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
                at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
                at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
                at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)
                - locked <0x00000000ec8c1e70> (a org.apache.tomcat.util.net.SocketWrapper)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                at java.lang.Thread.run(Thread.java:745)

Did someone already encountered this issue ?

Thanks,
Thomas

------------------------------------------------------------------------------
Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
Francisco, CA to explore cutting-edge tech and listen to tech luminaries
present their vision of the future. This family event has something for
everyone, including kids. Get more information and register today.
http://sdm.link/attshape
_______________________________________________
Webassembletool-users mailing list
Web...@li...<mailto:Web...@li...>
https://lists.sourceforge.net/lists/listinfo/webassembletool-users