Re: [EsiGate-users] Connection thread leak

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

EsiGate is following automatically redirects when performing an ESI include
so this should work unless the response from the server is invalid. There
should be more details about the 502 in the logs or in the error page.

2016-07-12 17:30 GMT+02:00 Poupon Emilie <EP...@vo...>:

> We found out something !
>
>
>
> It seems that we implemented a too complex error management with bad
> redirections :
>
> If an error occurs on an ESI inclusion request (ex : "resource not found"
> on application site), our app returns a 302 to rederect users on a “clean
> 404 error page” (which is on DRUPAL side).
>
> At this stage, EsiGate don’t manage the 302 and responds with a 502 + the
> thread is blocked.
>
>
>
> We are going to improve our application and the error management to not
> respond a 302 to an ESI request.
>
>
>
> Do you know about EsiGate problems when getting a 302 response in the
> context of an ESI tag request? Do you have ideas to manage these cases.
>
> Thx,
>
> Emilie Poupon Adam
>
> Responsable Delivery Web Mobile
>
> VSC Technologies
>
>
>
> *De :* fx....@gm... [mailto:fx....@gm...] *De la part de*
> Francois-Xavier Bonnet
> *Envoyé :* mardi 12 juillet 2016 15:41
>
> *À :* Poupon Emilie <EP...@VO...>
> *Cc :* Francois-Xavier Bonnet <fra...@ce...>;
> Auffredou Thomas <TAu...@vo...>;
> web...@li...
> *Objet :* Re: [EsiGate-users] Connection thread leak
>
>
>
> It looks like you might have an issue with your DNS server. Did you keep
> the full thread dump where you took the abstract in your first mail?
>
>
>
> 2016-07-12 12:23 GMT+02:00 Poupon Emilie <EP...@vo...>:
>
>
>
> The complete stack trace for the UnknownHostException
>
>
>
> 2016-07-12 12:19:53,161 ERROR http-bio-10.98.64.220-51125-exec-10
> org.esigate.HttpErrorPage - Error retrieving URL
>
> java.net.UnknownHostException: preprod1.tri.vsct.fr: unknown error
>
>         at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
>
>         at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:907)
>
>         at
> java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1302)
>
>         at java.net.InetAddress.getAllByName0(InetAddress.java:1255)
>
>         at java.net.InetAddress.getAllByName(InetAddress.java:1171)
>
>         at java.net.InetAddress.getAllByName(InetAddress.java:1105)
>
>         at
> org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:44)
>
>         at
> org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:101)
>
>         at
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:318)
>
>         at
> org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:363)
>
>         at
> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:219)
>
>         at
> org.esigate.http.ProxyingHttpClientBuilder$1.execute(ProxyingHttpClientBuilder.java:105)
>
>         at
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)
>
>         at
> org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)
>
>         at
> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
>
>         at
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
>
>         at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
>
>         at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
>
>         at
> org.esigate.http.HttpClientRequestExecutor.execute(HttpClientRequestExecutor.java:307)
>
>         at org.esigate.Driver.render(Driver.java:188)
>
>         at
> org.esigate.esi.IncludeElement.processPage(IncludeElement.java:193)
>
>         at org.esigate.esi.IncludeElement.onTagEnd(IncludeElement.java:81)
>
>         at
> org.esigate.parser.ParserContextImpl.endElement(ParserContextImpl.java:92)
>
>         at org.esigate.parser.Parser.parse(Parser.java:76)
>
>         at org.esigate.esi.EsiRenderer.render(EsiRenderer.java:122)
>
>         at org.esigate.Driver.performRendering(Driver.java:401)
>
>         at org.esigate.Driver.performRendering(Driver.java:360)
>
>         at org.esigate.Driver.proxy(Driver.java:295)
>
>         at org.esigate.DriverFactory.proxy(DriverFactory.java:403)
>
>         at org.esigate.servlet.ProxyFilter.doFilter(ProxyFilter.java:60)
>
>         at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
>
>         at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
>
>         at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
>
>         at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
>
>         at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
>
>         at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
>
>         at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
>
>         at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
>
>         at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
>
>         at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
>
>         at
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
>
>         at
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
>
>         at
> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)
>
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
>         at java.lang.Thread.run(Thread.java:745)
>
>
>
>
>
> activate FetchLogging : Ok, we are doing that right now.
>
>
>
> Emilie
>
>
>
> *De :* fx....@gm... [mailto:fx....@gm...] *De la part de*
> Francois-Xavier Bonnet
> *Envoyé :* mardi 12 juillet 2016 11:57
>
>
> *À :* Poupon Emilie <EP...@VO...>
> *Cc :* Francois-Xavier Bonnet <fra...@ce...>;
> Auffredou Thomas <TAu...@vo...>;
> web...@li...
> *Objet :* Re: [EsiGate-users] Connection thread leak
>
>
>
> Could you send the complete stack trace for the UnknownHostException?
>
> It would also be helpful to activate FetchLogging as explained here:
> http://www.esigate.org/reference.html#FragmentLogging_and_FetchLogging_extensions
>
>
>
> 2016-07-12 11:46 GMT+02:00 Poupon Emilie <EP...@vo...>:
>
>
>
> 1/ When falling this morning (with no user), we had no error in our
> EsiGate logs.
>
> What we see now is that the Java Application responds with some 302 to the
> EsiGate (bloc inclusion) : we are searching around these responses if they
> could explain the instance “death”.
>
>
>
> 2/ When falling past night (with some users), we had one error message we
> didn’t explain yet :
>
> org.esigate.HttpErrorPage - Error retrieving URL
>
> java.net.UnknownHostException: www.transilien.com: unknown error
>
>             at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
>
>             at
> java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:907)
>
>             at
> java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1302)
>
>             at java.net.InetAddress.getAllByName0(InetAddress.java:1255)
>
>             at java.net.InetAddress.getAllByName(InetAddress.java:1171)
>
> The EsiGate should not access the www.transilien.com domain as an
> external domain, in fact it is the website itself.
>
>
>
>
>
> Emilie Poupon Adam
>
> Responsable Delivery Web Mobile
>
> VSC Technologies
>
>
>
> *De :* fx....@gm... [mailto:fx....@gm...] *De la part de*
> Francois-Xavier Bonnet
> *Envoyé :* mardi 12 juillet 2016 10:44
>
>
> *À :* Poupon Emilie <EP...@VO...>
> *Cc :* Francois-Xavier Bonnet <fra...@ce...>;
> Auffredou Thomas <TAu...@vo...>;
> web...@li...
> *Objet :* Re: [EsiGate-users] Connection thread leak
>
>
>
> Do you have some errors in the log files of EsiGate?
>
>
>
> 2016-07-12 9:33 GMT+02:00 Poupon Emilie <EP...@vo...>:
>
> Hi,
>
>
>
> I did suscribe yesterday (but after sendind my e-mail) !
>
>
>
> What is the value of {{socketTimeout}} ?
>
> socketTimeout = 45000
>
>
>
> I can see in your configuration that you let the default value for
> maxConnectionsPerHost which is 20.
>
> Yes, we saw that point and agreed to put an upper value (70) since the
> number of connections is limited at entry.
>
> We are going to test this today.
>
>
>
> When the problem occurs, do you have a lot of users on the application?
> How many hits/s?
>
> First time we saw the problem we had an several users connected, but
> yesterday problem occurred on a server not used at all (only life check was
> running).
>
> We didn't find out how make the problem occurs, we tried with several
> stress tests and not succeed.
>
>
>
> Does the problem fix itself when the load decreases or not?
>
> NO ! We had to reboot the servers, even after several hours, number of
> open thread didn’t decrease.
>
>
>
> Is EsiGate used also for static files (images, js, css) from the drupal
> backend? If so why did you disable the cache.
>
> EsiGate mix transactional content from a Java application and editorial
> content from a drupal backend.
>
> We are using a VARNISH cache between the EsiGate and the Drupal backend.
>
>
>
> Find here the JVM / Tomcat parameters set on our ESIGate server :
>
> connectionTimeout = 20000
>
> maxThreads = 70
>
> acceptCount = 50
>
> compression no value
>
> compressionMinSize no value
>
> compressableMimeType no value
>
> JDK = 1.8u45 64bits
>
> Tomcat = 7.0.53
>
> JVM params = Xms512m Xmx512m XX:+UseConcMarkSweepGC
>
>
>
> Thx,
>
> Emilie Poupon Adam
>
>
>
> *De :* fx....@gm... [mailto:fx....@gm...] *De la part de*
> Francois-Xavier Bonnet
> *Envoyé :* mardi 12 juillet 2016 08:36
> *À :* Poupon Emilie <EP...@VO...>
> *Cc :* Francois-Xavier Bonnet <fra...@ce...>;
> Auffredou Thomas <TAu...@vo...>;
> web...@li...
>
>
> *Objet :* Re: [EsiGate-users] Connection thread leak
>
>
>
> Hi,
>
>
>
> Please subscribe to the mailing list for faster answers.
>
>
>
> What is the value of {{socketTimeout}} ?
>
> I can see in your configuration that you let the default value for
> maxConnectionsPerHost which is 20. When the problem occurs, do you have a
> lot of users on the application? How many hits/s? Does the problem fix
> itself when the load decreases or not?
>
> Is EsiGate used also for static files (images, js, css) from the drupal
> backend? If so why did you disable the cache.
>
>
>
> 2016-07-11 20:02 GMT+02:00 Poupon Emilie <EP...@vo...>:
>
> Hi,
>
>
>
> I’m working with Thomas, he left few minutes ago (he is coming back on
> wednesday).
>
> We still have the issue and no idea about the causes.
>
>
>
> ESI Gate 5.1 + esigate.properties
>
> Thx,
>
> Emilie Poupon Adam
>
>
>
> *De :* fx....@gm... [mailto:fx....@gm...] *De la part de*
> Francois-Xavier Bonnet
> *Envoyé :* lundi 11 juillet 2016 18:22
> *À :* Auffredou Thomas <TAu...@vo...>
> *Cc :* web...@li...; Poupon Emilie <
> EP...@VO...>
> *Objet :* Re: [EsiGate-users] Connection thread leak
>
>
>
> Hi,
>
>
>
> In order to help me understand your issue, could you give more
> information: Which version of EsiGate are you using? Could you share your
> esigate.properties configuration file?
>
>
>
> Please subscribe to the mailing so that your future messages will be
> forwarded directly without requiring an administrator to validate them.
>
>
>
> 2016-07-11 17:36 GMT+02:00 Auffredou Thomas <
> TAu...@vo...>:
>
> Hi,
>
>
>
> I’m experimenting issues with the ESI clientPool.
>
> At some point (I can’t figure out what are the causes), threads start to
> grow until reaching the thread pool saturation.
>
> The leaks happens linearly when requesting a healthcheck url, and any
> other request accelerates the behavior.
>
>
>
> The following threaddump extract shows it hangs while acquiring the http
> client.
>
>
>
> "http-bio-10.98.48.138-51125-exec-1" #40 daemon prio=5 os_prio=0
> tid=0x00007fb760001800 nid=0x6ff4 waiting on condition [0x00007fb7dc507000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>                 at sun.misc.Unsafe.park(Native Method)
>
>                 - parking to wait for  <0x00000000eb970880> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>                 at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>
>                 at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>                 at
> org.apache.http.pool.PoolEntryFuture.await(PoolEntryFuture.java:133)
>
>                 at
> org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:282)
>
>                 at
> org.apache.http.pool.AbstractConnPool.access$000(AbstractConnPool.java:64)
>
>                 at
> org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:177)
>
>                 at
> org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:170)
>
>                 at
> org.apache.http.pool.PoolEntryFuture.get(PoolEntryFuture.java:102)
>
>                 at
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:244)
>
>                 at
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:231)
>
>                 at
> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:173)
>
>                 at
> org.esigate.http.ProxyingHttpClientBuilder$1.execute(ProxyingHttpClientBuilder.java:105)
>
>                 at
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)
>
>                 at
> org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)
>
>                 at
> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
>
>                 at
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
>
>                 at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
>
>                 at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
>
>                 at
> org.esigate.http.HttpClientRequestExecutor.execute(HttpClientRequestExecutor.java:307)
>
>                 at org.esigate.Driver.render(Driver.java:188)
>
>                 at
> org.esigate.esi.IncludeElement.processPage(IncludeElement.java:193)
>
>                 at
> org.esigate.esi.IncludeElement.onTagEnd(IncludeElement.java:81)
>
>                 at
> org.esigate.parser.ParserContextImpl.endElement(ParserContextImpl.java:92)
>
>                 at org.esigate.parser.Parser.parse(Parser.java:76)
>
>                 at
> org.esigate.esi.EsiRenderer.render(EsiRenderer.java:122)
>
>                 at org.esigate.Driver.performRendering(Driver.java:401)
>
>                 at org.esigate.Driver.performRendering(Driver.java:360)
>
>                 at org.esigate.Driver.proxy(Driver.java:295)
>
>                 at org.esigate.DriverFactory.proxy(DriverFactory.java:403)
>
>                 at
> org.esigate.servlet.ProxyFilter.doFilter(ProxyFilter.java:60)
>
>                 at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
>
>                 at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
>
>                 at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
>
>                 at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
>
>                 at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
>
>                 at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
>
>                 at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
>
>                 at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
>
>                 at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
>
>                 at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
>
>                 at
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
>
>                 at
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
>
>                 at
> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)
>
>                 - locked <0x00000000ec8c1e70> (a
> org.apache.tomcat.util.net.SocketWrapper)
>
>                 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
>                 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
>                 at java.lang.Thread.run(Thread.java:745)
>
>
>
> Did someone already encountered this issue ?
>
>
>
> Thanks,
>
> Thomas
>
>
>
> ------------------------------------------------------------------------------
> Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
> Francisco, CA to explore cutting-edge tech and listen to tech luminaries
> present their vision of the future. This family event has something for
> everyone, including kids. Get more information and register today.
> http://sdm.link/attshape
> _______________________________________________
> Webassembletool-users mailing list
> Web...@li...
> https://lists.sourceforge.net/lists/listinfo/webassembletool-users
>
>
>
>
>
>
>
>
>
>
>