#1715 Using WebClient for specific websites is causing increasing number of live threads which "hangs" in memory.

[Maksym G.]

Hi,
I found that using WebClient for specific websites is causing increasing number of live threads which "hangs" in memory.

You can reproduce this bug for next website: http://lmktec.co.uk
I attached thread chart from visualvm before using webclient, after using webclient for geting one page from that website and after getting 20 different pages from that website.
Before there were 62 live threads, after getting one page it were 70 live threads, after 20 - more than 300 hundred, after 100 pages - more than 1 thousend.
The threads stay in memory forever, and can be killed only after tomcat restart.

Here are WebClient options that I an using:

final WebClient webClient = new WebClient(BrowserVersion.FIREFOX_38);
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
webClient.getOptions().setJavaScriptEnabled(true);
webClient.getCookieManager().setCookiesEnabled(true);
webClient.getOptions().setUseInsecureSSL(true);

webClient.getOptions().setTimeout(PAGE_LOAD_TIMEOUT); // 20 sec
webClient.setJavaScriptTimeout(JAVASCRIPT_TIMEOUT); // 10 sec
webClient.getOptions().setRedirectEnabled(true);
webClient.getOptions().setCssEnabled(false);

I am "cleaning resources" using next code:

webClient.getJavaScriptEngine().shutdown();
webClient.close();
webClient.getCache().clear();

Note: this bug is not reproducible if you will turn off the javascript.

I assume that this bug is reproducible for websites which are using websockets. Maybe WebClient opens websocket connections and didn't close them, but just listening to channel?

[Marc]

I've also encountered this leak on 2.18. It's quite easy to reproduce in fact, just run WebSocketTest from the testsuite, it will leak many threads for each browser that supports websockets. The problem is that the WebSocket class does WebSocketClient.connect, which ends up creating a QueueThreadPool. This threadpool will not be closed until application shutdown.

Here's an example stack trace of the threads that are generated and never die (I end up with thousands of them):

WebSocketClient@2119568777-190401-selector-WebSocketClientSelectorManager@55bba3e8/0" prio=10 tid=0x00007fd68c184800 nid=0x40f1 runnable [0x00007fd8c85ee000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x000000062a2a09f0> (a sun.nio.ch.Util$2)
- locked <0x000000062a2a0a00> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000062a2a09a8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:600)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:744)

I tried to address it by calling client.stop(); from onWebSocketClose() but apprently this is not always called.

[Ahmed Ashour]

Thanks for reporting, fixed in SVN.

You can get latest snapshot or build [1] after it succeeds.

[1] https://ci.canoo.com/teamcity/viewLog.html?buildTypeId=HtmlUnit_FastBuild&buildId=lastSuccessful&tab=artifacts

[Marc]

Sorry but it's still happenning on 2.19, maybe more than before. It happens with http://www.hirschdruck.de/ for example. Is there by any chance an easy way to disable websocket support without shutting down the whole javascript engine while these leaks are fixed?

[Ahmed Ashour]

Currently, the WebSocket are closed, when the HtmlPage.cleanUp() is called.

• Do you call HtmlPage.cleanUp()?
• Do you expect HtmlUnit to close the websocket without HtmlPage.cleanUp()?

[Marc]

Hi,

• Yes, I call cleanup.
• No, I don't expect websockets to be closed if cleanup is not called.

Unfortunately I cannot get any more thread dumps from this issue because I'm currently using a browser in which websockets are disabled. I tried to set up a reproductible test case with websockets and i wasn't able to reproduce the leak. So feel free to close this issue for the time being.

[Ahmed Ashour]

Great, please advise if you face any further issue.