There seems to be a situation on the code which sometimes leads to Deadlocks when clicking on a DomElement. A similar deadlock was described in ticket #1259 comments.
I'm currently unsure on whether this is page specific or it can happen on any situation due to a race condition.
Here's part of a thread dump which describes the deadlock:
...
Found one Java-level deadlock:
"JS executor for com.gargoylesoftware.htmlunit.WebClient@3970bb27":
waiting to lock monitor 0x00007fbb5dc5ff18 (object 0x0000000670d3fe10, a com.gargoylesoftware.htmlunit.html.HtmlPage),
which is held by “main”
“main“:
waiting to lock monitor 0x00007fbbac777b28 (object 0x00000006749e6fd0, a com.gargoylesoftware.htmlunit.html.HtmlPage),
which is held by "JS executor for com.gargoylesoftware.htmlunit.WebClient@3970bb27"
Java stack information for the threads listed above:
"JS executor for com.gargoylesoftware.htmlunit.WebClient@3970bb27":
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:935)
- waiting to lock <0x0000000670d3fe10> (a com.gargoylesoftware.htmlunit.html.HtmlPage)
at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:628)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:513)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.callFunction(JavaScriptEngine.java:885)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.callFunction(JavaScriptEngine.java:857)
at com.gargoylesoftware.htmlunit.InteractivePage.executeJavaScriptFunctionIfPossible(InteractivePage.java:216)
at com.gargoylesoftware.htmlunit.javascript.host.event.EventListenersContainer.executeEventListeners(EventListenersContainer.java:260)
at com.gargoylesoftware.htmlunit.javascript.host.event.EventListenersContainer.executeListeners(EventListenersContainer.java:392)
at com.gargoylesoftware.htmlunit.javascript.host.event.EventTarget.executeEventLocally(EventTarget.java:119)
at com.gargoylesoftware.htmlunit.html.HtmlImage.doOnLoad(HtmlImage.java:201)
at com.gargoylesoftware.htmlunit.html.HtmlImage.onAddedToPage(HtmlImage.java:103)
at com.gargoylesoftware.htmlunit.html.HtmlPage.notifyNodeAdded(HtmlPage.java:1733)
at com.gargoylesoftware.htmlunit.html.DomNode.fireAddition(DomNode.java:1118)
at com.gargoylesoftware.htmlunit.html.DomNode.appendChild(DomNode.java:1011)
at com.gargoylesoftware.htmlunit.javascript.host.dom.Node.appendChild(Node.java:235)
at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at net.sourceforge.htmlunit.corejs.javascript.MemberBox.invoke(MemberBox.java:153)
at net.sourceforge.htmlunit.corejs.javascript.FunctionObject.call(FunctionObject.java:449)
at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1536)
at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:798)
at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:105)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.doTopCall(ContextFactory.java:411)
at com.gargoylesoftware.htmlunit.javascript.HtmlUnitContextFactory.doTopCall(HtmlUnitContextFactory.java:309)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3286)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$4.doRun(JavaScriptEngine.java:878)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:939)
- locked <0x00000006749e6fd0> (a com.gargoylesoftware.htmlunit.html.HtmlPage)
at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:628)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:513)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.callFunction(JavaScriptEngine.java:885)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.callFunction(JavaScriptEngine.java:857)
at com.gargoylesoftware.htmlunit.InteractivePage.executeJavaScriptFunctionIfPossible(InteractivePage.java:216)
at com.gargoylesoftware.htmlunit.javascript.host.event.EventListenersContainer.executeEventListeners(EventListenersContainer.java:260)
at com.gargoylesoftware.htmlunit.javascript.host.event.EventListenersContainer.executeListeners(EventListenersContainer.java:392)
at com.gargoylesoftware.htmlunit.javascript.host.event.EventTarget.executeEventLocally(EventTarget.java:119)
at com.gargoylesoftware.htmlunit.html.HtmlScript.executeEventIfBrowserHasFeature(HtmlScript.java:425)
at com.gargoylesoftware.htmlunit.html.HtmlScript.executeScriptIfNeeded(HtmlScript.java:401)
at com.gargoylesoftware.htmlunit.html.HtmlScript$3.execute(HtmlScript.java:277)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.doProcessPostponedActions(JavaScriptEngine.java:1002)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.access$500(JavaScriptEngine.java:103)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:949)
at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:628)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:513)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.callFunction(JavaScriptEngine.java:885)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.callFunction(JavaScriptEngine.java:857)
at com.gargoylesoftware.htmlunit.InteractivePage.executeJavaScriptFunctionIfPossible(InteractivePage.java:216)
at com.gargoylesoftware.htmlunit.javascript.background.JavaScriptFunctionJob.runJavaScript(JavaScriptFunctionJob.java:52)
at com.gargoylesoftware.htmlunit.javascript.background.JavaScriptExecutionJob.run(JavaScriptExecutionJob.java:102)
at com.gargoylesoftware.htmlunit.javascript.background.JavaScriptJobManagerImpl.runSingleJob(JavaScriptJobManagerImpl.java:427)
at com.gargoylesoftware.htmlunit.javascript.background.DefaultJavaScriptExecutor.run(DefaultJavaScriptExecutor.java:156)
at java.lang.Thread.run(Thread.java:744)
“main”:
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:935)
- waiting to lock <0x00000006749e6fd0> (a com.gargoylesoftware.htmlunit.html.HtmlPage)
at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:628)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:513)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.callFunction(JavaScriptEngine.java:885)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.callFunction(JavaScriptEngine.java:857)
at com.gargoylesoftware.htmlunit.InteractivePage.executeJavaScriptFunctionIfPossible(InteractivePage.java:216)
at com.gargoylesoftware.htmlunit.javascript.host.event.EventListenersContainer.executeEventListeners(EventListenersContainer.java:260)
at com.gargoylesoftware.htmlunit.javascript.host.event.EventListenersContainer.executeBubblingListeners(EventListenersContainer.java:324)
at com.gargoylesoftware.htmlunit.javascript.host.event.EventTarget.fireEvent(EventTarget.java:221)
at com.gargoylesoftware.htmlunit.html.DomElement$2.run(DomElement.java:1160)
at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:628)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:513)
at com.gargoylesoftware.htmlunit.html.DomElement.fireEvent(DomElement.java:1165)
at com.gargoylesoftware.htmlunit.html.HtmlPage.executeEventHandlersIfNeeded(HtmlPage.java:1237)
at com.gargoylesoftware.htmlunit.html.HtmlPage.cleanUp(HtmlPage.java:332)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:474)
at com.gargoylesoftware.htmlunit.WebClient.loadDownloadedResponses(WebClient.java:2167)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.doProcessPostponedActions(JavaScriptEngine.java:982)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.processPostponedActions(JavaScriptEngine.java:1072)
at com.gargoylesoftware.htmlunit.html.DomElement.click(DomElement.java:789)
at com.gargoylesoftware.htmlunit.html.DomElement.click(DomElement.java:732)
- locked <0x0000000670d3fe10> (a com.gargoylesoftware.htmlunit.html.HtmlPage)
at com.gargoylesoftware.htmlunit.html.DomElement.click(DomElement.java:679)
…
at java.lang.Thread.run(Thread.java:744)
Diff:
From what I've seen, this problem on the stacktrace basically relates to the execution of onLoad event of an HtmlImage object.
I've also found deadlocks when executing onUnload and onBeforeUnload of whole HtmlPages.
Could this be related to frames and event propagation (bubbling)? If a page contains an iframe, it might get notified about events that happen inside this frame. So there could be multiple threads locking on two different HtmlPage instances: The instance with the "parent" page and the one with the iframe.
Maybe it would be a fix to always lock on the parent HtmlPage (code in question: JavascriptEngine:875)
Apart from that, I'm not sure if an event from an iframe should bubble up to the parent document at all (referring to http://stackoverflow.com/a/2046913).
I just got another JVM (1.8) dead lock from htmlunit 2.25.
Found one Java-level deadlock:
"JS executor for com.gargoylesoftware.htmlunit.WebClient@14abd28e":
waiting to lock monitor 0x00007f69e00259d8 (object 0x00007f6ddfc45070, a com.gargoylesoftware.htmlunit.html.HtmlPage),
which is held by "pool-30545-thread-1"
"pool-30545-thread-1":
waiting to lock monitor 0x00007f69901b6668 (object 0x00007f6de04873b8, a com.gargoylesoftware.htmlunit.html.HtmlPage),
which is held by "JS executor for com.gargoylesoftware.htmlunit.WebClient@14abd28e"
Java stack information for the threads listed above:
"JS executor for com.gargoylesoftware.htmlunit.WebClient@14abd28e":
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:875)
"pool-30545-thread-1":
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:875)
- waiting to lock <0x00007f6de04873b8> (a com.gargoylesoftware.htmlunit.html.HtmlPage)
at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:628)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:515)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.compile(JavaScriptEngine.java:718)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.compile(JavaScriptEngine.java:683)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadJavaScriptFromUrl(HtmlPage.java:1069)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadExternalJavaScriptFile(HtmlPage.java:966)
at com.gargoylesoftware.htmlunit.html.HtmlScript.executeScriptIfNeeded(HtmlScript.java:352)
at com.gargoylesoftware.htmlunit.html.HtmlScript$2.execute(HtmlScript.java:239)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.doProcessPostponedActions(JavaScriptEngine.java:939)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.access$500(JavaScriptEngine.java:100)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:889)
at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:628)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:515)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:775)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:751)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadExternalJavaScriptFile(HtmlPage.java:981)
at com.gargoylesoftware.htmlunit.html.HtmlScript.executeScriptIfNeeded(HtmlScript.java:352)
at com.gargoylesoftware.htmlunit.html.HtmlScript$2.execute(HtmlScript.java:239)
at com.gargoylesoftware.htmlunit.html.HtmlScript.onAllChildrenAddedToPage(HtmlScript.java:258)
at com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.endElement(HTMLParser.java:781)
at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source)
at com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.endElement(HTMLParser.java:738)
at net.sourceforge.htmlunit.cyberneko.HTMLTagBalancer.callEndElement(HTMLTagBalancer.java:1243)
at net.sourceforge.htmlunit.cyberneko.HTMLTagBalancer.endElement(HTMLTagBalancer.java:1143)
at net.sourceforge.htmlunit.cyberneko.filters.DefaultFilter.endElement(DefaultFilter.java:226)
at net.sourceforge.htmlunit.cyberneko.filters.NamespaceBinder.endElement(NamespaceBinder.java:345)
at net.sourceforge.htmlunit.cyberneko.HTMLScanner$ContentScanner.scanEndElement(HTMLScanner.java:3154)
at net.sourceforge.htmlunit.cyberneko.HTMLScanner$ContentScanner.scan(HTMLScanner.java:2117)
at net.sourceforge.htmlunit.cyberneko.HTMLScanner.scanDocument(HTMLScanner.java:945)
at net.sourceforge.htmlunit.cyberneko.HTMLConfiguration.parse(HTMLConfiguration.java:521)
at net.sourceforge.htmlunit.cyberneko.HTMLConfiguration.parse(HTMLConfiguration.java:472)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.parse(HTMLParser.java:988)
at com.gargoylesoftware.htmlunit.html.HTMLParser.parse(HTMLParser.java:246)
at com.gargoylesoftware.htmlunit.html.HTMLParser.parseHtml(HTMLParser.java:188)
at com.gargoylesoftware.htmlunit.DefaultPageCreator.createHtmlPage(DefaultPageCreator.java:267)
at com.gargoylesoftware.htmlunit.DefaultPageCreator.createPage(DefaultPageCreator.java:155)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:517)
at com.gargoylesoftware.htmlunit.javascript.host.html.HTMLDocument.close(HTMLDocument.java:1019)
at sun.reflect.GeneratedMethodAccessor429.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at net.sourceforge.htmlunit.corejs.javascript.MemberBox.invoke(MemberBox.java:153)
at net.sourceforge.htmlunit.corejs.javascript.FunctionObject.call(FunctionObject.java:448)
at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1540)
at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:800)
at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:105)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.doTopCall(ContextFactory.java:413)
at com.gargoylesoftware.htmlunit.javascript.HtmlUnitContextFactory.doTopCall(HtmlUnitContextFactory.java:292)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3264)
at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.exec(InterpretedFunction.java:115)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$3.doRun(JavaScriptEngine.java:766)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:879)
- locked <0x00007f6ddfc45070> (a com.gargoylesoftware.htmlunit.html.HtmlPage)
at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:628)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:515)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:775)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:751)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadExternalJavaScriptFile(HtmlPage.java:981)
at com.gargoylesoftware.htmlunit.html.HtmlScript.executeScriptIfNeeded(HtmlScript.java:352)
at com.gargoylesoftware.htmlunit.html.HtmlScript$2.execute(HtmlScript.java:239)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.doProcessPostponedActions(JavaScriptEngine.java:939)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.access$500(JavaScriptEngine.java:100)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:889)
at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:628)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:515)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.callFunction(JavaScriptEngine.java:824)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.callFunction(JavaScriptEngine.java:796)
at com.gargoylesoftware.htmlunit.html.HtmlPage.executeJavaScriptFunctionIfPossible(HtmlPage.java:2454)
at com.gargoylesoftware.htmlunit.javascript.host.event.EventListenersContainer.executeEventListeners(EventListenersContainer.java:259)
at com.gargoylesoftware.htmlunit.javascript.host.event.EventListenersContainer.executeCapturingListeners(EventListenersContainer.java:339)
at com.gargoylesoftware.htmlunit.javascript.host.event.EventTarget.fireEvent(EventTarget.java:147)
at com.gargoylesoftware.htmlunit.html.DomElement$2.run(DomElement.java:1389)
at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:628)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:515)
at com.gargoylesoftware.htmlunit.html.DomElement.fireEvent(DomElement.java:1394)
at com.gargoylesoftware.htmlunit.html.HtmlPage.executeEventHandlersIfNeeded(HtmlPage.java:1201)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:286)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:524)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:391)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:309)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:456)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:441)
at com.pan.utils.SingleCrawler$HtmlunitCrawl.call(SingleCrawler.java:1087)
at com.pan.utils.SingleCrawler$HtmlunitCrawl.call(SingleCrawler.java:1007)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Found 1 deadlock.
Heap
PSYoungGen total 9638400K, used 314955K [0x00007f7941000000, 0x00007f7bbf300000, 0x00007f7f41000000)
eden space 8888320K, 1% used [0x00007f7941000000,0x00007f7946af1298,0x00007f7b5f800000)
from space 750080K, 29% used [0x00007f7b91680000,0x00007f7b9ef21c80,0x00007f7bbf300000)
to space 783872K, 0% used [0x00007f7b5f800000,0x00007f7b5f800000,0x00007f7b8f580000)
ParOldGen total 14238208K, used 3357826K [0x00007f6d41000000, 0x00007f70a6080000, 0x00007f7941000000)
object space 14238208K, 23% used [0x00007f6d41000000,0x00007f6e0df20868,0x00007f70a6080000)
Metaspace used 51105K, capacity 53855K, committed 58880K, reserved 59392K