Recently, HtmlUnit did not return from a certain HTTPS request. The thread dump below suggests that it was stuck right in the middle of the SSL handshake, after the connect. Usually, a certain timeout ensures that in this case the code will return after a while with a timeout exception. But during SSL handshake that timeout is currently 0 (ie. no timeout).
This can be checked by hacking a "System.out.println(socket.getSoTimeout());" as the first line into HtmlUnitSSLConnectionSocketFactory.connectSocket().
HtmlUnit does configure a socket timeout, but it does so only for the RequestConfig, whose settings will be used only after the connection is established. During connect/handshake the SocketConfig will be used, which is currently not set up specially, hence the timeout value is 0.
So when it comes to (re-)configuring the timeout, HtmlUnit should not only (re-)configure the default RequestConfig, but also the default SocketConfig, at the HttpClientBuilder.
Thanks,
J.
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:422)
com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:460)
com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)
- locked java.lang.Object@79ec41c0
com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1188)
- locked java.lang.Object@5d0e2c9
com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1215)
com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1199)
org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:275)
org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:254)
com.gargoylesoftware.htmlunit.HtmlUnitSSLConnectionSocketFactory.connectSocket(HtmlUnitSSLConnectionSocketFactory.java:155)
org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:117)
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:314)
org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:363)
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:219)
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)
org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
com.gargoylesoftware.htmlunit.HttpWebConnection.getResponse(HttpWebConnection.java:173)
com.gargoylesoftware.htmlunit.WebClient.loadWebResponseFromWebConnection(WebClient.java:1312)
com.gargoylesoftware.htmlunit.WebClient.loadWebResponse(WebClient.java:1229)
com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:338)
com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:407)
com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:392)
Hi Joerg,
hopefully this is fixed now in SVN. Please check and report back.
I do not currently have time to test the new snapshot.
But as the fix in svn is the same as the one we have been using for a month (without any problems) I say it is working.
Hey, Ronald, works like a charm. Thanks!
Hi Carsten/Joerg,
thanks for the fast feedback.