htmlunit-user

[Htmlunit-user] java.net.SocketException: Connection reset

[Gabriel E. S. M. <gab...@gm...>]

Hi,

I want to thank the developers of htmlunit for their time and effort.  
I'm hoping you will be able to help me with an issue that I'm stuck in.  
I've been using htmlunit for a few months to download a file from a 
password-protected site, without issues.  This week, without any changes 
to the code, it stopped working.  I am getting a 
"java.net.SocketException: Connection reset" error as soon as I call 
WebClient.getPage(), before trying to log in.  However, I am able to 
access the page on Firefox.  I can also access other pages on htmlunit 
without issues.  Any help you can give me is going to be greatly 
appreciated!

The code below reproduces the error:

final String path = "https://passprogram.mbta.com";
try {
     final WebClient webClient = new WebClient();
     final HtmlPage page1 = webClient.getPage(path);
     System.out.println(page1.getTitleText());
} catch (IOException | FailingHttpStatusCodeException ex) {
     Logger.getLogger(Main.class.getName()).log(Level.SEVERE, null, ex);
}

I get the following output:

Dec 28, 2015 11:35:15 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing 
request to {s}->https://passprogram.mbta.com:443: Connection reset
Dec 28, 2015 11:35:15 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {s}->https://passprogram.mbta.com:443
Dec 28, 2015 11:35:16 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing 
request to {s}->https://passprogram.mbta.com:443: Connection reset
Dec 28, 2015 11:35:16 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {s}->https://passprogram.mbta.com:443
Dec 28, 2015 11:35:16 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing 
request to {s}->https://passprogram.mbta.com:443: Connection reset
Dec 28, 2015 11:35:16 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {s}->https://passprogram.mbta.com:443
Dec 28, 2015 11:35:16 PM main.Main test
SEVERE: null
java.net.SocketException: Connection reset
     at java.net.SocketInputStream.read(SocketInputStream.java:196)
     at java.net.SocketInputStream.read(SocketInputStream.java:122)
     at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
     at sun.security.ssl.InputRecord.read(InputRecord.java:480)
     at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:946)
     at 
sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1344)
     at 
sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1371)
     at 
sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1355)
     at 
org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394)
     at 
org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353)
     at 
com.gargoylesoftware.htmlunit.httpclient.HtmlUnitSSLConnectionSocketFactory.connectSocket(HtmlUnitSSLConnectionSocketFactory.java:188)
     at 
org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
     at 
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
     at 
org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
     at 
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
     at 
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
     at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
     at 
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
     at 
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
     at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:71)
     at 
com.gargoylesoftware.htmlunit.HttpWebConnection.getResponse(HttpWebConnection.java:177)
     at 
com.gargoylesoftware.htmlunit.WebClient.loadWebResponseFromWebConnection(WebClient.java:1324)
     at 
com.gargoylesoftware.htmlunit.WebClient.loadWebResponse(WebClient.java:1241)
     at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:348)
     at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:417)
     at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:402)
     at main.Main.test(Main.java:160)
     at main.Main.main(Main.java:65)


I am running Ubuntu 14.04 and have the following Java installed:

java version "1.7.0_91"
OpenJDK Runtime Environment (IcedTea 2.6.3) (7u91-2.6.3-0ubuntu0.14.04.1)
OpenJDK 64-Bit Server VM (build 24.91-b01, mixed mode)


Thanks,
Gabriel

Re: [Htmlunit-user] java.net.SocketException: Connection reset

[Gabriel S. <gab...@gm...>]

I contacted the webmasters of the site and found out that they upgraded
their server to require TLS 1.1 or higher.  Apparently htmlunit doesn't try
these protocols by default.  I resolved the problem by adding the following
line after instantiating the WebClient.

webClient.getOptions().setSSLClientProtocols(new String[] { "TLSv1.2",
"TLSv1.1" });

Perhaps the developers should consider changing the defaults of htmlunit so
that it tries the newest and strongest protocols first, mimicking the
behavior of common browsers such as Firefox and Chrome.

Thanks again to the developers and the user community.

- Gabriel



On Mon, Dec 28, 2015 at 11:43 PM, Gabriel E. Sánchez Martínez <
gab...@gm...> wrote:

> Hi,
>
> I want to thank the developers of htmlunit for their time and effort.  I'm
> hoping you will be able to help me with an issue that I'm stuck in.  I've
> been using htmlunit for a few months to download a file from a
> password-protected site, without issues.  This week, without any changes to
> the code, it stopped working.  I am getting a "java.net.SocketException:
> Connection reset" error as soon as I call WebClient.getPage(), before
> trying to log in.  However, I am able to access the page on Firefox.  I can
> also access other pages on htmlunit without issues.  Any help you can give
> me is going to be greatly appreciated!
>
> The code below reproduces the error:
>
> final String path = "https://passprogram.mbta.com";
> try {
>     final WebClient webClient = new WebClient();
>     final HtmlPage page1 = webClient.getPage(path);
>     System.out.println(page1.getTitleText());
> } catch (IOException | FailingHttpStatusCodeException ex) {
>     Logger.getLogger(Main.class.getName()).log(Level.SEVERE, null, ex);
> }
>
> I get the following output:
>
> Dec 28, 2015 11:35:15 PM org.apache.http.impl.execchain.RetryExec execute
> INFO: I/O exception (java.net.SocketException) caught when processing
> request to {s}->https://passprogram.mbta.com:443: Connection reset
> Dec 28, 2015 11:35:15 PM org.apache.http.impl.execchain.RetryExec execute
> INFO: Retrying request to {s}->https://passprogram.mbta.com:443
> Dec 28, 2015 11:35:16 PM org.apache.http.impl.execchain.RetryExec execute
> INFO: I/O exception (java.net.SocketException) caught when processing
> request to {s}->https://passprogram.mbta.com:443: Connection reset
> Dec 28, 2015 11:35:16 PM org.apache.http.impl.execchain.RetryExec execute
> INFO: Retrying request to {s}->https://passprogram.mbta.com:443
> Dec 28, 2015 11:35:16 PM org.apache.http.impl.execchain.RetryExec execute
> INFO: I/O exception (java.net.SocketException) caught when processing
> request to {s}->https://passprogram.mbta.com:443: Connection reset
> Dec 28, 2015 11:35:16 PM org.apache.http.impl.execchain.RetryExec execute
> INFO: Retrying request to {s}->https://passprogram.mbta.com:443
> Dec 28, 2015 11:35:16 PM main.Main test
> SEVERE: null
> java.net.SocketException: Connection reset
>     at java.net.SocketInputStream.read(SocketInputStream.java:196)
>     at java.net.SocketInputStream.read(SocketInputStream.java:122)
>     at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
>     at sun.security.ssl.InputRecord.read(InputRecord.java:480)
>     at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:946)
>     at
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1344)
>     at
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1371)
>     at
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1355)
>     at
> org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394)
>     at
> org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353)
>     at
> com.gargoylesoftware.htmlunit.httpclient.HtmlUnitSSLConnectionSocketFactory.connectSocket(HtmlUnitSSLConnectionSocketFactory.java:188)
>     at
> org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
>     at
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
>     at
> org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
>     at
> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
>     at
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
>     at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
>     at
> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
>     at
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
>     at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:71)
>     at
> com.gargoylesoftware.htmlunit.HttpWebConnection.getResponse(HttpWebConnection.java:177)
>     at
> com.gargoylesoftware.htmlunit.WebClient.loadWebResponseFromWebConnection(WebClient.java:1324)
>     at
> com.gargoylesoftware.htmlunit.WebClient.loadWebResponse(WebClient.java:1241)
>     at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:348)
>     at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:417)
>     at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:402)
>     at main.Main.test(Main.java:160)
>     at main.Main.main(Main.java:65)
>
>
> I am running Ubuntu 14.04 and have the following Java installed:
>
> java version "1.7.0_91"
> OpenJDK Runtime Environment (IcedTea 2.6.3) (7u91-2.6.3-0ubuntu0.14.04.1)
> OpenJDK 64-Bit Server VM (build 24.91-b01, mixed mode)
>
>
> Thanks,
> Gabriel
>
>
>