Re: [Htmlunit-user] getPage throwing UnknownHostException's

[David N. <dav...@gm...>]

On 3/14/14, Ahmed Ashour <asa...@ya...> wrote:
> The only known issue is 1577 where HtmlUnit makes new socket connection per
> call to getPage(), which is not happening in 2.13
>
> I suggest two things: revert to 2.13, and directly use HttpClient or Java
> net with very minimal logic, so that we know if it's HtmlUnit or Java.

Hi, thanks for the reply.

I'm currently using HTMLUnit 2.11.

The relevant code from inside the threads is as follows:

			WebClient client = new WebClient(BrowserVersion.FIREFOX_10);
			client.setRefreshHandler(new ThreadedRefreshHandler());
			client.getCookieManager().setCookiesEnabled(true);
			client.getOptions().setThrowExceptionOnFailingStatusCode(false);
			client.getOptions().setThrowExceptionOnScriptError(false);
			client.getOptions().setJavaScriptEnabled(false);
			client.getOptions().setCssEnabled(false);
			client.getOptions().setPopupBlockerEnabled(true);
			client.getOptions().setPrintContentOnFailingStatusCode(false);
			client.getOptions().setUseInsecureSSL(true);
			client.setRefreshHandler(new ThreadedRefreshHandler());
			client.getOptions().setTimeout(120000);
			client.setJavaScriptTimeout(20000);
			client.getOptions().setPrintContentOnFailingStatusCode(false);
			
			HtmlPage p = client.getPage(url);
			client.closeAllWindows();

----------

		WebClient client = new WebClient(BrowserVersion.FIREFOX_10);
		client.setRefreshHandler(new ThreadedRefreshHandler());
		client.getCookieManager().setCookiesEnabled(true);
		client.getOptions().setThrowExceptionOnFailingStatusCode(false);
		client.getOptions().setThrowExceptionOnScriptError(false);
		client.getOptions().setJavaScriptEnabled(false);
		client.getOptions().setCssEnabled(false);
		client.getOptions().setPopupBlockerEnabled(true);
		client.getOptions().setPrintContentOnFailingStatusCode(false);
		client.getOptions().setTimeout(20000);
		client.getOptions().setPrintContentOnFailingStatusCode(false);
		client.getOptions().setUseInsecureSSL(true);
		
		WebRequest r = new WebRequest(new URL(iframeURL));
		r.setAdditionalHeader("Referer", referUrl);
		Page p = client.getPage(r);
		String readFileToString = p.getWebResponse().getContentAsString();
		client.closeAllWindows();

----------

		WebClient client = new WebClient(BrowserVersion.FIREFOX_10);
		client.setRefreshHandler(new ThreadedRefreshHandler());
		client.getCookieManager().setCookiesEnabled(true);
		client.getOptions().setThrowExceptionOnFailingStatusCode(false);
		client.getOptions().setThrowExceptionOnScriptError(false);
		client.getOptions().setJavaScriptEnabled(false);
		client.getOptions().setCssEnabled(false);
		client.getOptions().setPopupBlockerEnabled(true);
		client.getOptions().setPrintContentOnFailingStatusCode(false);
		client.getOptions().setTimeout(20000);
		client.getOptions().setPrintContentOnFailingStatusCode(false);
		client.getOptions().setUseInsecureSSL(true);
		
		Page p = client.getPage(url);
		Pattern patt = Pattern.compile("[0-9]+");
		Matcher match = patt.matcher(p.getUrl().toString());

		if (match.find()) {
			String tempNumber = match.group();
			p = client.getPage(url2);
			String temp = p.getWebResponse().getContentAsString();
			patt = Pattern.compile("apiKey:\"(.*?)\",");
			match = patt.matcher(temp);
			if (match.find() && match.group().split(":").length > 0) {
				ret = match.group().split(":")[1].replace("\"", "").replace(",", "");
			}
		}
		client.closeAllWindows();

----------

-David

PS: What's the etiquette here? Top posting or bottom posting? Gmail
makes it to much easier to top post and that's what I prefer, but some
lists insist on bottom posting.