Hi,
Downloaded latest HtmlUnit (great stuff!) but unfortunately I'm encountering a NullPointerException when I call the WebClient.closeAllWindows() method (approx. once every 5 calls).
Executing the same code without the closeAllWindows() call generates a heap memory overflow after a while (the code snipet below gets called ~a hundred times).
Please see below for stack trace and code snipet (search for 'EXCEPTION OCCURS HERE').
Any help greatly appreciated!
Cheers,
Thomas
Exception trace is:
Caused by: java.lang.NullPointerException
at com.gargoylesoftware.htmlunit.javascript.host.Event.<init>(Event.java:280)
at com.gargoylesoftware.htmlunit.javascript.host.Event.<init>(Event.java:267)
at com.gargoylesoftware.htmlunit.html.HtmlPage.executeEventHandlersIfNeeded(HtmlPage.java:1138)
at com.gargoylesoftware.htmlunit.html.HtmlPage.isOnbeforeunloadAccepted(HtmlPage.java:2070)
at com.gargoylesoftware.htmlunit.TopLevelWindow.close(TopLevelWindow.java:110)
at com.gargoylesoftware.htmlunit.WebClient.closeAllWindows(WebClient.java:2017)
Source code:
private String getUrl(UrlRequestVO urlVO, ProxyVO proxy, boolean checkRss){
String result = null;
String exceptStr = null;
WebClient webClient = null;
HtmlPage page1 = null;
Page page2 = null;
System.out.println("**Getting "+urlVO.getUrl()+" via "+proxy.url);
try{ //Gets start page webClient = new WebClient(BrowserVersion.FIREFOX_3); webClient.setThrowExceptionOnScriptError(false); webClient.setJavaScriptTimeout(10000); //Javascript timeout in ms page1 = webClient.getPage(proxy.url); boolean foundForm = false; HtmlSubmitInput button = null; HtmlTextInput text = null; //Searches for appropriate form to fill: should have a 'submit' and a 'text' field and should redirect to current website for (int formCount=0; (formCount<page1.getForms().size()) && (!foundForm); ++formCount){ HtmlForm form = page1.getForms().get(formCount); button = null; text = null; //Checks elements in form and identifies a submit button and a text field for (int elementCount=0; (elementCount<form.getElementsByTagName("input").size()) && ((text == null) || (button == null)); ++elementCount){ if ("submit".equals(form.getElementsByTagName("input").get(elementCount).getAttribute("type").toLowerCase())) button = (HtmlSubmitInput) form.getElementsByTagName("input").get(elementCount); if ("text".equals(form.getElementsByTagName("input").get(elementCount).getAttribute("type").toLowerCase())) text = (HtmlTextInput) form.getElementsByTagName("input").get(elementCount); } foundForm = (text != null) && (button != null); //Checks that it is not a foreign form (ie, that it redirects on same website) if (foundForm){ foundForm = (form.getActionAttribute().toLowerCase().startsWith("http") == false) && (form.getActionAttribute().toLowerCase().equals("#") == false); } } //If appropriate form found if ((text != null) && (button != null)){ //Fills text element with target url text.setValueAttribute(urlVO.getUrl()); //Clicks on submit page2 = button.click(); //Gets result result = page2.getWebResponse().getContentAsString(); //Checks valid rss if needed if (checkRss){ if ((! result.toLowerCase().contains("<rss")) || (! result.toLowerCase().contains("</rss"))){ result = null; exceptStr = "Invalid RSS content"; } } } else exceptStr = "No appropriate form found"; text = null; button = null; } catch (Exception ex){ exceptStr = ex.toString(); } //Cleans memory try{ if (page2 != null) page2.cleanUp(); if (page1 != null) page1.cleanUp(); if ((webClient != null) && (webClient.getTopLevelWindows().size() > 0)) webClient.closeAllWindows(); //<== EXCEPTION OCCURS HERE webClient = null; } catch (Exception ex){ throw new RuntimeException("Error while cleaning memory.", ex); } //Updates proxy DB with success/failure data if (result == null){ ++proxy.reqFailed; ++proxy.reqTotal; proxy.reqLastException = exceptStr; } else{ ++proxy.reqTotal; proxy.reqLastSuccess = new Date(); } ProxyListDAO.updateProxy(proxy); System.out.println("Done"); return result; }
View and moderate all "bugs Discussion" comments posted by this user
Mark all as spam, and block user from posting to "Bugs"
P.S.: A walk-around is to deactivate javascript: webClient.setJavaScriptEnabled(false);
Can you give a try to the latest snapshot? The problem is perhaps already solved and if not, it would give us better indication on the cause.
This Tracker item was closed automatically by the system. It was
previously set to a Pending status, and the original submitter
did not respond within 30 days (the time period specified by
the administrator of this Tracker).
I have a similar problem.
While testing a site, which is unfortunately not under my control, sporadic NullPointerExceptions show up, resulting from the: "closeable"-loop of the method:
* com.gargoylesoftware.htmlunit.html.HtmlPage#cleanUp
The reason is that the ArrayList: "autoCloseableList" may contain null-Elements, eventually.
My problem is that the null pointer exceptions are propagated through the whole call-stack. While it probably is an error that the array-list contains null elements, for me the fact that there might be some remnants left, does not justify discarding all results from prior operations. To overcome my problem I cloned the 3.20-SNAPSHOT branch and built my own version that gracefully ignores the null-elements, logging the event but continuing execution.