Menu

#1006 java.lang.NullPointerException when calling closeAllWindows

2.6
closed
nobody
None
5
2018-01-06
2009-12-29
Anonymous
No

Hi,

Downloaded latest HtmlUnit (great stuff!) but unfortunately I'm encountering a NullPointerException when I call the WebClient.closeAllWindows() method (approx. once every 5 calls).

Executing the same code without the closeAllWindows() call generates a heap memory overflow after a while (the code snipet below gets called ~a hundred times).

Please see below for stack trace and code snipet (search for 'EXCEPTION OCCURS HERE').

Any help greatly appreciated!
Cheers,
Thomas

Exception trace is:
Caused by: java.lang.NullPointerException
at com.gargoylesoftware.htmlunit.javascript.host.Event.<init>(Event.java:280)
at com.gargoylesoftware.htmlunit.javascript.host.Event.<init>(Event.java:267)
at com.gargoylesoftware.htmlunit.html.HtmlPage.executeEventHandlersIfNeeded(HtmlPage.java:1138)
at com.gargoylesoftware.htmlunit.html.HtmlPage.isOnbeforeunloadAccepted(HtmlPage.java:2070)
at com.gargoylesoftware.htmlunit.TopLevelWindow.close(TopLevelWindow.java:110)
at com.gargoylesoftware.htmlunit.WebClient.closeAllWindows(WebClient.java:2017)

Source code:
private String getUrl(UrlRequestVO urlVO, ProxyVO proxy, boolean checkRss){
String result = null;
String exceptStr = null;
WebClient webClient = null;
HtmlPage page1 = null;
Page page2 = null;
System.out.println("**Getting "+urlVO.getUrl()+" via "+proxy.url);

    try{
        //Gets start page
        webClient = new WebClient(BrowserVersion.FIREFOX_3);
        webClient.setThrowExceptionOnScriptError(false);
        webClient.setJavaScriptTimeout(10000); //Javascript timeout in ms
        page1 = webClient.getPage(proxy.url);

        boolean foundForm = false;
        HtmlSubmitInput button = null;
        HtmlTextInput text = null;

        //Searches for appropriate form to fill: should have a 'submit' and a 'text' field and should redirect to current website
        for (int formCount=0; (formCount<page1.getForms().size()) && (!foundForm); ++formCount){
            HtmlForm form = page1.getForms().get(formCount);
            button = null;
            text = null;

            //Checks elements in form and identifies a submit button and a text field
            for (int elementCount=0; (elementCount<form.getElementsByTagName("input").size()) && ((text == null) || (button == null)); ++elementCount){
                if ("submit".equals(form.getElementsByTagName("input").get(elementCount).getAttribute("type").toLowerCase())) button = (HtmlSubmitInput) form.getElementsByTagName("input").get(elementCount);
                if ("text".equals(form.getElementsByTagName("input").get(elementCount).getAttribute("type").toLowerCase())) text = (HtmlTextInput) form.getElementsByTagName("input").get(elementCount);
            }
            foundForm = (text != null) && (button != null);

            //Checks that it is not a foreign form (ie, that it redirects on same website)
            if (foundForm){
                foundForm = (form.getActionAttribute().toLowerCase().startsWith("http") == false) &&
                            (form.getActionAttribute().toLowerCase().equals("#") == false);
            }
        }

        //If appropriate form found
        if ((text != null) && (button != null)){
            //Fills text element with target url
            text.setValueAttribute(urlVO.getUrl());

            //Clicks on submit
            page2 = button.click();

            //Gets result
            result = page2.getWebResponse().getContentAsString();

            //Checks valid rss if needed
            if (checkRss){
                if ((! result.toLowerCase().contains("<rss")) || (! result.toLowerCase().contains("</rss"))){
                    result = null;
                    exceptStr = "Invalid RSS content";
                }
            }
        }
        else exceptStr = "No appropriate form found";
        text = null;
        button = null;
    }
    catch (Exception ex){
        exceptStr = ex.toString();
    }

    //Cleans memory
    try{
        if (page2 != null) page2.cleanUp();
        if (page1 != null) page1.cleanUp();
        if ((webClient != null) && (webClient.getTopLevelWindows().size() > 0)) webClient.closeAllWindows(); //<== EXCEPTION OCCURS HERE
        webClient = null;
    }
    catch (Exception ex){ throw new RuntimeException("Error while cleaning memory.", ex); }

    //Updates proxy DB with success/failure data
    if (result == null){
        ++proxy.reqFailed;
        ++proxy.reqTotal;
        proxy.reqLastException = exceptStr;
    }
    else{
        ++proxy.reqTotal;
        proxy.reqLastSuccess = new Date();
    }
    ProxyListDAO.updateProxy(proxy);

    System.out.println("Done");

    return result;
}

Discussion

  • Anonymous

    Anonymous - 2009-12-30

    P.S.: A walk-around is to deactivate javascript: webClient.setJavaScriptEnabled(false);

     
  • Marc Guillemot

    Marc Guillemot - 2010-01-07

    Can you give a try to the latest snapshot? The problem is perhaps already solved and if not, it would give us better indication on the cause.

     
  • SourceForge Robot

    This Tracker item was closed automatically by the system. It was
    previously set to a Pending status, and the original submitter
    did not respond within 30 days (the time period specified by
    the administrator of this Tracker).

     
  • Mark Kahl

    Mark Kahl - 2018-01-06

    I have a similar problem.
    While testing a site, which is unfortunately not under my control, sporadic NullPointerExceptions show up, resulting from the: "closeable"-loop of the method:
    * com.gargoylesoftware.htmlunit.html.HtmlPage#cleanUp
    The reason is that the ArrayList: "autoCloseableList" may contain null-Elements, eventually.
    My problem is that the null pointer exceptions are propagated through the whole call-stack. While it probably is an error that the array-list contains null elements, for me the fact that there might be some remnants left, does not justify discarding all results from prior operations. To overcome my problem I cloned the 3.20-SNAPSHOT branch and built my own version that gracefully ignores the null-elements, logging the event but continuing execution.

     

Log in to post a comment.