Menu ▾ ▴

#1799 HtmlAnchor click() and getPage() not working

2.22
accepted
nobody
None
1
2019-04-25
2016-06-12
Timothy Nessus
No

I'm trying to analyze a website (shop.lego.com), but when using either click or getPage on the anchor (corresponding to the text "View All") the HtmlPage produced is the same as the original page. I believe the problem has to do with a HtmlUnit running a javascript on the page (javascript is enabled on the webclient).

The specific webpage is http://shop.lego.com/en-CA/Sales-And-Deals. I create a webClient with the following settings:

WebClient webClient = new WebClient(BrowserVersion.FIREFOX_38);
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.getOptions().setJavaScriptEnabled(true);
webClient.getOptions().setActiveXNative(true);
webClient.getOptions().setAppletEnabled(true);
webClient.getOptions().setActiveXNative(true);
webClient.getOptions().setCssEnabled(true);
webClient.getOptions().setDoNotTrackEnabled(false);
webClient.getOptions().setGeolocationEnabled(true);
webClient.getOptions().setPopupBlockerEnabled(false);
webClient.getOptions().setPrintContentOnFailingStatusCode(true);
webClient.getOptions().setRedirectEnabled(true);
webClient.getCache().clear();
webClient.getCache().setMaxSize(0);
webClient.setAjaxController(new NicelyResynchronizingAjaxController());
webClient.getCookieManager().setCookiesEnabled(true);

Then I open the page (I put loop in place to ensure that the page returned is an Html page), and retrieve the anchor.

final URL url = new URL("http://shop.lego.com/en-CA/Sales-And-Deals");

int count = 0;
Page pageOri = webClient.getPage(url);
while (count < 5 && !(pageOri instanceof HtmlPage)) {
    pageOri = webClient.getPage(url);
    count++;
    System.out.println("Read Attempt #" + count + " failed.");
}
final HtmlPage page = (HtmlPage) pageOri;
HtmlAnchor choosen = page.getAnchorByText("View All");

Then I click on the anchor and print the text representation, but the text representation is the same as the original.

HtmlPage allResults = (HtmlPage) choosen.getPage();
System.out.println(allResults.asText());

Discussion

  • Ahmed Ashour

    Ahmed Ashour - 2016-06-13
    • Description has changed:

    Diff:

    --- old
+++ new
@@ -2,38 +2,44 @@

 The specific webpage is http://shop.lego.com/en-CA/Sales-And-Deals. I create a webClient with the following settings:

+~~~
 WebClient webClient = new WebClient(BrowserVersion.FIREFOX_38);
-    webClient.getOptions().setThrowExceptionOnScriptError(false);
-    webClient.getOptions().setJavaScriptEnabled(true);
-    webClient.getOptions().setActiveXNative(true);
-    webClient.getOptions().setAppletEnabled(true);
-    webClient.getOptions().setActiveXNative(true);
-    webClient.getOptions().setCssEnabled(true);
-    webClient.getOptions().setDoNotTrackEnabled(false);
-    webClient.getOptions().setGeolocationEnabled(true);
-    webClient.getOptions().setPopupBlockerEnabled(false);
-    webClient.getOptions().setPrintContentOnFailingStatusCode(true);
-    webClient.getOptions().setRedirectEnabled(true);
-    webClient.getCache().clear();
-    webClient.getCache().setMaxSize(0);
-    webClient.setAjaxController(new NicelyResynchronizingAjaxController());
-    webClient.getCookieManager().setCookiesEnabled(true);
+webClient.getOptions().setThrowExceptionOnScriptError(false);
+webClient.getOptions().setJavaScriptEnabled(true);
+webClient.getOptions().setActiveXNative(true);
+webClient.getOptions().setAppletEnabled(true);
+webClient.getOptions().setActiveXNative(true);
+webClient.getOptions().setCssEnabled(true);
+webClient.getOptions().setDoNotTrackEnabled(false);
+webClient.getOptions().setGeolocationEnabled(true);
+webClient.getOptions().setPopupBlockerEnabled(false);
+webClient.getOptions().setPrintContentOnFailingStatusCode(true);
+webClient.getOptions().setRedirectEnabled(true);
+webClient.getCache().clear();
+webClient.getCache().setMaxSize(0);
+webClient.setAjaxController(new NicelyResynchronizingAjaxController());
+webClient.getCookieManager().setCookiesEnabled(true);
+~~~

-    Then I open the page (I put loop in place to ensure that the page returned is an Html page), and retrieve the anchor.
+Then I open the page (I put loop in place to ensure that the page returned is an Html page), and retrieve the anchor.

-    final URL url = new URL("http://shop.lego.com/en-CA/Sales-And-Deals");
+~~~
+final URL url = new URL("http://shop.lego.com/en-CA/Sales-And-Deals");

-    int count = 0;
-    Page pageOri = webClient.getPage(url);
-    while (count < 5 && !(pageOri instanceof HtmlPage)) {
-      pageOri = webClient.getPage(url);
-      count++;
-      System.out.println("Read Attempt #" + count + " failed.");
-    }
-   final HtmlPage page = (HtmlPage) pageOri;
-  HtmlAnchor choosen = page.getAnchorByText("View All");
+int count = 0;
+Page pageOri = webClient.getPage(url);
+while (count < 5 && !(pageOri instanceof HtmlPage)) {
+    pageOri = webClient.getPage(url);
+    count++;
+    System.out.println("Read Attempt #" + count + " failed.");
+}
+final HtmlPage page = (HtmlPage) pageOri;
+HtmlAnchor choosen = page.getAnchorByText("View All");
+~~~

   Then I click on the anchor and print the text representation, but the text representation is the same as the original.

-      HtmlPage allResults = (HtmlPage) choosen.getPage();
-      System.out.println(allResults.asText());
+~~~
+HtmlPage allResults = (HtmlPage) choosen.getPage();
+System.out.println(allResults.asText());
+~~~
     

  • Ahmed Ashour

    Ahmed Ashour - 2016-06-13
    • status: open --> accepted
     

  • Ahmed Ashour

    Ahmed Ashour - 2016-06-13

    setThrowExceptionOnScriptError(false) hides an underlying error.

    Test case added in NativeObjectTest.setPrototype()

     

  • RBRi

    RBRi - 2019-04-25

    test was moved to XMLHttpRequest2Test.readPropertyFromPrototypeShouldThrow()

     

Log in to post a comment.