Menu ▾ ▴

#1975 Some urls are missing while getting request from WebClient using HtmlUnit

2.31
open
nobody
None
1
2018-07-25
2018-07-24
Ibrahim S
No

List of Issues:

Not Loading urls like facebook.com with type json, google analytics urls with type javascript, gif and xhr.

For example in Google Analytics out of 5 fired 4 urls. 1 urls is not firing. Please check our code and let us know how to fire all urls at a time in conversion.async.js

WebClient webClient = new WebClient(BrowserVersion.FIREFOX_52);
webClient.getCookieManager().clearCookies();
webClient.getCache().clear();

webClient.setAjaxController(new NicelyResynchronizingAjaxController());
webClient.setCssErrorHandler(new SilentCssErrorHandler());
webClient.getOptions().setTimeout(120000);
// to wait for AJAX

webClient.waitForBackgroundJavaScript(60000);

webClient.getOptions().setRedirectEnabled(true);
webClient.getOptions().setJavaScriptEnabled(true);
webClient.getOptions().setPrintContentOnFailingStatusCode(false);
            webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.getOptions().setUseInsecureSSL(true);
webClient.getOptions().setDoNotTrackEnabled(false);

new WebConnectionWrapper(webClient) {
                int conversionUrlCount = 0;

                @Override
                public WebResponse getResponse(WebRequest request) throws IOException {
                    System.out.println(request.getUrl());
                    WebResponse response = super.getResponse(request);
                    System.out.println(response.getStatusCode());
                    if (response.getStatusCode() < 400) {
                        resourceUrls.add(request.getUrl());
                    }
                    return response;
                }
};

String url = "",
HtmlPage page = webClient.getPage(url);

// to forcibly load the link
HtmlLink link = page.getFirstByXPath("//link");
link.getWebResponse(true);

// to forcibly load the image
try {
        HtmlImage image = page.getFirstByXPath("//img");
        image.getImageReader();
} catch (IOException e) {
        //don't need to crash at this point,
        //just let the user know that a wrong file has been passed.
}

Discussion

  • RBRi

    RBRi - 2018-07-24
    • Description has changed:

    Diff:

    --- old
+++ new
@@ -5,6 +5,7 @@
 For example in Google Analytics out of 5 fired 4 urls. 1 urls is not firing. Please check our code and let us know how to fire all urls at a time in conversion.async.js


+~~~
  WebClient webClient = new WebClient(BrowserVersion.FIREFOX_52);
             webClient.getCookieManager().clearCookies();
             webClient.getCache().clear();
@@ -54,5 +55,4 @@
                 //don't need to crash at this point,
                 //just let the user know that a wrong file has been passed.
             }
-
-
+~~~
     

  • RBRi

    RBRi - 2018-07-24
    • Description has changed:

    Diff:

    --- old
+++ new
@@ -7,25 +7,25 @@

 ~~~
  WebClient webClient = new WebClient(BrowserVersion.FIREFOX_52);

-            webClient.getCookieManager().clearCookies();
-            webClient.getCache().clear();
+webClient.getCookieManager().clearCookies();
+webClient.getCache().clear();


-            webClient.setAjaxController(new NicelyResynchronizingAjaxController());
-            webClient.setCssErrorHandler(new SilentCssErrorHandler());
-            webClient.getOptions().setTimeout(120000);
-            // to wait for AJAX
+webClient.setAjaxController(new NicelyResynchronizingAjaxController());
+webClient.setCssErrorHandler(new SilentCssErrorHandler());
+webClient.getOptions().setTimeout(120000);
+// to wait for AJAX


-            webClient.waitForBackgroundJavaScript(60000);
+webClient.waitForBackgroundJavaScript(60000);


-            webClient.getOptions().setRedirectEnabled(true);
-            webClient.getOptions().setJavaScriptEnabled(true);
-            webClient.getOptions().setPrintContentOnFailingStatusCode(false);
+webClient.getOptions().setRedirectEnabled(true);
+webClient.getOptions().setJavaScriptEnabled(true);
+webClient.getOptions().setPrintContentOnFailingStatusCode(false);
             webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
-            webClient.getOptions().setThrowExceptionOnScriptError(false);
-            webClient.getOptions().setUseInsecureSSL(true);
-            webClient.getOptions().setDoNotTrackEnabled(false);
+webClient.getOptions().setThrowExceptionOnScriptError(false);
+webClient.getOptions().setUseInsecureSSL(true);
+webClient.getOptions().setDoNotTrackEnabled(false);


-            new WebConnectionWrapper(webClient) {
+new WebConnectionWrapper(webClient) {
                 int conversionUrlCount = 0;

                 @Override
@@ -38,21 +38,21 @@
                     }
                     return response;
                 }

-            };
+};

 String url = "",

-            HtmlPage page = webClient.getPage(url);
+HtmlPage page = webClient.getPage(url);


-            // to forcibly load the link
-            HtmlLink link = page.getFirstByXPath("//link");
-            link.getWebResponse(true);
+// to forcibly load the link
+HtmlLink link = page.getFirstByXPath("//link");
+link.getWebResponse(true);


-          // to forcibly load the image
-            try {
-                HtmlImage image = page.getFirstByXPath("//img");
-                image.getImageReader();
-            } catch (IOException e) {
-                //don't need to crash at this point,
-                //just let the user know that a wrong file has been passed.
-            }
+// to forcibly load the image
+try {
+        HtmlImage image = page.getFirstByXPath("//img");
+        image.getImageReader();
+} catch (IOException e) {
+        //don't need to crash at this point,
+        //just let the user know that a wrong file has been passed.
+}
 ~~~
     

  • RBRi

    RBRi - 2018-07-24
    • Description has changed:

    Diff:

    --- old
+++ new
@@ -6,7 +6,7 @@


 ~~~

- WebClient webClient = new WebClient(BrowserVersion.FIREFOX_52);
+WebClient webClient = new WebClient(BrowserVersion.FIREFOX_52);
 webClient.getCookieManager().clearCookies();
 webClient.getCache().clear();
     

  • RBRi

    RBRi - 2018-07-24

    Sorry, for asking again. I need a way to reproduce this, any help to understand the problem is welcome.

    • maybe you are able to provide the url your are calling with your initial call
    • maybe you can provide some html pieces showing, how this missing stuff is called by the real browsers - is there a script tag that refers them or does some javascript do the call

    And finally please place your
    webClient.waitForBackgroundJavaScript(60000);
    call after the get class (this is not an option).

     

Log in to post a comment.

Auth0 Logo