[HtmlUnit] [htmlunit:bugs] #1975 Some urls are missing while getting request from WebClient using HtmlUnit

[RBRi <rb...@us...>]

- Description has changed:

Diff:

~~~~

--- old
+++ new
@@ -6,7 +6,7 @@
 
 
 ~~~
- WebClient webClient = new WebClient(BrowserVersion.FIREFOX_52);
+WebClient webClient = new WebClient(BrowserVersion.FIREFOX_52);
 webClient.getCookieManager().clearCookies();
 webClient.getCache().clear();
 

~~~~




---

** [bugs:#1975] Some urls are missing while getting request from WebClient using HtmlUnit**

**Status:** open
**Group:** 2.31
**Created:** Tue Jul 24, 2018 07:50 AM UTC by Ibrahim S
**Last Updated:** Tue Jul 24, 2018 04:26 PM UTC
**Owner:** nobody


List of Issues:

Not Loading urls like facebook.com with type json, google analytics urls with type javascript, gif and xhr.

For example in Google Analytics out of 5 fired 4 urls. 1 urls is not firing. Please check our code and let us know how to fire all urls at a time in conversion.async.js


~~~
WebClient webClient = new WebClient(BrowserVersion.FIREFOX_52);
webClient.getCookieManager().clearCookies();
webClient.getCache().clear();

webClient.setAjaxController(new NicelyResynchronizingAjaxController());
webClient.setCssErrorHandler(new SilentCssErrorHandler());
webClient.getOptions().setTimeout(120000);
// to wait for AJAX

webClient.waitForBackgroundJavaScript(60000);

webClient.getOptions().setRedirectEnabled(true);
webClient.getOptions().setJavaScriptEnabled(true);
webClient.getOptions().setPrintContentOnFailingStatusCode(false);
            webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.getOptions().setUseInsecureSSL(true);
webClient.getOptions().setDoNotTrackEnabled(false);

new WebConnectionWrapper(webClient) {
                int conversionUrlCount = 0;

                @Override
                public WebResponse getResponse(WebRequest request) throws IOException {
                    System.out.println(request.getUrl());
                    WebResponse response = super.getResponse(request);
                    System.out.println(response.getStatusCode());
                    if (response.getStatusCode() < 400) {
                        resourceUrls.add(request.getUrl());
                    }
                    return response;
                }
};

String url = "",
HtmlPage page = webClient.getPage(url);

// to forcibly load the link
HtmlLink link = page.getFirstByXPath("//link");
link.getWebResponse(true);

// to forcibly load the image
try {
        HtmlImage image = page.getFirstByXPath("//img");
        image.getImageReader();
} catch (IOException e) {
        //don't need to crash at this point,
        //just let the user know that a wrong file has been passed.
}
~~~



---

Sent from sourceforge.net because htm...@li... is subscribed to https://sourceforge.net/p/htmlunit/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/htmlunit/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.