#2016 JavaScript taking way too long time

[Abhishek Thakur]

Hi There,

I am new to HtmlUnit and I am using this library to login/geo login with postcodes to different websites. I am able to login to websites but it is taking way too long(up to 5 mins). I need to have Javascript enabled to download the forms asyncronously.

I am using below webclient configuration

WebClient webClient = new WebClient(BrowserVersion.CHROME);

WebClientOptions webClientOption = webClient.getOptions();
webClientOption.setCssEnabled(false);
webClientOption.setJavaScriptEnabled(true);
webClientOption.setUseInsecureSSL(true);
webClientOption.setTimeout(30000);
webClientOption.setThrowExceptionOnScriptError(false);
webClientOption.setThrowExceptionOnFailingStatusCode(false);
webClientOption.setUseInsecureSSL(true);
webClientOption.setActiveXNative(true);
webClientOption.setAppletEnabled(true);
webClientOption.setPopupBlockerEnabled(false);
webClientOption.setRedirectEnabled(true);

webClient.waitForBackgroundJavaScript(30000);
webClient.waitForBackgroundJavaScriptStartingBefore(10000);
webClient.setJavaScriptTimeout(30000);
webClient.getCookieManager().setCookiesEnabled(true);
webClient.setAjaxController(new NicelyResynchronizingAjaxController());

  JavaScriptEngine scriptEngine = (JavaScriptEngine) webClient.getJavaScriptEngine();
HtmlUnitContextFactory factory = scriptEngine.getContextFactory();
Context context = factory.enterContext();
context.setOptimizationLevel(9);

This is happening with almost all the website I am trying to login. Any help in improving the time to run javascript will be greatly appreciated.

Regards,
Abhishek Thakur

[RBRi]

Without a concret URL it is a bit tricky to analyze the problem.

But two points

1. waitForBackgroundJavaScript / waitForBackgroundJavaScriptStartingBefore are not options. You have to call these methods after every action that might trigger js (e.g. openUrl).
2. do not set the AjaxController if there is no real need.

If you can provide a real complet sample i will have a look.

[Abhishek Thakur]

Hi There,

Thank you for coming back to me and I will use the suggestions in my application.

Also, I am trying to do two things
1. Change the country of url https://www.instacart.com/ to CA and then use the cookies for subsequent request.

My sample code will be as below

WebClient webClient = new WebClient(BrowserVersion.CHROME);

WebClientOptions webClientOption = webClient.getOptions();
webClientOption.setCssEnabled(false);
webClientOption.setJavaScriptEnabled(true);
webClientOption.setUseInsecureSSL(true);
webClientOption.setTimeout(30000);
webClientOption.setThrowExceptionOnScriptError(false);
webClientOption.setThrowExceptionOnFailingStatusCode(false);
webClientOption.setUseInsecureSSL(true);
webClientOption.setActiveXNative(true);
webClientOption.setAppletEnabled(true);
webClientOption.setPopupBlockerEnabled(false);
webClientOption.setRedirectEnabled(true);

webClient.setJavaScriptTimeout(30000);
webClient.getCookieManager().setCookiesEnabled(true);

  JavaScriptEngine scriptEngine = (JavaScriptEngine) webClient.getJavaScriptEngine();
HtmlUnitContextFactory factory = scriptEngine.getContextFactory();
Context context = factory.enterContext();
context.setOptimizationLevel(9);

 WebRequest request = new WebRequest(new URL( https://www.instacart.com/));
HtmlPage page = webClient.getPage(request);

HtmlSelect select = page.getElementByName("country_alpha_2");
HtmlOption option = select.getOptionByValue("CA");
 page = select.setSelectedAttribute(option, true);

1. Also for https://www.amazon.co.uk/, I am trying to fix Delivery location to "W1J 5DL" and get the cookies after that.

I found that without javascript enabled pages can be accessed fast but I can't change the location in the above sites as I think the location part was downloaded through some asynchronous javascript call.

Kindly suggest further on this issue as at the moment total time taken in above procedures are more than 5 mins. Thanking you in advance.

Regards,
Abhishek Thakur

[Abhishek Thakur]

Hi There,

When I used the above code it took me 3 mins just to load the url when JavaScript was enabled as you can see in the below logs

""2019-05-30 08:21:52 [main] WARN  c.g.htmlunit.html.HtmlLink - Link type 'stylesheet' not supported.
""2019-05-30 08:21:52 [main] WARN  c.g.htmlunit.html.HtmlLink - Link type 'apple-touch-icon-precomposed' not supported.
""2019-05-30 08:21:52 [main] WARN  c.g.htmlunit.html.HtmlLink - Link type 'manifest' not supported.
""2019-05-30 08:21:52 [main] WARN  c.g.htmlunit.html.HtmlLink - Link type 'icon' not supported.
""2019-05-30 08:21:52 [main] WARN  c.g.htmlunit.html.HtmlLink - Link type 'apple-touch-icon-precomposed' not supported.
""2019-05-30 08:21:52 [main] WARN  c.g.htmlunit.html.HtmlLink - Link type 'apple-touch-icon-precomposed' not supported.
""2019-05-30 08:21:52 [main] WARN  c.g.htmlunit.html.HtmlLink - Link type 'apple-touch-icon-precomposed' not supported.
""2019-05-30 08:21:52 [main] WARN  c.g.htmlunit.html.HtmlLink - Link type 'apple-touch-icon-precomposed' not supported.
""2019-05-30 08:21:52 [main] WARN  c.g.htmlunit.html.HtmlLink - Link type 'apple-touch-icon-precomposed' not supported.
""2019-05-30 08:21:52 [main] WARN  c.g.htmlunit.html.HtmlLink - Link type 'canonical' not supported.
""2019-05-30 08:21:57 [main] WARN  c.g.h.IncorrectnessListenerImpl - Obsolete content type encountered: 'text/javascript'.
""2019-05-30 08:21:57 [main] WARN  c.g.htmlunit.html.HtmlLink - Link type '' not supported.
""2019-05-30 08:22:18 [main] WARN  c.g.h.IncorrectnessListenerImpl - Obsolete content type encountered: 'text/javascript'.
""2019-05-30 08:22:23 [main] WARN  c.g.h.IncorrectnessListenerImpl - Obsolete content type encountered: 'text/javascript'.
""2019-05-30 08:22:24 [main] WARN  c.g.htmlunit.html.HtmlScript - Script is not JavaScript (type: application/ld+json, language: ). Skipping execution.
""2019-05-30 08:22:24 [main] WARN  c.g.htmlunit.html.HtmlLink - Link type 'stylesheet' not supported.
""2019-05-30 08:22:49 [main] WARN  c.g.h.IncorrectnessListenerImpl - Obsolete content type encountered: 'application/x-javascript'.
""2019-05-30 08:22:53 [main] WARN  c.g.h.IncorrectnessListenerImpl - Obsolete content type encountered: 'application/x-javascript'.
""2019-05-30 08:24:26 [main] ERROR c.g.h.j.DefaultJavaScriptErrorListener - Timeout during JavaScript execution after 88809ms; allowed only 30000ms

Altogether it took me 5 mins to update the country in the url https://www.instacart.com/ to CA! I think this is too much and HtmlUnit should be faster than that.

Regards,
Abhishek Thakur

[Abhishek Thakur]

Hi There,

I investigated further and found that without proxy it takes 4-5 secs to load a page but with proxy it takes ~30sec to load the same page. But still it takes long enough time!

Regards,
Abhishek Thakur