htmlunit-user — Discussion of the use of HtmlUnit

[Htmlunit-user] Encrypted password value generated by HTMLUnit Javascript is different from the value sent by Browser

[Vasudevan C. <vco...@gm...>]

All,

   I was entering the username and password into the respective fields
   and clicked the button. The password was encrypted (by HtmlUnit JS
engine)
   and the host rejected with the message "Invalid login credentials".

   Whereas the login is successful from the browser. The encrypted password
   length is more from the browser compared to HtmlUnit generated values.

   Looks like the execution of JS for password encryption in HtmlUnit is
having
    some issues.

   I am copying the encrypted password of HTMlUnit & actual browser.

   HtmlUnit Value:

 v="0384l01600l11696l23456l312416l425088l525600l651200l7720l8880l9912l10848l11896l12720l13832l14848l151568l16800l17720l18896l19768l201616l21848l22720l23912l24912l251584l26912l27848l28880l29800l30896l31768l321600l33800l34816l35~0"

   Browser Value:
v="0808l0816l11824l23136l36784l413312l528160l649152l7720l81616l91600l10768l111616l12720l13832l14800l15800l16800l17720l18912l191600l20896l21880l22720l23864l241600l251600l26848l27784l281552l29832l30912l31832l32768l331632l34864l35~0688l01744l13680l24096l36912l429696l553248l6117760l71616l81760l91840l101616l11"

   I tried simulating all browser version options supported by HtmlUnit and
it
   fails always.

   I appreciate your help or any pointers to solve this issue.

Regards
 Vasu

Re: [Htmlunit-user] HTMLUnit is sending a POST request with content-type set as application/x-www-form-urlencoded whereas browser sets the content-type as text/plain

[<rb...@rb...>]

Am 10. Februar 2020 12:38:16 MEZ schrieb Vasudevan Comandur <vco...@gm...>:
>Hi,
>
>   I have enabled JS to access the following site:
>  https://allegis.ramcocloud.com/RVW/extui/vwrt/LaunchPanel.htm
>
>  What I saw was that there is a difference in setting the content-type
>between
>browser and HTMLUnit. HTMLUnit sets as
>application/x-www-form-urlencoded
> where as browser is setting as text/plain. I am using HTMLUnit version
>2.37.
>  The server rejects the request,
>
>  The URL where the difference happened is copied below for reference
>
>https://allegis.ramcocloud.com/RVW/ilrt/dispatcher.aspx?RVWRTQS_ACTIVITY=login&RVWRTQS_ILBO=ezelogin&RVWRTQS_TAB=&RVWRTQS_EVENTNAME=tskstdlaunch&RVWRTQS_PCONTROL=&RVWRTQS_PVIEW=&RVWRTQS_EVENTDETAILS=&RVWRTQS_REQID=522d4031-300d-46d5-8d53-064979e6b948&RVWRTQS_PROTO=false&RVWRTQS_TRACE=false&RVWRTQS_XMLHTTP=TRUE&RVWRTQS_COMPONENT=security&RVWRTQS_LANGID=1
>
>  Appreciate your help on this.
>
>Regards
> Vasu

Can you please open an issue at github

[Htmlunit-user] HTMLUnit is sending a POST request with content-type set as application/x-www-form-urlencoded whereas browser sets the content-type as text/plain

[Vasudevan C. <vco...@gm...>]

Hi,

   I have enabled JS to access the following site:
  https://allegis.ramcocloud.com/RVW/extui/vwrt/LaunchPanel.htm

  What I saw was that there is a difference in setting the content-type
between
   browser and HTMLUnit. HTMLUnit sets as application/x-www-form-urlencoded
  where as browser is setting as text/plain. I am using HTMLUnit version
2.37.
  The server rejects the request,

  The URL where the difference happened is copied below for reference

https://allegis.ramcocloud.com/RVW/ilrt/dispatcher.aspx?RVWRTQS_ACTIVITY=login&RVWRTQS_ILBO=ezelogin&RVWRTQS_TAB=&RVWRTQS_EVENTNAME=tskstdlaunch&RVWRTQS_PCONTROL=&RVWRTQS_PVIEW=&RVWRTQS_EVENTDETAILS=&RVWRTQS_REQID=522d4031-300d-46d5-8d53-064979e6b948&RVWRTQS_PROTO=false&RVWRTQS_TRACE=false&RVWRTQS_XMLHTTP=TRUE&RVWRTQS_COMPONENT=security&RVWRTQS_LANGID=1

  Appreciate your help on this.

Regards
 Vasu

[Htmlunit-user] PrimeFaces.addSubmitParam is not called using click()

[guich <gu...@su...>]

Hi,

I'm trying to invoke the following ahref using HtmlUnit.

<li> Create <#>  
		
The code i'm using is:

         try (final WebClient webClient = new
WebClient(BrowserVersion.CHROME)) 
         {
            webClient.setAjaxController(new
NicelyResynchronizingAjaxController());
//            webClient.setCssErrorHandler(new SilentCssErrorHandler());
            webClient.getOptions().setUseInsecureSSL(true);
            webClient.getOptions().setJavaScriptEnabled(true);
            webClient.getOptions().setCssEnabled(false);
            webClient.waitForBackgroundJavaScript(10000);
//            webClient.getOptions().setThrowExceptionOnScriptError(false);
            ...
            HtmlListItem item = (HtmlListItem)spans.get(idx3);
            HtmlAnchor a = (HtmlAnchor)item.getFirstElementChild();
            System.out.println("a: "+a);
            HtmlPage pag3 = item.click();
            System.out.println(pag3);
         }

This is not working: the returned page is the same one. 



--
Sent from: http://htmlunit.10904.n7.nabble.com/HtmlUnit-General-f20847.html

[Htmlunit-user] [ANN] HtmlUnit 2.37.0 released

[Ronald B. <rb...@rb...>]

Hi all,

it is a pleasure to announce the availability of HtmlUnit 2.37.0.
The main enhancements are:
  - many js improvements done in Rhino
  - CHROME 79
  - FF52 removed
  - FF68 added

The full list of changes can be found in [1]
Thanks to all the contributors.

Happy Testing/Scraping!
   The HtmlUnit team

[1] http://htmlunit.sourceforge.net/changes-report.html#a2.37.0

[Htmlunit-user] htmlunit 2.35 and 2.36 OSGi release (htmlunit-2.36.0-OSGi.jar) leaks threads and hangs the applications

[Wayne X. <wx...@ya...>]

Hi,
I recently upgraded to 2.35 and 2.36 using htmlunit as a crawling unit. After crawling a lot of pages, I started to see tons of the following threads in the thread dump and eventually it eats up system resource and hangs the container (in docker).
The problem looks very much like what's mentioned in this link:
'Re: [Htmlunit-user] Help with UpgradeException' - MARC


| 
| 
|  | 
'Re: [Htmlunit-user] Help with UpgradeException' - MARC


 |

 |

 |


I wonder if anybody else has seen it. I'm a little surprised that I couldn't find another recent report about this problem giving that 2.35 has been out for a while.
Any pointers would be highly appreciated.


"WebSocketClient@1404341633-126276" #126276 daemon prio=5 os_prio=0 tid=0x00007faba4228800 nid=0xbe7 runnable [0x00007fa1ec102000]   java.lang.Thread.State: RUNNABLE        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)        - locked <0x00007fba17c3b6f0> (a sun.nio.ch.Util$3)        - locked <0x00007fba17c3b6d8> (a java.util.Collections$UnmodifiableSet)        - locked <0x00007fba17c3b528> (a sun.nio.ch.EPollSelectorImpl)        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:464)        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:401)        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)        at org.eclipse.jetty.io.ManagedSelector$$Lambda$71/1759199424.run(Unknown Source)        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:786)        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:743)        at java.lang.Thread.run(Thread.java:748)
"WebSocketClient@1404341633-126275" #126275 daemon prio=5 os_prio=0 tid=0x00007faba403b000 nid=0xbdb waiting on condition [0x00007fa28901c000]   java.lang.Thread.State: TIMED_WAITING (parking)        at sun.misc.Unsafe.park(Native Method)        - parking to wait for  <0x00007fba17d68fd8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:727)        at java.lang.Thread.run(Thread.java:748)


-Wayne

Re: [Htmlunit-user] HTMLUnit does not seem to execute google Recaptcha JS to fetch a Token

[Vasudevan C. <vco...@gm...>]

Hi Ronald,

   I have raised this issue in github.

Regards
 Vasu

On Sun, 15 Dec 2019 at 22:33, Ronald Brill <rb...@rb...> wrote:

> Hi Vasu,
>
> will have a look at this. But please open an issue at github - i fear
> otherwise this will be lost in the stack of open tasks on my desk.
>
>         RBRi
>
> On Fri, 13 Dec 2019 04:15:02 +0530 Vasudevan Comandur wrote:
> >
> >Hi,
> >
> >   I am using the URL to get the login screen
> >https://recruiter.cwjobs.co.uk/login/.
> >   There is a Recaptcha JS which is supposed to populate the value
> reCaptcha
> >    token within a IFrame. I did not see the request to fetch the token
> >data from
> >    google going out of HTMLUnit.
> >
> >   When I try it directly with Chrome Browser, I see the request to fetch
> >token data
> >    going and the value is populated inside an iFrame.
> >
> >   I am using the latest version of HTMLUnit 2.36.
> >
> >   Any help would be appreciated.
> >
> >Regards
> > Vasu
> >
> >
> >
> >----< Inline text [text-plain-04.txt] >------------------
> >
> >
> >
> >
> >----< Inline text [text-plain-05.txt] >------------------
> >
> >_______________________________________________
> >Htmlunit-user mailing list
> >Htm...@li...
> >https://lists.sourceforge.net/lists/listinfo/htmlunit-user
> >
> >
>
>

Re: [Htmlunit-user] HTMLUnit does not seem to execute google Recaptcha JS to fetch a Token

[Ronald B. <rb...@rb...>]

Hi Vasu,

will have a look at this. But please open an issue at github - i fear otherwise this will be lost in the stack of open tasks on my desk.

	RBRi

On Fri, 13 Dec 2019 04:15:02 +0530 Vasudevan Comandur wrote:
>
>Hi,
>
>   I am using the URL to get the login screen
>https://recruiter.cwjobs.co.uk/login/.
>   There is a Recaptcha JS which is supposed to populate the value reCaptcha
>    token within a IFrame. I did not see the request to fetch the token
>data from
>    google going out of HTMLUnit.
>
>   When I try it directly with Chrome Browser, I see the request to fetch
>token data
>    going and the value is populated inside an iFrame.
>
>   I am using the latest version of HTMLUnit 2.36.
>
>   Any help would be appreciated.
>
>Regards
> Vasu
>
>
>
>----< Inline text [text-plain-04.txt] >------------------
>
>
>
>
>----< Inline text [text-plain-05.txt] >------------------
>
>_______________________________________________
>Htmlunit-user mailing list
>Htm...@li...
>https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>
>

Re: [Htmlunit-user] IncompatibleClassChangeError when using HtmlUnit in java in Intelli

[Ronald B. <rb...@rb...>]

This is usually a class path problem, looks like you have two different versions of jetty in your classpath.

	RBRi

On Sun, 8 Dec 2019 07:24:23 -0500 Jiechao Wang wrote:
>
>Hi, I am trying to use HtmlUnit with the following code in java in IntelliJ
>Ultimate 2018.3.6
>
>public static void main(String[] args){
>    WebClient webClient = new WebClient();}
>
>It is a gradle project so I have this in my build.gradle
>
>compile 'net.sourceforge.htmlunit:htmlunit:2.36.0'
>
>But when I ran the above one line program from Intellij, I got such error:
>
>Exception in thread "main" java.lang.IncompatibleClassChangeError:
>class org.eclipse.jetty.util.component.ContainerLifeCycle can not
>implement org.eclipse.jetty.util.component.Container, because it is
>not an interface (org.eclipse.jetty.util.component.Container is in
>unnamed module of loader 'app')
>at java.base/java.lang.ClassLoader.defineClass1(Native Method)
>at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1016)
>at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
>at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:802)
>at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:700)
>at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:623)
>at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
>at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
>at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
>at java.base/java.lang.ClassLoader.defineClass1(Native Method)
>at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1016)
>at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
>at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:802)
>at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:700)
>at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:623)
>at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
>at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
>at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
>at java.base/java.lang.Class.getDeclaredFields0(Native Method)
>at java.base/java.lang.Class.privateGetDeclaredFields(Class.java:3062)
>at java.base/java.lang.Class.getDeclaredFields(Class.java:2249)
>at com.gargoylesoftware.htmlunit.javascript.configuration.AbstractJavaScriptConfiguration.process(AbstractJavaScriptConfiguration.java:298)
>at com.gargoylesoftware.htmlunit.javascript.configuration.AbstractJavaScriptConfiguration.getClassConfiguration(AbstractJavaScriptConfiguration.java:204)
>at com.gargoylesoftware.htmlunit.javascript.configuration.AbstractJavaScriptConfiguration.buildUsageMap(AbstractJavaScriptConfiguration.java:85)
>at com.gargoylesoftware.htmlunit.javascript.configuration.AbstractJavaScriptConfiguration.<init>(AbstractJavaScriptConfiguration.java:65)
>at com.gargoylesoftware.htmlunit.javascript.configuration.JavaScriptConfiguration.<init>(JavaScriptConfiguration.java:603)
>at com.gargoylesoftware.htmlunit.javascript.configuration.JavaScriptConfiguration.getInstance(JavaScriptConfiguration.java:619)
>at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.<init>(JavaScriptEngine.java:141)
>at com.gargoylesoftware.htmlunit.WebClient.<init>(WebClient.java:249)
>at com.gargoylesoftware.htmlunit.WebClient.<init>(WebClient.java:227)
>at com.gargoylesoftware.htmlunit.WebClient.<init>(WebClient.java:219)
>at com.tripadvisor.flights.partners.travix.TravixProcessor.main(TravixProcessor.java:10)
>
>Process finished with exit code 1
>
>Any help is appreciated! Thanks!
>
>
>
>----< Inline text [text-plain-04.txt] >------------------
>
>
>
>
>----< Inline text [text-plain-05.txt] >------------------
>
>_______________________________________________
>Htmlunit-user mailing list
>Htm...@li...
>https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>
>

[Htmlunit-user] HTMLUnit does not seem to execute google Recaptcha JS to fetch a Token

[Vasudevan C. <vco...@gm...>]

Hi,

   I am using the URL to get the login screen
https://recruiter.cwjobs.co.uk/login/.
   There is a Recaptcha JS which is supposed to populate the value reCaptcha
    token within a IFrame. I did not see the request to fetch the token
data from
    google going out of HTMLUnit.

   When I try it directly with Chrome Browser, I see the request to fetch
token data
    going and the value is populated inside an iFrame.

   I am using the latest version of HTMLUnit 2.36.

   Any help would be appreciated.

Regards
 Vasu

[Htmlunit-user] IncompatibleClassChangeError when using HtmlUnit in java in Intelli

[Jiechao W. <jie...@gm...>]

Hi, I am trying to use HtmlUnit with the following code in java in IntelliJ
Ultimate 2018.3.6

public static void main(String[] args){
    WebClient webClient = new WebClient();}

It is a gradle project so I have this in my build.gradle

compile 'net.sourceforge.htmlunit:htmlunit:2.36.0'

But when I ran the above one line program from Intellij, I got such error:

Exception in thread "main" java.lang.IncompatibleClassChangeError:
class org.eclipse.jetty.util.component.ContainerLifeCycle can not
implement org.eclipse.jetty.util.component.Container, because it is
not an interface (org.eclipse.jetty.util.component.Container is in
unnamed module of loader 'app')
at java.base/java.lang.ClassLoader.defineClass1(Native Method)
at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1016)
at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:802)
at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:700)
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:623)
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
at java.base/java.lang.ClassLoader.defineClass1(Native Method)
at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1016)
at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:802)
at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:700)
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:623)
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
at java.base/java.lang.Class.getDeclaredFields0(Native Method)
at java.base/java.lang.Class.privateGetDeclaredFields(Class.java:3062)
at java.base/java.lang.Class.getDeclaredFields(Class.java:2249)
at com.gargoylesoftware.htmlunit.javascript.configuration.AbstractJavaScriptConfiguration.process(AbstractJavaScriptConfiguration.java:298)
at com.gargoylesoftware.htmlunit.javascript.configuration.AbstractJavaScriptConfiguration.getClassConfiguration(AbstractJavaScriptConfiguration.java:204)
at com.gargoylesoftware.htmlunit.javascript.configuration.AbstractJavaScriptConfiguration.buildUsageMap(AbstractJavaScriptConfiguration.java:85)
at com.gargoylesoftware.htmlunit.javascript.configuration.AbstractJavaScriptConfiguration.<init>(AbstractJavaScriptConfiguration.java:65)
at com.gargoylesoftware.htmlunit.javascript.configuration.JavaScriptConfiguration.<init>(JavaScriptConfiguration.java:603)
at com.gargoylesoftware.htmlunit.javascript.configuration.JavaScriptConfiguration.getInstance(JavaScriptConfiguration.java:619)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.<init>(JavaScriptEngine.java:141)
at com.gargoylesoftware.htmlunit.WebClient.<init>(WebClient.java:249)
at com.gargoylesoftware.htmlunit.WebClient.<init>(WebClient.java:227)
at com.gargoylesoftware.htmlunit.WebClient.<init>(WebClient.java:219)
at com.tripadvisor.flights.partners.travix.TravixProcessor.main(TravixProcessor.java:10)

Process finished with exit code 1

Any help is appreciated! Thanks!

[Htmlunit-user] Problems with input elements in iframe

[David L. <leo...@gm...>]

Hello

I am new to htmlunit and I am using it for the first time.

Currently I try to automatically fill and submit details in an iframe as
part of some automated test suite.
The problem I have:
It seems that the text I add and the selections I made are not applied at
all.
I verified this by running a local web server with the same html and
exchanged the javascript which is executed when submitting the data. I
added debug output into the javascript and it shows me that all of the
input elements still have their initial values.

Here is the code I use (its scala code but as u can see its similar to
java):

import com.gargoylesoftware.htmlunit.{Page, WebClient}
import com.gargoylesoftware.htmlunit.html._
import java.nio.file.{Files, Paths}
import com.gargoylesoftware.htmlunit.CollectingAlertHandler
import java.util

object WebsiteAutofill {
  val webClient = new WebClient()
  webClient.getOptions.setJavaScriptEnabled(true)
  webClient.getOptions.setRedirectEnabled(true)
  webClient.getOptions.setCssEnabled(true)

  val collectedAlerts = new util.ArrayList[String]
  webClient.setAlertHandler(new CollectingAlertHandler(collectedAlerts))

  val url = "http://localhost:8000/secupay.html"

  def main(args: Array[String]): Unit = {
    val iframe: HtmlPage = webClient.getPage(url)

    val cardName: HtmlTextInput =
iframe.getElementById("card_name").asInstanceOf[HtmlTextInput]
    val cardNumber: HtmlTextInput =
iframe.getElementById("card_number").asInstanceOf[HtmlTextInput]

    val cardExpiryMonth: HtmlSelect =
iframe.getElementById("card_expiry_month").asInstanceOf[HtmlSelect]
    val cardExpiryYear: HtmlSelect =
iframe.getElementById("card_expiry_year").asInstanceOf[HtmlSelect]
    val cardCvv: HtmlTextInput =
iframe.getElementById("card_cvc").asInstanceOf[HtmlTextInput]

    val submitButton: HtmlSubmitInput =
iframe.getElementById("btn_submit").asInstanceOf[HtmlSubmitInput]

    cardName.setValueAttribute("Card Owner")
    cardName.setText("Card Owner")

    cardNumber.setValueAttribute("4024007186180153")
    cardNumber.setText("4024007186180153")

    cardExpiryMonth.setSelectedAttribute(
      cardExpiryMonth.getOptionByValue("1"),
      true
    ): Page
    cardExpiryMonth.setAttribute("value", "1")

    cardExpiryYear.setSelectedAttribute(
      cardExpiryYear.getOptionByValue("2021"),
      true
    ): Page
    cardExpiryYear.setAttribute("value", "2021")

    cardCvv.setValueAttribute("123")
    cardCvv.setText("123")

    println(s"""
         |card name value: ${cardName
                 .getAttribute("value")} | ${cardName.getText}
         |numbervalue: ${cardNumber.getAttribute("value")}
         |expire month: ${cardExpiryMonth.getAttribute("value")}
         |expire year: ${cardExpiryYear.getAttribute("value")}
         |cvv: ${cardCvv.getAttribute("value")}
         |""".stripMargin)

    val redirectPage: HtmlPage = submitButton.click().asInstanceOf[HtmlPage]
    println(s"Redirect url: ${redirectPage.getUrl.toString}")

    val htmlBody = redirectPage.getWebResponse.getContentAsString
    Files.write(Paths.get("out.html"), htmlBody.getBytes("utf-8"))

    println(s"CollectedAlerts: ${collectedAlerts.size()}")
    collectedAlerts.forEach(println(_))
  }
}

In the println call I can see that the values are set but later when I
print the debug alert outputs from the javascript the values seem to be
lost.

I am not experienced with using mailing lists so I dont know if I can
attach files here, I will just try to attach also the html and javascript
file which I am using so it can be fully reproduced.

I would be really glad if somebody can help me cos I am struggling with
this for days already.

Br
David

[Htmlunit-user] Option to set or configure the Experimental flags for chrome

[Sandeepa R. <san...@gm...>]

Hi,

As chrome has introduced  same-site cookie setting as experimental feature
,wondering if there is a way to enable these flags from html unit when
setting browser version to CHROME.

Thanks,
Sandeep.

[Htmlunit-user] Not executing Javascript on load, checking things, then executing Javascript

[Aaron B. <Aar...@ad...>]

So I've got an unusual case, not sure if it can be done. I want to load up a page, NOT execute the Javascript. Let me poke around the headers and HTML DOM returned, etc. Then I want to enable the Javascript and have it load/execute. 

Reason I want to do it this way is the page I'm loading to test has some JS that runs, and then automatically sends the user on to another URL. However I need to test/check the first page that is loaded, _before_ the Javascript in the WebClient executes to send on the Page. Any way to do this?

--Aaron

Re: [Htmlunit-user] HtmlUnit and JUnit

[Oscar B. <oba...@um...>]

Ok, sounds good.  Thanks.

Oscar

Oscar Bastidas, Ph.D.
Postdoctoral Research Associate
University of Minnesota


On Sun, Oct 27, 2019 at 8:59 AM Ronald Brill <rb...@rb...> wrote:

> Hi Oscar,
>
> HtmlUnit has its roots as tool for unit testing (and this is the reason
> for the name).
> But there is no restriction - you can use if for web scrapping also.
> As of today HtmlUnit is a headless browser (written in java) you can use
> from your java program.
> And from my knowledge HtmlUnit is today used in many web scrappers.
>
>         RBRi
>
> On Thu, 24 Oct 2019 14:52:12 -0500 Oscar Bastidas wrote:
> >
> >Hello,
> >
> >I am a novice in using this software and I have read that in using
> >HTMLUnit, there is some association with JUnit (unfortunately, I am not
> too
> >clear as to what JUnit does in the first place that ties it to HTMLUnit in
> >so many articles on HTMLUnit).
> >
> >Will HTMLUnit not work if I were to try to use its jar files alone for,
> >say, a web-scraping program run on a command-line without using JUnit?
> >Thanks.
> >
> >Oscar
> >
> >
> >
> >----< Inline text [text-plain-04.txt] >------------------
> >
> >
> >
> >
> >----< Inline text [text-plain-05.txt] >------------------
> >
> >_______________________________________________
> >Htmlunit-user mailing list
> >Htm...@li...
> >https://lists.sourceforge.net/lists/listinfo/htmlunit-user
> >
> >
>
>

Re: [Htmlunit-user] HtmlUnit and JUnit

[beshoo <be...@gm...>]

I tested it agents Instagram, and it didn't work.

It seems dos not support java script worker's.

Please confirm.

Regard

On Sun, 27 Oct 2019, 4:18 pm Ronald Brill, <rb...@rb...> wrote:

> Hi Oscar,
>
> HtmlUnit has its roots as tool for unit testing (and this is the reason
> for the name).
> But there is no restriction - you can use if for web scrapping also.
> As of today HtmlUnit is a headless browser (written in java) you can use
> from your java program.
> And from my knowledge HtmlUnit is today used in many web scrappers.
>
>         RBRi
>
> On Thu, 24 Oct 2019 14:52:12 -0500 Oscar Bastidas wrote:
> >
> >Hello,
> >
> >I am a novice in using this software and I have read that in using
> >HTMLUnit, there is some association with JUnit (unfortunately, I am not
> too
> >clear as to what JUnit does in the first place that ties it to HTMLUnit in
> >so many articles on HTMLUnit).
> >
> >Will HTMLUnit not work if I were to try to use its jar files alone for,
> >say, a web-scraping program run on a command-line without using JUnit?
> >Thanks.
> >
> >Oscar
> >
> >
> >
> >----< Inline text [text-plain-04.txt] >------------------
> >
> >
> >
> >
> >----< Inline text [text-plain-05.txt] >------------------
> >
> >_______________________________________________
> >Htmlunit-user mailing list
> >Htm...@li...
> >https://lists.sourceforge.net/lists/listinfo/htmlunit-user
> >
> >
>
>
>
> _______________________________________________
> Htmlunit-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>

Re: [Htmlunit-user] HtmlUnit and JUnit

[Ronald B. <rb...@rb...>]

Hi Oscar,

HtmlUnit has its roots as tool for unit testing (and this is the reason for the name).
But there is no restriction - you can use if for web scrapping also.
As of today HtmlUnit is a headless browser (written in java) you can use from your java program.
And from my knowledge HtmlUnit is today used in many web scrappers.

	RBRi

On Thu, 24 Oct 2019 14:52:12 -0500 Oscar Bastidas wrote:
>
>Hello,
>
>I am a novice in using this software and I have read that in using
>HTMLUnit, there is some association with JUnit (unfortunately, I am not too
>clear as to what JUnit does in the first place that ties it to HTMLUnit in
>so many articles on HTMLUnit).
>
>Will HTMLUnit not work if I were to try to use its jar files alone for,
>say, a web-scraping program run on a command-line without using JUnit?
>Thanks.
>
>Oscar
>
>
>
>----< Inline text [text-plain-04.txt] >------------------
>
>
>
>
>----< Inline text [text-plain-05.txt] >------------------
>
>_______________________________________________
>Htmlunit-user mailing list
>Htm...@li...
>https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>
>

[Htmlunit-user] HtmlUnit and JUnit

[Oscar B. <oba...@um...>]

Hello,

I am a novice in using this software and I have read that in using
HTMLUnit, there is some association with JUnit (unfortunately, I am not too
clear as to what JUnit does in the first place that ties it to HTMLUnit in
so many articles on HTMLUnit).

Will HTMLUnit not work if I were to try to use its jar files alone for,
say, a web-scraping program run on a command-line without using JUnit?
Thanks.

Oscar

[Htmlunit-user] [ANN] HtmlUnit 2.36.0 released

[Ronald B. <rb...@rb...>]

Hi all,

it is a pleasure to announce the availability of HtmlUnit 2.36.0.
The main enhancements are:
  - again resync with Rhino and various Javascript improvements
  - minor Neko and CssParser improvements
  - as always various fixes, specially with JavaScript
  - Chrome 76 and Brotli support
  - Firefox 52 is deprecated

The full list of changes can be found in [1]
Thanks to all the contributors.

Happy Testing/Scraping!
   The HtmlUnit team

[1] http://htmlunit.sourceforge.net/changes-report.html#a2.36.0

Re: [Htmlunit-user] Cannot get links added by Javascript

[Markus J. <mar...@op...>]

Hello,

Just curious, is there some progress on this issue? Or could there be some work around?

Thanks,
Markus 
 
-----Original message-----
> From:Ronald Brill <rb...@rb...>
> Sent: Saturday 27th July 2019 19:59
> To: htm...@li...; Markus Jelsma <mar...@op...>
> Subject: RE: [Htmlunit-user] Cannot get links added by Javascript
> 
> >I have created issue 63 https://github.com/HtmlUnit/htmlunit/issues/63
> 
> Thanks, but i can't promis a fast solution.
> 
> >Has HtmlUnit moved all issue handling from Sourceforge to Github? 
> 
> Not really, but the code was moved and i look at both issue trackers.
> 
> 
>     RBRi
>     --------------------------
>     WETATOR
>     Smart web application testing
>     http://www.wetator.org
> 

Re: [Htmlunit-user] Cannot get links added by Javascript

[Ronald B. <rb...@rb...>]

>I have created issue 63 https://github.com/HtmlUnit/htmlunit/issues/63

Thanks, but i can't promis a fast solution.

>Has HtmlUnit moved all issue handling from Sourceforge to Github? 

Not really, but the code was moved and i look at both issue trackers.


    RBRi
    --------------------------
    WETATOR
    Smart web application testing
    http://www.wetator.org

Re: [Htmlunit-user] Cannot get links added by Javascript

[Markus J. <mar...@op...>]

Hello Ronald,

I have created issue 63 https://github.com/HtmlUnit/htmlunit/issues/63

Has HtmlUnit moved all issue handling from Sourceforge to Github? 

Many thanks,
Markus
 
-----Original message-----
> From:Ronald Brill <rb...@rb...>
> Sent: Saturday 27th July 2019 17:09
> To: htm...@li...; Markus Jelsma <mar...@op...>
> Subject: Re: [Htmlunit-user] Cannot get links added by Javascript
> 
> Hello,
> 
> have done a look at this problem. Looks like some js does not get loaded/executed. 
> Can you please open an issue on github for this.
> 
> 	RBRi
> 
> On Tue, 23 Jul 2019 12:18:04 +0000 Markus Jelsma wrote:
> >
> >Hello again,
> >
> >We need to get links from web pages, added by Javascript in this case. Using HtmlUnit 2.35.0 this works well for most sites, but since recently we have trouble with a specific group of websites, sharing the same CMS.
> >
> >I tried all available BrowserVersions, increased waiting time outs but i cannot get the links from the DOM. 
> >
> >https://www.afas.nl/nieuwsberichten/nieuws
> >This website since recently, assuming the site has changed, is throwing a EcmaError, not being able to read a property. Is that error also actually the real reason the links are not being added?
> >
> >https://www.werkenbijrocwestbrabant.nl/
> >This site, same CMS, is partially working and i get no error. But i cannot get all the links.
> >
> >In case of the first site, is the error responsible for the problem, and if so, how could we work around it? And in case of the second site, if there is no error, but also no desired outcome, where to look for the solution?
> >
> >Many thanks,
> >Markus
> >
> >
> >_______________________________________________
> >Htmlunit-user mailing list
> >Htm...@li...
> >https://lists.sourceforge.net/lists/listinfo/htmlunit-user
> >
> 
> 

Re: [Htmlunit-user] Cannot get links added by Javascript

[Ronald B. <rb...@rb...>]

Hello,

have done a look at this problem. Looks like some js does not get loaded/executed. 
Can you please open an issue on github for this.

	RBRi

On Tue, 23 Jul 2019 12:18:04 +0000 Markus Jelsma wrote:
>
>Hello again,
>
>We need to get links from web pages, added by Javascript in this case. Using HtmlUnit 2.35.0 this works well for most sites, but since recently we have trouble with a specific group of websites, sharing the same CMS.
>
>I tried all available BrowserVersions, increased waiting time outs but i cannot get the links from the DOM. 
>
>https://www.afas.nl/nieuwsberichten/nieuws
>This website since recently, assuming the site has changed, is throwing a EcmaError, not being able to read a property. Is that error also actually the real reason the links are not being added?
>
>https://www.werkenbijrocwestbrabant.nl/
>This site, same CMS, is partially working and i get no error. But i cannot get all the links.
>
>In case of the first site, is the error responsible for the problem, and if so, how could we work around it? And in case of the second site, if there is no error, but also no desired outcome, where to look for the solution?
>
>Many thanks,
>Markus
>
>
>_______________________________________________
>Htmlunit-user mailing list
>Htm...@li...
>https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>

[Htmlunit-user] Cannot get links added by Javascript

[Markus J. <mar...@op...>]

Hello again,

We need to get links from web pages, added by Javascript in this case. Using HtmlUnit 2.35.0 this works well for most sites, but since recently we have trouble with a specific group of websites, sharing the same CMS.

I tried all available BrowserVersions, increased waiting time outs but i cannot get the links from the DOM. 

https://www.afas.nl/nieuwsberichten/nieuws
This website since recently, assuming the site has changed, is throwing a EcmaError, not being able to read a property. Is that error also actually the real reason the links are not being added?

https://www.werkenbijrocwestbrabant.nl/
This site, same CMS, is partially working and i get no error. But i cannot get all the links.

In case of the first site, is the error responsible for the problem, and if so, how could we work around it? And in case of the second site, if there is no error, but also no desired outcome, where to look for the solution?

Many thanks,
Markus

Re: [Htmlunit-user] ClassCastException, unable to obtain hyperlinks added to the DOM by Javascript

[Markus J. <mar...@op...>]

Ah, that works indeed. It was a case of removing an xml-apis jar from somewhere on the classpath.

Thanks,
Markus

 
 
-----Original message-----
> From:Ronald Brill <rb...@rb...>
> Sent: Sunday 21st July 2019 14:24
> To: htm...@li...; Markus Jelsma <mar...@op...>
> Subject: Re: [Htmlunit-user] ClassCastException, unable to obtain hyperlinks added to the DOM by Javascript
> 
> On Mon, 15 Jul 2019 16:21:57 +0000 Markus Jelsma wrote:
> >
> >Caused by: java.lang.ClassCastException: org.apache.xerces.jaxp.DocumentBuilderFactoryImpl cannot be cast to javax.xml.parsers.DocumentBuilderFactory
> >
> 
> This usuall point to a classpath problem (or some other module instanziating a different XMLParser as the one (Xerxes) HtmlUnit requires.
> 
> 
> 
>     RBRi
>     --------------------------
>     WETATOR
>     Smart web application testing
>     http://www.wetator.org
>