htmlunit-user — Discussion of the use of HtmlUnit

[Htmlunit-user] runtimeError: message=[Exceeded maximum stack depth]

[Filip P. <fil...@gm...>]

Hello, I am new to htmlunit.
I need to load page "https://www.t-mobile.cz/sms/closed.jsp" which has new
JS.
I cant load full page. Get result with <noscript> tag.
I tried all BrowserVersion and I am using last htmlunit 2.33.

My code is:
========
WebClient webClient = new WebClient(BrowserVersion.FIREFOX_60);

CookieManager cookieMan = new CookieManager();
cookieMan = webClient.getCookieManager();
cookieMan.setCookiesEnabled(true);

webClient.getOptions().setJavaScriptEnabled(true);
webClient.getOptions().setRedirectEnabled(true);
webClient.getOptions().setCssEnabled(true);
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.getOptions().setCssEnabled(true);

HtmlPage page = webClient.getPage("https://www.t-mobile.cz/sms/closed.jsp");
webClient.waitForBackgroundJavaScript(30 * 1000);
String pageAsXml = page.asXml();
========

I get debug:
com.gargoylesoftware.htmlunit.IncorrectnessListenerImpl notify
WARNING: Obsolete content type encountered: 'text/javascript'.

Sometimes I get debug:
SEVERE: runtimeError: message=[Exceeded maximum stack depth]


Is it wrong JS on a page? When I load page via Firefox, debugger gets some
errors too, but the page loads fine.
Any suggestions?

Thank you for very much your reply.

Best regards

Filip

Re: [Htmlunit-user] ContentType gzip-only-text/html

[Vasudevan C. <vco...@gm...>]

Thanks Ronald for your reply.
Look forward to a new snapshot.

Regards
 Vasu

On Fri, 14 Sep 2018 at 11:56 PM, Ronald Brill <rb...@rb...> wrote:

> Yes this a bug, will fix this and make a new snapshot available
>
>         RBRi
>
> On Thu, 13 Sep 2018 02:44:39 +0530 Vasudevan Comandur wrote:
> >Hi,
> >
> >   Will HTMLUnit fail when the response header contains
> >Content-Encoding=gzip-only-text/html and the Content-Type is
> >application/pdf?
> >   Looks like it is reading the webResponseData as gZIP stream. Is it an
> >issue?
> >
> >Regards
> > Vasu
> >
> >On Thu, 13 Sep 2018 at 01:10, Ronald Brill <rb...@rb...> wrote:
> >
> >> Hi all,
> >>
> >> it is a pleasure to announce the availability of HtmlUnit 2.33.
> >> The main enhancements are:
> >>   - the source code is now at github
> >>   - finally we are using the TypedArray impl from Rhino instead of our
> own
> >> one
> >>   - reworked js event handling
> >>   - as always various fixes, specially with JavaScript
> >>
> >> The full list of changes can be found in [1]
> >> Thanks to all the contributors.
> >>
> >> Happy Testing/Scraping!
> >>    The HtmlUnit team
> >>
> >> [1] http://htmlunit.sourceforge.net/changes-report.html#a2.33
> >>
> >>
> >>
> >> _______________________________________________
> >> Htmlunit-user mailing list
> >> Htm...@li...
> >> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
> >>
> >
> >
>
>

[Htmlunit-user] ContentType gzip-only-text/html

[Ronald B. <rb...@rb...>]

Yes this a bug, will fix this and make a new snapshot available

	RBRi

On Thu, 13 Sep 2018 02:44:39 +0530 Vasudevan Comandur wrote:
>Hi,
>
>   Will HTMLUnit fail when the response header contains
>Content-Encoding=gzip-only-text/html and the Content-Type is
>application/pdf?
>   Looks like it is reading the webResponseData as gZIP stream. Is it an
>issue?
>
>Regards
> Vasu
>
>On Thu, 13 Sep 2018 at 01:10, Ronald Brill <rb...@rb...> wrote:
>
>> Hi all,
>>
>> it is a pleasure to announce the availability of HtmlUnit 2.33.
>> The main enhancements are:
>>   - the source code is now at github
>>   - finally we are using the TypedArray impl from Rhino instead of our own
>> one
>>   - reworked js event handling
>>   - as always various fixes, specially with JavaScript
>>
>> The full list of changes can be found in [1]
>> Thanks to all the contributors.
>>
>> Happy Testing/Scraping!
>>    The HtmlUnit team
>>
>> [1] http://htmlunit.sourceforge.net/changes-report.html#a2.33
>>
>>
>>
>> _______________________________________________
>> Htmlunit-user mailing list
>> Htm...@li...
>> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>>
>
>

Re: [Htmlunit-user] HtmlUnit 2.33 released

[Vasudevan C. <vco...@gm...>]

Hi,

   Will HTMLUnit fail when the response header contains
Content-Encoding=gzip-only-text/html and the Content-Type is
application/pdf?
   Looks like it is reading the webResponseData as gZIP stream. Is it an
issue?

Regards
 Vasu

On Thu, 13 Sep 2018 at 01:10, Ronald Brill <rb...@rb...> wrote:

> Hi all,
>
> it is a pleasure to announce the availability of HtmlUnit 2.33.
> The main enhancements are:
>   - the source code is now at github
>   - finally we are using the TypedArray impl from Rhino instead of our own
> one
>   - reworked js event handling
>   - as always various fixes, specially with JavaScript
>
> The full list of changes can be found in [1]
> Thanks to all the contributors.
>
> Happy Testing/Scraping!
>    The HtmlUnit team
>
> [1] http://htmlunit.sourceforge.net/changes-report.html#a2.33
>
>
>
> _______________________________________________
> Htmlunit-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>

[Htmlunit-user] HtmlUnit 2.33 released

[Ronald B. <rb...@rb...>]

Hi all,

it is a pleasure to announce the availability of HtmlUnit 2.33.
The main enhancements are:
  - the source code is now at github
  - finally we are using the TypedArray impl from Rhino instead of our own one
  - reworked js event handling
  - as always various fixes, specially with JavaScript

The full list of changes can be found in [1]
Thanks to all the contributors.

Happy Testing/Scraping!
   The HtmlUnit team

[1] http://htmlunit.sourceforge.net/changes-report.html#a2.33

[Htmlunit-user] [ANN] HtmlUnit 2.32 released

[Ronald B. <rb...@rb...>]

Hi all,

it is a pleasure to announce the availability of HtmlUnit 2.32.
The main enhancements are:
  - again resync with Rhino and various Javascript improvements
  - Firefox 45 is gone, Firefox 60 added
  - Chrome 68
  - as always various fixes, specially with JavaScript

Even if the list of changes is not that impressive, at least the FF60 support is another great step forward.

The full list of changes can be found in [1]
Thanks to all the contributors.

Happy Testing/Scraping!
   The HtmlUnit team

[1] http://htmlunit.sourceforge.net/changes-report.html#a2.32

[Htmlunit-user] invoke echart api error

[<li...@ce...>]

Hi all

I accessed a html which including echart code by htmlunit.

The div content in the java output is different with html output.

Please help ~

 

------------------


刘超 技术中心
Mobile：18240256086
E-mail：li...@ce...
沈阳世纪高通科技有限公司 
地址：沈阳市沈河区青年大街1-1号沈阳市府恒隆广场办公楼1座12层1201室
F12 No.1 Office tower, No.1 Qingnian Da Jie, Shenhe District, Shenyang 110063, China
http://www.cennavi.com.cn/

[Htmlunit-user] Content loaded by JavaScript

[Felipe S. <fel...@gm...>]

Hi, i'm new in HtmlUnit trying to parse a div content thats loaded by
JavaScript.

My code:

WebClient client = new WebClient(BrowserVersion.CHROME);
HtmlPage pagina = client.getPage("https://www.rico.com.vc/renda-fixa/cdb");
client.getOptions().setThrowExceptionOnScriptError(false);
client.getOptions().setJavaScriptEnabled(true);
client.setAjaxController(new NicelyResynchronizingAjaxController());
client.waitForBackgroundJavaScript(60000);
System.out.println(pagina.asText());

But i'm getting this errors:

INFORMAÇÕES:
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "
http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">


<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
<title>Rico.com.vc</title>
        <link href='
https://fonts.googleapis.com/css?family=Montserrat:400,700|Open+Sans:400,700&subset=latin,latin-ext'
rel='stylesheet' type='text/css'>

<style type="text/css">
body
{
margin:0;
padding:0;
font-family:'Trebuchet Ms', Arial, Helvetica;
font-size:12px;
}
.text-center { text-align: center; }
.uppercase { text-transform: uppercase; }
.font-montserrat { font-family: 'Montserrat'; }
.font-open { font-family: 'Open Sans'; }
.font-size-1 { font-size: 16px; }
.font-size-2 { font-size: 28px; }
.padding-bottom-1 { padding-bottom: 10px; }
.padding-bottom-2 { padding-bottom: 30px; }
.msg-error
{
    margin-top: -145px;
    padding-left: 235px;
    font-size: 60px;
    color: #FFF;
    text-shadow:
    -1px -1px 0 #F18719,
    1px -1px 0 #F18719,
    -1px 1px 0 #F18719,
    1px 1px 0 #F18719;
}
.margin-top-1 { margin-top: 145px; }
.font-grey { color: grey; }
.font-bold { font-weight: bold; }
.img_logo { width: 50%; }
.img { width: 110%; padding: 20px 0 30px 0; }
.content { width: 500px; text-align: center; margin: 20px auto; }
.button-wrapper
{
    background-color: #EF8A32;
    padding: 15px 15px;
    border: 1px solid #EF8A32;
    border-radius: 4px;
}
            .home-link
            {
            color: white;
text-decoration: none;
font-weight: bold;
}
</style>

</head>
<body onload="document_onload()">
    <script type="text/javascript">
    function document_onload()
    {
        lblErro.innerHTML = "";
    }
    </script>
<div class="content">
            <a href="/"><img src="//www.rico.com.vc/rico-base/Rico_logo.jpg"
class="img_logo" border="0" alt="Logo Rico" /></a>
            <img src="//www.rico.com.vc/dashboard/img/404.png" class="img"
alt="Computador com mensagem de erro" />
            <div>
                <p class="msg-error">404</p>
            </div>
</div>
        <div class="text-center">
    <p class="uppercase font-montserrat font-size-2 font-grey font-bold
padding-bottom-1 margin-top-1">A p&aacute;gina solicitada n&atilde;o foi
encontrada.</p>
            <p class="font-open font-size-1 font-grey
padding-bottom-2">Caso o problema persista, favor entrar em contato com<br
/> nossa central de <a href="
https://www.rico.com.vc/servicos/atendimento/contato" class="font-grey
font-bold">atendimento.</a></p>
<button class="button-wrapper" type="button">
<a href="/" class="home-link uppercase" title="Voltar para a Home">Ir para
a P&aacute;gina Inicial ></a>
</button>
</div>
</body>
</html>
jun 06, 2018 12:45:12 PM
com.gargoylesoftware.htmlunit.javascript.DefaultJavaScriptErrorListener
loadScriptError
GRAVE: Error loading JavaScript from [
https://www.rico.com.vc:443/WebResource.axd?d=p-e1U0PJjdGCHIHBWiD1_mnNyd8XXQ5baJIt17nqS_Wf552pOyyqkjGu6pxXAZ0QL3vedCpP0awH9-IXEKTmPIHCFcY_2PgSBqh3-Kt13gLbD5Wx8QQ_xVePbKJbgc7Nt5QmnNlvk1_kvJEpqvYH5nDIR3o1&t=636177466400000000
].
com.gargoylesoftware.htmlunit.FailingHttpStatusCodeException: 404 Not Found
for
https://www.rico.com.vc:443/WebResource.axd?d=p-e1U0PJjdGCHIHBWiD1_mnNyd8XXQ5baJIt17nqS_Wf552pOyyqkjGu6pxXAZ0QL3vedCpP0awH9-IXEKTmPIHCFcY_2PgSBqh3-Kt13gLbD5Wx8QQ_xVePbKJbgc7Nt5QmnNlvk1_kvJEpqvYH5nDIR3o1&t=636177466400000000
at
com.gargoylesoftware.htmlunit.WebClient.throwFailingHttpStatusCodeExceptionIfNecessary(WebClient.java:590)
at
com.gargoylesoftware.htmlunit.html.HtmlPage.loadJavaScriptFromUrl(HtmlPage.java:1034)
at
com.gargoylesoftware.htmlunit.html.HtmlPage.loadExternalJavaScriptFile(HtmlPage.java:975)
at
com.gargoylesoftware.htmlunit.html.HtmlScript.executeScriptIfNeeded(HtmlScript.java:371)
at
com.gargoylesoftware.htmlunit.html.HtmlScript$2.execute(HtmlScript.java:246)
at
com.gargoylesoftware.htmlunit.html.HtmlScript.onAllChildrenAddedToPage(HtmlScript.java:267)
at
com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.endElement(HTMLParser.java:805)
at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source)
at
com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.endElement(HTMLParser.java:761)
at
net.sourceforge.htmlunit.cyberneko.HTMLTagBalancer.callEndElement(HTMLTagBalancer.java:1236)
at
net.sourceforge.htmlunit.cyberneko.HTMLTagBalancer.endElement(HTMLTagBalancer.java:1136)
at
net.sourceforge.htmlunit.cyberneko.filters.DefaultFilter.endElement(DefaultFilter.java:226)
at
net.sourceforge.htmlunit.cyberneko.filters.NamespaceBinder.endElement(NamespaceBinder.java:345)
at
net.sourceforge.htmlunit.cyberneko.HTMLScanner$ContentScanner.scanEndElement(HTMLScanner.java:3189)
at
net.sourceforge.htmlunit.cyberneko.HTMLScanner$ContentScanner.scan(HTMLScanner.java:2141)
at
net.sourceforge.htmlunit.cyberneko.HTMLScanner.scanDocument(HTMLScanner.java:945)
at
net.sourceforge.htmlunit.cyberneko.HTMLConfiguration.parse(HTMLConfiguration.java:521)
at
net.sourceforge.htmlunit.cyberneko.HTMLConfiguration.parse(HTMLConfiguration.java:472)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at
com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.parse(HTMLParser.java:1004)
at com.gargoylesoftware.htmlunit.html.HTMLParser.parse(HTMLParser.java:253)
at
com.gargoylesoftware.htmlunit.html.HTMLParser.parseHtml(HTMLParser.java:195)
at
com.gargoylesoftware.htmlunit.DefaultPageCreator.createHtmlPage(DefaultPageCreator.java:267)
at
com.gargoylesoftware.htmlunit.DefaultPageCreator.createPage(DefaultPageCreator.java:158)
at
com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:529)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:463)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:448)
at br.com.controller.RoboRico.capturarConteudo(RoboRico.java:26)
at org.Robo.Main.main(Main.java:14)

Exception in thread "main"
com.gargoylesoftware.htmlunit.FailingHttpStatusCodeException: 404 Not Found
for
https://www.rico.com.vc:443/WebResource.axd?d=p-e1U0PJjdGCHIHBWiD1_mnNyd8XXQ5baJIt17nqS_Wf552pOyyqkjGu6pxXAZ0QL3vedCpP0awH9-IXEKTmPIHCFcY_2PgSBqh3-Kt13gLbD5Wx8QQ_xVePbKJbgc7Nt5QmnNlvk1_kvJEpqvYH5nDIR3o1&t=636177466400000000
at
com.gargoylesoftware.htmlunit.WebClient.throwFailingHttpStatusCodeExceptionIfNecessary(WebClient.java:590)
at
com.gargoylesoftware.htmlunit.html.HtmlPage.loadJavaScriptFromUrl(HtmlPage.java:1034)
at
com.gargoylesoftware.htmlunit.html.HtmlPage.loadExternalJavaScriptFile(HtmlPage.java:975)
at
com.gargoylesoftware.htmlunit.html.HtmlScript.executeScriptIfNeeded(HtmlScript.java:371)
at
com.gargoylesoftware.htmlunit.html.HtmlScript$2.execute(HtmlScript.java:246)
at
com.gargoylesoftware.htmlunit.html.HtmlScript.onAllChildrenAddedToPage(HtmlScript.java:267)
at
com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.endElement(HTMLParser.java:805)
at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source)
at
com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.endElement(HTMLParser.java:761)
at
net.sourceforge.htmlunit.cyberneko.HTMLTagBalancer.callEndElement(HTMLTagBalancer.java:1236)
at
net.sourceforge.htmlunit.cyberneko.HTMLTagBalancer.endElement(HTMLTagBalancer.java:1136)
at
net.sourceforge.htmlunit.cyberneko.filters.DefaultFilter.endElement(DefaultFilter.java:226)
at
net.sourceforge.htmlunit.cyberneko.filters.NamespaceBinder.endElement(NamespaceBinder.java:345)
at
net.sourceforge.htmlunit.cyberneko.HTMLScanner$ContentScanner.scanEndElement(HTMLScanner.java:3189)
at
net.sourceforge.htmlunit.cyberneko.HTMLScanner$ContentScanner.scan(HTMLScanner.java:2141)
at
net.sourceforge.htmlunit.cyberneko.HTMLScanner.scanDocument(HTMLScanner.java:945)
at
net.sourceforge.htmlunit.cyberneko.HTMLConfiguration.parse(HTMLConfiguration.java:521)
at
net.sourceforge.htmlunit.cyberneko.HTMLConfiguration.parse(HTMLConfiguration.java:472)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at
com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.parse(HTMLParser.java:1004)
at com.gargoylesoftware.htmlunit.html.HTMLParser.parse(HTMLParser.java:253)
at
com.gargoylesoftware.htmlunit.html.HTMLParser.parseHtml(HTMLParser.java:195)
at
com.gargoylesoftware.htmlunit.DefaultPageCreator.createHtmlPage(DefaultPageCreator.java:267)
at
com.gargoylesoftware.htmlunit.DefaultPageCreator.createPage(DefaultPageCreator.java:158)
at
com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:529)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:463)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:448)
at br.com.controller.RoboRico.capturarConteudo(RoboRico.java:26)
at org.Robo.Main.main(Main.java:14)

Thanks.

[Htmlunit-user] Error: cannot find symbol after fresh install of HtmlUnit

[Serg R. <rr...@ya...>]

Hello all
I'm new user, just downloaded and set-up
HtmlUnit in the first time.
1) 
My classpath set up as follows:(and all the jars are in this folder D:\java\jar\htmlunit\)
set CLASSPATH=C:\jdk9\bin\;C:\jdk9\jre\bin\;C:\jdk9\lib\;d:\java\servlet.jar;d:\java\JAVAX\;D:\CosmosLib\;D:\java\jar\htmlunit\commons-codec-1.10.jar;D:\java\jar\htmlunit\commons-io-2.6.jar;D:\java\jar\htmlunit\commons-lang3-3.7.jar;D:\java\jar\htmlunit\commons-logging-1.2.jar;D:\java\jar\htmlunit\commons-net-3.6.jar;D:\java\jar\htmlunit\commons-text-1.3.jar;D:\java\jar\htmlunit\htmlunit-2.31.jar;D:\java\jar\htmlunit\htmlunit-core-js-2.31.jar;D:\java\jar\htmlunit\htmlunit-cssparser-1.0.0.jar;D:\java\jar\htmlunit\httpclient-4.5.5.jar;D:\java\jar\htmlunit\httpcore-4.4.9.jar;D:\java\jar\htmlunit\httpmime-4.5.5.jar;D:\java\jar\htmlunit\jetty-client-9.4.10.v20180503.jar;D:\java\jar\htmlunit\jetty-http-9.4.10.v20180503.jar;D:\java\jar\htmlunit\jetty-io-9.4.10.v20180503.jar;D:\java\jar\htmlunit\jetty-util-9.4.10.v20180503.jar;D:\java\jar\htmlunit\jetty-xml-9.4.10.v20180503.jar;D:\java\jar\htmlunit\neko-htmlunit-2.31.jar;D:\java\jar\htmlunit\serializer-2.7.2.jar;D:\java\jar\htmlunit\websocket-api-9.4.10.v20180503.jar;D:\java\jar\htmlunit\websocket-client-9.4.10.v20180503.jar;D:\java\jar\htmlunit\websocket-common-9.4.10.v20180503.jar;D:\java\jar\htmlunit\xalan-2.7.2.jar;D:\java\jar\htmlunit\xercesImpl-2.11.0.jar;D:\java\jar\htmlunit\xml-apis-1.4.01.jar;.
2)
Have this line to import htmlunit package:
import com.gargoylesoftware.htmlunit.*;

3) 
Put this line in my java code (from example):
final HtmlPage page1 = webClient.getPage("http://...login_2.html");

However, compliling under Java 1.9 gives this error
for this particular line:error: cannot find symbol HtmlPage page1 = ....
                          ^
Cannot figure out, why it is not imported.
Could you help.
Thank you
Serg

[Htmlunit-user] [ANN] HtmlUnit 2.31 released

[Ronald B. <rb...@rb...>]

Hi all,

it is a pleasure to announce the availability of HtmlUnit 2.31.
The main enhancements are:
  - resync with Rhino and various Javascript improvements
  - Firefox 45 is now deprecated
  - as always various fixes, specially with JavaScript

The full list of changes can be found in [1]
Thanks to all the contributors.

Happy Testing/Scraping!
   The HtmlUnit team

[1] http://htmlunit.sourceforge.net/changes-report.html#a2.31

Re: [Htmlunit-user] HTMLUnit crashes loading a magazines homepage

[Michael M. <mic...@su...>]

Nevermind! I was able to keep going by adding: 

 

 
webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);

 
webClient.getOptions().setThrowExceptionOnScriptError(false);

 

Sorry for the bandwidth consumed!

Michael

 

 

From: Michael Moser <mic...@su...> 
Sent: Wednesday, April 11, 2018 11:24 PM
To: htm...@li...
Subject: [Htmlunit-user] HTMLUnit crashes loading a magazines homepage

 

Hi - I was experimenting with HTMLUnit. I would like to write myself a small
utility that always downloads me the newest editions of a magazine that I
have subscribed to.

 

However, HTMLUnit always fails (i.e. throws an exception) when simply
loading the landing page of the magazine's shop site, i.e. already in the
second statement of:

 

       WebClient webClient = new WebClient();

       HtmlPage page1 = webClient.getPage("
<https://shop.heise.de/customer/account>
https://shop.heise.de/customer/account");

 

Is there anything I can do to prevent that crash and keep going?

 

Regards,

Michael

[Htmlunit-user] HTMLUnit crashes loading a magazines homepage

[Michael M. <mic...@su...>]

Hi - I was experimenting with HTMLUnit. I would like to write myself a small
utility that always downloads me the newest editions of a magazine that I
have subscribed to.

 

However, HTMLUnit always fails (i.e. throws an exception) when simply
loading the landing page of the magazine's shop site, i.e. already in the
second statement of:

 

       WebClient webClient = new WebClient();

       HtmlPage page1 = webClient.getPage("
<https://shop.heise.de/customer/account>
https://shop.heise.de/customer/account");

 

Is there anything I can do to prevent that crash and keep going?

 

Regards,

Michael

Re: [Htmlunit-user] Is it possible to set content-type as multipart/form-data in HTTPHEader and POST?

[Vasudevan C. <vco...@gm...>]

Hi All,

    I am creating a Web Request object with the URL and the method is set
to POST and setting the FormEncoding Type to MULTIPART.

   How do I upload a file?. I know the name of the file and its path. DO I
need create a dummy form with a submit Button?

   I appreciate your help.

Regards
 Vasu

On 25 January 2018 at 01:11, Ronald Brill <rb...@rb...> wrote:

> yourWebRequest.setEncodingType(FormEncodingType.MULTIPART);
>
> Hope the same methods are available in your old version also
>
>         RBRi
>
> On Thu, 25 Jan 2018 01:08:33 +0530 Vasudevan Comandur wrote:
> >
> >Hi Ronald,
> >
> >    I am constructing my own WebRequest with HTTPMethod set to POST.
> >
> >    I construct the NameValue Pair and POST the request. I see HTMLUnit
> set
> >the content-type to x-form-encoded in this case.
> >
> >    However, I want the similar feature to construct a
> >multi-part/formencoded content-type in HTMLUnit. Is it possible?
> >
> >     I have another question. For one of the sites, the client(javascript)
> >is setting a cookie value and sending it to the server.  Is it possible
> >     in HTMLUnit to add that cookie to the cookie manager of Webclient
> >using addCookie Method?. Will that work. Because, such a call does
> >     not set the Cookie in HTMLUnit in later version. Earlier versions of
> >HtmlUnit 2.9, I was able to do that.
> >
> >     Looking forward to your reply.
> >
> >Regards
> > Vasu
> >
> >
> >
> >
> >On 25 January 2018 at 00:57, Ronald Brill <rb...@rb...> wrote:
> >
> >> Hi Vasu,
> >>
> >> i'm not sure that i got your point.
> >> Do you like
> >> * to manipulate the form before clicking the submit button or
> >> * have you created your own WebRequest and like to change the encoding
> >> type?
> >>
> >> Maybe you can illustrate your ide with some (pseudo) code
> >>
> >>         RBRi
> >>
> >> BTW: 2.29 and 2.30 is on the way
> >>
> >>
> >> >Hi,
> >> >
> >> >   I have been using htmlunit 2.23 version. I wanted to know if it is
> >> >possible to set
> >> >   the content-type as multipart/form-data in HTTPRequestHEader and
> post a
> >> >form.
> >> >
> >> >    Is it possible to achieve the above scenario without creating a
> sample
> >> >html file
> >> >    and click on the submit button?
> >> >
> >> >   Thanks for your input & suggestions in advance.
> >> >
> >> >Regards
> >> > Vasu
> >>
> >>
> >
> >
>
>

Re: [Htmlunit-user] Our Application to support Google Recaptcha V2

[jp l. <jp....@gm...>]

Any updates on this ? I am stuck with the same.  But from here
http://www.bcra.gob.ar/BCRAyVos/Situacion_crediticia.asp

On Wed, Feb 28, 2018 at 3:18 PM, Vasudevan Comandur <vco...@gm...> wrote:
> All,
>
>     One of the sites use google recaptcha v2. Our application uses HTMLUnit
>     to scrap the contents. Now, they throw recaptcha V2 having the check-box
>     inside a widget.
>
>     The widget construction happens inside the Javascript. I waited for
> about 20
>     secs and read the DOM as XML.
>
>      I was checking if the div class that I am looking for is available. But
> it was not.
>
>      The starting URL is
> https://verify.naukri.com/captcha?URL=https://resdex.naukri.com:443/v2/search/advSearch%3f
>
>      The Div class name is "rc-anchor-center-container".
>
>      Am I doing anything wrong?. I want to click the DIV element.
>
>       Looking forward to your help and I appreciate your help in advance.
>
> Regards
>  Vasu
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Htmlunit-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>

[Htmlunit-user] recaptcha

[jp l. <jp....@gm...>]

Can someone solved how to pass reCaptcha using HtmlUnit 2.30 ?

https://www.google.com/recaptcha/api2/demo

thank you

[Htmlunit-user] [ANN] HtmlUnit 2.30 released

[Ronald B. <rb...@rb...>]

Hi all,

it is a pleasure to announce the availability of HtmlUnit 2.30.
The main enhancements are:
  - using our own css parser for faster css processing
  - URLSearchParams implemented
  - Support Chrome 66
  - as always various fixes, specially with JavaScript

The full list of changes can be found in [1]
Thanks to all the contributors.

Happy Testing/Scraping!
   The HtmlUnit team

[1] http://htmlunit.sourceforge.net/changes-report.html#a2.30

[Htmlunit-user] login to instagram

[beshoo <be...@gm...>]

Dear all , is there any one can use html-unit to log to instagram website
and click on like button?

Regards

Re: [Htmlunit-user] org.eclipse.jetty.websocket.api.UpgradeException: 0 null

[Ian C. <ian...@gm...>]

Hi Ronald,

I just wanted to check in and see if there was any additional information I
could provide to assist with diagnosing this issue.

Many thanks,

Ian.

On Sun, Mar 18, 2018 at 4:12 PM Ian Clarke <ian...@gm...> wrote:

> Thanks for the quick response.  Here is the relevant test:
>
>
> https://github.com/kwebio/core/blob/spek/src/test/kotlin/io/kweb/KwebSpec.kt
>
> The server is started with this line:
>
> val kweb = Kweb(port = 12243) {
> It serves a HTML page, with some JavaScript that immediately opens a
> websocket back to the server.  In this simple test the websocket isn't
> actually used for sending/receiving any message.
>
> I use a simple function to create the WebClient and handle some setup, it
> is here:
>
> https://github.com/kwebio/core/blob/spek/src/test/kotlin/io/kweb/util.kt
>
> I'm using htmlunit version 2.29 - which I believe is the latest.
>
> Please don't hesitate to let me know if you have any questions.
>
> Many thanks,
>
> Ian.
>
> Ian Clarke
> http://blog.locut.us/
>
> On Sun, Mar 18, 2018 at 2:35 PM Ronald Brill <rb...@rb...> wrote:
>
>> Hi Ian,
>>
>> what version of HtmlUnit do you use and can you please provide some code
>> to give me a chance to reproduce your problem.
>>
>> There was a bug in the past with a similar error message.
>> https://sourceforge.net/p/htmlunit/bugs/1865/
>>
>>         RBRi
>>
>> On Sun, 18 Mar 2018 16:39:02 +0000 Ian Clarke wrote:
>> >
>> >I'm attempting to use HtmlUnit to test a web framework I'm writing that
>> >relies on websockets.
>> >
>> >Everything seems to work, HTMLUnit connects to my webserver and retrieves
>> >and verifies that a simple H1 element is present, however it then
>> produces
>> >this exception:
>> >https://gist.github.com/sanity/4695c64196494d36e09cbb7a036125d3
>> >
>> >After this my tests fail to terminate (I'm using Spek, a popular Kotlin
>> >test framework), I've paused the vm and all I see is the main thread
>> >"parked".
>> >
>> >I'm a bit lost about where to go next with this, would appreciate any
>> >suggestions.
>> >
>> >Thanks,
>> >
>> >Ian.
>> >
>> >Ian Clarke
>> >http://blog.locut.us/
>>
>> --
>
> Uprizer Labs LLC
>
> http://uprizer.com/
>
-- 

Uprizer Labs LLC

http://uprizer.com/

Re: [Htmlunit-user] org.eclipse.jetty.websocket.api.UpgradeException: 0 null

[Ian C. <ian...@gm...>]

Thanks for the quick response.  Here is the relevant test:

https://github.com/kwebio/core/blob/spek/src/test/kotlin/io/kweb/KwebSpec.kt

The server is started with this line:

val kweb = Kweb(port = 12243) {
It serves a HTML page, with some JavaScript that immediately opens a
websocket back to the server.  In this simple test the websocket isn't
actually used for sending/receiving any message.

I use a simple function to create the WebClient and handle some setup, it
is here:

https://github.com/kwebio/core/blob/spek/src/test/kotlin/io/kweb/util.kt

I'm using htmlunit version 2.29 - which I believe is the latest.

Please don't hesitate to let me know if you have any questions.

Many thanks,

Ian.

Ian Clarke
http://blog.locut.us/

On Sun, Mar 18, 2018 at 2:35 PM Ronald Brill <rb...@rb...> wrote:

> Hi Ian,
>
> what version of HtmlUnit do you use and can you please provide some code
> to give me a chance to reproduce your problem.
>
> There was a bug in the past with a similar error message.
> https://sourceforge.net/p/htmlunit/bugs/1865/
>
>         RBRi
>
> On Sun, 18 Mar 2018 16:39:02 +0000 Ian Clarke wrote:
> >
> >I'm attempting to use HtmlUnit to test a web framework I'm writing that
> >relies on websockets.
> >
> >Everything seems to work, HTMLUnit connects to my webserver and retrieves
> >and verifies that a simple H1 element is present, however it then produces
> >this exception:
> >https://gist.github.com/sanity/4695c64196494d36e09cbb7a036125d3
> >
> >After this my tests fail to terminate (I'm using Spek, a popular Kotlin
> >test framework), I've paused the vm and all I see is the main thread
> >"parked".
> >
> >I'm a bit lost about where to go next with this, would appreciate any
> >suggestions.
> >
> >Thanks,
> >
> >Ian.
> >
> >Ian Clarke
> >http://blog.locut.us/
>
> --

Uprizer Labs LLC

http://uprizer.com/

Re: [Htmlunit-user] org.eclipse.jetty.websocket.api.UpgradeException: 0 null

[Ronald B. <rb...@rb...>]

Hi Ian,

what version of HtmlUnit do you use and can you please provide some code to give me a chance to reproduce your problem.

There was a bug in the past with a similar error message.
https://sourceforge.net/p/htmlunit/bugs/1865/

	RBRi

On Sun, 18 Mar 2018 16:39:02 +0000 Ian Clarke wrote:
>
>I'm attempting to use HtmlUnit to test a web framework I'm writing that
>relies on websockets.
>
>Everything seems to work, HTMLUnit connects to my webserver and retrieves
>and verifies that a simple H1 element is present, however it then produces
>this exception:
>https://gist.github.com/sanity/4695c64196494d36e09cbb7a036125d3
>
>After this my tests fail to terminate (I'm using Spek, a popular Kotlin
>test framework), I've paused the vm and all I see is the main thread
>"parked".
>
>I'm a bit lost about where to go next with this, would appreciate any
>suggestions.
>
>Thanks,
>
>Ian.
>
>Ian Clarke
>http://blog.locut.us/

[Htmlunit-user] org.eclipse.jetty.websocket.api.UpgradeException: 0 null

[Ian C. <ian...@gm...>]

I'm attempting to use HtmlUnit to test a web framework I'm writing that
relies on websockets.

Everything seems to work, HTMLUnit connects to my webserver and retrieves
and verifies that a simple H1 element is present, however it then produces
this exception:
https://gist.github.com/sanity/4695c64196494d36e09cbb7a036125d3

After this my tests fail to terminate (I'm using Spek, a popular Kotlin
test framework), I've paused the vm and all I see is the main thread
"parked".

I'm a bit lost about where to go next with this, would appreciate any
suggestions.

Thanks,

Ian.

Ian Clarke
http://blog.locut.us/
-- 

Uprizer Labs LLC

http://uprizer.com/

[Htmlunit-user] Our Application to support Google Recaptcha V2

[Vasudevan C. <vco...@gm...>]

All,

    One of the sites use google recaptcha v2. Our application uses HTMLUnit
    to scrap the contents. Now, they throw recaptcha V2 having the check-box
    inside a widget.

    The widget construction happens inside the Javascript. I waited for
about 20
    secs and read the DOM as XML.

     I was checking if the div class that I am looking for is available.
But it was not.

     The starting URL is
https://verify.naukri.com/captcha?URL=https://resdex.naukri.com:443/v2/search/advSearch%3f

     The Div class name is "rc-anchor-center-container".

     Am I doing anything wrong?. I want to click the DIV element.

      Looking forward to your help and I appreciate your help in advance.

Regards
 Vasu

Re: [Htmlunit-user] The mouseout event also fires an onblur?

[Ronald B. <rb...@rb...>]

Hi Karl,

did some tests but was too stupid to make a testcase that shows the problem. Please open a bug and provide a sample to give us a chance to fix this.

	RBRi

> Heho, regards bugs the rule is simple: if it differs from real browsers it is a bug. Please open one. 
> Even if your description seems to be really clear maybe you can attach some of your test pages. Will have a look. Focus handling still has some 
> flaws...
> Thanks RBRi

Re: [Htmlunit-user] The mouseout event also fires an onblur?

[<rb...@rb...>]

Am 2. Februar 2018 07:03:42 MEZ schrieb "Karl M. Davis" <ka...@ju...>:
>Howdy again, 
>
>Wow, I just reread that and thought about the problem some more, and
>boy did I botch it all. Apologies. Here's a better attempt: 
>
>Here's an odd problem I'd been seeing in HtmlUnit: an onblur listener
>I've got attached to a form is getting fired whenever I click the
>form's submit button. It didn't make a lot of sense to me at first,
>because 1) it worked in HtmlUnit v2.14 and only appeared when I updated
>to v2.29.0, and 2) it works in FF/Chrome/etc. 
>
>After digging through HtmlUnit's guts with a debugger for a couple of
>days, I've tracked down the problem to some HtmlUnit code that doesn't
>seem quite right to me. In
>'org.openqa.selenium.htmlunit.HtmlUnitMouse.moveOutIfNeeded(DomElement)',
>around line 113, it seems that every click() event first fires, in
>order, a mouseOver event, a mouseOut event, and the a blur event.
>Drilling down a bit deeper, as part of the blur event, HtmlUnit first
>clears the page's focus entirely. Instead of focus changing from
>OldFocusedThing-->NewFocusedThing, it instead goes
>OldFocusedThing-->null-->NewFocusedThing. That seems not quite correct
>to me? 
>
>This probably isn't a big deal for most folks, except that the form I'm
>trying to submit has a blur handler that hides the form. So, when
>HtmlUnit does to click the button, halfway through that it's hidden the
>button and can't click it. 
>
>So what I'm wondering is whether my analysis is correct and this is a
>bug in HtmlUnit? Or is this just one of those things it's reasonable to
>expect me to work around? I honestly don't know -- not familiar enough
>with the relevant specs. 
>
>Thanks in advance, 
>Karl M. Davis 
>
>
>
>From: "Karl M. Davis" <ka...@ju...> 
>To: "htmlunit-user" <htm...@li...> 
>Sent: Thursday, February 1, 2018 10:39:39 PM 
>Subject: [Htmlunit-user] The mouseout event also fires an onblur? 
>
>Howdy all, 
>
>Finally tracked down the cause of an odd problem I'd been seeing in
>HtmlUnit, but not in FF/Chrome/etc.: I've got a form that hides itself
>when the user clicks out of it, via an onblur event listener. 
>
>Turns out, the culprit is here: 
>org.openqa.selenium.htmlunit.HtmlUnitMouse.moveOutIfNeeded(DomElement)
>line 113 
>
>I don't think firing an onblur as part of a mouseOut is expected? I'm
>definitely not seeing it in my desktop browsers. (This also wasn't
>happening with older versions of HtmlUnit.) 
>
>Anyone know if this is a bug in my expectations, or in HtmlUnit? 
>
>Thanks! 
>Karl M. Davis 
>
>------------------------------------------------------------------------------
>
>Check out the vibrant tech community on one of the world's most 
>engaging tech sites, Slashdot.org! http://sdm.link/slashdot 
>_______________________________________________ 
>Htmlunit-user mailing list 
>Htm...@li... 
>https://lists.sourceforge.net/lists/listinfo/htmlunit-user 

Heho, regards bugs the rule is simple: if it differs from real browsers it is a bug. Please open one. Even if your description seems to be really clear maybe you can attach some of your test pages. Will have a look. Focus handling still has some flaws...
Thanks RBRi

Re: [Htmlunit-user] The mouseout event also fires an onblur?

[Karl M. D. <ka...@ju...>]

Howdy again, 

Wow, I just reread that and thought about the problem some more, and boy did I botch it all. Apologies. Here's a better attempt: 

Here's an odd problem I'd been seeing in HtmlUnit: an onblur listener I've got attached to a form is getting fired whenever I click the form's submit button. It didn't make a lot of sense to me at first, because 1) it worked in HtmlUnit v2.14 and only appeared when I updated to v2.29.0, and 2) it works in FF/Chrome/etc. 

After digging through HtmlUnit's guts with a debugger for a couple of days, I've tracked down the problem to some HtmlUnit code that doesn't seem quite right to me. In 'org.openqa.selenium.htmlunit.HtmlUnitMouse.moveOutIfNeeded(DomElement)', around line 113, it seems that every click() event first fires, in order, a mouseOver event, a mouseOut event, and the a blur event. Drilling down a bit deeper, as part of the blur event, HtmlUnit first clears the page's focus entirely. Instead of focus changing from OldFocusedThing-->NewFocusedThing, it instead goes OldFocusedThing-->null-->NewFocusedThing. That seems not quite correct to me? 

This probably isn't a big deal for most folks, except that the form I'm trying to submit has a blur handler that hides the form. So, when HtmlUnit does to click the button, halfway through that it's hidden the button and can't click it. 

So what I'm wondering is whether my analysis is correct and this is a bug in HtmlUnit? Or is this just one of those things it's reasonable to expect me to work around? I honestly don't know -- not familiar enough with the relevant specs. 

Thanks in advance, 
Karl M. Davis 



From: "Karl M. Davis" <ka...@ju...> 
To: "htmlunit-user" <htm...@li...> 
Sent: Thursday, February 1, 2018 10:39:39 PM 
Subject: [Htmlunit-user] The mouseout event also fires an onblur? 

Howdy all, 

Finally tracked down the cause of an odd problem I'd been seeing in HtmlUnit, but not in FF/Chrome/etc.: I've got a form that hides itself when the user clicks out of it, via an onblur event listener. 

Turns out, the culprit is here: 
org.openqa.selenium.htmlunit.HtmlUnitMouse.moveOutIfNeeded(DomElement) line 113 

I don't think firing an onblur as part of a mouseOut is expected? I'm definitely not seeing it in my desktop browsers. (This also wasn't happening with older versions of HtmlUnit.) 

Anyone know if this is a bug in my expectations, or in HtmlUnit? 

Thanks! 
Karl M. Davis 

------------------------------------------------------------------------------ 
Check out the vibrant tech community on one of the world's most 
engaging tech sites, Slashdot.org! http://sdm.link/slashdot 
_______________________________________________ 
Htmlunit-user mailing list 
Htm...@li... 
https://lists.sourceforge.net/lists/listinfo/htmlunit-user