htmlunit-user — Discussion of the use of HtmlUnit

[Htmlunit-user] How do I use XPATH within the CDATA section of XML in HtmlUnit

[Vasudevan C. <vco...@gm...>]

Hi Ahmed,

   Thanks for your reply. Now I have a CDATA section which has Html tags
and I
    would like to access some of the data from the CDATA section using
Xpath.

   Is it possible to do?. If so, how can I do that?

   Any code samples or pointers would be of great help.

Regards
 Vasu

Re: [Htmlunit-user] Clarification Needed on HTMLUnit to process XML payload

[Ahmed A. <asa...@ya...>]

Hi Vasu,
XmlPage page = webClient.getPage(abc);
page.getByXPath(), .getFirstByXPath()


Ahmed

      From: Vasudevan Comandur <vco...@gm...>
 To: "htm...@li..." <htm...@li...> 
 Sent: Tuesday, May 10, 2016 10:56 AM
 Subject: [Htmlunit-user] Clarification Needed on HTMLUnit to process XML payload
   
Hi,

 I am using HTMLUnit to extract data from web pages. One of the sites is
 sending text/xml as the content type. Is there a way to extract data from the
  xml payload using Xpath?. If so, can you tell me the APIs that can be
    used to  extract data from the XML page?

       I appreciate your help in this regard.

    Regards
     Vasu
  

[Htmlunit-user] Clarification Needed on HTMLUnit to process XML payload

[Vasudevan C. <vco...@gm...>]

Hi,

 I am using HTMLUnit to extract data from web pages. One of the sites is
  sending text/xml as the content type. Is there a way to extract data from
the
  xml payload using Xpath?. If so, can you tell me the APIs that can be
    used to  extract data from the XML page?

       I appreciate your help in this regard.

    Regards
     Vasu

Re: [Htmlunit-user] HTMLunit fails to execute onclick mojarra script in proforma href - bug report ?

[Ahmed A. <asa...@ya...>]

Hi Helmut,
- With SVN version, I don't see NullPointerException. Please update to latest version/build [1].- I get: "Error: Label ID for 'European Union' not found.", because label is an empty string.- You shouldn't set throwsingExceptionOnScriptError to false, otherwise this can hide errors from being reported.
- Try to isolate a minimal case as hinted in [2]
Ahmed
[1] https://ci.canoo.com/teamcity/viewLog.html?buildTypeId=HtmlUnit_FastBuild&buildId=lastSuccessful&tab=artifacts[2] http://htmlunit.sourceforge.net/submittingJSBugs.html



      From: Helmut Zollner [General] <hj...@al...>
 To: HTMLunit mailinglist <htm...@li...> 
 Sent: Monday, May 2, 2016 12:13 AM
 Subject: [Htmlunit-user] HTMLunit fails to execute onclick mojarra script in proforma href - bug report ?
   
  Hi, Currently I am writing a kind of web-scraper based on HTMLunit for harvesting company details from the Hannovermesse trade fair website. I seem to have encountered a showstopper in my efforts, because I cannot get the page forward button on the search results page to work in HTMLunit.
 
 The entry website is
http:/www.hannovermesse.de/en/exhibition/exhibitors-products/advanced-search/
  After setting a few Search filters in Checkboxes (EU Region, Industrial Automation/Robotics) it and submitting the form. I am able to load the search results. With the above selection I get around 400 hits, if I select Exhibitors tab. I receive the first results page with the first page. These Search results are displayed on
http://www.hannovermesse.de/en/exhibition/exhibitors-products/search . Mojarra jacascripts are used to fill in the data opn the page. 
 NOTE: You need to run the whole sequence to get to the results screen! The host seems to identify the session and thus determines what to display and by default it displays nothing.
 
 This gives me first 20 hits on this first page. It also displays on the bottom of the page a page selector, with page 1 selected.
 "[<][1] 2 ... | n [>]"
 In order to harvest all contacts I need to click through all the screens indicated in the search results. Until the Right Arrow is no longer an href.
 
 The idea was to use the right hand button to iterate through the pages and harvest the company details on each page as I go along. I located the right button with getXPath, verified it. I even modified the DOM Node by adding a Name attribute with the same name as the class attribute so HTMLunit could find it with the more standard HTMLanchor returning functions.
 This are the details of the right button: [href           ] : '#'
 [onclick        ] :'mojarra.jsfcljs(document.getElementById('searchResult:search'),{'searchResult:j_idt489':'searchResult:j_idt489'},'');return false'
 [class          ] : 'button buttonS buttonSecondary next-page icon-arrow-right-desktop '
 [name           ] : 'button buttonS buttonSecondary next-page icon-arrow-right-desktop'
  The result was still a runtime error with a NULL pointer Exception.
 
 The log messages are:
----------------------------------------------------------------------------------
 Mai 01, 2016 6:05:11 PM com.gargoylesoftware.htmlunit.html.HtmlScript isExecutionNeeded WARNING: Script is not JavaScript (type: text/html, language: ). Skipping execution.
 Mai 01, 2016 6:05:12 PM com.gargoylesoftware.htmlunit.javascript.StrictErrorReporter runtimeError SEVERE: runtimeError: message=[An invalid or illegal selector was specified (selector: '*,:x' error: Invalid selector: :x).] sourceName=[http://www.hannovermesse.de/files/001-fs5/media/layout/js/dmag.min.js] line=[2] lineSource=[null] lineOffset=[0]
 Mai 01, 2016 6:05:12 PM com.gargoylesoftware.htmlunit.javascript.StrictErrorReporter runtimeError SEVERE: runtimeError: message=[An invalid or illegal selector was specified (selector: '[id='sizzle-1462118712173'] :selected' error: Invalid selector: [id="sizzle-1462118712173"] :selected).]sourceName=[http://www.hannovermesse.de/files/001-fs5/media/layout/js/dmag.min.js] line=[2] lineSource=[null] lineOffset=[0]
 Mai 01, 2016 6:05:12 PM com.gargoylesoftware.htmlunit.IncorrectnessListenerImpl notify WARNING: Obsolete content type encountered: 'text/javascript'.
 Mai 01, 2016 6:05:17 PM com.gargoylesoftware.htmlunit.IncorrectnessListenerImpl notify WARNING: Obsolete content type encountered: 'application/x-javascript'.
----------------------------------------------------------------------------------
 
 Have tried various browser option settings (see setup method), but no joy. I found that this 
     "invalid or illegal selector was specified (selector: '*,:x' error: Invalid selector: :x)."-error 
 has cropped up sometimes with arachnid and another headless browsers. 
  There the insertion of a "webClient().waitForBackgroundJavaScriptStartingBefore(5000);" fixed the issue. I tried that, but it hasn't worked for me.
 
 I am enclosing my quick and dirty proof-of-concept Java program for your reference. For a development environment I am using Eclipse MARS (Version: Mars Release (4.5.0) Build id: 20150621-1200) with Java JRE 1.8, JUnit4 and HTMLunit 2.22 libs.
 
 Does anyone have an idea what is going on, or what to change to make it work? I can't believe i am the first one to stumble across this!
 
 My Java code is attached to this email. 
  Thank you for your ideas or suggestions.
  Kind Regards Helmut
  
------------------------------------------------------------------------------
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
_______________________________________________
Htmlunit-user mailing list
Htm...@li...
https://lists.sourceforge.net/lists/listinfo/htmlunit-user


  

[Htmlunit-user] HTMLunit fails to execute onclick mojarra script in proforma href - bug report ?

[Helmut Z. [General] <hj...@al...>]

import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;

import org.w3c.dom.NamedNodeMap;
import org.w3c.dom.Node;

import com.gargoylesoftware.htmlunit.BrowserVersion;
import com.gargoylesoftware.htmlunit.WebClient;
import com.gargoylesoftware.htmlunit.WebClientOptions;
import com.gargoylesoftware.htmlunit.html.HtmlAnchor;
import com.gargoylesoftware.htmlunit.html.HtmlCheckBoxInput;
import com.gargoylesoftware.htmlunit.html.HtmlElement;
import com.gargoylesoftware.htmlunit.html.HtmlForm;
import com.gargoylesoftware.htmlunit.html.HtmlOption;
import com.gargoylesoftware.htmlunit.html.HtmlPage;
import com.gargoylesoftware.htmlunit.html.HtmlSelect;
import com.gargoylesoftware.htmlunit.html.HtmlSubmitInput;

public class App {
	static WebClient webClient;
	
	static String[] countries = {
				"European Union"
									};

	static String[] categories = {
				"Robotics"			
	};
	
	@SuppressWarnings("deprecation")
	public static void main(String[] args) throws Exception {

		setUp();
		
	    HtmlPage currentPage = webClient.getPage("http://www.hannovermesse.de/en/exhibition/exhibitors-products/advanced-search/");
	    System.out.println(currentPage.getTitleText()+"Web page open\n------------------------------------------------------------------\n");

	    registerCountries(currentPage);
		registerCategories(currentPage);
	    System.out.println("Search filters registered\n------------------------------------------------------------------\n");

	    currentPage = submitSearchRequest(currentPage);
	    System.out.println("Search filters submitted and results loaded\n------------------------------------------------------------------\n");

	    selectExhibitorView(currentPage);
	    System.out.println("Exhibitor View selected\n------------------------------------------------------------------\n");
	
	    showCriteria(currentPage);
		
	    showResultsCount(currentPage);
	    
	    HtmlPage backupPage = currentPage;
	    
	    for(int n=0, tn=0; n<1; n++){
    		System.out.println("========================================================================================");
    		System.out.println(" Results page "+n+1);
    		
        	HtmlAnchor nextPageButton = (HtmlAnchor) currentPage.getFirstByXPath(".//div[@class=\"col s-col12 m-col12 l-col12\"]/ul/following-sibling::a");
    		String classValue = nextPageButton.getAttribute("class");
    		nextPageButton.setAttribute("name", nextPageButton.getAttribute("class").trim());

    		NamedNodeMap attribList = nextPageButton.getAttributes();
    		for (int i=0; i < attribList.getLength(); i++) {
    			Node node = attribList.item(i);
    			String key=node.getLocalName();
    			String val=node.getNodeValue();    			
	    		System.out.printf("[%-15s] : '%s'\n", key, val);
    		}
    		
	    	List <HtmlElement> elementList = (List<HtmlElement>)currentPage.getByXPath(".//h4[@itemprop=\"name\"]/text()");	    	
	    	int i=0;
	    	for(; i<elementList.size();i++){
	    		System.out.printf("[%3d] '%s'\n", +(tn+i), elementList.get(i));
	    	}
	    	tn=i;
	    	
    		System.out.println("Next Button :");
			final HtmlAnchor newPageLink  = (HtmlAnchor) currentPage.getAnchorByName(classValue.trim());
			currentPage = (HtmlPage) newPageLink.click();
            currentPage = nextPageButton.click();
     		System.out.println("===========>[13]");

	    }
	    currentPage = backupPage;	       
	    
	    System.out.println("Done");
	    webClient.close();
	}
	
	private static void showResultsCount(HtmlPage currentPage) {
	    String results = "";
	    int count;
		results = (String) currentPage.getByXPath("String("+".//div[@class=\"col l-col8 m-col7 s-col12\"]/p[@class=\"query-text\"]/text()"+")").get(0);
	    publish("Raw results : "+results);
		count= Integer.parseInt(results.split(" ")[0]);
	    publish("Results : "+count+" found.\n");
	}

	private static void selectExhibitorView(HtmlPage currentPage) {
		HtmlSelect select = (HtmlSelect) currentPage.getElementById("searchResult:resultType");
		HtmlOption option = select.getOptionByValue("1");
		select.setSelectedAttribute(option, true);		
	}

	private static HtmlPage submitSearchRequest(HtmlPage currentPage) {
		try {	    
			final HtmlForm form  = (HtmlForm) currentPage.getFormByName("searchAP:search");
			final HtmlSubmitInput button = form.getInputByName("searchAP:searchButton2");			
			currentPage = (HtmlPage) button.click();
	    	System.out.println(currentPage.getTitleText());
		} catch (Exception e) {
	    	System.out.println("===> Cannot submit Search Form, no submit button found!");
		}
		return currentPage;	
	}

	private static void showCriteria(HtmlPage currentPage) {
	    publish("Filtercriteria for this search:");
	    String results = "";
		results = (String) currentPage.getByXPath("String(.//h1[contains(text(), \"Search Result\")]/following-sibling::p)").get(0);

		String[] criteria= results.split(",");
		String key = "";
		Map<String, ArrayList<String>> cMap = new LinkedHashMap<String, ArrayList<String>>();
		ArrayList<String> value = new ArrayList<String>();
		cMap.put(key, value);
		
	    for(int i=0; i<criteria.length; i++){
	    	if(criteria[i].contains(":")){
	    		String workCopy = new String(criteria[i]);
	    		String[] bits= workCopy.split(":");
	    		key = bits[0].trim();
	    		criteria[i]=bits[1].trim();
	    		value = new ArrayList<String>();
	    		cMap.put(key, value);
	    	}
			value.add(criteria[i].trim());
	    }  
	    
		for (Map.Entry<String, ArrayList<String>> entry : cMap.entrySet()) {
	        key = entry.getKey();
	        value = entry.getValue();
	        if(!value.isEmpty()){
		        System.out.println(key+": ");
				for (int i = 0; i < value.size(); i++) {
					System.out.println("  "+value.get(i));
				}
	        }
	    }
	}

	public static void publish(String text) {
	    System.out.println(text);		
	}
	public static void registerCountries(HtmlPage currentPage) {
		for(int i=0;i < countries.length; i++){
			setCountryCheckbox(currentPage, countries[i]);
		}
	}

	public static void registerCategories(HtmlPage currentPage) {
		for(int i=0;i < categories.length; i++){
			setCategoryCheckbox(currentPage, categories[i]);
		}		
	}
	
	public static void setCountryCheckbox(HtmlPage currentPage, String text) {
	    String label="";
	    HtmlCheckBoxInput input;
	    
	    try {
			label = (String) currentPage.getByXPath("String(.//label[contains(text(), \""+text+"\")]/@for)").get(0);
		    System.out.print(text);
		    input = currentPage.getHtmlElementById(label);
		    input.setChecked(true);
		    System.out.println(": "+(input.isChecked()?"SET":""));
		} catch (Exception e) {
		    System.out.println("\rError: Label ID for '"+text+"' not found. ");
		}
	}
	
	public static void setCategoryCheckbox(HtmlPage currentPage, String text) {
	    String label="";
	    HtmlCheckBoxInput input;
	    String XPathXpression = ".//strong[contains(text(), \""+text+"\")]/parent::div/input/@id";
	    
	    try {
			label = (String) currentPage.getByXPath("String("+XPathXpression+")").get(0);
		    System.out.print(text+" : "+"'"+label+"' ");
		    input = currentPage.getHtmlElementById(label);
		    input.setChecked(true);
		    System.out.println(": "+(input.isChecked()?"SET":""));
		} catch (Exception e) {
		    System.out.println("\rError: Label ID for '"+text+"' not found. ");
		}
	}

	public static void setUp() throws InterruptedException {
	      webClient = new WebClient(BrowserVersion.FIREFOX_45);
	      WebClientOptions options = webClient.getOptions();
	      options.setPrintContentOnFailingStatusCode(true);
	      options.setJavaScriptEnabled(true);
		  options.setThrowExceptionOnScriptError(false);
		  options.setThrowExceptionOnFailingStatusCode(false);
		  webClient.waitForBackgroundJavaScriptStartingBefore(5000);	      
	  }
}

Re: [Htmlunit-user] How do I provide a value for a text field that does not have the value attribute in Htmlunit?

[Ahmed A. <asa...@ya...>]

Hi Marvin,
You can do something like:
            HtmlPage page = webClient.getPage(url);            HtmlInput input = page.getHtmlElementById("code");            input.setValueAttribute("something");
If you want to intercept the server response, please read http://htmlunit.sourceforge.net/faq.html#HowToModifyRequestOrResponse
Ahmed
      From: Marvin Desrosiers <mar...@gm...>
 To: htm...@li... 
 Sent: Thursday, April 28, 2016 6:26 PM
 Subject: [Htmlunit-user] How do I provide a value for a text field that does not have the value attribute in Htmlunit?
   
The element below was received in a form:
<input id="code" type="text" name="code" placeholder="Enter code here" autocomplete="off"/>
I don't know how to pass a string to this input element.
Thank you,

Marvin DesrosiersM (862) 368-2327E mar...@gm...
------------------------------------------------------------------------------
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
_______________________________________________
Htmlunit-user mailing list
Htm...@li...
https://lists.sourceforge.net/lists/listinfo/htmlunit-user


  

[Htmlunit-user] How do I provide a value for a text field that does not have the value attribute in Htmlunit?

[Marvin D. <mar...@gm...>]

The element below was received in a form:

<input id="code" type="text" name="code" placeholder="Enter code here" autocomplete="off"/>

I don't know how to pass a string to this input element.

Thank you,

Marvin Desrosiers
M (862) 368-2327
E mar...@gm...

[Htmlunit-user] [ANN] HtmlUnit 2.21 Released

[Ahmed A. <asa...@ya...>]

Hi all,
It is a pleasure to announce the availability of HtmlUnit 2.21.
The major changes are:    - Introduction of Firefox 45, and removal of Firefox 31.    - INTERNET_EXPLORER has 'IE' nickname.    - JavaScript and CSS fixes
The full list of modifications are in [1].
Thanks to all the contributors.
Happy Testing,The HtmlUnit team

[1] http://htmlunit.sourceforge.net/changes-report.html#a2.21

Re: [Htmlunit-user] Working with React js DOM elements

[Ahmed A. <asa...@ya...>]

Hi Natasha,
- Please use latest version (SVN/build [1])- Provide your complete code (Java/Html/JavaScript).- If possible, try to isolate a minimal test case as hinted in HtmlUnit – Submitting JavaScript Bugs
Yours,Ahmed
[1] https://ci.canoo.com/teamcity/viewLog.html?buildTypeId=HtmlUnit_FastBuild&buildId=lastSuccessful&tab=artifacts



      From: Natasha L. <laz...@gm...>
 To: Htmlunit-user <htm...@li...> 
 Sent: Tuesday, April 5, 2016 11:24 AM
 Subject: Re: [Htmlunit-user] Working with React js DOM elements
   
Hi,
seems that I have been wrong about 2.19 supporting React events (same goes for 2.20). From what I could debug, seems that no event handlers are attached to the React elements, but it may be that I'm looking in the wrong place, since the JavaScriptEngine and Event related classes are tricky to debug. Can I get any help regarding this? 
Have you considered how the current implementation of the javascript events handles these kinds of frameworks that work with event wrappers? Is it something that is planned to be fixed/implemented?
On Wed, Mar 30, 2016 at 12:03 PM, Natasha L. <laz...@gm...> wrote:

Hi,
I've been working with HtmlUnit to try to interact with React js elements. >From what I know about React, it uses syntetic events that wrap around the native javascript events. My problem was interacting with React elements that throw these events, using HtmlUnit 2.18. Basically no events were bound to their handlers. I tried with 2.19, and it seems to work.
I use 2.18 with some local patches, and I guess a lot has changed since 2.18, but is there a (or a few) specfic merge(s) that tackle(s) this issue, so that I can see what's changed?


Thanks,


------------------------------------------------------------------------------

_______________________________________________
Htmlunit-user mailing list
Htm...@li...
https://lists.sourceforge.net/lists/listinfo/htmlunit-user


  

Re: [Htmlunit-user] Working with React js DOM elements

[Natasha L. <laz...@gm...>]

Hi,

seems that I have been wrong about 2.19 supporting React events (same goes
for 2.20). From what I could debug, seems that no event handlers are
attached to the React elements, but it may be that I'm looking in the wrong
place, since the JavaScriptEngine and Event related classes are tricky to
debug. Can I get any help regarding this?

Have you considered how the current implementation of the javascript events
handles these kinds of frameworks that work with event wrappers? Is it
something that is planned to be fixed/implemented?

On Wed, Mar 30, 2016 at 12:03 PM, Natasha L. <
laz...@gm...> wrote:

> Hi,
>
> I've been working with HtmlUnit to try to interact with React js elements.
> From what I know about React, it uses syntetic events that wrap around the
> native javascript events. My problem was interacting with React elements
> that throw these events, using HtmlUnit 2.18. Basically no events were
> bound to their handlers. I tried with 2.19, and it seems to work.
>
> I use 2.18 with some local patches, and I guess a lot has changed since
> 2.18, but is there a (or a few) specfic merge(s) that tackle(s) this issue,
> so that I can see what's changed?
>
>
> Thanks,
>

Re: [Htmlunit-user] Working with React js DOM elements

[Ahmed A. <asa...@ya...>]

Hi Natasha,
- It is much better to upgrade to latest version, actually the developers work with SVN version only.
- If you have useful patches, you are welcome to submit them.
- You can search for 'Event' in 2.19 changes [1]
Ahmed
[1] http://htmlunit.sourceforge.net/changes-report.html#a2.19


      From: Natasha L. <laz...@gm...>
 To: Htmlunit-user <htm...@li...> 
 Sent: Wednesday, March 30, 2016 12:03 PM
 Subject: [Htmlunit-user] Working with React js DOM elements
   
Hi,
I've been working with HtmlUnit to try to interact with React js elements. >From what I know about React, it uses syntetic events that wrap around the native javascript events. My problem was interacting with React elements that throw these events, using HtmlUnit 2.18. Basically no events were bound to their handlers. I tried with 2.19, and it seems to work.
I use 2.18 with some local patches, and I guess a lot has changed since 2.18, but is there a (or a few) specfic merge(s) that tackle(s) this issue, so that I can see what's changed?


Thanks,  

[Htmlunit-user] Working with React js DOM elements

[Natasha L. <laz...@gm...>]

Hi,

I've been working with HtmlUnit to try to interact with React js elements.
>From what I know about React, it uses syntetic events that wrap around the
native javascript events. My problem was interacting with React elements
that throw these events, using HtmlUnit 2.18. Basically no events were
bound to their handlers. I tried with 2.19, and it seems to work.

I use 2.18 with some local patches, and I guess a lot has changed since
2.18, but is there a (or a few) specfic merge(s) that tackle(s) this issue,
so that I can see what's changed?


Thanks,

Re: [Htmlunit-user] Working with React js DOM elements

[Ahmed A. <asa...@ya...>]

Hi,

I've been working with HtmlUnit to try to interact with React js elements.
>From what I know about React, it uses syntetic events that wrap around the
native javascript events. My problem was interacting with React elements
that throw these events, using HtmlUnit 2.18. Basically no events were
bound to their handlers. I tried with 2.19, and it seems to work.

I use 2.18 with some local patches, and I guess a lot has changed since
2.18, but is there a (or a few) specfic merge(s) that tackle(s) this issue,
so that I can see what's changed?


Thanks,

Re: [Htmlunit-user] Invalid Cookie Header

[Ahmed A. <asa...@ya...>]

Hi Marvin,
You can set it to 'off' in the log4j.properties, an example in https://sourceforge.net/p/htmlunit/code/HEAD/tree/trunk/htmlunit/src/test/resources/log4j.properties
Ahmed

      From: Marvin Desrosiers <mar...@gm...>
 To: htm...@li... 
 Sent: Tuesday, March 22, 2016 3:16 PM
 Subject: [Htmlunit-user] Invalid Cookie Header
   
Hi,How do I turn off the logger providing the Invalid Cookie Header?

Mar 22, 2016 9:59:56 AM org.apache.http.client.protocol.ResponseProcessCookies processCookies 
WARNING: Invalid cookie header: "Set-Cookie: box_redirect_url=value; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=-1458655231; path=/; domain=.app.box.com". Negative max-age attribute: -1458655231 
Mar 22, 2016 9:59:56 AM org.apache.http.client.protocol.ResponseProcessCookies processCookies 
WARNING: Invalid cookie header: "Set-Cookie: box_redirect_rm=value; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=-1458655231; path=/; domain=.app.box.com". Negative max-age attribute: -1458655231 
Mar 22, 2016 9:59:56 AM org.apache.http.client.protocol.ResponseProcessCookies processCookies 
WARNING: Invalid cookie header: "Set-Cookie: box_referrer_url=value; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=-1458655231; path=/; domain=.app.box.com". Negative max-age attribute: -1458655231Thank you,
Marvin Desrosiers
  

Re: [Htmlunit-user] JBoss EAP 6 error

[Ronald B. <rb...@rb...>]

On Tue, 29 Mar 2016 08:38:14 -0300 Adrian wrote:
>
>com.gargoylesoftware.htmlunit.FailingHttpStatusCodeException: 404 Not Found
>for http://webhml:80/etc/httpd/conf/adrum.js
>

Is this url/page available when calling from real browser?

	RBRi

BTW: If yes it seems you apache configuration has a real security issue because the configuration is readable from the outside...

[Htmlunit-user] JBoss EAP 6 error

[Adrian <adr...@gm...>]

Hi all.

I´m using htmlunit in a corporate server (no admin rights) in order to run
some js code embed in html (need "run" a html/js with Arcgis query and
catch the result - just java allowed there), but it´s giving the error
below.

Any help?

Strack trace:

com.gargoylesoftware.htmlunit.FailingHttpStatusCodeException: 404 Not Found
for http://webhml:80/etc/httpd/conf/adrum.js
at
com.gargoylesoftware.htmlunit.WebClient.throwFailingHttpStatusCodeExceptionIfNecessary(WebClient.java:535)
at
com.gargoylesoftware.htmlunit.html.HtmlPage.loadJavaScriptFromUrl(HtmlPage.java:1076)
at
com.gargoylesoftware.htmlunit.html.HtmlPage.loadExternalJavaScriptFile(HtmlPage.java:1009)
at
com.gargoylesoftware.htmlunit.html.HtmlScript.executeScriptIfNeeded(HtmlScript.java:395)
at
com.gargoylesoftware.htmlunit.html.HtmlScript$3.execute(HtmlScript.java:276)
at
com.gargoylesoftware.htmlunit.html.HtmlScript.onAllChildrenAddedToPage(HtmlScript.java:290)
at
com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.endElement(HTMLParser.java:800)
at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source)
at
com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.endElement(HTMLParser.java:757)
at
org.cyberneko.html.HTMLTagBalancer.callEndElement(HTMLTagBalancer.java:1170)
at org.cyberneko.html.HTMLTagBalancer.endElement(HTMLTagBalancer.java:1072)
at
org.cyberneko.html.filters.DefaultFilter.endElement(DefaultFilter.java:206)
at
org.cyberneko.html.filters.NamespaceBinder.endElement(NamespaceBinder.java:330)
at
org.cyberneko.html.HTMLScanner$ContentScanner.scanEndElement(HTMLScanner.java:3126)
at org.cyberneko.html.HTMLScanner$ContentScanner.scan(HTMLScanner.java:2093)
at org.cyberneko.html.HTMLScanner.scanDocument(HTMLScanner.java:920)
at org.cyberneko.html.HTMLConfiguration.parse(HTMLConfiguration.java:499)
at org.cyberneko.html.HTMLConfiguration.parse(HTMLConfiguration.java:452)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at
com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.parse(HTMLParser.java:1040)
at com.gargoylesoftware.htmlunit.html.HTMLParser.parse(HTMLParser.java:253)
at
com.gargoylesoftware.htmlunit.html.HTMLParser.parseHtml(HTMLParser.java:199)
at
com.gargoylesoftware.htmlunit.DefaultPageCreator.createHtmlPage(DefaultPageCreator.java:272)
at
com.gargoylesoftware.htmlunit.DefaultPageCreator.createPage(DefaultPageCreator.java:160)
at
com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:476)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:350)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:415)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:400)
at
org.apache.jsp.getCelulaLatLng_jsp._jspService(getCelulaLatLng_jsp.java:175)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:69)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
at
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:365)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:309)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:242)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149)
at
org.jboss.modcluster.container.jbossweb.JBossWebContext$RequestListenerValve.event(JBossWebContext.java:91)
at
org.jboss.modcluster.container.jbossweb.JBossWebContext$RequestListenerValve.invoke(JBossWebContext.java:72)
at
org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:490)
at
org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926)
at java.lang.Thread.run(Thread.java:745)

Que Deus te abençoe / God bless you,

Adrian Roberto Barbosa
Fone: +55 44 9119-6989
VoIP: sip:adr...@ip...
Skype: adrian.barbosa71

Cremos que todas as coisas cooperam
para o bem daqueles que amam a Deus,
daqueles que são chamados segundo o seu propósito"
(Romanos 8.28)

And we know that in all things God works for the good of those who love
him,
who have been called according to his purpose.
(Romans 8.28)

[Htmlunit-user] Invalid Cookie Header

[Marvin D. <mar...@gm...>]

Hi,
How do I turn off the logger providing the Invalid Cookie Header?

Mar 22, 2016 9:59:56 AM org.apache.http.client.protocol.ResponseProcessCookies processCookies 
WARNING: Invalid cookie header: "Set-Cookie: box_redirect_url=value; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=-1458655231; path=/; domain=.app.box.com". Negative max-age attribute: -1458655231 
Mar 22, 2016 9:59:56 AM org.apache.http.client.protocol.ResponseProcessCookies processCookies 
WARNING: Invalid cookie header: "Set-Cookie: box_redirect_rm=value; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=-1458655231; path=/; domain=.app.box.com". Negative max-age attribute: -1458655231 
Mar 22, 2016 9:59:56 AM org.apache.http.client.protocol.ResponseProcessCookies processCookies 
WARNING: Invalid cookie header: "Set-Cookie: box_referrer_url=value; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=-1458655231; path=/; domain=.app.box.com". Negative max-age attribute: -1458655231
Thank you,
Marvin Desrosiers

Re: [Htmlunit-user] Sending Cookies in HTTP header

[Marvin D. <mar...@gm...>]

Ahmed,
My goal is to use htmlunit to login into my account and complete the Oauth2
process and receive the authorization code. This process consists of a GET
(receiving login page), a POST(submitting the form from the login page) and
another POST (submitting the form granting access to my account) and
returning the Authorization code in the response. When I complete the
process on Chrome the second POST returns a response (Status Code: 302
Found )with a location header that contains the Authorization Code.

I'm having trouble submitting the second post as I receive:

java.lang.RuntimeException: org.apache.http.conn.HttpHostConnectException:
Connect to 127.0.0.1:80 [/127.0.0.1] failed: Connection refused: connect
at com.gargoylesoftware.htmlunit.WebClient.download(WebClient.java:2018)
at com.gargoylesoftware.htmlunit.html.HtmlForm.submit(HtmlForm.java:140)
at
com.gargoylesoftware.htmlunit.html.HtmlButton.doClickStateUpdate(HtmlButton.java:87)
at
com.gargoylesoftware.htmlunit.html.HtmlElement.click(HtmlElement.java:1321)
at
com.gargoylesoftware.htmlunit.html.HtmlElement.click(HtmlElement.java:1268)
at
com.gargoylesoftware.htmlunit.html.HtmlElement.click(HtmlElement.java:1216)
at com.ricoh.box.App.main(App.java:244)
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to
127.0.0.1:80 [/127.0.0.1] failed: Connection refused: connect
at
org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:140)
at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:314)
at
org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:363)
at
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:219)
at
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)
at
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
at
com.gargoylesoftware.htmlunit.HttpWebConnection.getResponse(HttpWebConnection.java:178)
at
com.gargoylesoftware.htmlunit.WebClient.loadWebResponseFromWebConnection(WebClient.java:1313)
at
com.gargoylesoftware.htmlunit.WebClient.loadWebResponseFromWebConnection(WebClient.java:1371)
at
com.gargoylesoftware.htmlunit.WebClient.loadWebResponse(WebClient.java:1230)
at com.gargoylesoftware.htmlunit.WebClient.download(WebClient.java:2014)
... 6 more
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at
org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:72)
at
org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:117)
... 19 more



Hi Marvin,
You usually put log4j.properties in your project path, and configure it
accordingly.
Anyhow, your case fails because of 404, I am not sure if you get a
different result.
Also, by enabling HtmlUnit to throw JavaScript errors, there is error
because NativeError.stack is not yet implements [1], and this is a known
issue.
Yours,Ahmed
[1]
https://sourceforge.net/p/htmlunit/code/HEAD/tree/trunk/htmlunit/src/test/java/com/gargoylesoftware/htmlunit/javascript/NativeErrorTest.java#l34


      From: Marvin Desrosiers <marvin.desrosiers@...>
 To: htmlunit-user@...
 Sent: Friday, March 18, 2016 9:44 PM
 Subject: [Htmlunit-user] Fwd: Emailing: HTML-Email.html

Ahmed,I tried enabling the logger by adding:Logger.
getLogger("org.apache.http.headers").setLevel(Level.ALL);
But I didn't see any change. I've provided the credentials below.Thank
you,-Marvin
USER_EMAIL = ricohusa23@... = htmlunit AUTHORIZE_LOCATION =
https://app.box.com/api/oauth2/authorize?response_type=code&client_id=tr0z0beizpf2zutquoy0heainecgq7s0&redirect_uri=http://127.0.0.1&state=unused
Begin forwarded message:


To: marvin.desrosiers@...
Subject: Emailing: HTML-Email.html



Hi Marvin,

You can enable the logger of "org.apache.http.headers"in log4j.properties
[1] so see what HtmlUnit is sending.

Otherwise you need to provide more details,including the URL and
credentials, for others to look reproduce the issue.

Ahmed

[1]
https://sourceforge.net/p/htmlunit/code/HEAD/tree/trunk/htmlunit/src/test/resources/log4j.properties



From: Marvin Desrosiers <marvin.desrosiers@...>
To: htmlunit-user@...
Sent: Friday, March 18, 2016 6:43 PM
Subject: [Htmlunit-user] Sending Cookies in HTTP header

I'm trying to understand why the cookies are not being sent back to the
server. htmlunit is set with setUseInsecureSSLby default to false for the
secure tag.

try {
                WebClient webClient= newWebClient(BrowserVersion.CHROME);

                /*webClient.getOptions().setUseInsecureSSL(false);*/

/*webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);*/
                webClient.getOptions().setCssEnabled(false);
                webClient.getOptions().setJavaScriptEnabled(false);

                CookieManager cookieManager=webClient.getCookieManager();

                out.println(cookieManager.getCookies().toString());

            out.println("start");

            final HtmlPage loginPage= webClient.getPage(AUTHORIZE_URL);
            String requestToken =
loginPage.getElementByName("request_token").getAttribute("value");
            out.println("requesttoken :" + requestToken);


            out.println("afterloginPage");



            out.println(cookieManager.getCookies().toString());
            Set<Cookie> boxCookies= newHashSet<Cookie>();
            boxCookies.addAll(webClient.getCookieManager().getCookies());
            StringBuilder cookieHeader =newStringBuilder();

            /*Iterator<Cookie> ite= boxCookies.iterator();
            while (ite.hasNext()){
                    Cookiecookie = ite.next();
                    cookie.getDomain().substring(1);
                    Stringname = cookie.getName();
                    Stringvalue = cookie.getValue();

                    System.out.println("Cookie:"+ name + "=" +value);
                    if()){
                           cookieHeader.append(name + "=" +value);
                    }
                    cookieHeader.append(";"+ name + "=" +value);
            }
            */
            webClient.addRequestHeader("Accept-Encoding","gzip, deflate,
sdch");
            webClient.addRequestHeader("Accept-Language","en-US,en;q=0.8");
            webClient.addRequestHeader("Cache-Control","no-cache");
            webClient.addRequestHeader("Upgrade-Insecure-Requests","1");
            webClient.addRequestHeader("DNT","1");
            webClient.addRequestHeader("Host","app.company.com");

            for (int index= 0; index < boxCookies.size(); index++){
                    Stringcookie = boxCookies.toArray()[index].toString();
                    StringcookieNameValue=cookie.substring(0,
cookie.indexOf(";"));
                String name =
cookieNameValue.substring(0,cookieNameValue.indexOf("="));
                String value
=cookieNameValue.substring(cookieNameValue.indexOf("=") + 1);

                    if(index == 0){
                          cookieHeader.append(name + "=" +value);
                    }else {
                          cookieHeader.append("; "+ name + "="+value);
                    }

            }
            WebRequest secondLoginPage =newWebRequest(AUTHORIZE_URL);

secondLoginPage.setAdditionalHeader("Cookie",cookieHeader.toString());
            HtmlPage loginPage2 = webClient.getPage(secondLoginPage);


out.println(loginPage2.getWebResponse().getWebRequest().getRequestParameters().toString());
            out.println("\n");
            Map<?, ?> additionalRequest1=
loginPage2.getWebResponse().getWebRequest().getAdditionalHeaders();

            Iterator<?> ite0 = additionalRequest1.entrySet().iterator();
             while(ite0.hasNext()){
                    out.println(ite0.next());
             }
             out.println("\n");


            final HtmlTextInput login= (HtmlTextInput)
loginPage2.getElementById("login");
            login.setValueAttribute(USER_EMAIL);

            final HtmlPasswordInputpassword= (HtmlPasswordInput)
loginPage2.getElementById("password");
            password.setValueAttribute(USER_PASS);

            final HtmlSubmitInputbutton_submit =
loginPage2.getElementByName("login_submit");

            final HtmlPage accessGrantingPage= button_submit.click();

            final HtmlForm  requestForm=
(HtmlForm)accessGrantingPage.getElementById("consent_form");

            Map<?, ?> additionalRequest=
accessGrantingPage.getWebResponse().getWebRequest().getAdditionalHeaders();
            out.println("\n");
            Iterator<?> ite2 = additionalRequest.entrySet().iterator();
             while(ite2.hasNext()){
                    out.println(ite2.next());
             }
            out.println("\n");

            out.println("afteraccessGrantingPage");

            out.println(cookieManager.getCookies().toString());
            final HtmlButton consent_accept_button=
accessGrantingPage.getElementByName("consent_accept");

            try {
                    finalHtmlPage authorizationPage =
consent_accept_button.click();
                    out.println("afterauthorizationPage");
                    out.println(authorizationPage.getUrl().toString());

out.println(authorizationPage.getWebResponse().getStatusMessage());


out.println(authorizationPage.getWebResponse().getResponseHeaders());

            } catch (RuntimeExceptionre){
                    re.printStackTrace();
            }
            webClient.closeAllWindows();
        } catch (IOException ioe){
                ioe.printStackTrace();
        } finally {

        }
Your message is ready to be sent with the following file or link
attachments:
HTML-Email.html

Note: To protect against computer viruses, e-mail programs may
preventsending or receiving certain types of file attachments.  Check
youre-mail security settings to determine how attachments are handled.


------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.

_______________________________________________
Htmlunit-user mailing list
Htmlunit-user@...
https://lists.sourceforge.net/lists/listinfo/htmlunit-user

Re: [Htmlunit-user] Fwd: Emailing: HTML-Email.html

[Ahmed A. <asa...@ya...>]

Hi Marvin,
You usually put log4j.properties in your project path, and configure it accordingly.
Anyhow, your case fails because of 404, I am not sure if you get a different result.
Also, by enabling HtmlUnit to throw JavaScript errors, there is error because NativeError.stack is not yet implements [1], and this is a known issue.
Yours,Ahmed
[1] https://sourceforge.net/p/htmlunit/code/HEAD/tree/trunk/htmlunit/src/test/java/com/gargoylesoftware/htmlunit/javascript/NativeErrorTest.java#l34


      From: Marvin Desrosiers <mar...@gm...>
 To: htm...@li... 
 Sent: Friday, March 18, 2016 9:44 PM
 Subject: [Htmlunit-user] Fwd: Emailing: HTML-Email.html
   
Ahmed,I tried enabling the logger by adding:Logger. getLogger("org.apache.http.headers").setLevel(Level.ALL); 
But I didn't see any change. I've provided the credentials below.Thank you,-Marvin
USER_EMAIL = ric...@gm...USER_PASS = htmlunit AUTHORIZE_LOCATION = https://app.box.com/api/oauth2/authorize?response_type=code&client_id=tr0z0beizpf2zutquoy0heainecgq7s0&redirect_uri=http://127.0.0.1&state=unused
Begin forwarded message:


To: mar...@gm...
Subject: Emailing: HTML-Email.html



Hi Marvin,

You can enable the logger of "org.apache.http.headers"in log4j.properties [1] so see what HtmlUnit is sending.

Otherwise you need to provide more details,including the URL and credentials, for others to look reproduce the issue.

Ahmed

[1] https://sourceforge.net/p/htmlunit/code/HEAD/tree/trunk/htmlunit/src/test/resources/log4j.properties



From: Marvin Desrosiers <mar...@gm...>
To: htm...@li... 
Sent: Friday, March 18, 2016 6:43 PM
Subject: [Htmlunit-user] Sending Cookies in HTTP header

I'm trying to understand why the cookie arenot being sent back to the server. htmlunit is set with setUseInsecureSSLby default to false for the secure tag. 

try { 
                WebClient webClient= newWebClient(BrowserVersion.CHROME); 
                
                /*webClient.getOptions().setUseInsecureSSL(false);*/
                /*webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);*/
                webClient.getOptions().setCssEnabled(false);
                webClient.getOptions().setJavaScriptEnabled(false);
                
                CookieManager cookieManager=webClient.getCookieManager(); 
            
                out.println(cookieManager.getCookies().toString());
                
            out.println("start");
            
            final HtmlPage loginPage= webClient.getPage(AUTHORIZE_URL); 
            String requestToken = loginPage.getElementByName("request_token").getAttribute("value");
            out.println("requesttoken :" + requestToken); 
            
            
            out.println("afterloginPage"); 
            
            
            
            out.println(cookieManager.getCookies().toString());
            Set<Cookie> boxCookies= newHashSet<Cookie>(); 
            boxCookies.addAll(webClient.getCookieManager().getCookies());
            StringBuilder cookieHeader =newStringBuilder(); 
            
            /*Iterator<Cookie> ite= boxCookies.iterator(); 
            while (ite.hasNext()){ 
                    Cookiecookie = ite.next(); 
                    cookie.getDomain().substring(1);
                    Stringname = cookie.getName(); 
                    Stringvalue = cookie.getValue(); 
                    
                    System.out.println("Cookie:"+ name + "=" +value); 
                    if()){ 
                           cookieHeader.append(name + "=" +value);
                    }
                    cookieHeader.append(";"+ name + "=" +value); 
            } 
            */ 
            webClient.addRequestHeader("Accept-Encoding","gzip, deflate, sdch"); 
            webClient.addRequestHeader("Accept-Language","en-US,en;q=0.8"); 
            webClient.addRequestHeader("Cache-Control","no-cache");
            webClient.addRequestHeader("Upgrade-Insecure-Requests","1"); 
            webClient.addRequestHeader("DNT","1"); 
            webClient.addRequestHeader("Host","app.company.com");
            
            for (int index= 0; index < boxCookies.size(); index++){ 
                    Stringcookie = boxCookies.toArray()[index].toString(); 
                    StringcookieNameValue=cookie.substring(0, cookie.indexOf(";")); 
                String name = cookieNameValue.substring(0,cookieNameValue.indexOf("="));
                String value =cookieNameValue.substring(cookieNameValue.indexOf("=") + 1);
                
                    if(index == 0){ 
                          cookieHeader.append(name + "=" +value);
                    }else { 
                          cookieHeader.append("; "+ name + "="+value);
                    }
                    
            } 
            WebRequest secondLoginPage =newWebRequest(AUTHORIZE_URL); 
            secondLoginPage.setAdditionalHeader("Cookie",cookieHeader.toString()); 
            HtmlPage loginPage2 = webClient.getPage(secondLoginPage);
            
            out.println(loginPage2.getWebResponse().getWebRequest().getRequestParameters().toString());
            out.println("\n");
            Map<?, ?> additionalRequest1= loginPage2.getWebResponse().getWebRequest().getAdditionalHeaders(); 
            
            Iterator<?> ite0 = additionalRequest1.entrySet().iterator();
             while(ite0.hasNext()){
                    out.println(ite0.next()); 
             } 
             out.println("\n");
            
            
            final HtmlTextInput login= (HtmlTextInput) loginPage2.getElementById("login"); 
            login.setValueAttribute(USER_EMAIL);
            
            final HtmlPasswordInputpassword= (HtmlPasswordInput) loginPage2.getElementById("password");
            password.setValueAttribute(USER_PASS);
            
            final HtmlSubmitInputbutton_submit = loginPage2.getElementByName("login_submit");
            
            final HtmlPage accessGrantingPage= button_submit.click(); 
            
            final HtmlForm  requestForm= (HtmlForm)accessGrantingPage.getElementById("consent_form");
            
            Map<?, ?> additionalRequest= accessGrantingPage.getWebResponse().getWebRequest().getAdditionalHeaders();
            out.println("\n");
            Iterator<?> ite2 = additionalRequest.entrySet().iterator();
             while(ite2.hasNext()){
                    out.println(ite2.next()); 
             } 
            out.println("\n");
            
            out.println("afteraccessGrantingPage"); 
            
            out.println(cookieManager.getCookies().toString());
            final HtmlButton consent_accept_button= accessGrantingPage.getElementByName("consent_accept"); 
            
            try { 
                    finalHtmlPage authorizationPage = consent_accept_button.click(); 
                    out.println("afterauthorizationPage"); 
                    out.println(authorizationPage.getUrl().toString());
                    out.println(authorizationPage.getWebResponse().getStatusMessage());
        
                    out.println(authorizationPage.getWebResponse().getResponseHeaders());
                    
            } catch (RuntimeExceptionre){ 
                    re.printStackTrace();
            } 
            webClient.closeAllWindows();
        } catch (IOException ioe){ 
                ioe.printStackTrace();
        } finally { 
                
        }
Your message is ready to be sent with the following file or link attachments:
HTML-Email.html

Note: To protect against computer viruses, e-mail programs may preventsending or receiving certain types of file attachments.  Check youre-mail security settings to determine how attachments are handled.


------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140
_______________________________________________
Htmlunit-user mailing list
Htm...@li...
https://lists.sourceforge.net/lists/listinfo/htmlunit-user


  

[Htmlunit-user] Fwd: Emailing: HTML-Email.html

[Marvin D. <mar...@gm...>]

Ahmed,
I tried enabling the logger by adding:
Logger. getLogger("org.apache.http.headers").setLevel(Level.ALL); 

But I didn't see any change. I've provided the credentials below.
Thank you,
-Marvin

USER_EMAIL = ric...@gm...
USER_PASS = htmlunit 
AUTHORIZE_LOCATION = https://app.box.com/api/oauth2/authorize?response_type=code&client_id=tr0z0beizpf2zutquoy0heainecgq7s0&redirect_uri=http://127.0.0.1&state=unused

Begin forwarded message:

> To: mar...@gm...
> Subject: Emailing: HTML-Email.html
> 
> Hi Marvin, 
> 
> You can enable the logger of "org.apache.http.headers" in log4j.properties [1] so see what HtmlUnit is sending. 
> 
> Otherwise you need to provide more details, including the URL and credentials, for others to look reproduce the issue. 
> 
> Ahmed 
> 
> [1] https://sourceforge.net/p/htmlunit/code/HEAD/tree/trunk/htmlunit/src/test/resources/log4j.properties 
> 
> 
> 
> From: Marvin Desrosiers <mar...@gm...>
> To: htm...@li... 
> Sent: Friday, March 18, 2016 6:43 PM
> Subject: [Htmlunit-user] Sending Cookies in HTTP header 
> 
> I'm trying to understand why the cookie are not being sent back to the server. htmlunit is set with setUseInsecureSSL by default to false for the secure tag. 
> 
> try { 
>                WebClient webClient = newWebClient(BrowserVersion.CHROME); 
>                
>                /*webClient.getOptions().setUseInsecureSSL(false);*/ 
>                /*webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);*/ 
>                webClient.getOptions().setCssEnabled(false); 
>                webClient.getOptions().setJavaScriptEnabled(false); 
>                
>                CookieManager cookieManager= webClient.getCookieManager(); 
>            
>                out.println(cookieManager.getCookies().toString()); 
>                
>            out.println("start"); 
>            
>            final HtmlPage loginPage = webClient.getPage(AUTHORIZE_URL); 
>            String requestToken = loginPage.getElementByName("request_token").getAttribute("value"); 
>            out.println("request token :" + requestToken); 
>            
>            
>            out.println("after loginPage"); 
>            
>            
>            
>            out.println(cookieManager.getCookies().toString()); 
>            Set<Cookie> boxCookies = newHashSet<Cookie>(); 
>            boxCookies.addAll(webClient.getCookieManager().getCookies()); 
>            StringBuilder cookieHeader = newStringBuilder(); 
>            
>            /*Iterator<Cookie> ite = boxCookies.iterator(); 
>            while (ite.hasNext()){ 
>                    Cookie cookie = ite.next(); 
>                    cookie.getDomain().substring(1); 
>                    String name = cookie.getName(); 
>                    String value = cookie.getValue(); 
>                    
>                    System.out.println("Cookie:" + name + "=" +value); 
>                    if ()){ 
>                            cookieHeader.append(name + "=" +value); 
>                    } 
>                    cookieHeader.append("; "+ name + "=" +value); 
>            } 
>            */ 
>            webClient.addRequestHeader("Accept-Encoding", "gzip, deflate, sdch"); 
>            webClient.addRequestHeader("Accept-Language", "en-US,en;q=0.8"); 
>            webClient.addRequestHeader("Cache-Control","no-cache"); 
>            webClient.addRequestHeader("Upgrade-Insecure-Requests", "1"); 
>            webClient.addRequestHeader("DNT", "1"); 
>            webClient.addRequestHeader("Host","app.company.com"); 
>            
>            for (int index = 0; index < boxCookies.size(); index++){ 
>                    String cookie = boxCookies.toArray()[index].toString(); 
>                    String cookieNameValue=cookie.substring(0, cookie.indexOf(";")); 
>                String name = cookieNameValue.substring(0,cookieNameValue.indexOf("=")); 
>                String value = cookieNameValue.substring(cookieNameValue.indexOf("=") + 1); 
>                
>                    if (index == 0){ 
>                           cookieHeader.append(name + "=" +value); 
>                    } else { 
>                           cookieHeader.append("; "+ name + "="+value); 
>                    } 
>                    
>            } 
>            WebRequest secondLoginPage = newWebRequest(AUTHORIZE_URL); 
>            secondLoginPage.setAdditionalHeader("Cookie", cookieHeader.toString()); 
>            HtmlPage loginPage2 = webClient.getPage(secondLoginPage); 
>            
>            out.println(loginPage2.getWebResponse().getWebRequest().getRequestParameters().toString()); 
>            out.println("\n"); 
>            Map<?, ?> additionalRequest1 = loginPage2.getWebResponse().getWebRequest().getAdditionalHeaders(); 
>            
>            Iterator<?> ite0 = additionalRequest1.entrySet().iterator(); 
>             while(ite0.hasNext()){ 
>                     out.println(ite0.next()); 
>             } 
>             out.println("\n"); 
>            
>            
>            final HtmlTextInput login = (HtmlTextInput) loginPage2.getElementById("login"); 
>            login.setValueAttribute(USER_EMAIL); 
>            
>            final HtmlPasswordInput password= (HtmlPasswordInput) loginPage2.getElementById("password"); 
>            password.setValueAttribute(USER_PASS); 
>            
>            final HtmlSubmitInput button_submit = loginPage2.getElementByName("login_submit"); 
>            
>            final HtmlPage accessGrantingPage = button_submit.click(); 
>            
>            final HtmlForm  requestForm = (HtmlForm)accessGrantingPage.getElementById("consent_form"); 
>            
>            Map<?, ?> additionalRequest = accessGrantingPage.getWebResponse().getWebRequest().getAdditionalHeaders(); 
>            out.println("\n"); 
>            Iterator<?> ite2 = additionalRequest.entrySet().iterator(); 
>             while(ite2.hasNext()){ 
>                     out.println(ite2.next()); 
>             } 
>            out.println("\n"); 
>            
>            out.println("after accessGrantingPage"); 
>            
>            out.println(cookieManager.getCookies().toString()); 
>            final HtmlButton consent_accept_button = accessGrantingPage.getElementByName("consent_accept"); 
>            
>            try { 
>                    final HtmlPage authorizationPage = consent_accept_button.click(); 
>                    out.println("after authorizationPage"); 
>                    out.println(authorizationPage.getUrl().toString()); 
>                    out.println(authorizationPage.getWebResponse().getStatusMessage()); 
>        
>                    out.println(authorizationPage.getWebResponse().getResponseHeaders()); 
>                    
>            } catch (RuntimeException re){ 
>                    re.printStackTrace(); 
>            } 
>            webClient.closeAllWindows(); 
>        } catch (IOException ioe){ 
>                ioe.printStackTrace(); 
>        } finally { 
>                
>        }
> Your message is ready to be sent with the following file or link attachments: 
> HTML-Email.html
> 
> Note: To protect against computer viruses, e-mail programs may prevent sending or receiving certain types of file attachments.  Check your e-mail security settings to determine how attachments are handled.

Re: [Htmlunit-user] Sending Cookies in HTTP header

[Ahmed A. <asa...@ya...>]

Hi Marvin,
You can enable the logger of "org.apache.http.headers" in log4j.properties [1] so see what HtmlUnit is sending.
Otherwise you need to provide more details, including the URL and credentials, for others to look reproduce the issue.

Ahmed
[1] https://sourceforge.net/p/htmlunit/code/HEAD/tree/trunk/htmlunit/src/test/resources/log4j.properties


      From: Marvin Desrosiers <mar...@gm...>
 To: htm...@li... 
 Sent: Friday, March 18, 2016 6:43 PM
 Subject: [Htmlunit-user] Sending Cookies in HTTP header
   
I'm trying to understand why the cookie are not being sent back to the server. htmlunit is set with setUseInsecureSSL by default to false for the secure tag. 
try { 
                WebClient webClient = newWebClient(BrowserVersion.CHROME); 
                
                /*webClient.getOptions().setUseInsecureSSL(false);*/ 
                /*webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);*/ 
                webClient.getOptions().setCssEnabled(false); 
                webClient.getOptions().setJavaScriptEnabled(false); 
                
                CookieManager cookieManager= webClient.getCookieManager(); 
            
                out.println(cookieManager.getCookies().toString()); 
                
            out.println("start"); 
            
            final HtmlPage loginPage = webClient.getPage(AUTHORIZE_URL); 
            String requestToken = loginPage.getElementByName("request_token").getAttribute("value"); 
            out.println("request token :" + requestToken); 
            
            
            out.println("after loginPage"); 
            
            
            
            out.println(cookieManager.getCookies().toString()); 
            Set<Cookie> boxCookies = newHashSet<Cookie>(); 
            boxCookies.addAll(webClient.getCookieManager().getCookies()); 
            StringBuilder cookieHeader = newStringBuilder(); 
            
            /*Iterator<Cookie> ite = boxCookies.iterator(); 
            while (ite.hasNext()){ 
                    Cookie cookie = ite.next(); 
                    cookie.getDomain().substring(1); 
                    String name = cookie.getName(); 
                    String value = cookie.getValue(); 
                    
                    System.out.println("Cookie:" + name + "=" +value); 
                    if ()){ 
                            cookieHeader.append(name + "=" +value); 
                    } 
                    cookieHeader.append("; "+ name + "=" +value); 
            } 
            */ 
            webClient.addRequestHeader("Accept-Encoding", "gzip, deflate, sdch"); 
            webClient.addRequestHeader("Accept-Language", "en-US,en;q=0.8"); 
            webClient.addRequestHeader("Cache-Control","no-cache"); 
            webClient.addRequestHeader("Upgrade-Insecure-Requests", "1"); 
            webClient.addRequestHeader("DNT", "1"); 
            webClient.addRequestHeader("Host","app.company.com"); 
            
            for (int index = 0; index < boxCookies.size(); index++){ 
                    String cookie = boxCookies.toArray()[index].toString(); 
                    String cookieNameValue=cookie.substring(0, cookie.indexOf(";")); 
                String name = cookieNameValue.substring(0,cookieNameValue.indexOf("=")); 
                String value = cookieNameValue.substring(cookieNameValue.indexOf("=") + 1); 
                
                    if (index == 0){ 
                           cookieHeader.append(name + "=" +value); 
                    } else { 
                           cookieHeader.append("; "+ name + "="+value); 
                    } 
                    
            } 
            WebRequest secondLoginPage = newWebRequest(AUTHORIZE_URL); 
            secondLoginPage.setAdditionalHeader("Cookie", cookieHeader.toString()); 
            HtmlPage loginPage2 = webClient.getPage(secondLoginPage); 
            
            out.println(loginPage2.getWebResponse().getWebRequest().getRequestParameters().toString()); 
            out.println("\n"); 
            Map<?, ?> additionalRequest1 = loginPage2.getWebResponse().getWebRequest().getAdditionalHeaders(); 
            
            Iterator<?> ite0 = additionalRequest1.entrySet().iterator(); 
             while(ite0.hasNext()){ 
                     out.println(ite0.next()); 
             } 
             out.println("\n"); 
            
            
            final HtmlTextInput login = (HtmlTextInput) loginPage2.getElementById("login"); 
            login.setValueAttribute(USER_EMAIL); 
            
            final HtmlPasswordInput password= (HtmlPasswordInput) loginPage2.getElementById("password"); 
            password.setValueAttribute(USER_PASS); 
            
            final HtmlSubmitInput button_submit = loginPage2.getElementByName("login_submit"); 
            
            final HtmlPage accessGrantingPage = button_submit.click(); 
            
            final HtmlForm  requestForm = (HtmlForm)accessGrantingPage.getElementById("consent_form"); 
            
            Map<?, ?> additionalRequest = accessGrantingPage.getWebResponse().getWebRequest().getAdditionalHeaders(); 
            out.println("\n"); 
            Iterator<?> ite2 = additionalRequest.entrySet().iterator(); 
             while(ite2.hasNext()){ 
                     out.println(ite2.next()); 
             } 
            out.println("\n"); 
            
            out.println("after accessGrantingPage"); 
            
            out.println(cookieManager.getCookies().toString()); 
            final HtmlButton consent_accept_button = accessGrantingPage.getElementByName("consent_accept"); 
            
            try { 
                    final HtmlPage authorizationPage = consent_accept_button.click(); 
                    out.println("after authorizationPage"); 
                    out.println(authorizationPage.getUrl().toString()); 
                    out.println(authorizationPage.getWebResponse().getStatusMessage()); 
        
                    out.println(authorizationPage.getWebResponse().getResponseHeaders()); 
                    
            } catch (RuntimeException re){ 
                    re.printStackTrace(); 
            } 
            webClient.closeAllWindows(); 
        } catch (IOException ioe){ 
                ioe.printStackTrace(); 
        } finally { 
                
        }  

[Htmlunit-user] Sending Cookies in HTTP header

[Marvin D. <mar...@gm...>]

I'm trying to understand why the cookie are not being sent back to the server. htmlunit is set with setUseInsecureSSL by default to false for the secure tag. 



try { 
                WebClient webClient = newWebClient(BrowserVersion.CHROME); 
                
                /*webClient.getOptions().setUseInsecureSSL(false);*/ 
                /*webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);*/ 
                webClient.getOptions().setCssEnabled(false); 
                webClient.getOptions().setJavaScriptEnabled(false); 
                
                CookieManager cookieManager= webClient.getCookieManager(); 
            
                out.println(cookieManager.getCookies().toString()); 
                
            out.println("start"); 
            
            final HtmlPage loginPage = webClient.getPage(AUTHORIZE_URL); 
            String requestToken = loginPage.getElementByName("request_token").getAttribute("value"); 
            out.println("request token :" + requestToken); 
            
            
            out.println("after loginPage"); 
            
            
            
            out.println(cookieManager.getCookies().toString()); 
            Set<Cookie> boxCookies = newHashSet<Cookie>(); 
            boxCookies.addAll(webClient.getCookieManager().getCookies()); 
            StringBuilder cookieHeader = newStringBuilder(); 
            
            /*Iterator<Cookie> ite = boxCookies.iterator(); 
            while (ite.hasNext()){ 
                    Cookie cookie = ite.next(); 
                    cookie.getDomain().substring(1); 
                    String name = cookie.getName(); 
                    String value = cookie.getValue(); 
                    
                    System.out.println("Cookie:" + name + "=" +value); 
                    if ()){ 
                            cookieHeader.append(name + "=" +value); 
                    } 
                    cookieHeader.append("; "+ name + "=" +value); 
            } 
            */ 
            webClient.addRequestHeader("Accept-Encoding", "gzip, deflate, sdch"); 
            webClient.addRequestHeader("Accept-Language", "en-US,en;q=0.8"); 
            webClient.addRequestHeader("Cache-Control","no-cache"); 
            webClient.addRequestHeader("Upgrade-Insecure-Requests", "1"); 
            webClient.addRequestHeader("DNT", "1"); 
            webClient.addRequestHeader("Host","app.company.com"); 
            
            for (int index = 0; index < boxCookies.size(); index++){ 
                    String cookie = boxCookies.toArray()[index].toString(); 
                    String cookieNameValue=cookie.substring(0, cookie.indexOf(";")); 
                String name = cookieNameValue.substring(0,cookieNameValue.indexOf("=")); 
                String value = cookieNameValue.substring(cookieNameValue.indexOf("=") + 1); 
                
                    if (index == 0){ 
                           cookieHeader.append(name + "=" +value); 
                    } else { 
                           cookieHeader.append("; "+ name + "="+value); 
                    } 
                    
            } 
            WebRequest secondLoginPage = newWebRequest(AUTHORIZE_URL); 
            secondLoginPage.setAdditionalHeader("Cookie", cookieHeader.toString()); 
            HtmlPage loginPage2 = webClient.getPage(secondLoginPage); 
            
            out.println(loginPage2.getWebResponse().getWebRequest().getRequestParameters().toString()); 
            out.println("\n"); 
            Map<?, ?> additionalRequest1 = loginPage2.getWebResponse().getWebRequest().getAdditionalHeaders(); 
            
            Iterator<?> ite0 = additionalRequest1.entrySet().iterator(); 
             while(ite0.hasNext()){ 
                     out.println(ite0.next()); 
             } 
             out.println("\n"); 
            
            
            final HtmlTextInput login = (HtmlTextInput) loginPage2.getElementById("login"); 
            login.setValueAttribute(USER_EMAIL); 
            
            final HtmlPasswordInput password= (HtmlPasswordInput) loginPage2.getElementById("password"); 
            password.setValueAttribute(USER_PASS); 
            
            final HtmlSubmitInput button_submit = loginPage2.getElementByName("login_submit"); 
            
            final HtmlPage accessGrantingPage = button_submit.click(); 
            
            final HtmlForm  requestForm = (HtmlForm)accessGrantingPage.getElementById("consent_form"); 
            
            Map<?, ?> additionalRequest = accessGrantingPage.getWebResponse().getWebRequest().getAdditionalHeaders(); 
            out.println("\n"); 
            Iterator<?> ite2 = additionalRequest.entrySet().iterator(); 
             while(ite2.hasNext()){ 
                     out.println(ite2.next()); 
             } 
            out.println("\n"); 
            
            out.println("after accessGrantingPage"); 
            
            out.println(cookieManager.getCookies().toString()); 
            final HtmlButton consent_accept_button = accessGrantingPage.getElementByName("consent_accept"); 
            
            try { 
                    final HtmlPage authorizationPage = consent_accept_button.click(); 
                    out.println("after authorizationPage"); 
                    out.println(authorizationPage.getUrl().toString()); 
                    out.println(authorizationPage.getWebResponse().getStatusMessage()); 
        
                    out.println(authorizationPage.getWebResponse().getResponseHeaders()); 
                    
            } catch (RuntimeException re){ 
                    re.printStackTrace(); 
            } 
            webClient.closeAllWindows(); 
        } catch (IOException ioe){ 
                ioe.printStackTrace(); 
        } finally { 
                
        }

[Htmlunit-user] Cannot select dynamic selection SOLVED

[<r.b...@af...>]

SOLVED! .. I just added browser.waitForBackgroundJavaScript(8000); after
page = (HtmlPage) search_submit.click();.. thanks !

-----Messaggio originale-----
Da: r.b...@af... [mailto:r.b...@af...] 
Inviato: venerdì 18 marzo 2016 11:20
A: htm...@li...
Oggetto: [Htmlunit-user] Cannot select dynamic selection


Hi, how can I select the field "Search By" in this site with HTmlUnit? :
http://web.molpower.com/NaviGator/Tracking/home the field is not a normal
tag This is my code :

      String searchLink = "http://web.molpower.com/NaviGator/Tracking/home";
      System.out.println(" searchLink BN : [" + searchLink + "]");

      HtmlPage page = (HtmlPage) browser.getPage(searchLink);
     
      // <input id="SearchBy" class="valid" type="text" style="width: 100%;
display: none;" name="SearchBy" data-role="dropdownlist">
      String xpath = "//input[contains(@id, 'SearchBy')]";
      HtmlInput input_select = (HtmlInput) page.getFirstByXPath(xpath);
      input_select.setValueAttribute("BKG");
      
      // <textarea id="SearchString" class="valid"
style="width:500px;height:75px;" spellcheck="false" rows="2"
name="SearchString" cols="20"></textarea>
      xpath = "//textarea[contains(@id, 'SearchString')]";
      HtmlTextArea input_area = (HtmlTextArea) page.getFirstByXPath(xpath);
      input_area.setText("28010665376A"); // booking number
      System.out.println("input_area : [" + input_area.asXml() + "]");

      // <input id="btnSearch" class="k-button" type="submit"
value="Search">
      xpath = "//input[contains(@id, 'btnSearch')]";
      HtmlSubmitInput search_submit = (HtmlSubmitInput)
page.getFirstByXPath(xpath);


      HtmlUnorderedList ul = (HtmlUnorderedList)
page.getElementById("SearchBy_listbox");
      List<HtmlListItem> listLi = (List<HtmlListItem>)
page.getByXPath(ul.getCanonicalXPath() + "/li");

      // Get the default selected li element
      HtmlListItem liDefault = listLi.get(0);
      System.out.println("Default item selected : [" + liDefault.asXml() +
"]");

      for (HtmlListItem li : listLi) {
        // Select "Booking"
        if (li.asXml().contains("Booking")) {
          System.out.println("Booking : [" + li.asXml() + "]");
          li.setAttribute("aria-selected", "true");
          li.setAttribute("class", "k-item k-state-selected
k-state-focused");
          li.setAttribute("id", "SearchBy_option_selected");

          liDefault.setAttribute("class", "k-item");
          liDefault.removeAttribute("id");
          liDefault.removeAttribute("aria-selected");
        }
      }

      page = (HtmlPage) search_submit.click();


I cannot get the result page but the same search page..

Thanks.

Roberto Bottoni
--------------------------------------------
AfterBit s.r.l.
Via Valdera P, 67 - 56038 Ponsacco (Pisa) - ITALY Tel. +39 0587 734173  Fax.
+39 0587 734173 E-mail : r.b...@af... - Internet :
www.afterbit.com



----------------------------------------------------------------------------
--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140
_______________________________________________
Htmlunit-user mailing list
Htm...@li...
https://lists.sourceforge.net/lists/listinfo/htmlunit-user

[Htmlunit-user] Cannot select dynamic selection

[<r.b...@af...>]

Hi, how can I select the field "Search By" in this site with HTmlUnit? :
http://web.molpower.com/NaviGator/Tracking/home the field is not a normal
tag
This is my code :

      String searchLink = "http://web.molpower.com/NaviGator/Tracking/home";
      System.out.println(" searchLink BN : [" + searchLink + "]");

      HtmlPage page = (HtmlPage) browser.getPage(searchLink);
     
      // <input id="SearchBy" class="valid" type="text" style="width: 100%;
display: none;" name="SearchBy" data-role="dropdownlist">
      String xpath = "//input[contains(@id, 'SearchBy')]";
      HtmlInput input_select = (HtmlInput) page.getFirstByXPath(xpath);
      input_select.setValueAttribute("BKG");
      
      // <textarea id="SearchString" class="valid"
style="width:500px;height:75px;" spellcheck="false" rows="2"
name="SearchString" cols="20"></textarea>
      xpath = "//textarea[contains(@id, 'SearchString')]";
      HtmlTextArea input_area = (HtmlTextArea) page.getFirstByXPath(xpath);
      input_area.setText("28010665376A"); // booking number
      System.out.println("input_area : [" + input_area.asXml() + "]");

      // <input id="btnSearch" class="k-button" type="submit"
value="Search">
      xpath = "//input[contains(@id, 'btnSearch')]";
      HtmlSubmitInput search_submit = (HtmlSubmitInput)
page.getFirstByXPath(xpath);


      HtmlUnorderedList ul = (HtmlUnorderedList)
page.getElementById("SearchBy_listbox");
      List<HtmlListItem> listLi = (List<HtmlListItem>)
page.getByXPath(ul.getCanonicalXPath() + "/li");

      // Get the default selected li element
      HtmlListItem liDefault = listLi.get(0);
      System.out.println("Default item selected : [" + liDefault.asXml() +
"]");

      for (HtmlListItem li : listLi) {
        // Select "Booking"
        if (li.asXml().contains("Booking")) {
          System.out.println("Booking : [" + li.asXml() + "]");
          li.setAttribute("aria-selected", "true");
          li.setAttribute("class", "k-item k-state-selected
k-state-focused");
          li.setAttribute("id", "SearchBy_option_selected");

          liDefault.setAttribute("class", "k-item");
          liDefault.removeAttribute("id");
          liDefault.removeAttribute("aria-selected");
        }
      }

      page = (HtmlPage) search_submit.click();


I cannot get the result page but the same search page..

Thanks.

Roberto Bottoni
--------------------------------------------
AfterBit s.r.l.
Via Valdera P, 67 - 56038 Ponsacco (Pisa) - ITALY 
Tel. +39 0587 734173  Fax. +39 0587 734173
E-mail : r.b...@af... - Internet : www.afterbit.com

Re: [Htmlunit-user] Nested getByXPath Has Me All Confused

[Ahmed A. <asa...@ya...>]

Cool!
>> why the implementation of getByXPath is demanding I use the wildcards for the searches instead of being able to specify the elements.
It doesn't demand, but it is easier. You can also use "div/span[last()-1]" or "/html/body/span/div/input", etc.
Keep in mind, you always need to know the exact HTML (generated by HtmlUnit or by real browser, which may differ), to have relevant XPath expressions.
Ahmed

      From: Stephen Paulsen <st...@lo...>
 To: htm...@li...; Ahmed Ashour <asa...@ya...> 
 Sent: Tuesday, March 8, 2016 8:19 PM
 Subject: Re: [Htmlunit-user] Nested getByXPath Has Me All Confused
   
AT LONG LAST... This works:


Ahmed, I question why the implementation of getByXPath is demanding I use the wildcards for the searches instead of being able to specify the elements.  But, for now, as long as it works, I'm going to run with it.

Thank you all for the input and suggestions.