htmlunit-develop — General discussion of the development of HtmlUnit

Re: [HtmlUnit] Redirects

[Mike B. <mb...@Ga...>]

I've read the http spec again and it isn't really clear what the correct 
behaviour is so I'm going to do what the common browsers do.

I've changed redirect logic so that on a 302 or 303 status, if the 
method changes from post to get, the parameters will be thrown away for 
the second request.  This code is now in cvs.

-- 
Mike Bowler
Principal, Gargoyle Software Inc.
Voice: (416) 822-0973 | Email  : mb...@Ga...
Fax  : (416) 822-0975 | Website: http://www.GargoyleSoftware.com

Re: [HtmlUnit] Redirects

[Thomas B. <th...@cy...>]

Hello again,=20

thanks a lot for your quick answer.=20

Am Montag, 23. September 2002 15:57 schrieb Mike Bowler:
> > In HtmlUnit the submit of the
> > form-parameters via is POST is correct. But when it tries to
> > follow the redirect, it somehow converts the POST parameters
> > into GET parameters and appends them to the URL-string. That's
> > not right behavior and our application can't handle that.
> > Is there a possibility to switch that off or does anyone know a
> > workaround?
>
> You can disable automatic redirection handling with
> WebClient.setRedirectEnabled(false).

Thats not exactly what I had in mind ;)

> It's certainly possible that I've made a mistake in the redirection
> handling but as far as I know, it is correct according to the HTTP/1.1
> specification.
>
> http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3
>
> If you think that HtmlUnit is handling this incorrectly then please
> open a bug report.  I'll need to know the exact status code being
> returned from the server as the redirection logic is different for
> different status codes.
>
> Since you mentioned the method switching from post to get, I assume
> that the status code you got was either a 302 or a 303.

So after I got your mail I checked it all a little bit and found out the=20
following: The given status code was a 302 and the HTTP/1.1 specification=
=20
doesn't really mention what to do with POST-Request-Parameters on a 302, =
does=20
it? As far as I understand, it says to keep the original Request-URI for=20
future requests, not convert parameters for the redirect-URI. But maybe I=
'm=20
wrong with this, after reading the specifications a few times I was prett=
y=20
puzzled how to understand it at all ;)
However, I tried something different. I compared the behavior of an inter=
net=20
browser (Mozilla) with the behavior of HtmlUnit. It is in fact so, that t=
he=20
browser doesnt convert the original POST-Parameters into GET-Parameters w=
hile=20
doing the redirect and HtmlUnit does. I tried this out with two simple=20
PHP-scripts. The first one contains a form, which points to itself (PHP_S=
ELF=20
as action), but redirects to the second when the variable for the=20
submit-button is set. The second one just prints out the value of the=20
form-datas if accessible, like the vaue of the submit-button for instance=
=2E So=20
with browser I don't get any output on the second page, like I get with=20
HtmlUnit.=20
Well, as I said, I'm not sure how it really is specified. But however it =
is,=20
HtmlUnit does here something uncommon. The most user agents (IE, NS,=20
Opera...) handle it like described above and different from the HtmlUnit-=
way.

It's up to you if it's a bug in HtmlUnit or actually in all other clients=
=2E In=20
case its not a bug, I'd like to see a possibility to configure browser-li=
ke=20
behavior on redirects.

Thanks for your help anyways.=20
Kind regards,
Thomas
--=20
Thomas Bartz
th...@cy...
G=E4rtnerstr. 28, 10245 Berlin; Tel.: 030 29005717
http://www.cyainhell.de

[HtmlUnit] Re: select options bug

[Mike B. <mb...@Ga...>]

 > We have a page with two html select tags, both listing the US states
>  (one for origin and the other for destination).  Using htmlunit when I
>  select the first state it is throwing the following exception:
> 
>  IllegalStateException: Found more than one matching element

This one has already been fixed in cvs.  I expect to package up the next 
pre-release fairly soon but I still have some work to do with the 
documentation since the package structure has changed.

If you need it in a hurry then I'd recommend getting the latest code 
from CVS.  Don't forget that you pull module "htmlunit" now, not "src", 
"lib" and "docs".

-- 
Mike Bowler
Principal, Gargoyle Software Inc.
Voice: (416) 822-0973 | Email  : mb...@Ga...
Fax  : (416) 822-0975 | Website: http://www.GargoyleSoftware.com

[HtmlUnit] select options bug

[Brian K. <bt...@ne...>]

We have a page with two html select tags, both listing the US states
(one for origin and the other for destination).  Using htmlunit when I
select the first state it is throwing the following exception:

=20

IllegalStateException: Found more than one matching element

=20

I looked at the source and found that htmlunit is looking at the entire
page for all the options across all select tags and combining them into
one big list.  This is why it's finding a duplicate entry.  The bug
makes htmlunit unusable for our team... any idea on when a fix can be
released?

=20

Thanks,

=20

Brian

Re: [HtmlUnit] Redirects

[Harm de L. <ha...@ww...>]

Hi there,

I'm having the same problem. We use Lotus Domino as our webserver.
I'm curious if somebody has a workaround or fix for this problem.

Thanks,

Harm de Laat
Informatiefabriek
The Netherlands.


On Mon, Sep 23, 2002 at 03:29:23PM +0200, th...@cy... wrote:
> Hi, 
> 
> I encountered another problem with HtmUnit, which seems to be a little
> tricky. In our app we use lots of redirects. So after submitting a form
> the page containing the form will redirect to the next page (after
> checking submitted form-parameters). In HtmlUnit the submit of the
> form-parameters via is POST is correct. But when it tries to follow the
> redirect, it somehow converts the POST parameters into GET parameters
> and appends them to the URL-string. That's not right behavior and our
> application can't handle that.
> Is there a possibility to switch that off or does anyone know a
> workaround?
> Let me know if there isn't, so that I can open a new feature request.
> 
> Thanks, 
> Thomas Bartz
> 
> 
> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> HtmlUnit-develop mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlunit-develop

-- 
Harm de Laat
--
  echo "tooL ed mroH" | rev | tr o a

[HtmlUnit] Redirects

[Mike B. <gar...@ya...>]

> In HtmlUnit the submit of the
> form-parameters via is POST is correct. But when it tries to 
> follow the redirect, it somehow converts the POST parameters 
> into GET parameters and appends them to the URL-string. That's 
> not right behavior and our application can't handle that.
> Is there a possibility to switch that off or does anyone know a
> workaround?

You can disable automatic redirection handling with
WebClient.setRedirectEnabled(false).

It's certainly possible that I've made a mistake in the redirection
handling but as far as I know, it is correct according to the HTTP/1.1
specification.

http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3

If you think that HtmlUnit is handling this incorrectly then please
open a bug report.  I'll need to know the exact status code being
returned from the server as the redirection logic is different for
different status codes.

Since you mentioned the method switching from post to get, I assume
that the status code you got was either a 302 or a 303.

-- 
Mike Bowler
Principal, Gargoyle Software Inc.
Voice: (416) 822-0973 | Email  : mb...@Ga...
Fax  : (416) 822-0975 | Website: http://www.GargoyleSoftware.com

______________________________________________________________________ 
Post your free ad now! http://personals.yahoo.ca

[HtmlUnit] Redirects

[<th...@cy...>]

Hi, 

I encountered another problem with HtmUnit, which seems to be a little
tricky. In our app we use lots of redirects. So after submitting a form
the page containing the form will redirect to the next page (after
checking submitted form-parameters). In HtmlUnit the submit of the
form-parameters via is POST is correct. But when it tries to follow the
redirect, it somehow converts the POST parameters into GET parameters
and appends them to the URL-string. That's not right behavior and our
application can't handle that.
Is there a possibility to switch that off or does anyone know a
workaround?
Let me know if there isn't, so that I can open a new feature request.

Thanks, 
Thomas Bartz

Re: [HtmlUnit] Click on HtmlImageInput

[Mike B. <mb...@Ga...>]

>
>
>I need to "click" on an image submit button (<input type="image" ...>)
>but the method seems not to be implemented yet in class HtmlImageInput
>like it is in HtmlSubmitInput. I'd like to see this feature as soon as
>possible in HtmlUnit, since we got a lot of these buttons in our app ;)
>  
>

Please open a feature request on sourceforge so this doesn't get lost 
and I'll see if I can get it in on the weekend.  It's unlikely I'll have 
time before then.

>Is there any quick workaround for that problem anyways?
>  
>

You could call submit() on the form directly.  It wouldn't execute 
onclick handlers (if any) but it would do the other behaviour.


-- 
Mike Bowler
Principal, Gargoyle Software Inc.
Voice: (416) 822-0973 | Email  : mb...@Ga...
Fax  : (416) 822-0975 | Website: http://www.GargoyleSoftware.com

[HtmlUnit] Click on HtmlImageInput

[<th...@cy...>]

Hi, 

I need to "click" on an image submit button (<input type="image" ...>)
but the method seems not to be implemented yet in class HtmlImageInput
like it is in HtmlSubmitInput. I'd like to see this feature as soon as
possible in HtmlUnit, since we got a lot of these buttons in our app ;)

Is there any quick workaround for that problem anyways?

Greetings from Germany
Thomas Bartz

[HtmlUnit] Checking out code - new directory structure

[Mike B. <mb...@Ga...>]

For anyone who is checking code out of cvs.....

The directory tree has changed quite a bit.  Previously, you had to 
check out src, lib and docs.  Now it's all been moved under one module - 
"htmlunit".  The three old modules are still there but aren't being used 
anymore.

The new layout follows the jakarta (apache) standard directory layout
http://jakarta.apache.org/site/dirlayout.html

The online docs are still referring to the old modules - I'll fix those 
up when I get a chance.

-- 
Mike Bowler
Principal, Gargoyle Software Inc.
Voice: (416) 822-0973 | Email  : mb...@Ga...
Fax  : (416) 822-0975 | Website: http://www.GargoyleSoftware.com

Re: [HtmlUnit] javascript in external files

[Mike B. <mb...@Ga...>]

Hi Asle,

 > I tested the click method of htmlanchor on this page:
 > http://www2.shell.com/home/Framework?siteId=investor-en
 > Depending on the anchor I click() I get a MalformedUrlException
 > or "The undefined value has no properties".

A couple of days ago, I found and fixed some spots where relative urls 
weren't being expanded correctly.  Mostly likely that's the cause of 
your MalformedUrlException.  The latest code in CVS has those fixes if 
you want to try it.

The "the undefined value has no properties" error means that the page is 
trying to reference a javascript property or function that I haven't 
implemented yet.  I'll have to do some investigation to figure that one 
out.  Please open a bug report for this one so that it doesen't get lost.

-- 
Mike Bowler
Principal, Gargoyle Software Inc.
Voice: (416) 822-0973 | Email  : mb...@Ga...
Fax  : (416) 822-0975 | Website: http://www.GargoyleSoftware.com

RE: RE: [HtmlUnit] javascript in external files

[Asle P. <ape...@ho...>]

I tested the click method of htmlanchor on this page:
http://www2.shell.com/home/Framework?siteId=investor-en
Depending on the anchor I click() I get a MalformedUrlException
or "The undefined value has no properties".

The reason for the exceptions could be that I use Jython to interface with 
htmlunit but I really cant imagine that it could cause those errors.

regards,
Asle


_________________________________________________________________
Send and receive Hotmail on your mobile device: http://mobile.msn.com

Re: [HtmlUnit] javascript in external files

[Mike B. <mb...@Ga...>]

Hi Asle,

 > Does htmlunit handle cases where javascript is not embedded
 > in the html sourcecode but located in an external file?

Yes, it handles this case.

 > I am using 1.1-pre4 and as far as I could tell it did not seem to
 > evaluate external javascripts.

That's strange - it's been working fine for me.    I wonder if it isn't 
able to resolve the url in the src tag.  If it isn't able to load the 
external file then you should see an error message like this:

[ERROR] HtmlPage - -Error loading javascript from 
[http://www.alphaworks.ibm.com/www.ibm.com/common/stats/stats.js] 
status=[404 Not found - file doesn't exist or is read protected [even 
tried multi]]

Are you seeing any log messages?

 > BTW: The getAnchors() solution worked fine.. thanks!

Glad to hear it.

-- 
Mike Bowler
Principal, Gargoyle Software Inc.
Voice: (416) 822-0973 | Email  : mb...@Ga...
Fax  : (416) 822-0975 | Website: http://www.GargoyleSoftware.com

[HtmlUnit] javascript in external files

[Asle P. <ape...@ho...>]

Does htmlunit handle cases where javascript is not embedded in the html 
sourcecode but located in an external file?

I guess that the js-code located outside the main html file should be 
fetched and evaluated. I am using 1.1-pre4 and as far as I could tell it did 
not seem to evaluate external javascripts.

BTW: The getAnchors() solution worked fine.. thanks!

regards,
Asle

_________________________________________________________________
Send and receive Hotmail on your mobile device: http://mobile.msn.com

Re: [HtmlUnit] HtmlTextArea

[Mike B. <mb...@Ga...>]

On Thu, 29 Aug 2002, Cort Schaefer wrote:

> This may be an obvious question, but is there some reason that HtmlTextArea
> does not extend HtmlInput?

Because it isn't an <input>.  From the javascript perspective, a text area
IS an input but for the HTML classes, the only subclasses of HtmlInput are
those that actually are <input> tags.

> Also, is there any way to ignore certificate warnings when trying to make an
> HTTPS connection? (i.e. cert for wrong domain or expired)

I don't know - I haven't seen any of those warnings yet.  Is there a
publicly accessible site that gives warnings?  ie something I can try?


-- 
Mike Bowler
Principal, Gargoyle Software Inc.
Voice: (416) 822-0973 | Email  : mb...@Ga...
Fax  : (416) 822-0975 | Website: http://www.GargoyleSoftware.com

[HtmlUnit] HtmlTextArea

[Cort S. <co...@xm...>]

This may be an obvious question, but is there some reason that HtmlTextArea
does not extend HtmlInput?

Also, is there any way to ignore certificate warnings when trying to make an
HTTPS connection? (i.e. cert for wrong domain or expired)

Cort 

Re: [HtmlUnit] htmlAnchors

[Mike B. <mb...@Ga...>]

> How can I retrieve htmlAnchors from a HtmlPage if there is no ID on 
> the anchors to adress them (similar with forms)? E.g. is there a way
> of getting a list of all the links/anchors from the page?

The next pre-release will have three methods that do what you want.

HtmlPage.getAnchorByHref()
HtmlPage.getAnchorByText()
HtmlPage.getAnchors() <- returns all anchors on the page

All three are already in CVS if you want to try the latest code.

-- 
Mike Bowler
Principal, Gargoyle Software Inc.
Voice: (416) 822-0973 | Email  : mb...@Ga...
Fax  : (416) 822-0975 | Website: http://www.GargoyleSoftware.com

Re: [HtmlUnit] htmlAnchors

[Simon S. <sm...@la...>]

On Thu, Aug 29, 2002 at 02:03:20PM +0000, Asle Pedersen wrote:
> How can I retrieve htmlAnchors from a HtmlPage if there is no ID on the 
> anchors to adress them (similar with forms)? E.g. is there a way of getting 
> a list of all the links/anchors from the page?

There's a feature request for this, I think, but I use a chunk of code
similar to this:

    /** Similar to HtmlPage.getAnchorById() but uses the label of the anchor to obtain the element
     * @param page The page containing the anchor being searched for
     * @param label The label (user visible part) of the anchor
     * @return The HtmlAnchor represented by the given label, or null if no label is found. */
    private HtmlAnchor getHtmlAnchor(HtmlPage page, String label) {
        // Make sure that the inputs are sane
        if (label == null)
            return null;

        List anchors = new ArrayList();
        anchors.add("a");
        List returnedAnchors = page.getHtmlElementsByTagNames( anchors );
        Iterator i = returnedAnchors.iterator();
        HtmlAnchor anchor = null;
        while(i.hasNext()) {
            anchor = (HtmlAnchor)i.next();
            if (label.equals(anchor.asText()))
                return anchor;
        }
        return null;
    }

Cheers,

Simon

-- 
Asynchronous inputs are at the root of our race problems.
		-- D. Winker and F. Prosser

[HtmlUnit] htmlAnchors

[Asle P. <ape...@ho...>]

How can I retrieve htmlAnchors from a HtmlPage if there is no ID on the 
anchors to adress them (similar with forms)? E.g. is there a way of getting 
a list of all the links/anchors from the page?

regards,
Asle

_________________________________________________________________
Chat with friends online, try MSN Messenger: http://messenger.msn.com

[HtmlUnit] connect to secure sites

[netbeans <net...@fl...>]

I want to use HTMLUnit to fetch
information from websites I use (that I login
to) and create a personal page of information
just the way I want it and save it to disk for
offline viewing.  I am hitting
some of these sites and am getting errors
back saying the page is not available.  I believe
they are detecting whether I am connecting to the
site with a recognized browser or not.

I changed all the HTTP request headers using HTMLUnit
to match that of my browser but was still unable to get
through.  What can I do to hit the web site login
with my information and get the data I need?

Does the mozilla browser for example connect in
such a way that these sites can tell?  Am I going
to have to dig into the mozilla code to do this
for me or is there a way to do it with HTMLUnit?

The headers I sent matched those of Mozilla exactly
except for 2 things:
    1.  order - could not get HTMLUnit to send in the
    order the header was added.

    2.  HTMLUnit adds "Content-Length" even when I requested
    that it be removed.

Besides these 2 items - the headers I sent were identical to
mozilla headers and I am able to hit the site using mozilla 1.0.

How do I specify the exact order and get the "Content-Length"
removed?  something tells me this may  not make a difference but
I should at least be able to do these two and try.

If a browser gets through - there has to be a way to get through
with htmlunit or some other program.

thanks

netbeans

[HtmlUnit] HtmlUnit-1.1-pre4 has been released

[Mike B. <mb...@Ga...>]

  HtmlUnit-1.1-pre4 has been released. Refer to the change log at 
http://htmlunit.sourceforge.net/changes.html for full details.

-- 
Mike Bowler
Principal, Gargoyle Software Inc.
Voice: (416) 822-0973 | Email  : mb...@Ga...
Fax  : (416) 822-0975 | Website: http://www.GargoyleSoftware.com

Re: [HtmlUnit] data driven interface

[Mike B. <mb...@Ga...>]

> Is there a data-driven interface to HtmlUnit?
> For instance, Canoo WebTest enables you to create an XML file 
> to drive HttpUnit.

No, the only interface to HtmlUnit is the java api.

-- 
Mike Bowler
Principal, Gargoyle Software Inc.
Voice: (416) 822-0973 | Email  : mb...@Ga...
Fax  : (416) 822-0975 | Website: http://www.GargoyleSoftware.com

[HtmlUnit] data driven interface

[<Eri...@al...>]

Is there a data-driven interface to HtmlUnit?
For instance, Canoo WebTest enables you to create an XML file to drive
HttpUnit.

RE: [HtmlUnit] organization of the cvs repository

[Cort S. <co...@xm...>]

When the time comes, I would be interested in knowing how long it really
does take them.  I have a goof in my project as well and will need them to
do some clean up for me as well - but need to put some thought into what I
want to do as well.

Cort

-----Original Message-----
From: Mike Bowler [mailto:mb...@Ga...] 
Sent: Thursday, August 15, 2002 7:45 PM
To: 'htm...@li...'
Subject: Re: [HtmlUnit] organization of the cvs repository

 > Can things be moved within the repository, while
 > still preserving the CVS history of these files?

Very interesting.  I'll have to keep that in mind when it's time to 
switch over to the new directory structure.  It'll be interesting how 
fast they are able to do the switchover - it took them 2 weeks to reset 
the administrator password for the mailing list.

-- 
Mike Bowler
Principal, Gargoyle Software Inc.
Voice: (416) 822-0973 | Email  : mb...@Ga...
Fax  : (416) 822-0975 | Website: http://www.GargoyleSoftware.com




-------------------------------------------------------
This sf.net email is sponsored by: OSDN - Tired of that same old
cell phone?  Get a new here for FREE!
https://www.inphonic.com/r.asp?r=sourceforge1&refcode1=vs3390
_______________________________________________
HtmlUnit-develop mailing list
Htm...@li...
https://lists.sourceforge.net/lists/listinfo/htmlunit-develop

Re: [HtmlUnit] form.elements missing radio inputs

[Mike B. <mb...@Ga...>]

 > I found that the JavaScript Form host does not add radio inputs to 
the elements collection.

Radio buttons have to be handled differently because when referencing 
them by name, you get back an array rather than one element.  Most 
likely I didn't take into account the fact that they still need to be 
expanded in the elements array.

Your fix may break the element referencing by name so watch out for 
that.  If you haven't already opened a bug for this one then please do 
so it doesn't get lost - I've got a couple of other fixes that I'm 
working on before this one.

I've been in a crunch over the last couple of days, wrapping up some 
presentations for a conference I'm speaking at, so I'm a little behind 
on fixes right now.

--
Mike Bowler
Principal, Gargoyle Software Inc.
Voice: (416) 822-0973 | Email  : mb...@Ga...
Fax  : (416) 822-0975 | Website: http://www.GargoyleSoftware.com