Menu
▾
▴
Re: [Htmlunit-user] Web pages that guard against browsing through html
|
From: Cameron M. <cam...@gm...> - 2011-10-20 20:24:34
|
Thanks again for the suggestions. I have done some more testing with Charles' web proxy and have discovered some differences in what the browser does vs. what htmlunit does when login occurs. When the browser logs in, the POST request contains a CSRF token in it. By the time the browser gets to the desired page, a new token has been issued, and the token that was contained in the login POST is included in the referrer. In htmlunit, the login POST does not include the CSRF token, but instead includes the sessionID (this despite the fact that a CSRF token is in fact issued during login). This seems to be the issue that is causing the automatic logout, as after the login stage, a new token is issued like in the browser, but there is no referrer listed. Is there a way to force the login POST request to use the CSRF token instead of the sessionID in htmlunit, like it does in the browser? Thanks, Cameron On Thu, Oct 20, 2011 at 3:14 AM, Marc Guillemot <mgu...@ya...> wrote: > Hi Cameron, > > I can't say how the token is lost :-( > You now have to understand where the request comes from and at which > place the problem occurs. The visual JavaScript debugger may be helpful > for that (see WebClientUtils). > > Cheers, > Marc. > -- > HtmlUnit support & consulting from the source > Blog: http://mguillem.wordpress.com > > Le 19/10/2011 22:55, Cameron McCormick a écrit : > > Thanks for the suggestions. I used Charles web proxy to monitor HTTP > > requests/response between the server and both the browser and HtmlUnit, > > and have found a difference. In the browser, when I click 'sign in' > > after entering my login credentials, one of the last GET requests is for > > a URL containing the session OWASP CRSF token in it. However, when I > > login through htmlunit, this step is different, as the URL in this GET > > request is missing the OWASP CRSF token. This results in my session > > getting logged out. It is worth noting that before this GET request is > > made, a different GET request is made for a different URL that again > > includes the OWASP CRSF token in it, and the request succeeds (200 OK) > > in both the browser and htmlunit. So my question is how does this > > token get lost between requests/ can it be recovered? > > > > Thanks, > > Cameron > > > > On Tue, Oct 18, 2011 at 2:47 PM, Ronald Brill <rb...@rb... > > <mailto:rb...@rb...>> wrote: > > > > On Tue, 18 Oct 2011 20:34:56 +0200 tommmmmm wrote: > > > > > >On 18 October 2011 18:49, Kellner, Matt <mke...@am... > > <mailto:mke...@am...>> wrote: > > > > > >> This sounds like a simple case of a session not being started > > properly. > > >> When you manually navigate to the secure page, you're likely > > skipping a > > >> step that starts or updates a session token with the current > > time, so it > > >> times out when you try to do something else requiring a reference > > to that > > >> token. The timeout could be either because the session doesn't > > exist at > > >> all, or because it has a timestamp on it that is longer ago than > the > > >> server's session timeout. (If it's not initialized properly, the > > session > > >> time may be set to the minimum value for the server's date/time > > object. > > >> > > >> Your HtmlUnit script should ensure that it goes through a proper > > sign-in > > >> procedure on the secure page to make sure sessions are started > > properly. > > >> > > >> To answer your more general question: Individual site owners may > > use all > > >> sorts of methods to detect non-human traffic, but there's no > single > > >> standard. Many sites look for "robots" by detecting the > > BrowserAgent on the > > >> connection, but HtmlUnit by default emulates Firefox 3, so its > > BrowserAgent > > >> tag should look just like a real instance of Firefox. Another > common > > >> practice is to look at traffic patterns and, through some > algorithm, > > >> determine if the connection is automated or not. But this is a > > difficult > > >> problem to solve, so I think it's more likely that you're simply > > running > > >> into a lack of session init. > > >> > > >> Hope this helps. > > >> ________________________________________ > > >> From: Cameron McCormick [cam...@gm... > > <mailto:cam...@gm...>] > > >> Sent: Tuesday, October 18, 2011 9:28 AM > > >> To: htm...@li... > > <mailto:htm...@li...> > > >> Subject: [Htmlunit-user] Web pages that guard against browsing > > through html > > >> > > >> Hello, > > >> > > >> I just have a general question about web page security. I am > using > > >> htmlunit to attempt to complete some tasks on an https web page > > used for > > >> email. I can navigate to the page, and then login using my > > credentials, but > > >> as soon as I attempt to click any meaningful link in the html (by > > meaningful > > >> I mean links other than the 'Help' or 'Sign Out' links) I am > > brought to the > > >> session timeout page. I investigated further and navigated to > > the page > > >> manually and viewed the Page Source, and discovered that clicking > > links in > > >> the Page Source manually also brings me to the Session Timeout > > page (well, > > >> it brings me to the source for the Session Timeout page). So my > > question is > > >> how do web pages guard against users browsing programmatically > > through the > > >> page's html, and is there a way around this? > > >> > > >> Thanks, > > >> Cameron > > >> > > >> > > >> > > > ------------------------------------------------------------------------------ > > >> All the data continuously generated in your IT infrastructure > > contains a > > >> definitive record of customers, application performance, security > > >> threats, fraudulent activity and more. Splunk takes this data and > > makes > > >> sense of it. Business sense. IT sense. Common sense. > > >> http://p.sf.net/sfu/splunk-d2d-oct > > >> _______________________________________________ > > >> Htmlunit-user mailing list > > >> Htm...@li... > > <mailto:Htm...@li...> > > >> https://lists.sourceforge.net/lists/listinfo/htmlunit-user > > >> > > > > > >I've came across that too ! With my own bank's page. It is so > > strict that > > >even clicking back or going through bad http referer makes me loose > the > > >session. I am very much interested into this topic too. > > > > > >As a side note I would like to add that some links launch external > > programs > > >like "BankBrowser" for money transfers. I have no idea how to > > handle such > > >situation in htmlUnit. > > > > > >Thanks, > > >Thomas > > > > > > > I think i have always the same suggestion. > > 1.) Use charles web proxy (or fiddler) and monitor the http > > requests/responses. > > 2.) Do it with the browser and HtmlUnit. > > 3.) Try to figure out, what the differnce is. > > > > If you can find something, there is a good chance that we can fix > > HtmlUnit. Don't wast your time with wild guesses. > > > > > > Second: > > I'm not sure what 'BankBrowsers' are. Real external programs or > Applets? > > At the moment i'm doing some development to make some more applets > > running with HtmlUnit. Maybe that helps. > > > > > > RBRi > > -------------------------- > > Wetator > > Smart web application testing > > http://www.wetator.org > > > ------------------------------------------------------------------------------ > The demand for IT networking professionals continues to grow, and the > demand for specialized networking skills is growing even more rapidly. > Take a complimentary Learning@Ciosco Self-Assessment and learn > about Cisco certifications, training, and career opportunities. > http://p.sf.net/sfu/cisco-dev2dev > _______________________________________________ > Htmlunit-user mailing list > Htm...@li... > https://lists.sourceforge.net/lists/listinfo/htmlunit-user > |
