Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

Hi Vasu,

>>   As you see the headers are identical. But Recaptcha service throws error image after around 200 refreshes done from

Well, they are not, there is a big cookie header in the second request of real browser, but HtmlUnit sends nothing.

CookiePREF=ID=27454a65a243cd70:U=508d4e3f9fa20406:FF=0:LD=en:TM=1406649235:LM=1407436763:GM=1:S=aCHb3CgmGRRcedMi; _ga=GA1.1.1138220839.1407426218; OGP=-4061129:; SID=DQAAAMkAAABEIRL4EB_u_34e9SPchv4-B1gv-ecyCxqE9yuqGcSCBx8fvByoEltivh8scqBkKd-g3_BS4QjJUAb54fr-Xl8SrIiSkJYVlD4dtJn2YB5JIgD7s_hCHT58eMV11jkFKy1NL3R91ibbv0inWCLlle59rF3VKaYOK1BhT_P0RFgbhttdSCq_sPccI_ejQe6Q4tem-CErtVlzQbnSq1idOr-XMoSHn6t0hzsRkrtFs7vzdShkayUlnnee5ruugolGON6gNN0gVQ2xY8XPdwYDBx_T; HSID=AvFUJATSD2LR-GOTz; APISID=_q3L-5Jckjza4xOL/AM5CzbFXam4moeOPb

I guess this is a much higher possibility than the header order.

Please investigate which request returns the response with that values of those cookies, and what are the corresponding HtmlUnit request/response.

Ahmed

________________________________
 From: Vasudevan Comandur <vco...@gm...>
To: Ahmed Ashour <asa...@ya...>; "htm...@li..." <htm...@li...> 
Sent: Thursday, August 21, 2014 11:46 AM
Subject: Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google

Hi Ahmed,

   Copying the header exchange between HTMLUnit and Recaptcha server.

   HTMLUnit (First Request)

 GET /challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 HTTP/1.1 
Host api.recaptcha.net 
User-Agent Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0 
Accept-Language en-US,en;q=0.5 
Accept-Encoding gzip, deflate 
Referer http://www.patcurtis.com/captcha/recaptcha_demo.php 
Connection keep-alive 
Accept */* 
  WebServer (Response)

 HTTP/1.1 302 Moved Temporarily 
Cache-Control no-cache, no-store, max-age=0, must-revalidate 
Pragma no-cache 
Expires Fri, 01 Jan 1990 00:00:00 GMT 
Date Thu, 21 Aug 2014 09:39:10 GMT 
Location http://www.google.com/recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 
Content-Type text/html; charset=UTF-8 
Content-Encoding gzip 
X-Content-Type-Options nosniff 
X-XSS-Protection 1; mode=block 
Content-Length 227 
Server GSE 
Alternate-Protocol 80:quic 
  HTMLUnit(Request)

 GET /recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 HTTP/1.1 
Host www.google.com 
User-Agent Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0 
Accept-Language en-US,en;q=0.5 
Accept-Encoding gzip, deflate 
Referer http://www.patcurtis.com/captcha/recaptcha_demo.php 
Connection keep-alive 
Accept */* 
  WebServer(Response)

 HTTP/1.1 200 OK 
Cache-Control no-cache, no-store, max-age=0, must-revalidate 
Pragma no-cache 
Expires Fri, 01 Jan 1990 00:00:00 GMT 
Date Thu, 21 Aug 2014 09:39:14 GMT 
Set-Cookie NID=67=Q5SsGkHHzte18yk1v8PHHpUlEMpWHhLKW1XJtIq9gql_ma_oLvAJ0W88OPBtn8JfDdLzeb7egYW1T8CyHiBuMviGYlRa7nh-wdFTVFCQqk0umJgym5WximGdBhfby_K5;Domain=.google.com;Path=/;Expires=Fri, 20-Feb-2015 09:39:14 GMT;HttpOnly 
P3P CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info." 
Content-Type text/javascript 
Content-Encoding gzip 
X-Content-Type-Options nosniff 
X-XSS-Protection 1; mode=block 
Content-Length 7048 
Server GSE 
Alternate-Protocol 80:quic 
  As you see the headers are identical. But Recaptcha service throws error image after around 200 refreshes done from
  the browser.

  Let me know if you need any additional information.

   Thanks for your help in advance

Regards
 Vasu

On Thu, Aug 21, 2014 at 12:10 PM, Ahmed Ashour <asa...@ya...> wrote:

Hi Vasu,
>
>
>There are two differences:
>    - Cookies in the second request of real browser, they seem to be stored from previous requests. Please clear cache and rerun the comparison. I also wonder from where did HtmlUnit send the cookies in the second request.
>    - 'Accept' header location: this could be a reason, but Cookie are the much more the possibility.
> 
>Ahmed
>
>________________________________
>
>From: Vasudevan Comandur <vco...@gm...>
>To: Ahmed Ashour <asa...@ya...>; "htm...@li..." <htm...@li...> 
>Sent: Wednesday, August 20, 2014 9:06 PM
>Subject: Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google
> 
>
>
>Hi Ahmed,
>
>    I am giving you the request details for REAL Browser & HTMLUnit below.
>
>REal Browser (First Request)
>
>
>
> GET /challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 HTTP/1.1 
>Host api.recaptcha.net 
>User-Agent Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0 
>Accept */* 
>Accept-Language en-US,en;q=0.5 
>Accept-Encoding gzip, deflate 
>Referer http://www.patcurtis.com/captcha/recaptcha_demo.php 
>Connection keep-alive 
>Cache-Control max-age=0 
>WebServer(Response)
>
>
> HTTP/1.1 302 Moved Temporarily 
>Cache-Control no-cache, no-store, max-age=0, must-revalidate 
>Pragma no-cache 
>Expires Fri, 01 Jan 1990 00:00:00 GMT 
>Date Wed, 20 Aug 2014 11:40:18 GMT 
>Location http://www.google.com/recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 
>Content-Type text/html; charset=UTF-8 
>Content-Encoding gzip 
>X-Content-Type-Options nosniff 
>X-XSS-Protection 1; mode=block 
>Content-Length 227 
>Server GSE 
>Alternate-Protocol 80:quic 
>RB(Request)
>
>
> GET /recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 HTTP/1.1 
>Host www.google.com 
>User-Agent Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0 
>Accept */* 
>Accept-Language en-US,en;q=0.5 
>Accept-Encoding gzip, deflate 
>Referer http://www.patcurtis.com/captcha/recaptcha_demo.php 
>Cookie PREF=ID=27454a65a243cd70:U=508d4e3f9fa20406:FF=0:LD=en:TM=1406649235:LM=1407436763:GM=1:S=aCHb3CgmGRRcedMi; _ga=GA1.1.1138220839.1407426218; OGP=-4061129:; NID=67=pLb98eXX7XshplSC25O1hUrM6VPe1bG9epaYJREF4EBGoV-TPIQ0MynwuZdb1ra4i_j8jfzeKgHyJRL917tqVvJ1drL2BTn2QE4HaRblzxd0lpITPiswWjYNWO3YMn8g0aV7bw3zFqdW9HrdT3h2iJxykpn7QTCJm7fFHZoSYVRR1zWL_jNpSHvs7u912M5K1skdqY-lGtU; SID=DQAAAMkAAAD1nK0mZZoRhWbUjiHM6nCXAWouJuka0s3NW3l9AO0Tp7vLrszmHqYEeOI2mbDn6Ew2DuOvg0DxTEDPsjQ_-lMPWoXn_8TQYCEpFyvdZUdqdYT1qb2jNjv7cb6192vCkFAjY4LPJo5tKDUvm8_UDsJGrrILWzwGEf9TMkyo8uMP_rZcL3JFbVWfsN7W8j2MZWRLlSoZ-wcpW8PiQc_se5XSVXoZKUdGd67OemZdbEBmX_VRZ66mJln6EtJeJdOWruXp_I72ldDaz9UZMvZJ7ySH; HSID=AbDchAUPI4DtlOcyV; APISID=xLgJV19gOJC9cCD_/ARvW0PuATO71HDPNc 
>Connection keep-alive 
>WebServer(Response)
>
>
> HTTP/1.1 200 OK 
>Cache-Control no-cache, no-store, max-age=0, must-revalidate 
>Pragma no-cache 
>Expires Fri, 01 Jan 1990 00:00:00 GMT 
>Date Wed, 20 Aug 2014 11:39:22 GMT 
>Content-Type text/javascript 
>Content-Encoding gzip 
>X-Content-Type-Options nosniff 
>X-XSS-Protection 1; mode=block 
>Content-Length 7090 
>Server GSE 
>Alternate-Protocol 80:quic 
>HTMLUnit (First Request)
>
>
> GET /challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 HTTP/1.1 
>Host api.recaptcha.net 
>User-Agent Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0 
>Accept-Language en-US,en;q=0.5 
>Accept-Encoding gzip, deflate 
>Referer http://www.patcurtis.com/captcha/recaptcha_demo.php 
>Connection keep-alive 
>Accept */* 
>Cache-Control max-age=0 
>WebServer(Response)
>
>
>
> HTTP/1.1 302 Moved Temporarily 
>Cache-Control no-cache, no-store, max-age=0, must-revalidate 
>Pragma no-cache 
>Expires Fri, 01 Jan 1990 00:00:00 GMT 
>Date Wed, 20 Aug 2014 18:22:58 GMT 
>Location http://www.google.com/recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 
>Content-Type text/html; charset=UTF-8 
>Content-Encoding gzip 
>X-Content-Type-Options nosniff 
>X-XSS-Protection 1; mode=block 
>Content-Length 227 
>Server GSE 
>Alternate-Protocol 80:quic 
>HTMLUnit(Request)
>
>
> GET /recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 HTTP/1.1 
>Host www.google.com 
>User-Agent Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0 
>Accept-Language en-US,en;q=0.5 
>Cookie NID=67=cA1Qro6B-PbS07zWxNtD3iJ0hHfuc0itJYr1CVnyDCL4PbJNELomp3zFICR6NGdC0C3rQ440_1Y743dl87W5cTtWbAadzMGOZ0-1hbYTqYOSqZrSeN922y0XQr5IcKGE 
>Accept-Encoding gzip, deflate 
>Referer http://www.patcurtis.com/captcha/recaptcha_demo.php 
>Connection keep-alive 
>Accept */* 
>Cache-Control max-age=0 
>WebServer(Response)
>
>
> HTTP/1.1 200 OK 
>Cache-Control no-cache, no-store, max-age=0, must-revalidate 
>Pragma no-cache 
>Expires Fri, 01 Jan 1990 00:00:00 GMT 
>Date Wed, 20 Aug 2014 18:22:58 GMT 
>Content-Type text/javascript 
>Content-Encoding gzip 
>X-Content-Type-Options nosniff 
>X-XSS-Protection 1; mode=block 
>Content-Length 6994 
>Server GSE 
>Alternate-Protocol 80:quic 
>*******************************************************************
>
>Kindly Let me know if you need any additional information. Please note that there are many request/response exchanges
>after the above steps till we get the actual CAPTCHA image
>
>Thanks for your help in advance.
>
>Regards
> Vasu
>
>
>
>------------------------------------------------------------------------------
>Slashdot TV.
>Video for Nerds.  Stuff that matters.
>http://tv.slashdot.org/
>_______________________________________________
>Htmlunit-user mailing list
>Htm...@li...
>https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>
>

Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google

Java GUI-Less browser, supporting JavaScript, to run against web pages

Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google