Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google

[Ahmed A. <asa...@ya...>]

Hi Vasu,

There are two differences:
    - Cookies in the second request of real browser, they seem to be stored from previous requests. Please clear cache and rerun the comparison. I also wonder from where did HtmlUnit send the cookies in the second request.
    - 'Accept' header location: this could be a reason, but Cookie are the much more the possibility.
 
Ahmed

________________________________
 From: Vasudevan Comandur <vco...@gm...>
To: Ahmed Ashour <asa...@ya...>; "htm...@li..." <htm...@li...> 
Sent: Wednesday, August 20, 2014 9:06 PM
Subject: Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google
 


Hi Ahmed,

    I am giving you the request details for REAL Browser & HTMLUnit below.

REal Browser (First Request)



 GET /challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 HTTP/1.1 
Host api.recaptcha.net 
User-Agent Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0 
Accept */* 
Accept-Language en-US,en;q=0.5 
Accept-Encoding gzip, deflate 
Referer http://www.patcurtis.com/captcha/recaptcha_demo.php 
Connection keep-alive 
Cache-Control max-age=0 
WebServer(Response)


 HTTP/1.1 302 Moved Temporarily 
Cache-Control no-cache, no-store, max-age=0, must-revalidate 
Pragma no-cache 
Expires Fri, 01 Jan 1990 00:00:00 GMT 
Date Wed, 20 Aug 2014 11:40:18 GMT 
Location http://www.google.com/recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 
Content-Type text/html; charset=UTF-8 
Content-Encoding gzip 
X-Content-Type-Options nosniff 
X-XSS-Protection 1; mode=block 
Content-Length 227 
Server GSE 
Alternate-Protocol 80:quic 
RB(Request)


 GET /recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 HTTP/1.1 
Host www.google.com 
User-Agent Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0 
Accept */* 
Accept-Language en-US,en;q=0.5 
Accept-Encoding gzip, deflate 
Referer http://www.patcurtis.com/captcha/recaptcha_demo.php 
Cookie PREF=ID=27454a65a243cd70:U=508d4e3f9fa20406:FF=0:LD=en:TM=1406649235:LM=1407436763:GM=1:S=aCHb3CgmGRRcedMi; _ga=GA1.1.1138220839.1407426218; OGP=-4061129:; NID=67=pLb98eXX7XshplSC25O1hUrM6VPe1bG9epaYJREF4EBGoV-TPIQ0MynwuZdb1ra4i_j8jfzeKgHyJRL917tqVvJ1drL2BTn2QE4HaRblzxd0lpITPiswWjYNWO3YMn8g0aV7bw3zFqdW9HrdT3h2iJxykpn7QTCJm7fFHZoSYVRR1zWL_jNpSHvs7u912M5K1skdqY-lGtU; SID=DQAAAMkAAAD1nK0mZZoRhWbUjiHM6nCXAWouJuka0s3NW3l9AO0Tp7vLrszmHqYEeOI2mbDn6Ew2DuOvg0DxTEDPsjQ_-lMPWoXn_8TQYCEpFyvdZUdqdYT1qb2jNjv7cb6192vCkFAjY4LPJo5tKDUvm8_UDsJGrrILWzwGEf9TMkyo8uMP_rZcL3JFbVWfsN7W8j2MZWRLlSoZ-wcpW8PiQc_se5XSVXoZKUdGd67OemZdbEBmX_VRZ66mJln6EtJeJdOWruXp_I72ldDaz9UZMvZJ7ySH; HSID=AbDchAUPI4DtlOcyV; APISID=xLgJV19gOJC9cCD_/ARvW0PuATO71HDPNc 
Connection keep-alive 
WebServer(Response)


 HTTP/1.1 200 OK 
Cache-Control no-cache, no-store, max-age=0, must-revalidate 
Pragma no-cache 
Expires Fri, 01 Jan 1990 00:00:00 GMT 
Date Wed, 20 Aug 2014 11:39:22 GMT 
Content-Type text/javascript 
Content-Encoding gzip 
X-Content-Type-Options nosniff 
X-XSS-Protection 1; mode=block 
Content-Length 7090 
Server GSE 
Alternate-Protocol 80:quic 
HTMLUnit (First Request)


 GET /challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 HTTP/1.1 
Host api.recaptcha.net 
User-Agent Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0 
Accept-Language en-US,en;q=0.5 
Accept-Encoding gzip, deflate 
Referer http://www.patcurtis.com/captcha/recaptcha_demo.php 
Connection keep-alive 
Accept */* 
Cache-Control max-age=0 
WebServer(Response)



 HTTP/1.1 302 Moved Temporarily 
Cache-Control no-cache, no-store, max-age=0, must-revalidate 
Pragma no-cache 
Expires Fri, 01 Jan 1990 00:00:00 GMT 
Date Wed, 20 Aug 2014 18:22:58 GMT 
Location http://www.google.com/recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 
Content-Type text/html; charset=UTF-8 
Content-Encoding gzip 
X-Content-Type-Options nosniff 
X-XSS-Protection 1; mode=block 
Content-Length 227 
Server GSE 
Alternate-Protocol 80:quic 
HTMLUnit(Request)


 GET /recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 HTTP/1.1 
Host www.google.com 
User-Agent Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0 
Accept-Language en-US,en;q=0.5 
Cookie NID=67=cA1Qro6B-PbS07zWxNtD3iJ0hHfuc0itJYr1CVnyDCL4PbJNELomp3zFICR6NGdC0C3rQ440_1Y743dl87W5cTtWbAadzMGOZ0-1hbYTqYOSqZrSeN922y0XQr5IcKGE 
Accept-Encoding gzip, deflate 
Referer http://www.patcurtis.com/captcha/recaptcha_demo.php 
Connection keep-alive 
Accept */* 
Cache-Control max-age=0 
WebServer(Response)


 HTTP/1.1 200 OK 
Cache-Control no-cache, no-store, max-age=0, must-revalidate 
Pragma no-cache 
Expires Fri, 01 Jan 1990 00:00:00 GMT 
Date Wed, 20 Aug 2014 18:22:58 GMT 
Content-Type text/javascript 
Content-Encoding gzip 
X-Content-Type-Options nosniff 
X-XSS-Protection 1; mode=block 
Content-Length 6994 
Server GSE 
Alternate-Protocol 80:quic 
*******************************************************************

Kindly Let me know if you need any additional information. Please note that there are many request/response exchanges
after the above steps till we get the actual CAPTCHA image

Thanks for your help in advance.

Regards
 Vasu