Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

Hi Ahmed,

   Thanks for your reply.

    I had removed the cookie name NID which is used by Recaptcha service.
You would not be seeing that cookie
    in the browser request. There are other cookies for "/" domain is being
sent.

   Whereas HTMLUnit does not have those cookies to start with. Once I get
the NID cookie, it is being sent on subsequent
   requests.

    I hope my analysis is fine.

Regards
 Vasu

On Thu, Aug 21, 2014 at 5:32 PM, Ahmed Ashour <asa...@ya...> wrote:

> Hi Vasu,
>
>
> >>   As you see the headers are identical. But Recaptcha service throws
> error image after around 200 refreshes done from
>
> Well, they are not, there is a big cookie header in the second request of
> real browser, but HtmlUnit sends nothing.
>
>
> Cookie PREF=ID=27454a65a243cd70:U=508d4e3f9fa20406:FF=0:LD=en:TM=1406649235:LM=1407436763:GM=1:S=aCHb3CgmGRRcedMi;
> _ga=GA1.1.1138220839.1407426218; OGP=-4061129:;
> SID=DQAAAMkAAABEIRL4EB_u_34e9SPchv4-B1gv-ecyCxqE9yuqGcSCBx8fvByoEltivh8scqBkKd-g3_BS4QjJUAb54fr-Xl8SrIiSkJYVlD4dtJn2YB5JIgD7s_hCHT58eMV11jkFKy1NL3R91ibbv0inWCLlle59rF3VKaYOK1BhT_P0RFgbhttdSCq_sPccI_ejQe6Q4tem-CErtVlzQbnSq1idOr-XMoSHn6t0hzsRkrtFs7vzdShkayUlnnee5ruugolGON6gNN0gVQ2xY8XPdwYDBx_T;
> HSID=AvFUJATSD2LR-GOTz; APISID=_q3L-5Jckjza4xOL/AM5CzbFXam4moeOPb
>
>
> I guess this is a much higher possibility than the header order.
>
> Please investigate which request returns the response with that values of
> those cookies, and what are the corresponding HtmlUnit request/response.
>
> Ahmed
>
>   ------------------------------
>  *From:* Vasudevan Comandur <vco...@gm...>
> *To:* Ahmed Ashour <asa...@ya...>; "
> htm...@li..." <htm...@li...>
>
> *Sent:* Thursday, August 21, 2014 11:46 AM
>
> *Subject:* Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google
>
> Hi Ahmed,
>
>    Copying the header exchange between HTMLUnit and Recaptcha server.
>
>    HTMLUnit (First Request)
>
> GET /challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 HTTP/1.1Host
> api.recaptcha.netUser-AgentMozilla/5.0 (Windows NT 5.1; rv:31.0)
> Gecko/20100101 Firefox/31.0Accept-Languageen-US,en;q=0.5Accept-Encodinggzip,
> deflateRefererhttp://www.patcurtis.com/captcha/recaptcha_demo.php
> Connectionkeep-aliveAccept*/*
>   WebServer (Response)
>
> HTTP/1.1 302 Moved TemporarilyCache-Controlno-cache, no-store, max-age=0,
> must-revalidatePragmano-cacheExpiresFri, 01 Jan 1990 00:00:00 GMTDateThu,
> 21 Aug 2014 09:39:10 GMTLocation
> http://www.google.com/recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83
> Content-Typetext/html; charset=UTF-8Content-Encodinggzip
> X-Content-Type-OptionsnosniffX-XSS-Protection1; mode=blockContent-Length
> 227ServerGSEAlternate-Protocol80:quic
>   HTMLUnit(Request)
>
>
> GET /recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83
> HTTP/1.1Hostwww.google.comUser-AgentMozilla/5.0 (Windows NT 5.1; rv:31.0)
> Gecko/20100101 Firefox/31.0Accept-Languageen-US,en;q=0.5Accept-Encodinggzip,
> deflateRefererhttp://www.patcurtis.com/captcha/recaptcha_demo.php
> Connectionkeep-aliveAccept*/*
>   WebServer(Response)
>
> HTTP/1.1 200 OKCache-Controlno-cache, no-store, max-age=0, must-revalidate
> Pragmano-cacheExpiresFri, 01 Jan 1990 00:00:00 GMTDateThu, 21 Aug 2014
> 09:39:14 GMTSet-Cookie
> NID=67=Q5SsGkHHzte18yk1v8PHHpUlEMpWHhLKW1XJtIq9gql_ma_oLvAJ0W88OPBtn8JfDdLzeb7egYW1T8CyHiBuMviGYlRa7nh-wdFTVFCQqk0umJgym5WximGdBhfby_K5;Domain=.
> google.com;Path=/;Expires=Fri, 20-Feb-2015 09:39:14 GMT;HttpOnlyP3PCP="This
> is not a P3P policy! See
> http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657
> for more info."Content-Typetext/javascriptContent-Encodinggzip
> X-Content-Type-OptionsnosniffX-XSS-Protection1; mode=blockContent-Length
> 7048ServerGSEAlternate-Protocol80:quic
>   As you see the headers are identical. But Recaptcha service throws error
> image after around 200 refreshes done from
>   the browser.
>
>   Let me know if you need any additional information.
>
>    Thanks for your help in advance
>
> Regards
>  Vasu
>
>
> On Thu, Aug 21, 2014 at 12:10 PM, Ahmed Ashour <asa...@ya...> wrote:
>
> Hi Vasu,
>
> There are two differences:
>     - Cookies in the second request of real browser, they seem to be
> stored from previous requests. Please clear cache and rerun the comparison.
> I also wonder from where did HtmlUnit send the cookies in the second
> request.
>     - 'Accept' header location: this could be a reason, but Cookie are the
> much more the possibility.
>
> Ahmed
>   ------------------------------
>  *From:* Vasudevan Comandur <vco...@gm...>
> *To:* Ahmed Ashour <asa...@ya...>; "
> htm...@li..." <htm...@li...>
>
> *Sent:* Wednesday, August 20, 2014 9:06 PM
> *Subject:* Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google
>
> Hi Ahmed,
>
>     I am giving you the request details for REAL Browser & HTMLUnit below.
>
> REal Browser (First Request)
>
>
> GET /challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 HTTP/1.1Host
> api.recaptcha.netUser-AgentMozilla/5.0 (Windows NT 5.1; rv:31.0)
> Gecko/20100101 Firefox/31.0Accept*/*Accept-Languageen-US,en;q=0.5
> Accept-Encodinggzip, deflateReferer
> http://www.patcurtis.com/captcha/recaptcha_demo.phpConnectionkeep-alive
> Cache-Controlmax-age=0
> WebServer(Response)
>
> HTTP/1.1 302 Moved TemporarilyCache-Controlno-cache, no-store, max-age=0,
> must-revalidatePragmano-cacheExpiresFri, 01 Jan 1990 00:00:00 GMT DateWed,
> 20 Aug 2014 11:40:18 GMTLocation
> http://www.google.com/recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83
> Content-Typetext/html; charset=UTF-8Content-Encodinggzip
> X-Content-Type-OptionsnosniffX-XSS-Protection1; mode=blockContent-Length
> 227ServerGSEAlternate-Protocol80:quic
> RB(Request)
>
> GET /recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83
> HTTP/1.1Hostwww.google.comUser-AgentMozilla/5.0 (Windows NT 5.1; rv:31.0)
> Gecko/20100101 Firefox/31.0Accept*/*Accept-Languageen-US,en;q=0.5
> Accept-Encodinggzip, deflateReferer
> http://www.patcurtis.com/captcha/recaptcha_demo.phpCookiePREF=ID=27454a65a243cd70:U=508d4e3f9fa20406:FF=0:LD=en:TM=1406649235:LM=1407436763:GM=1:S=aCHb3CgmGRRcedMi;
> _ga=GA1.1.1138220839.1407426218; OGP=-4061129:;
> NID=67=pLb98eXX7XshplSC25O1hUrM6VPe1bG9epaYJREF4EBGoV-TPIQ0MynwuZdb1ra4i_j8jfzeKgHyJRL917tqVvJ1drL2BTn2QE4HaRblzxd0lpITPiswWjYNWO3YMn8g0aV7bw3zFqdW9HrdT3h2iJxykpn7QTCJm7fFHZoSYVRR1zWL_jNpSHvs7u912M5K1skdqY-lGtU;
> SID=DQAAAMkAAAD1nK0mZZoRhWbUjiHM6nCXAWouJuka0s3NW3l9AO0Tp7vLrszmHqYEeOI2mbDn6Ew2DuOvg0DxTEDPsjQ_-lMPWoXn_8TQYCEpFyvdZUdqdYT1qb2jNjv7cb6192vCkFAjY4LPJo5tKDUvm8_UDsJGrrILWzwGEf9TMkyo8uMP_rZcL3JFbVWfsN7W8j2MZWRLlSoZ-wcpW8PiQc_se5XSVXoZKUdGd67OemZdbEBmX_VRZ66mJln6EtJeJdOWruXp_I72ldDaz9UZMvZJ7ySH;
> HSID=AbDchAUPI4DtlOcyV; APISID=xLgJV19gOJC9cCD_/ARvW0PuATO71HDPNc
> Connectionkeep-alive
> WebServer(Response)
>
> HTTP/1.1 200 OKCache-Controlno-cache, no-store, max-age=0, must-revalidate
> Pragmano-cacheExpiresFri, 01 Jan 1990 00:00:00 GMT DateWed, 20 Aug 2014
> 11:39:22 GMTContent-Typetext/javascriptContent-Encodinggzip
> X-Content-Type-OptionsnosniffX-XSS-Protection 1; mode=blockContent-Length
> 7090ServerGSE Alternate-Protocol80:quic
> HTMLUnit (First Request)
>
> GET /challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 HTTP/1.1Host
> api.recaptcha.netUser-AgentMozilla/5.0 (Windows NT 5.1; rv:31.0)
> Gecko/20100101 Firefox/31.0Accept-Languageen-US,en;q=0.5Accept-Encodinggzip,
> deflate Refererhttp://www.patcurtis.com/captcha/recaptcha_demo.php
> Connectionkeep-aliveAccept*/*Cache-Control max-age=0
> WebServer(Response)
>
>
> HTTP/1.1 302 Moved TemporarilyCache-Controlno-cache, no-store, max-age=0,
> must-revalidatePragmano-cacheExpiresFri, 01 Jan 1990 00:00:00 GMT DateWed,
> 20 Aug 2014 18:22:58 GMTLocation
> http://www.google.com/recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83
> Content-Typetext/html; charset=UTF-8Content-Encodinggzip
> X-Content-Type-OptionsnosniffX-XSS-Protection1; mode=blockContent-Length
> 227ServerGSEAlternate-Protocol80:quic
> HTMLUnit(Request)
>
> GET /recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83
> HTTP/1.1Hostwww.google.comUser-AgentMozilla/5.0 (Windows NT 5.1; rv:31.0)
> Gecko/20100101 Firefox/31.0Accept-Languageen-US,en;q=0.5Cookie
> NID=67=cA1Qro6B-PbS07zWxNtD3iJ0hHfuc0itJYr1CVnyDCL4PbJNELomp3zFICR6NGdC0C3rQ440_1Y743dl87W5cTtWbAadzMGOZ0-1hbYTqYOSqZrSeN922y0XQr5IcKGE
> Accept-Encodinggzip, deflateReferer
> http://www.patcurtis.com/captcha/recaptcha_demo.phpConnectionkeep-alive
> Accept*/*Cache-Controlmax-age=0
> WebServer(Response)
>
> HTTP/1.1 200 OKCache-Controlno-cache, no-store, max-age=0, must-revalidate
> Pragmano-cacheExpiresFri, 01 Jan 1990 00:00:00 GMT DateWed, 20 Aug 2014
> 18:22:58 GMTContent-Typetext/javascriptContent-Encodinggzip
> X-Content-Type-OptionsnosniffX-XSS-Protection 1; mode=blockContent-Length
> 6994ServerGSE Alternate-Protocol80:quic
> *******************************************************************
>
> Kindly Let me know if you need any additional information. Please note
> that there are many request/response exchanges
> after the above steps till we get the actual CAPTCHA image
>
> Thanks for your help in advance.
>
> Regards
>  Vasu
>
>
>
> ------------------------------------------------------------------------------
> Slashdot TV.
> Video for Nerds.  Stuff that matters.
> http://tv.slashdot.org/
> _______________________________________________
> Htmlunit-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>
>
>
>
>
>
> ------------------------------------------------------------------------------
> Slashdot TV.
> Video for Nerds.  Stuff that matters.
> http://tv.slashdot.org/
> _______________________________________________
> Htmlunit-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>
>

Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google

Java GUI-Less browser, supporting JavaScript, to run against web pages

Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google