htmlunit-user — Discussion of the use of HtmlUnit

Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google

[Ahmed A. <asa...@ya...>]

Hi Vasu,

Build #2824 was made 23/08/2014 22:23:10, which is two days back.

Currently the build is running after few failed ones "now building since 08/25/2014 15:30:06"

Ahmed

________________________________
 From: Vasudevan Comandur <vco...@gm...>
To: Ahmed Ashour <asa...@ya...> 
Cc: Html Unit Users <htm...@li...> 
Sent: Monday, August 25, 2014 4:10 PM
Subject: Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google
 


Hi Ahmed,

   Has the build been successful?. I looked into the file Httpwebconnection.java in the build # 2824 and I did not see
   any code for referer and cookie. 

   Have you made the changes in some other file?. Please confirm.

Regards
 Vasu




On Mon, Aug 25, 2014 at 2:25 PM, Ahmed Ashour <asa...@ya...> wrote:

Hi Vasu,
>
>
>Please always communicate through the user-list.
>
>
>Referer and Cookies are now fixed in SVN.
>
>
>Thanks,
>Ahmed
>
>________________________________
> 
>From: Vasudevan Comandur <vco...@gm...>
>To: Ahmed Ashour <asa...@ya...> 
>Sent: Friday, August 22, 2014 2:31 PM
>
>Subject: Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google
>
>
>
>Hi Ahmed,
>
>    How about Referer and Cookie headers?. Will it come after Connection?
>
>    But when I looked into the headers of FF, Connection is coming after Referer & Cookies. Copying the header for your 
>    reference.
>
>
>
> GET /recaptcha/api/reload?c=03AHJ_VuuDDJeKrmGjlz1qmHvOCg24MbFS8gIBET5ZEzoT9x6WArdJvsS081P9buyCrsG1V6sqIk18N1Dfmrx3CnkhJ6jWu1glZwILtg5GZDHQEZphmxiRv3_6bkOME83Ex2FeQ-KhgWyFUut1tAKS4lsve_EJwDy_b02fyGfoW77iFFxHK4_gUPGbyLUV5R56wfEJLtDK7Y7AyBkyiXKN3vlqBOiahdNP3OR9804dExS2zvm2CJR1_InMAZETw9xhCQiviGeZNKqc-bC9oiqUNl1MTRkNnZ5M5bdCGjen9oprPziN6Q3Uf2NRm_jp2CQJDRburqzVBKL_3fA1TcPiSEHG-F0Vt9Ka8g&k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83&reason=t&type=image&lang=en&th=,8bBAlXaKcbT6sLautLQmRqgA5vAAAABAoAAAACPYAMvWdozw4osKbFVJ1AqPclG0Mn06Z0JwwRajuOfW-9LduuzuoI2mWRjA33V033hmvp-6if1HJBQNP4Xosnkci6mAKwBGfcnnwCc7-Kl0iIZrrNIuIBxQGc42xY-obi9gQZ8m5pgQ5cJFRHGEwCGIUg7FH8986Rw3_AKT3N2wfujfNtmrv8NTchi9UDkiPJGb9b_6qFdDD3EfBM0VCjY6T3xN7GydIpTI4MnWa-iA3nWq5YAaIFJCuzH47VPNdFIHCgst-KNwwcK0kSIPQQ HTTP/1.1 
>Host www.google.com 
>User-Agent Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0 
>Accept */* 
>Accept-Language en-US,en;q=0.5 
>Accept-Encoding gzip, deflate 
>Referer http://www.patcurtis.com/captcha/recaptcha_demo.php 
>Cookie NID=67=UTuSYqnmGItItcnyg5Jg3dGLTxCNQIHEk0cQp6gUvl9lYhkHjTeqvSc1mowP1LvSmf_uSRvRb6PMmifh9gagVFuKJDS160nr6MdZzMFXPfJoAtl7ZpA53su_izSPPIW3D0hGnAyV97SSUKPAyvNnP8bkovl9CdmcN2TIPE2_LBkaEomtrbuRIKs4JNju_aEdgfKvDclCMw; SID=DQAAAPAAAAB5aQmSV5SJw0dntY7eIMRMjEFwqScqktfLB64G0TkCiNKGDxF23waqqXSuzUwuLOY_mC4n6QHLz3STedw42FWwsDCP10d7i3to2E5HrLe9SIn8E0W6mjclWwUp-pBHgOdZVqRGNHAXkraQg3P5bLznaEXL05lSGG7H3q7_lU9KIzRxlbOxmjwu4T0WCXm4O2dfL7qcZ0t8AhDEChNfyobvoloa11DxeyGeWCqEms_E1hYZNH8w4fqXVsaqFkrolO9VsC8Yyykg4CRnjMf9R4MhYBVXA-Y2KQri4WXgfsnLrpg0ebFWxKXt7cxPY3x-6-M; HSID=AGpI2QsnbQ1VCYYfW; APISID=652VROec_KUf4hKf/ABJNvaUeifvJVbTVk; PREF=ID=aafd265723a3ae2d:FF=0:LD=en:TM=1408702050:LM=1408702050:GM=1:S=y6H_PRt6O_BZFVb9; OGP=-4061129: 
>Connection keep-alive 
>
> Looking forward to your comments.
>
>Regards
> Vasu
>
>
>
>
>
>
>
>On Fri, Aug 22, 2014 at 4:28 PM, Ahmed Ashour <asa...@ya...> wrote:
>
>Hi Vasu,
>>
>>
>>HttpWebConnection as well.
>>
>>Ahmed
>>
>>
>>
>>On Aug 22, 2014, at 12:47 PM, Vasudevan Comandur <vco...@gm...> wrote:
>>
>>
>>Hi Ahmed,
>>> 
>>>   Thanks for the quick reply. 
>>>
>>>   Have you made the changes in only one file (BrowserVersion.Java) or in few other files as well?
>>>
>>>
   The reason is that if I have apply this fix to an older version of HTMLUnit, I could do that by myself.
>>>
>>>   Looking forward to your reply.
>>>
>>>Regards
>>> Vasu
>>>
>>>
>>>
>>>
>>>On Fri, Aug 22, 2014 at 4:08 PM, Ahmed Ashour <asa...@ya...> wrote:
>>>
>>>Hi Vasu,
>>>>
>>>>
>>>>Yes, the build usually takes one hour, and the team is monitoring it.
>>>>
>>>>
>>>>The artifacts are of the last successful build.
>>>>
>>>>
>>>>Ahmed
>>>>
>>>>________________________________
>>>> 
>>>>From: Vasudevan Comandur <vco...@gm...>
>>>>To: Ahmed Ashour <asa...@ya...>; "htm...@li..." <htm...@li...> 
>>>>Sent: Friday, August 22, 2014 12:35 PM
>>>>
>>>>Subject: Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google
>>>>
>>>>
>>>>
>>>>Hi Ahmed,
>>>>
>>>>   Thanks. At present, it says the build has failed. So I need to wait till the build is successful to get the latest.
>>>>
>>>>   Am I correct?
>>>>
>>>>Regards
>>>> Vasu
>>>>
>>>>
>>>>
>>>>
>>>>On Fri, Aug 22, 2014 at 3:54 PM, Ahmed Ashour <asa...@ya...> wrote:
>>>>
>>>>
>>>>>
>>>>>
>>>>>Hi Vasu,
>>>>>
>>>>>
>>>>>- HtmlUnit homepage: http://htmlunit.sourceforge.net/
>>>>>- Latest build: Cruise Control server
>>>>>- When we have a successful build: 'Build Artifacts', the artifacts as-of-now were built before the commit.
>>>>>
>>>>>
>>>>>Ahmed
>>>>>
>>>>>________________________________
>>>>> 
>>>>>From: Vasudevan Comandur <vco...@gm...>
>>>>>To: Ahmed Ashour <asa...@ya...>; "htm...@li..." <htm...@li...> 
>>>>>Sent: Friday, August 22, 2014 12:16 PM
>>>>>
>>>>>Subject: Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google
>>>>>
>>>>>
>>>>>
>>>>>Hi Ahmed,
>>>>>
>>>>>   How can I get the source & the jar for the fix that you put now from SVN?.
>>>>>
>>>>>Regards
>>>>> Vasu
>>>>>
>>>>>
>>>>>
>>>>>------------------------------------------------------------------------------
>>>>>Slashdot TV.
>>>>>Video for Nerds.  Stuff that matters.
>>>>>http://tv.slashdot.org/
>>>>>_______________________________________________
>>>>>Htmlunit-user mailing list
>>>>>Htm...@li...
>>>>>https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>
>
>

Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google

[Vasudevan C. <vco...@gm...>]

Hi Ahmed,

   Has the build been successful?. I looked into the file
Httpwebconnection.java in the build # 2824 and I did not see
   any code for referer and cookie.

   Have you made the changes in some other file?. Please confirm.

Regards
 Vasu


On Mon, Aug 25, 2014 at 2:25 PM, Ahmed Ashour <asa...@ya...> wrote:

> Hi Vasu,
>
> Please always communicate through the user-list.
>
> Referer and Cookies are now fixed in SVN.
>
> Thanks,
> Ahmed
>   ------------------------------
>  *From:* Vasudevan Comandur <vco...@gm...>
> *To:* Ahmed Ashour <asa...@ya...>
> *Sent:* Friday, August 22, 2014 2:31 PM
>
> *Subject:* Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google
>
> Hi Ahmed,
>
>     How about Referer and Cookie headers?. Will it come after Connection?
>
>     But when I looked into the headers of FF, Connection is coming after
> Referer & Cookies. Copying the header for your
>     reference.
>
>
> GET
> /recaptcha/api/reload?c=03AHJ_VuuDDJeKrmGjlz1qmHvOCg24MbFS8gIBET5ZEzoT9x6WArdJvsS081P9buyCrsG1V6sqIk18N1Dfmrx3CnkhJ6jWu1glZwILtg5GZDHQEZphmxiRv3_6bkOME83Ex2FeQ-KhgWyFUut1tAKS4lsve_EJwDy_b02fyGfoW77iFFxHK4_gUPGbyLUV5R56wfEJLtDK7Y7AyBkyiXKN3vlqBOiahdNP3OR9804dExS2zvm2CJR1_InMAZETw9xhCQiviGeZNKqc-bC9oiqUNl1MTRkNnZ5M5bdCGjen9oprPziN6Q3Uf2NRm_jp2CQJDRburqzVBKL_3fA1TcPiSEHG-F0Vt9Ka8g&k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83&reason=t&type=image&lang=en&th=,8bBAlXaKcbT6sLautLQmRqgA5vAAAABAoAAAACPYAMvWdozw4osKbFVJ1AqPclG0Mn06Z0JwwRajuOfW-9LduuzuoI2mWRjA33V033hmvp-6if1HJBQNP4Xosnkci6mAKwBGfcnnwCc7-Kl0iIZrrNIuIBxQGc42xY-obi9gQZ8m5pgQ5cJFRHGEwCGIUg7FH8986Rw3_AKT3N2wfujfNtmrv8NTchi9UDkiPJGb9b_6qFdDD3EfBM0VCjY6T3xN7GydIpTI4MnWa-iA3nWq5YAaIFJCuzH47VPNdFIHCgst-KNwwcK0kSIPQQ
> HTTP/1.1Hostwww.google.comUser-AgentMozilla/5.0 (Windows NT 5.1; rv:31.0)
> Gecko/20100101 Firefox/31.0Accept*/*Accept-Languageen-US,en;q=0.5
> Accept-Encodinggzip, deflateReferer
> http://www.patcurtis.com/captcha/recaptcha_demo.phpCookieNID=67=UTuSYqnmGItItcnyg5Jg3dGLTxCNQIHEk0cQp6gUvl9lYhkHjTeqvSc1mowP1LvSmf_uSRvRb6PMmifh9gagVFuKJDS160nr6MdZzMFXPfJoAtl7ZpA53su_izSPPIW3D0hGnAyV97SSUKPAyvNnP8bkovl9CdmcN2TIPE2_LBkaEomtrbuRIKs4JNju_aEdgfKvDclCMw;
> SID=DQAAAPAAAAB5aQmSV5SJw0dntY7eIMRMjEFwqScqktfLB64G0TkCiNKGDxF23waqqXSuzUwuLOY_mC4n6QHLz3STedw42FWwsDCP10d7i3to2E5HrLe9SIn8E0W6mjclWwUp-pBHgOdZVqRGNHAXkraQg3P5bLznaEXL05lSGG7H3q7_lU9KIzRxlbOxmjwu4T0WCXm4O2dfL7qcZ0t8AhDEChNfyobvoloa11DxeyGeWCqEms_E1hYZNH8w4fqXVsaqFkrolO9VsC8Yyykg4CRnjMf9R4MhYBVXA-Y2KQri4WXgfsnLrpg0ebFWxKXt7cxPY3x-6-M;
> HSID=AGpI2QsnbQ1VCYYfW; APISID=652VROec_KUf4hKf/ABJNvaUeifvJVbTVk;
> PREF=ID=aafd265723a3ae2d:FF=0:LD=en:TM=1408702050:LM=1408702050:GM=1:S=y6H_PRt6O_BZFVb9;
> OGP=-4061129:Connectionkeep-alive
>
>  Looking forward to your comments.
>
> Regards
>  Vasu
>
>
>
>
> On Fri, Aug 22, 2014 at 4:28 PM, Ahmed Ashour <asa...@ya...> wrote:
>
> Hi Vasu,
>
> HttpWebConnection as well.
>
> Ahmed
>
>
> On Aug 22, 2014, at 12:47 PM, Vasudevan Comandur <vco...@gm...>
> wrote:
>
> Hi Ahmed,
>
>    Thanks for the quick reply.
>
>    Have you made the changes in only one file (BrowserVersion.Java) or in
> few other files as well?
>
>    The reason is that if I have apply this fix to an older version of
> HTMLUnit, I could do that by myself.
>
>    Looking forward to your reply.
>
> Regards
>  Vasu
>
>
> On Fri, Aug 22, 2014 at 4:08 PM, Ahmed Ashour <asa...@ya...> wrote:
>
> Hi Vasu,
>
> Yes, the build usually takes one hour, and the team is monitoring it.
>
> The artifacts are of the last successful build.
>
> Ahmed
>   ------------------------------
>  *From:* Vasudevan Comandur <vco...@gm...>
> *To:* Ahmed Ashour <asa...@ya...>; "
> htm...@li..." <htm...@li...>
>
> *Sent:* Friday, August 22, 2014 12:35 PM
>
> *Subject:* Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google
>
> Hi Ahmed,
>
>    Thanks. At present, it says the build has failed. So I need to wait
> till the build is successful to get the latest.
>
>    Am I correct?
>
> Regards
>  Vasu
>
>
> On Fri, Aug 22, 2014 at 3:54 PM, Ahmed Ashour <asa...@ya...> wrote:
>
>
>
> Hi Vasu,
>
> - HtmlUnit homepage: http://htmlunit.sourceforge.net/
> - Latest build: Cruise Control server <http://build.canoo.com/htmlunit>
> - When we have a successful build: 'Build Artifacts', the artifacts
> as-of-now were built before the commit.
>
> Ahmed
>   ------------------------------
>  *From:* Vasudevan Comandur <vco...@gm...>
> *To:* Ahmed Ashour <asa...@ya...>; "
> htm...@li..." <htm...@li...>
>
> *Sent:* Friday, August 22, 2014 12:16 PM
>
> *Subject:* Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google
>
> Hi Ahmed,
>
>    How can I get the source & the jar for the fix that you put now from
> SVN?.
>
> Regards
>  Vasu
>
>
>
> ------------------------------------------------------------------------------
> Slashdot TV.
> Video for Nerds.  Stuff that matters.
> http://tv.slashdot.org/
> _______________________________________________
> Htmlunit-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>
>
>
>
>
>
>
>
>

Re: [Htmlunit-user] Webclient consuming more time than loading time.

[Ahmed A. <asa...@ya...>]

Hi Thashi,

- Adding user-list
- HTML parsing as well, but this is a must.
- CSS parsing

Ahmed

________________________________
 From: dhashi88 <dha...@ho...>
To: Ahmed Ashour <asa...@ya...> 
Sent: Monday, August 25, 2014 11:09 AM
Subject: Re: [Htmlunit-user] Webclient consuming more time than loading time.
 


Hi Ahmed,
Is there any other factors in webclient wich consumes more execution time?
Regards, 
Thashi

Sent from my 4G Ready Samsung Galaxy S4 


-------- Original message --------
From: Ahmed Ashour 
Date:25/08/2014 2:29 PM (GMT+05:30) 


To: htm...@li... 
Subject: Re: [Htmlunit-user] Webclient consuming more time than loading time. 

Hi,

I guess the time to execute the JavaScript.

Ahmed

________________________________
 From: dhashi88 <dha...@ho...>
To: htm...@li... 
Sent: Monday, August 25, 2014 10:15 AM
Subject: [Htmlunit-user] Webclient consuming more time than loading time.



Im using HtmlUnit to grab content of web page (having javascript)
   Stopwatch timer = new Stopwatch().start();
   final WebClient webClient = newWebClient(BrowserVersion.CHROME);
   webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
   webClient.getOptions().setJavaScriptEnabled(true);
   webClient.getOptions().setActiveXNative(false);
   webClient.getOptions().setAppletEnabled(false);
   webClient.getOptions().setCssEnabled(false);
   webClient.getOptions().setDoNotTrackEnabled(true);
   webClient.getOptions().setGeolocationEnabled(false);
   webClient.getOptions().setPopupBlockerEnabled(true);
   webClient.getOptions().setPrintContentOnFailingStatusCode(false);
   webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
   webClient.getOptions().setThrowExceptionOnScriptError(false);
   webClient.getOptions().setUseInsecureSSL(true);
   webClient.setCssErrorHandler(newSilentCssErrorHandler());
   webClient.getCookieManager().setCookiesEnabled(false);
   webClient.getOptions().setRedirectEnabled(false);
   webClient.getOptions().setTimeout(900);
 
   System.out.println("1.0 : " + timer.elapsed(TimeUnit.MILLISECONDS));
   final HtmlPage page = webClient.getPage(url);
  System.out.println(page.getWebResponse().getLoadTime());
   System.out.println("1.1 : " + timer.elapsed(TimeUnit.MILLISECONDS));
then the result i got is like this
   1.0 : 2
   707
   1.1 : 8003
Why webclient consuming more time (more than webresponse load time)? how can i reduce that time?
------------------------------------------------------------------------------
Slashdot TV.  
Video for Nerds.  Stuff that matters.
http://tv.slashdot.org/
_______________________________________________
Htmlunit-user mailing list
Htm...@li...
https://lists.sourceforge.net/lists/listinfo/htmlunit-user




------------------------------------------------------------------------------
Slashdot TV.  
Video for Nerds.  Stuff that matters.
http://tv.slashdot.org/

_______________________________________________
Htmlunit-user mailing list
Htm...@li...
https://lists.sourceforge.net/lists/listinfo/htmlunit-user

Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google

[Vasudevan C. <vco...@gm...>]

Hi Ahmed,

   THanks for the fix. I just replied to your mail without realizing that
it comes to you only and not to
   htmlunit users list.

   I have CCed to htmlunit user list as well.

   I will test it out and let you know.

Regards
 Vasu


On Mon, Aug 25, 2014 at 2:25 PM, Ahmed Ashour <asa...@ya...> wrote:

> Hi Vasu,
>
> Please always communicate through the user-list.
>
> Referer and Cookies are now fixed in SVN.
>
> Thanks,
> Ahmed
>   ------------------------------
>  *From:* Vasudevan Comandur <vco...@gm...>
> *To:* Ahmed Ashour <asa...@ya...>
> *Sent:* Friday, August 22, 2014 2:31 PM
>
> *Subject:* Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google
>
> Hi Ahmed,
>
>     How about Referer and Cookie headers?. Will it come after Connection?
>
>     But when I looked into the headers of FF, Connection is coming after
> Referer & Cookies. Copying the header for your
>     reference.
>
>
> GET
> /recaptcha/api/reload?c=03AHJ_VuuDDJeKrmGjlz1qmHvOCg24MbFS8gIBET5ZEzoT9x6WArdJvsS081P9buyCrsG1V6sqIk18N1Dfmrx3CnkhJ6jWu1glZwILtg5GZDHQEZphmxiRv3_6bkOME83Ex2FeQ-KhgWyFUut1tAKS4lsve_EJwDy_b02fyGfoW77iFFxHK4_gUPGbyLUV5R56wfEJLtDK7Y7AyBkyiXKN3vlqBOiahdNP3OR9804dExS2zvm2CJR1_InMAZETw9xhCQiviGeZNKqc-bC9oiqUNl1MTRkNnZ5M5bdCGjen9oprPziN6Q3Uf2NRm_jp2CQJDRburqzVBKL_3fA1TcPiSEHG-F0Vt9Ka8g&k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83&reason=t&type=image&lang=en&th=,8bBAlXaKcbT6sLautLQmRqgA5vAAAABAoAAAACPYAMvWdozw4osKbFVJ1AqPclG0Mn06Z0JwwRajuOfW-9LduuzuoI2mWRjA33V033hmvp-6if1HJBQNP4Xosnkci6mAKwBGfcnnwCc7-Kl0iIZrrNIuIBxQGc42xY-obi9gQZ8m5pgQ5cJFRHGEwCGIUg7FH8986Rw3_AKT3N2wfujfNtmrv8NTchi9UDkiPJGb9b_6qFdDD3EfBM0VCjY6T3xN7GydIpTI4MnWa-iA3nWq5YAaIFJCuzH47VPNdFIHCgst-KNwwcK0kSIPQQ
> HTTP/1.1Hostwww.google.comUser-AgentMozilla/5.0 (Windows NT 5.1; rv:31.0)
> Gecko/20100101 Firefox/31.0Accept*/*Accept-Languageen-US,en;q=0.5
> Accept-Encodinggzip, deflateReferer
> http://www.patcurtis.com/captcha/recaptcha_demo.phpCookieNID=67=UTuSYqnmGItItcnyg5Jg3dGLTxCNQIHEk0cQp6gUvl9lYhkHjTeqvSc1mowP1LvSmf_uSRvRb6PMmifh9gagVFuKJDS160nr6MdZzMFXPfJoAtl7ZpA53su_izSPPIW3D0hGnAyV97SSUKPAyvNnP8bkovl9CdmcN2TIPE2_LBkaEomtrbuRIKs4JNju_aEdgfKvDclCMw;
> SID=DQAAAPAAAAB5aQmSV5SJw0dntY7eIMRMjEFwqScqktfLB64G0TkCiNKGDxF23waqqXSuzUwuLOY_mC4n6QHLz3STedw42FWwsDCP10d7i3to2E5HrLe9SIn8E0W6mjclWwUp-pBHgOdZVqRGNHAXkraQg3P5bLznaEXL05lSGG7H3q7_lU9KIzRxlbOxmjwu4T0WCXm4O2dfL7qcZ0t8AhDEChNfyobvoloa11DxeyGeWCqEms_E1hYZNH8w4fqXVsaqFkrolO9VsC8Yyykg4CRnjMf9R4MhYBVXA-Y2KQri4WXgfsnLrpg0ebFWxKXt7cxPY3x-6-M;
> HSID=AGpI2QsnbQ1VCYYfW; APISID=652VROec_KUf4hKf/ABJNvaUeifvJVbTVk;
> PREF=ID=aafd265723a3ae2d:FF=0:LD=en:TM=1408702050:LM=1408702050:GM=1:S=y6H_PRt6O_BZFVb9;
> OGP=-4061129:Connectionkeep-alive
>
>  Looking forward to your comments.
>
> Regards
>  Vasu
>
>
>
>
> On Fri, Aug 22, 2014 at 4:28 PM, Ahmed Ashour <asa...@ya...> wrote:
>
> Hi Vasu,
>
> HttpWebConnection as well.
>
> Ahmed
>
>
> On Aug 22, 2014, at 12:47 PM, Vasudevan Comandur <vco...@gm...>
> wrote:
>
> Hi Ahmed,
>
>    Thanks for the quick reply.
>
>    Have you made the changes in only one file (BrowserVersion.Java) or in
> few other files as well?
>
>    The reason is that if I have apply this fix to an older version of
> HTMLUnit, I could do that by myself.
>
>    Looking forward to your reply.
>
> Regards
>  Vasu
>
>
> On Fri, Aug 22, 2014 at 4:08 PM, Ahmed Ashour <asa...@ya...> wrote:
>
> Hi Vasu,
>
> Yes, the build usually takes one hour, and the team is monitoring it.
>
> The artifacts are of the last successful build.
>
> Ahmed
>   ------------------------------
>  *From:* Vasudevan Comandur <vco...@gm...>
> *To:* Ahmed Ashour <asa...@ya...>; "
> htm...@li..." <htm...@li...>
>
> *Sent:* Friday, August 22, 2014 12:35 PM
>
> *Subject:* Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google
>
> Hi Ahmed,
>
>    Thanks. At present, it says the build has failed. So I need to wait
> till the build is successful to get the latest.
>
>    Am I correct?
>
> Regards
>  Vasu
>
>
> On Fri, Aug 22, 2014 at 3:54 PM, Ahmed Ashour <asa...@ya...> wrote:
>
>
>
> Hi Vasu,
>
> - HtmlUnit homepage: http://htmlunit.sourceforge.net/
> - Latest build: Cruise Control server <http://build.canoo.com/htmlunit>
> - When we have a successful build: 'Build Artifacts', the artifacts
> as-of-now were built before the commit.
>
> Ahmed
>   ------------------------------
>  *From:* Vasudevan Comandur <vco...@gm...>
> *To:* Ahmed Ashour <asa...@ya...>; "
> htm...@li..." <htm...@li...>
>
> *Sent:* Friday, August 22, 2014 12:16 PM
>
> *Subject:* Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google
>
> Hi Ahmed,
>
>    How can I get the source & the jar for the fix that you put now from
> SVN?.
>
> Regards
>  Vasu
>
>
>
> ------------------------------------------------------------------------------
> Slashdot TV.
> Video for Nerds.  Stuff that matters.
> http://tv.slashdot.org/
> _______________________________________________
> Htmlunit-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>
>
>
>
>
>
>
>
>

Re: [Htmlunit-user] Webclient consuming more time than loading time.

[Ahmed A. <asa...@ya...>]

Hi,

I guess the time to execute the JavaScript.

Ahmed

________________________________
 From: dhashi88 <dha...@ho...>
To: htm...@li... 
Sent: Monday, August 25, 2014 10:15 AM
Subject: [Htmlunit-user] Webclient consuming more time than loading time.
 


Im using HtmlUnit to grab content of web page (having javascript)
   Stopwatch timer = new Stopwatch().start();
   final WebClient webClient = newWebClient(BrowserVersion.CHROME);
   webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
   webClient.getOptions().setJavaScriptEnabled(true);
   webClient.getOptions().setActiveXNative(false);
   webClient.getOptions().setAppletEnabled(false);
   webClient.getOptions().setCssEnabled(false);
   webClient.getOptions().setDoNotTrackEnabled(true);
   webClient.getOptions().setGeolocationEnabled(false);
   webClient.getOptions().setPopupBlockerEnabled(true);
   webClient.getOptions().setPrintContentOnFailingStatusCode(false);
   webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
   webClient.getOptions().setThrowExceptionOnScriptError(false);
   webClient.getOptions().setUseInsecureSSL(true);
   webClient.setCssErrorHandler(newSilentCssErrorHandler());
   webClient.getCookieManager().setCookiesEnabled(false);
   webClient.getOptions().setRedirectEnabled(false);
   webClient.getOptions().setTimeout(900);
 
   System.out.println("1.0 : " + timer.elapsed(TimeUnit.MILLISECONDS));
   final HtmlPage page = webClient.getPage(url);
  System.out.println(page.getWebResponse().getLoadTime());
   System.out.println("1.1 : " + timer.elapsed(TimeUnit.MILLISECONDS));
then the result i got is like this
   1.0 : 2
   707
   1.1 : 8003
Why webclient consuming more time (more than webresponse load time)? how can i reduce that time?
------------------------------------------------------------------------------
Slashdot TV.  
Video for Nerds.  Stuff that matters.
http://tv.slashdot.org/
_______________________________________________
Htmlunit-user mailing list
Htm...@li...
https://lists.sourceforge.net/lists/listinfo/htmlunit-user

Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google

[Ahmed A. <asa...@ya...>]

Hi Vasu,

Please always communicate through the user-list.

Referer and Cookies are now fixed in SVN.

Thanks,
Ahmed

________________________________
 From: Vasudevan Comandur <vco...@gm...>
To: Ahmed Ashour <asa...@ya...> 
Sent: Friday, August 22, 2014 2:31 PM
Subject: Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google
 


Hi Ahmed,

    How about Referer and Cookie headers?. Will it come after Connection?

    But when I looked into the headers of FF, Connection is coming after Referer & Cookies. Copying the header for your 
    reference.



 GET /recaptcha/api/reload?c=03AHJ_VuuDDJeKrmGjlz1qmHvOCg24MbFS8gIBET5ZEzoT9x6WArdJvsS081P9buyCrsG1V6sqIk18N1Dfmrx3CnkhJ6jWu1glZwILtg5GZDHQEZphmxiRv3_6bkOME83Ex2FeQ-KhgWyFUut1tAKS4lsve_EJwDy_b02fyGfoW77iFFxHK4_gUPGbyLUV5R56wfEJLtDK7Y7AyBkyiXKN3vlqBOiahdNP3OR9804dExS2zvm2CJR1_InMAZETw9xhCQiviGeZNKqc-bC9oiqUNl1MTRkNnZ5M5bdCGjen9oprPziN6Q3Uf2NRm_jp2CQJDRburqzVBKL_3fA1TcPiSEHG-F0Vt9Ka8g&k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83&reason=t&type=image&lang=en&th=,8bBAlXaKcbT6sLautLQmRqgA5vAAAABAoAAAACPYAMvWdozw4osKbFVJ1AqPclG0Mn06Z0JwwRajuOfW-9LduuzuoI2mWRjA33V033hmvp-6if1HJBQNP4Xosnkci6mAKwBGfcnnwCc7-Kl0iIZrrNIuIBxQGc42xY-obi9gQZ8m5pgQ5cJFRHGEwCGIUg7FH8986Rw3_AKT3N2wfujfNtmrv8NTchi9UDkiPJGb9b_6qFdDD3EfBM0VCjY6T3xN7GydIpTI4MnWa-iA3nWq5YAaIFJCuzH47VPNdFIHCgst-KNwwcK0kSIPQQ HTTP/1.1 
Host www.google.com 
User-Agent Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0 
Accept */* 
Accept-Language en-US,en;q=0.5 
Accept-Encoding gzip, deflate 
Referer http://www.patcurtis.com/captcha/recaptcha_demo.php 
Cookie NID=67=UTuSYqnmGItItcnyg5Jg3dGLTxCNQIHEk0cQp6gUvl9lYhkHjTeqvSc1mowP1LvSmf_uSRvRb6PMmifh9gagVFuKJDS160nr6MdZzMFXPfJoAtl7ZpA53su_izSPPIW3D0hGnAyV97SSUKPAyvNnP8bkovl9CdmcN2TIPE2_LBkaEomtrbuRIKs4JNju_aEdgfKvDclCMw; SID=DQAAAPAAAAB5aQmSV5SJw0dntY7eIMRMjEFwqScqktfLB64G0TkCiNKGDxF23waqqXSuzUwuLOY_mC4n6QHLz3STedw42FWwsDCP10d7i3to2E5HrLe9SIn8E0W6mjclWwUp-pBHgOdZVqRGNHAXkraQg3P5bLznaEXL05lSGG7H3q7_lU9KIzRxlbOxmjwu4T0WCXm4O2dfL7qcZ0t8AhDEChNfyobvoloa11DxeyGeWCqEms_E1hYZNH8w4fqXVsaqFkrolO9VsC8Yyykg4CRnjMf9R4MhYBVXA-Y2KQri4WXgfsnLrpg0ebFWxKXt7cxPY3x-6-M; HSID=AGpI2QsnbQ1VCYYfW; APISID=652VROec_KUf4hKf/ABJNvaUeifvJVbTVk; PREF=ID=aafd265723a3ae2d:FF=0:LD=en:TM=1408702050:LM=1408702050:GM=1:S=y6H_PRt6O_BZFVb9; OGP=-4061129: 
Connection keep-alive 

 Looking forward to your comments.

Regards
 Vasu






On Fri, Aug 22, 2014 at 4:28 PM, Ahmed Ashour <asa...@ya...> wrote:

Hi Vasu,
>
>
>HttpWebConnection as well.
>
>Ahmed
>
>
>
>On Aug 22, 2014, at 12:47 PM, Vasudevan Comandur <vco...@gm...> wrote:
>
>
>Hi Ahmed,
>> 
>>   Thanks for the quick reply. 
>>
>>   Have you made the changes in only one file (BrowserVersion.Java) or in few other files as well?
>>
>>
   The reason is that if I have apply this fix to an older version of HTMLUnit, I could do that by myself.
>>
>>   Looking forward to your reply.
>>
>>Regards
>> Vasu
>>
>>
>>
>>
>>On Fri, Aug 22, 2014 at 4:08 PM, Ahmed Ashour <asa...@ya...> wrote:
>>
>>Hi Vasu,
>>>
>>>
>>>Yes, the build usually takes one hour, and the team is monitoring it.
>>>
>>>
>>>The artifacts are of the last successful build.
>>>
>>>
>>>Ahmed
>>>
>>>________________________________
>>> 
>>>From: Vasudevan Comandur <vco...@gm...>
>>>To: Ahmed Ashour <asa...@ya...>; "htm...@li..." <htm...@li...> 
>>>Sent: Friday, August 22, 2014 12:35 PM
>>>
>>>Subject: Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google
>>>
>>>
>>>
>>>Hi Ahmed,
>>>
>>>   Thanks. At present, it says the build has failed. So I need to wait till the build is successful to get the latest.
>>>
>>>   Am I correct?
>>>
>>>Regards
>>> Vasu
>>>
>>>
>>>
>>>
>>>On Fri, Aug 22, 2014 at 3:54 PM, Ahmed Ashour <asa...@ya...> wrote:
>>>
>>>
>>>>
>>>>
>>>>Hi Vasu,
>>>>
>>>>
>>>>- HtmlUnit homepage: http://htmlunit.sourceforge.net/
>>>>- Latest build: Cruise Control server
>>>>- When we have a successful build: 'Build Artifacts', the artifacts as-of-now were built before the commit.
>>>>
>>>>
>>>>Ahmed
>>>>
>>>>________________________________
>>>> 
>>>>From: Vasudevan Comandur <vco...@gm...>
>>>>To: Ahmed Ashour <asa...@ya...>; "htm...@li..." <htm...@li...> 
>>>>Sent: Friday, August 22, 2014 12:16 PM
>>>>
>>>>Subject: Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google
>>>>
>>>>
>>>>
>>>>Hi Ahmed,
>>>>
>>>>   How can I get the source & the jar for the fix that you put now from SVN?.
>>>>
>>>>Regards
>>>> Vasu
>>>>
>>>>
>>>>
>>>>------------------------------------------------------------------------------
>>>>Slashdot TV.
>>>>Video for Nerds.  Stuff that matters.
>>>>http://tv.slashdot.org/
>>>>_______________________________________________
>>>>Htmlunit-user mailing list
>>>>Htm...@li...
>>>>https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>>>>
>>>>
>>>
>>>
>>>
>>

[Htmlunit-user] Webclient consuming more time than loading time.

[dhashi88 <dha...@ho...>]

Im using HtmlUnit to grab content of web page (having javascript)

   Stopwatch timer = new Stopwatch().start();

   final WebClient webClient = newWebClient(BrowserVersion.CHROME);

   webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);

   webClient.getOptions().setJavaScriptEnabled(true);

   webClient.getOptions().setActiveXNative(false);

   webClient.getOptions().setAppletEnabled(false);

   webClient.getOptions().setCssEnabled(false);

   webClient.getOptions().setDoNotTrackEnabled(true);

   webClient.getOptions().setGeolocationEnabled(false);

   webClient.getOptions().setPopupBlockerEnabled(true);

   webClient.getOptions().setPrintContentOnFailingStatusCode(false);

   webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);

   webClient.getOptions().setThrowExceptionOnScriptError(false);

   webClient.getOptions().setUseInsecureSSL(true);

   webClient.setCssErrorHandler(newSilentCssErrorHandler());

   webClient.getCookieManager().setCookiesEnabled(false);

   webClient.getOptions().setRedirectEnabled(false);

   webClient.getOptions().setTimeout(900);

 

   System.out.println("1.0 : " + timer.elapsed(TimeUnit.MILLISECONDS));

   final HtmlPage page = webClient.getPage(url);

  System.out.println(page.getWebResponse().getLoadTime());

   System.out.println("1.1 : " + timer.elapsed(TimeUnit.MILLISECONDS));

then the result i got is like this

   1.0 : 2

   707

   1.1 : 8003

Why webclient consuming more time (more than webresponse load time)? how can i reduce that time?

Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google

[Ahmed A. <asa...@ya...>]

Hi Vasu,

Yes, the build usually takes one hour, and the team is monitoring it.

The artifacts are of the last successful build.

Ahmed

________________________________
 From: Vasudevan Comandur <vco...@gm...>
To: Ahmed Ashour <asa...@ya...>; "htm...@li..." <htm...@li...> 
Sent: Friday, August 22, 2014 12:35 PM
Subject: Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google
 


Hi Ahmed,

   Thanks. At present, it says the build has failed. So I need to wait till the build is successful to get the latest.

   Am I correct?

Regards
 Vasu




On Fri, Aug 22, 2014 at 3:54 PM, Ahmed Ashour <asa...@ya...> wrote:


>
>
>Hi Vasu,
>
>
>- HtmlUnit homepage: http://htmlunit.sourceforge.net/
>- Latest build: Cruise Control server
>- When we have a successful build: 'Build Artifacts', the artifacts as-of-now were built before the commit.
>
>
>Ahmed
>
>________________________________
> 
>From: Vasudevan Comandur <vco...@gm...>
>To: Ahmed Ashour <asa...@ya...>; "htm...@li..." <htm...@li...> 
>Sent: Friday, August 22, 2014 12:16 PM
>
>Subject: Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google
>
>
>
>Hi Ahmed,
>
>   How can I get the source & the jar for the fix that you put now from SVN?.
>
>Regards
> Vasu
>
>
>
>------------------------------------------------------------------------------
>Slashdot TV.
>Video for Nerds.  Stuff that matters.
>http://tv.slashdot.org/
>_______________________________________________
>Htmlunit-user mailing list
>Htm...@li...
>https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>
>

Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google

[Vasudevan C. <vco...@gm...>]

Hi Ahmed,

   Thanks. At present, it says the build has failed. So I need to wait till
the build is successful to get the latest.

   Am I correct?

Regards
 Vasu


On Fri, Aug 22, 2014 at 3:54 PM, Ahmed Ashour <asa...@ya...> wrote:

> Hi Vasu,
>
> - HtmlUnit homepage: http://htmlunit.sourceforge.net/
> - Latest build: Cruise Control server <http://build.canoo.com/htmlunit>
> - When we have a successful build: 'Build Artifacts', the artifacts
> as-of-now were built before the commit.
>
> Ahmed
>   ------------------------------
>  *From:* Vasudevan Comandur <vco...@gm...>
> *To:* Ahmed Ashour <asa...@ya...>; "
> htm...@li..." <htm...@li...>
>
> *Sent:* Friday, August 22, 2014 12:16 PM
>
> *Subject:* Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google
>
> Hi Ahmed,
>
>    How can I get the source & the jar for the fix that you put now from
> SVN?.
>
> Regards
>  Vasu
>
>
>
> ------------------------------------------------------------------------------
> Slashdot TV.
> Video for Nerds.  Stuff that matters.
> http://tv.slashdot.org/
> _______________________________________________
> Htmlunit-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>
>

Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google

[Ahmed A. <asa...@ya...>]

Hi Vasu,

- HtmlUnit homepage: http://htmlunit.sourceforge.net/
- Latest build: Cruise Control server
- When we have a successful build: 'Build Artifacts', the artifacts as-of-now were built before the commit.

Ahmed

________________________________
 From: Vasudevan Comandur <vco...@gm...>
To: Ahmed Ashour <asa...@ya...>; "htm...@li..." <htm...@li...> 
Sent: Friday, August 22, 2014 12:16 PM
Subject: Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google
 


Hi Ahmed,

   How can I get the source & the jar for the fix that you put now from SVN?.

Regards
 Vasu

Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google

[Vasudevan C. <vco...@gm...>]

Hi Ahmed,

   How can I get the source & the jar for the fix that you put now from
SVN?.

Regards
 Vasu


On Fri, Aug 22, 2014 at 12:55 PM, Ahmed Ashour <asa...@ya...> wrote:

> Hi Vasu,
>
> The order was just fixed in SVN (specially for FF).
>
> Now you can exactly match the values and the order of all headers.
>
> Ahmed
>   ------------------------------
>  *From:* Ahmed Ashour <asa...@ya...>
> *To:* "htm...@li..." <
> htm...@li...>
> *Sent:* Thursday, August 21, 2014 3:40 PM
>
> *Subject:* Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google
>
> Hi Vasu,
>
> Then it goes to the header order.
>
> Stay tuned.
>
> Ahmed
>
>
> ------------------------------------------------------------------------------
> Slashdot TV.
> Video for Nerds.  Stuff that matters.
> http://tv.slashdot.org/
> _______________________________________________
> Htmlunit-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>
>

Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google

[Vasudevan C. <vco...@gm...>]

Hi Ahmed,

    Thanks for your fixes. Let me take the source from SVN and test it out
and give you the feedback.

     I appreciate your quick response on this.

     How about for other browsers?. Just curious to know.

Regards
 Vasu


On Fri, Aug 22, 2014 at 12:55 PM, Ahmed Ashour <asa...@ya...> wrote:

> Hi Vasu,
>
> The order was just fixed in SVN (specially for FF).
>
> Now you can exactly match the values and the order of all headers.
>
> Ahmed
>   ------------------------------
>  *From:* Ahmed Ashour <asa...@ya...>
> *To:* "htm...@li..." <
> htm...@li...>
> *Sent:* Thursday, August 21, 2014 3:40 PM
> *Subject:* Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google
>
> Hi Vasu,
>
> Then it goes to the header order.
>
> Stay tuned.
>
> Ahmed
>
>
> ------------------------------------------------------------------------------
> Slashdot TV.
> Video for Nerds.  Stuff that matters.
> http://tv.slashdot.org/
> _______________________________________________
> Htmlunit-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>
>

Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google

[Ahmed A. <asa...@ya...>]

Hi Vasu,

The order was just fixed in SVN (specially for FF).

Now you can exactly match the values and the order of all headers.

Ahmed

________________________________
 From: Ahmed Ashour <asa...@ya...>
To: "htm...@li..." <htm...@li...> 
Sent: Thursday, August 21, 2014 3:40 PM
Subject: Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google
 


Hi Vasu,

Then it goes to the header order.

Stay tuned.

Ahmed

Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google

[Ahmed A. <asa...@ya...>]

Hi Vasu,

Then it goes to the header order.

Stay tuned.

Ahmed

________________________________
 From: Vasudevan Comandur <vco...@gm...>
To: Ahmed Ashour <asa...@ya...>; "htm...@li..." <htm...@li...> 
Sent: Thursday, August 21, 2014 3:36 PM
Subject: Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google
 


Hi Ahmed,

    Given below is the headers from Real Browser after Cleaning all the cookies to Google DOmain.

   Real Browser (First Request)


 GET /challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 HTTP/1.1 
Host api.recaptcha.net 
User-Agent Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0 
Accept */* 
Accept-Language en-US,en;q=0.5 
Accept-Encoding gzip, deflate 
Referer http://www.patcurtis.com/captcha/recaptcha_demo.php 
Connection keep-alive 
  WebServer (REsponse)



 HTTP/1.1 302 Moved Temporarily 
Cache-Control no-cache, no-store, max-age=0, must-revalidate 
Pragma no-cache 
Expires Fri, 01 Jan 1990 00:00:00 GMT 
Date Thu, 21 Aug 2014 13:03:52 GMT 
Location http://www.google.com/recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 
Content-Type text/html; charset=UTF-8 
Content-Encoding gzip 
X-Content-Type-Options nosniff 
X-XSS-Protection 1; mode=block 
Content-Length 227 
Server GSE 
Alternate-Protocol 80:quic 
  Real Browser (Request)


 GET /recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 HTTP/1.1 
Host www.google.com 
User-Agent Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0 
Accept */* 
Accept-Language en-US,en;q=0.5 
Accept-Encoding gzip, deflate 
Referer http://www.patcurtis.com/captcha/recaptcha_demo.php 
Connection keep-alive 
  WebServer (Response)



 HTTP/1.1 200 OK 
Cache-Control no-cache, no-store, max-age=0, must-revalidate 
Pragma no-cache 
Expires Fri, 01 Jan 1990 00:00:00 GMT 
Date Thu, 21 Aug 2014 13:03:52 GMT 
Set-Cookie NID=67=A_R74sa5dgTU0ms8LNjCLzfOwICle-mGKh64KGxg-NW37H5t3OLdmeB738prGT8f4UHk_dTG4erieysRFx0mT-RIlgZRY-L02ZEOPyNw9FE9jSJ96Tga1AYcamWp4fdo;Domain=.google.com;Path=/;Expires=Fri, 20-Feb-2015 13:03:52 GMT;HttpOnly 
P3P CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info." 
Content-Type text/javascript 
Content-Encoding gzip 
X-Content-Type-Options nosniff 
X-XSS-Protection 1; mode=block 
Content-Length 7081 
Server GSE 
Alternate-Protocol 80:quic 
   With this I tried more than 350 times and there was no error image thrown by Google.

   I Hope this helps to solve the issue. 

REgards
 Vasu




On Thu, Aug 21, 2014 at 5:32 PM, Ahmed Ashour <asa...@ya...> wrote:

Hi Vasu,
>
>
>
>
>>>   As you see the headers are identical. But Recaptcha service throws error image after around 200 refreshes done from
>
>
>
>Well, they are not, there is a big cookie header in the second request of real browser, but HtmlUnit sends nothing.
>
>
>
>
>CookiePREF=ID=27454a65a243cd70:U=508d4e3f9fa20406:FF=0:LD=en:TM=1406649235:LM=1407436763:GM=1:S=aCHb3CgmGRRcedMi; _ga=GA1.1.1138220839.1407426218; OGP=-4061129:; SID=DQAAAMkAAABEIRL4EB_u_34e9SPchv4-B1gv-ecyCxqE9yuqGcSCBx8fvByoEltivh8scqBkKd-g3_BS4QjJUAb54fr-Xl8SrIiSkJYVlD4dtJn2YB5JIgD7s_hCHT58eMV11jkFKy1NL3R91ibbv0inWCLlle59rF3VKaYOK1BhT_P0RFgbhttdSCq_sPccI_ejQe6Q4tem-CErtVlzQbnSq1idOr-XMoSHn6t0hzsRkrtFs7vzdShkayUlnnee5ruugolGON6gNN0gVQ2xY8XPdwYDBx_T; HSID=AvFUJATSD2LR-GOTz; APISID=_q3L-5Jckjza4xOL/AM5CzbFXam4moeOPb
>
> 
>
>
>I guess this is a much higher possibility than the header order.
>
>
>Please investigate which request returns the response with that values of those cookies, and what are the corresponding HtmlUnit request/response.
>
>
>Ahmed
>
>
>
>
>________________________________
>
>

Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google

[Vasudevan C. <vco...@gm...>]

Hi Ahmed,

    Given below is the headers from Real Browser after Cleaning all the
cookies to Google DOmain.

   Real Browser (First Request)

GET /challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 HTTP/1.1  Host
api.recaptcha.net  User-Agent Mozilla/5.0 (Windows NT 5.1; rv:31.0)
Gecko/20100101 Firefox/31.0  Accept */*  Accept-Language en-US,en;q=0.5
Accept-Encoding gzip, deflate  Referer
http://www.patcurtis.com/captcha/recaptcha_demo.php  Connection keep-alive
  WebServer (REsponse)


HTTP/1.1 302 Moved Temporarily  Cache-Control no-cache, no-store,
max-age=0, must-revalidate  Pragma no-cache  Expires Fri, 01 Jan 1990
00:00:00 GMT  Date Thu, 21 Aug 2014 13:03:52 GMT  Location
http://www.google.com/recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83
Content-Type text/html; charset=UTF-8  Content-Encoding gzip
X-Content-Type-Options nosniff  X-XSS-Protection 1; mode=block
Content-Length 227  Server GSE  Alternate-Protocol 80:quic
  Real Browser (Request)

GET /recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83
HTTP/1.1  Host www.google.com  User-Agent Mozilla/5.0 (Windows NT 5.1;
rv:31.0) Gecko/20100101 Firefox/31.0  Accept */*  Accept-Language
en-US,en;q=0.5  Accept-Encoding gzip, deflate  Referer
http://www.patcurtis.com/captcha/recaptcha_demo.php  Connection keep-alive
  WebServer (Response)


HTTP/1.1 200 OK  Cache-Control no-cache, no-store, max-age=0,
must-revalidate  Pragma no-cache  Expires Fri, 01 Jan 1990 00:00:00 GMT
Date Thu, 21 Aug 2014 13:03:52 GMT  Set-Cookie
NID=67=A_R74sa5dgTU0ms8LNjCLzfOwICle-mGKh64KGxg-NW37H5t3OLdmeB738prGT8f4UHk_dTG4erieysRFx0mT-RIlgZRY-L02ZEOPyNw9FE9jSJ96Tga1AYcamWp4fdo;Domain=.
google.com;Path=/;Expires=Fri, 20-Feb-2015 13:03:52 GMT;HttpOnly  P3P CP="This
is not a P3P policy! See
http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657
for more info."  Content-Type text/javascript  Content-Encoding gzip
X-Content-Type-Options nosniff  X-XSS-Protection 1; mode=block
Content-Length 7081  Server GSE  Alternate-Protocol 80:quic
   With this I tried more than 350 times and there was no error image
thrown by Google.

   I Hope this helps to solve the issue.

REgards
 Vasu


On Thu, Aug 21, 2014 at 5:32 PM, Ahmed Ashour <asa...@ya...> wrote:

> Hi Vasu,
>
>
> >>   As you see the headers are identical. But Recaptcha service throws
> error image after around 200 refreshes done from
>
> Well, they are not, there is a big cookie header in the second request of
> real browser, but HtmlUnit sends nothing.
>
>
> Cookie PREF=ID=27454a65a243cd70:U=508d4e3f9fa20406:FF=0:LD=en:TM=1406649235:LM=1407436763:GM=1:S=aCHb3CgmGRRcedMi;
> _ga=GA1.1.1138220839.1407426218; OGP=-4061129:;
> SID=DQAAAMkAAABEIRL4EB_u_34e9SPchv4-B1gv-ecyCxqE9yuqGcSCBx8fvByoEltivh8scqBkKd-g3_BS4QjJUAb54fr-Xl8SrIiSkJYVlD4dtJn2YB5JIgD7s_hCHT58eMV11jkFKy1NL3R91ibbv0inWCLlle59rF3VKaYOK1BhT_P0RFgbhttdSCq_sPccI_ejQe6Q4tem-CErtVlzQbnSq1idOr-XMoSHn6t0hzsRkrtFs7vzdShkayUlnnee5ruugolGON6gNN0gVQ2xY8XPdwYDBx_T;
> HSID=AvFUJATSD2LR-GOTz; APISID=_q3L-5Jckjza4xOL/AM5CzbFXam4moeOPb
>
>
> I guess this is a much higher possibility than the header order.
>
> Please investigate which request returns the response with that values of
> those cookies, and what are the corresponding HtmlUnit request/response.
>
> Ahmed
>
>   ------------------------------
>
>

Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google

[Vasudevan C. <vco...@gm...>]

Hi Ahmed,

   Thanks for your reply.

    I had removed the cookie name NID which is used by Recaptcha service.
You would not be seeing that cookie
    in the browser request. There are other cookies for "/" domain is being
sent.

   Whereas HTMLUnit does not have those cookies to start with. Once I get
the NID cookie, it is being sent on subsequent
   requests.

    I hope my analysis is fine.

Regards
 Vasu


On Thu, Aug 21, 2014 at 5:32 PM, Ahmed Ashour <asa...@ya...> wrote:

> Hi Vasu,
>
>
> >>   As you see the headers are identical. But Recaptcha service throws
> error image after around 200 refreshes done from
>
> Well, they are not, there is a big cookie header in the second request of
> real browser, but HtmlUnit sends nothing.
>
>
> Cookie PREF=ID=27454a65a243cd70:U=508d4e3f9fa20406:FF=0:LD=en:TM=1406649235:LM=1407436763:GM=1:S=aCHb3CgmGRRcedMi;
> _ga=GA1.1.1138220839.1407426218; OGP=-4061129:;
> SID=DQAAAMkAAABEIRL4EB_u_34e9SPchv4-B1gv-ecyCxqE9yuqGcSCBx8fvByoEltivh8scqBkKd-g3_BS4QjJUAb54fr-Xl8SrIiSkJYVlD4dtJn2YB5JIgD7s_hCHT58eMV11jkFKy1NL3R91ibbv0inWCLlle59rF3VKaYOK1BhT_P0RFgbhttdSCq_sPccI_ejQe6Q4tem-CErtVlzQbnSq1idOr-XMoSHn6t0hzsRkrtFs7vzdShkayUlnnee5ruugolGON6gNN0gVQ2xY8XPdwYDBx_T;
> HSID=AvFUJATSD2LR-GOTz; APISID=_q3L-5Jckjza4xOL/AM5CzbFXam4moeOPb
>
>
> I guess this is a much higher possibility than the header order.
>
> Please investigate which request returns the response with that values of
> those cookies, and what are the corresponding HtmlUnit request/response.
>
> Ahmed
>
>   ------------------------------
>
>

Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google

[Vasudevan C. <vco...@gm...>]

Hi Ahmed,

   Thanks for your reply.

    I had removed the cookie name NID which is used by Recaptcha service.
You would not be seeing that cookie
    in the browser request. There are other cookies for "/" domain is being
sent.

   Whereas HTMLUnit does not have those cookies to start with. Once I get
the NID cookie, it is being sent on subsequent
   requests.

    I hope my analysis is fine.

Regards
 Vasu


On Thu, Aug 21, 2014 at 5:32 PM, Ahmed Ashour <asa...@ya...> wrote:

> Hi Vasu,
>
>
> >>   As you see the headers are identical. But Recaptcha service throws
> error image after around 200 refreshes done from
>
> Well, they are not, there is a big cookie header in the second request of
> real browser, but HtmlUnit sends nothing.
>
>
> Cookie PREF=ID=27454a65a243cd70:U=508d4e3f9fa20406:FF=0:LD=en:TM=1406649235:LM=1407436763:GM=1:S=aCHb3CgmGRRcedMi;
> _ga=GA1.1.1138220839.1407426218; OGP=-4061129:;
> SID=DQAAAMkAAABEIRL4EB_u_34e9SPchv4-B1gv-ecyCxqE9yuqGcSCBx8fvByoEltivh8scqBkKd-g3_BS4QjJUAb54fr-Xl8SrIiSkJYVlD4dtJn2YB5JIgD7s_hCHT58eMV11jkFKy1NL3R91ibbv0inWCLlle59rF3VKaYOK1BhT_P0RFgbhttdSCq_sPccI_ejQe6Q4tem-CErtVlzQbnSq1idOr-XMoSHn6t0hzsRkrtFs7vzdShkayUlnnee5ruugolGON6gNN0gVQ2xY8XPdwYDBx_T;
> HSID=AvFUJATSD2LR-GOTz; APISID=_q3L-5Jckjza4xOL/AM5CzbFXam4moeOPb
>
>
> I guess this is a much higher possibility than the header order.
>
> Please investigate which request returns the response with that values of
> those cookies, and what are the corresponding HtmlUnit request/response.
>
> Ahmed
>
>   ------------------------------
>  *From:* Vasudevan Comandur <vco...@gm...>
> *To:* Ahmed Ashour <asa...@ya...>; "
> htm...@li..." <htm...@li...>
>
> *Sent:* Thursday, August 21, 2014 11:46 AM
>
> *Subject:* Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google
>
> Hi Ahmed,
>
>    Copying the header exchange between HTMLUnit and Recaptcha server.
>
>    HTMLUnit (First Request)
>
> GET /challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 HTTP/1.1Host
> api.recaptcha.netUser-AgentMozilla/5.0 (Windows NT 5.1; rv:31.0)
> Gecko/20100101 Firefox/31.0Accept-Languageen-US,en;q=0.5Accept-Encodinggzip,
> deflateRefererhttp://www.patcurtis.com/captcha/recaptcha_demo.php
> Connectionkeep-aliveAccept*/*
>   WebServer (Response)
>
> HTTP/1.1 302 Moved TemporarilyCache-Controlno-cache, no-store, max-age=0,
> must-revalidatePragmano-cacheExpiresFri, 01 Jan 1990 00:00:00 GMTDateThu,
> 21 Aug 2014 09:39:10 GMTLocation
> http://www.google.com/recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83
> Content-Typetext/html; charset=UTF-8Content-Encodinggzip
> X-Content-Type-OptionsnosniffX-XSS-Protection1; mode=blockContent-Length
> 227ServerGSEAlternate-Protocol80:quic
>   HTMLUnit(Request)
>
>
> GET /recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83
> HTTP/1.1Hostwww.google.comUser-AgentMozilla/5.0 (Windows NT 5.1; rv:31.0)
> Gecko/20100101 Firefox/31.0Accept-Languageen-US,en;q=0.5Accept-Encodinggzip,
> deflateRefererhttp://www.patcurtis.com/captcha/recaptcha_demo.php
> Connectionkeep-aliveAccept*/*
>   WebServer(Response)
>
> HTTP/1.1 200 OKCache-Controlno-cache, no-store, max-age=0, must-revalidate
> Pragmano-cacheExpiresFri, 01 Jan 1990 00:00:00 GMTDateThu, 21 Aug 2014
> 09:39:14 GMTSet-Cookie
> NID=67=Q5SsGkHHzte18yk1v8PHHpUlEMpWHhLKW1XJtIq9gql_ma_oLvAJ0W88OPBtn8JfDdLzeb7egYW1T8CyHiBuMviGYlRa7nh-wdFTVFCQqk0umJgym5WximGdBhfby_K5;Domain=.
> google.com;Path=/;Expires=Fri, 20-Feb-2015 09:39:14 GMT;HttpOnlyP3PCP="This
> is not a P3P policy! See
> http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657
> for more info."Content-Typetext/javascriptContent-Encodinggzip
> X-Content-Type-OptionsnosniffX-XSS-Protection1; mode=blockContent-Length
> 7048ServerGSEAlternate-Protocol80:quic
>   As you see the headers are identical. But Recaptcha service throws error
> image after around 200 refreshes done from
>   the browser.
>
>   Let me know if you need any additional information.
>
>    Thanks for your help in advance
>
> Regards
>  Vasu
>
>
> On Thu, Aug 21, 2014 at 12:10 PM, Ahmed Ashour <asa...@ya...> wrote:
>
> Hi Vasu,
>
> There are two differences:
>     - Cookies in the second request of real browser, they seem to be
> stored from previous requests. Please clear cache and rerun the comparison.
> I also wonder from where did HtmlUnit send the cookies in the second
> request.
>     - 'Accept' header location: this could be a reason, but Cookie are the
> much more the possibility.
>
> Ahmed
>   ------------------------------
>  *From:* Vasudevan Comandur <vco...@gm...>
> *To:* Ahmed Ashour <asa...@ya...>; "
> htm...@li..." <htm...@li...>
>
> *Sent:* Wednesday, August 20, 2014 9:06 PM
> *Subject:* Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google
>
> Hi Ahmed,
>
>     I am giving you the request details for REAL Browser & HTMLUnit below.
>
> REal Browser (First Request)
>
>
> GET /challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 HTTP/1.1Host
> api.recaptcha.netUser-AgentMozilla/5.0 (Windows NT 5.1; rv:31.0)
> Gecko/20100101 Firefox/31.0Accept*/*Accept-Languageen-US,en;q=0.5
> Accept-Encodinggzip, deflateReferer
> http://www.patcurtis.com/captcha/recaptcha_demo.phpConnectionkeep-alive
> Cache-Controlmax-age=0
> WebServer(Response)
>
> HTTP/1.1 302 Moved TemporarilyCache-Controlno-cache, no-store, max-age=0,
> must-revalidatePragmano-cacheExpiresFri, 01 Jan 1990 00:00:00 GMT DateWed,
> 20 Aug 2014 11:40:18 GMTLocation
> http://www.google.com/recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83
> Content-Typetext/html; charset=UTF-8Content-Encodinggzip
> X-Content-Type-OptionsnosniffX-XSS-Protection1; mode=blockContent-Length
> 227ServerGSEAlternate-Protocol80:quic
> RB(Request)
>
> GET /recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83
> HTTP/1.1Hostwww.google.comUser-AgentMozilla/5.0 (Windows NT 5.1; rv:31.0)
> Gecko/20100101 Firefox/31.0Accept*/*Accept-Languageen-US,en;q=0.5
> Accept-Encodinggzip, deflateReferer
> http://www.patcurtis.com/captcha/recaptcha_demo.phpCookiePREF=ID=27454a65a243cd70:U=508d4e3f9fa20406:FF=0:LD=en:TM=1406649235:LM=1407436763:GM=1:S=aCHb3CgmGRRcedMi;
> _ga=GA1.1.1138220839.1407426218; OGP=-4061129:;
> NID=67=pLb98eXX7XshplSC25O1hUrM6VPe1bG9epaYJREF4EBGoV-TPIQ0MynwuZdb1ra4i_j8jfzeKgHyJRL917tqVvJ1drL2BTn2QE4HaRblzxd0lpITPiswWjYNWO3YMn8g0aV7bw3zFqdW9HrdT3h2iJxykpn7QTCJm7fFHZoSYVRR1zWL_jNpSHvs7u912M5K1skdqY-lGtU;
> SID=DQAAAMkAAAD1nK0mZZoRhWbUjiHM6nCXAWouJuka0s3NW3l9AO0Tp7vLrszmHqYEeOI2mbDn6Ew2DuOvg0DxTEDPsjQ_-lMPWoXn_8TQYCEpFyvdZUdqdYT1qb2jNjv7cb6192vCkFAjY4LPJo5tKDUvm8_UDsJGrrILWzwGEf9TMkyo8uMP_rZcL3JFbVWfsN7W8j2MZWRLlSoZ-wcpW8PiQc_se5XSVXoZKUdGd67OemZdbEBmX_VRZ66mJln6EtJeJdOWruXp_I72ldDaz9UZMvZJ7ySH;
> HSID=AbDchAUPI4DtlOcyV; APISID=xLgJV19gOJC9cCD_/ARvW0PuATO71HDPNc
> Connectionkeep-alive
> WebServer(Response)
>
> HTTP/1.1 200 OKCache-Controlno-cache, no-store, max-age=0, must-revalidate
> Pragmano-cacheExpiresFri, 01 Jan 1990 00:00:00 GMT DateWed, 20 Aug 2014
> 11:39:22 GMTContent-Typetext/javascriptContent-Encodinggzip
> X-Content-Type-OptionsnosniffX-XSS-Protection 1; mode=blockContent-Length
> 7090ServerGSE Alternate-Protocol80:quic
> HTMLUnit (First Request)
>
> GET /challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 HTTP/1.1Host
> api.recaptcha.netUser-AgentMozilla/5.0 (Windows NT 5.1; rv:31.0)
> Gecko/20100101 Firefox/31.0Accept-Languageen-US,en;q=0.5Accept-Encodinggzip,
> deflate Refererhttp://www.patcurtis.com/captcha/recaptcha_demo.php
> Connectionkeep-aliveAccept*/*Cache-Control max-age=0
> WebServer(Response)
>
>
> HTTP/1.1 302 Moved TemporarilyCache-Controlno-cache, no-store, max-age=0,
> must-revalidatePragmano-cacheExpiresFri, 01 Jan 1990 00:00:00 GMT DateWed,
> 20 Aug 2014 18:22:58 GMTLocation
> http://www.google.com/recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83
> Content-Typetext/html; charset=UTF-8Content-Encodinggzip
> X-Content-Type-OptionsnosniffX-XSS-Protection1; mode=blockContent-Length
> 227ServerGSEAlternate-Protocol80:quic
> HTMLUnit(Request)
>
> GET /recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83
> HTTP/1.1Hostwww.google.comUser-AgentMozilla/5.0 (Windows NT 5.1; rv:31.0)
> Gecko/20100101 Firefox/31.0Accept-Languageen-US,en;q=0.5Cookie
> NID=67=cA1Qro6B-PbS07zWxNtD3iJ0hHfuc0itJYr1CVnyDCL4PbJNELomp3zFICR6NGdC0C3rQ440_1Y743dl87W5cTtWbAadzMGOZ0-1hbYTqYOSqZrSeN922y0XQr5IcKGE
> Accept-Encodinggzip, deflateReferer
> http://www.patcurtis.com/captcha/recaptcha_demo.phpConnectionkeep-alive
> Accept*/*Cache-Controlmax-age=0
> WebServer(Response)
>
> HTTP/1.1 200 OKCache-Controlno-cache, no-store, max-age=0, must-revalidate
> Pragmano-cacheExpiresFri, 01 Jan 1990 00:00:00 GMT DateWed, 20 Aug 2014
> 18:22:58 GMTContent-Typetext/javascriptContent-Encodinggzip
> X-Content-Type-OptionsnosniffX-XSS-Protection 1; mode=blockContent-Length
> 6994ServerGSE Alternate-Protocol80:quic
> *******************************************************************
>
> Kindly Let me know if you need any additional information. Please note
> that there are many request/response exchanges
> after the above steps till we get the actual CAPTCHA image
>
> Thanks for your help in advance.
>
> Regards
>  Vasu
>
>
>
> ------------------------------------------------------------------------------
> Slashdot TV.
> Video for Nerds.  Stuff that matters.
> http://tv.slashdot.org/
> _______________________________________________
> Htmlunit-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>
>
>
>
>
>
> ------------------------------------------------------------------------------
> Slashdot TV.
> Video for Nerds.  Stuff that matters.
> http://tv.slashdot.org/
> _______________________________________________
> Htmlunit-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>
>

Re: [Htmlunit-user] How to enable my (htmlunit) browser work as Flash enabled?

[Ahmed A. <asa...@ya...>]

Hi,

If the below doesn't work, then you need to investigate further in HtmlUnit code, which part that doesn't identity flash with your website.

Ahmed

    private static BrowserVersion firefox31WithUptoDateFlash = new BrowserVersion(
            BrowserVersion.FIREFOX_31.getApplicationName(),
            BrowserVersion.FIREFOX_31.getApplicationVersion(),
            BrowserVersion.FIREFOX_31.getUserAgent(),
            BrowserVersion.FIREFOX_31.getBrowserVersionNumeric(),
            new BrowserVersionFeatures[] {
            BrowserVersionFeatures.STYLESHEET_HREF_EXPANDURL,
            BrowserVersionFeatures.STYLESHEET_HREF_STYLE_NULL });
            static {
            PluginConfiguration plugin = new PluginConfiguration("Shockwave Flash",
            "Shockwave Flash 14.0 r0", "NPSWF32_14_0_0_145.dll");
            plugin.getMimeTypes().add(new PluginConfiguration.MimeType("application/x-shockwave-flash", "Adobe Flash movie","swf"));
            //plugin.getMimeTypes().add(new PluginConfiguration.MimeType("application/futuresplash", "FutureSplash movie", "spl"));
            firefox31WithUptoDateFlash.getPlugins().add(plugin);
            }


________________________________
 From: dhashi88 <dha...@ho...>
To: htm...@li... 
Sent: Thursday, August 21, 2014 9:41 AM
Subject: [Htmlunit-user] How to enable my (htmlunit) browser work as Flash	enabled?
 


i was able to do in HtmlUnit 2.13 version by
private static BrowserVersion firefox17WithUptoDateFlash = new BrowserVersion(
BrowserVersion.FIREFOX_17.getApplicationName(),
BrowserVersion.FIREFOX_17.getApplicationVersion(),
BrowserVersion.FIREFOX_17.getUserAgent(),
BrowserVersion.FIREFOX_17.getBrowserVersionNumeric(),
new BrowserVersionFeatures[] {
BrowserVersionFeatures.JS_FRAME_RESOLVE_URL_WITH_PARENT_WINDOW,
BrowserVersionFeatures.STYLESHEET_HREF_EXPANDURL,
BrowserVersionFeatures.STYLESHEET_HREF_STYLE_NULL });
static {
PluginConfiguration plugin = new PluginConfiguration("Shockwave Flash",
"Shockwave Flash 14.0 r0", "NPSWF32_14_0_0_145.dll");
plugin.getMimeTypes().add(new PluginConfiguration.MimeType("application/x-shockwave-flash", "Adobe Flash movie","swf"));
//plugin.getMimeTypes().add(new PluginConfiguration.MimeType("application/futuresplash", "FutureSplash movie", "spl"));
firefox17WithUptoDateFlash.getPlugins().add(plugin);
}
but same is not working for HtmlUnit 2.15 . what is the reason. How can i achieve this?
------------------------------------------------------------------------------
Slashdot TV.  
Video for Nerds.  Stuff that matters.
http://tv.slashdot.org/
_______________________________________________
Htmlunit-user mailing list
Htm...@li...
https://lists.sourceforge.net/lists/listinfo/htmlunit-user

Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google

[Ahmed A. <asa...@ya...>]

Hi Vasu,


>>   As you see the headers are identical. But Recaptcha service throws error image after around 200 refreshes done from


Well, they are not, there is a big cookie header in the second request of real browser, but HtmlUnit sends nothing.


CookiePREF=ID=27454a65a243cd70:U=508d4e3f9fa20406:FF=0:LD=en:TM=1406649235:LM=1407436763:GM=1:S=aCHb3CgmGRRcedMi; _ga=GA1.1.1138220839.1407426218; OGP=-4061129:; SID=DQAAAMkAAABEIRL4EB_u_34e9SPchv4-B1gv-ecyCxqE9yuqGcSCBx8fvByoEltivh8scqBkKd-g3_BS4QjJUAb54fr-Xl8SrIiSkJYVlD4dtJn2YB5JIgD7s_hCHT58eMV11jkFKy1NL3R91ibbv0inWCLlle59rF3VKaYOK1BhT_P0RFgbhttdSCq_sPccI_ejQe6Q4tem-CErtVlzQbnSq1idOr-XMoSHn6t0hzsRkrtFs7vzdShkayUlnnee5ruugolGON6gNN0gVQ2xY8XPdwYDBx_T; HSID=AvFUJATSD2LR-GOTz; APISID=_q3L-5Jckjza4xOL/AM5CzbFXam4moeOPb

 

I guess this is a much higher possibility than the header order.

Please investigate which request returns the response with that values of those cookies, and what are the corresponding HtmlUnit request/response.

Ahmed



________________________________
 From: Vasudevan Comandur <vco...@gm...>
To: Ahmed Ashour <asa...@ya...>; "htm...@li..." <htm...@li...> 
Sent: Thursday, August 21, 2014 11:46 AM
Subject: Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google
 


Hi Ahmed,

   Copying the header exchange between HTMLUnit and Recaptcha server.

   HTMLUnit (First Request)


 GET /challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 HTTP/1.1 
Host api.recaptcha.net 
User-Agent Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0 
Accept-Language en-US,en;q=0.5 
Accept-Encoding gzip, deflate 
Referer http://www.patcurtis.com/captcha/recaptcha_demo.php 
Connection keep-alive 
Accept */* 
  WebServer (Response)


 HTTP/1.1 302 Moved Temporarily 
Cache-Control no-cache, no-store, max-age=0, must-revalidate 
Pragma no-cache 
Expires Fri, 01 Jan 1990 00:00:00 GMT 
Date Thu, 21 Aug 2014 09:39:10 GMT 
Location http://www.google.com/recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 
Content-Type text/html; charset=UTF-8 
Content-Encoding gzip 
X-Content-Type-Options nosniff 
X-XSS-Protection 1; mode=block 
Content-Length 227 
Server GSE 
Alternate-Protocol 80:quic 
  HTMLUnit(Request)



 GET /recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 HTTP/1.1 
Host www.google.com 
User-Agent Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0 
Accept-Language en-US,en;q=0.5 
Accept-Encoding gzip, deflate 
Referer http://www.patcurtis.com/captcha/recaptcha_demo.php 
Connection keep-alive 
Accept */* 
  WebServer(Response)


 HTTP/1.1 200 OK 
Cache-Control no-cache, no-store, max-age=0, must-revalidate 
Pragma no-cache 
Expires Fri, 01 Jan 1990 00:00:00 GMT 
Date Thu, 21 Aug 2014 09:39:14 GMT 
Set-Cookie NID=67=Q5SsGkHHzte18yk1v8PHHpUlEMpWHhLKW1XJtIq9gql_ma_oLvAJ0W88OPBtn8JfDdLzeb7egYW1T8CyHiBuMviGYlRa7nh-wdFTVFCQqk0umJgym5WximGdBhfby_K5;Domain=.google.com;Path=/;Expires=Fri, 20-Feb-2015 09:39:14 GMT;HttpOnly 
P3P CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info." 
Content-Type text/javascript 
Content-Encoding gzip 
X-Content-Type-Options nosniff 
X-XSS-Protection 1; mode=block 
Content-Length 7048 
Server GSE 
Alternate-Protocol 80:quic 
  As you see the headers are identical. But Recaptcha service throws error image after around 200 refreshes done from
  the browser.

  Let me know if you need any additional information.


   Thanks for your help in advance

Regards
 Vasu




On Thu, Aug 21, 2014 at 12:10 PM, Ahmed Ashour <asa...@ya...> wrote:

Hi Vasu,
>
>
>There are two differences:
>    - Cookies in the second request of real browser, they seem to be stored from previous requests. Please clear cache and rerun the comparison. I also wonder from where did HtmlUnit send the cookies in the second request.
>    - 'Accept' header location: this could be a reason, but Cookie are the much more the possibility.
> 
>Ahmed
>
>________________________________
>
>From: Vasudevan Comandur <vco...@gm...>
>To: Ahmed Ashour <asa...@ya...>; "htm...@li..." <htm...@li...> 
>Sent: Wednesday, August 20, 2014 9:06 PM
>Subject: Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google
> 
>
>
>Hi Ahmed,
>
>    I am giving you the request details for REAL Browser & HTMLUnit below.
>
>REal Browser (First Request)
>
>
>
> GET /challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 HTTP/1.1 
>Host api.recaptcha.net 
>User-Agent Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0 
>Accept */* 
>Accept-Language en-US,en;q=0.5 
>Accept-Encoding gzip, deflate 
>Referer http://www.patcurtis.com/captcha/recaptcha_demo.php 
>Connection keep-alive 
>Cache-Control max-age=0 
>WebServer(Response)
>
>
> HTTP/1.1 302 Moved Temporarily 
>Cache-Control no-cache, no-store, max-age=0, must-revalidate 
>Pragma no-cache 
>Expires Fri, 01 Jan 1990 00:00:00 GMT 
>Date Wed, 20 Aug 2014 11:40:18 GMT 
>Location http://www.google.com/recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 
>Content-Type text/html; charset=UTF-8 
>Content-Encoding gzip 
>X-Content-Type-Options nosniff 
>X-XSS-Protection 1; mode=block 
>Content-Length 227 
>Server GSE 
>Alternate-Protocol 80:quic 
>RB(Request)
>
>
> GET /recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 HTTP/1.1 
>Host www.google.com 
>User-Agent Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0 
>Accept */* 
>Accept-Language en-US,en;q=0.5 
>Accept-Encoding gzip, deflate 
>Referer http://www.patcurtis.com/captcha/recaptcha_demo.php 
>Cookie PREF=ID=27454a65a243cd70:U=508d4e3f9fa20406:FF=0:LD=en:TM=1406649235:LM=1407436763:GM=1:S=aCHb3CgmGRRcedMi; _ga=GA1.1.1138220839.1407426218; OGP=-4061129:; NID=67=pLb98eXX7XshplSC25O1hUrM6VPe1bG9epaYJREF4EBGoV-TPIQ0MynwuZdb1ra4i_j8jfzeKgHyJRL917tqVvJ1drL2BTn2QE4HaRblzxd0lpITPiswWjYNWO3YMn8g0aV7bw3zFqdW9HrdT3h2iJxykpn7QTCJm7fFHZoSYVRR1zWL_jNpSHvs7u912M5K1skdqY-lGtU; SID=DQAAAMkAAAD1nK0mZZoRhWbUjiHM6nCXAWouJuka0s3NW3l9AO0Tp7vLrszmHqYEeOI2mbDn6Ew2DuOvg0DxTEDPsjQ_-lMPWoXn_8TQYCEpFyvdZUdqdYT1qb2jNjv7cb6192vCkFAjY4LPJo5tKDUvm8_UDsJGrrILWzwGEf9TMkyo8uMP_rZcL3JFbVWfsN7W8j2MZWRLlSoZ-wcpW8PiQc_se5XSVXoZKUdGd67OemZdbEBmX_VRZ66mJln6EtJeJdOWruXp_I72ldDaz9UZMvZJ7ySH; HSID=AbDchAUPI4DtlOcyV; APISID=xLgJV19gOJC9cCD_/ARvW0PuATO71HDPNc 
>Connection keep-alive 
>WebServer(Response)
>
>
> HTTP/1.1 200 OK 
>Cache-Control no-cache, no-store, max-age=0, must-revalidate 
>Pragma no-cache 
>Expires Fri, 01 Jan 1990 00:00:00 GMT 
>Date Wed, 20 Aug 2014 11:39:22 GMT 
>Content-Type text/javascript 
>Content-Encoding gzip 
>X-Content-Type-Options nosniff 
>X-XSS-Protection 1; mode=block 
>Content-Length 7090 
>Server GSE 
>Alternate-Protocol 80:quic 
>HTMLUnit (First Request)
>
>
> GET /challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 HTTP/1.1 
>Host api.recaptcha.net 
>User-Agent Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0 
>Accept-Language en-US,en;q=0.5 
>Accept-Encoding gzip, deflate 
>Referer http://www.patcurtis.com/captcha/recaptcha_demo.php 
>Connection keep-alive 
>Accept */* 
>Cache-Control max-age=0 
>WebServer(Response)
>
>
>
> HTTP/1.1 302 Moved Temporarily 
>Cache-Control no-cache, no-store, max-age=0, must-revalidate 
>Pragma no-cache 
>Expires Fri, 01 Jan 1990 00:00:00 GMT 
>Date Wed, 20 Aug 2014 18:22:58 GMT 
>Location http://www.google.com/recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 
>Content-Type text/html; charset=UTF-8 
>Content-Encoding gzip 
>X-Content-Type-Options nosniff 
>X-XSS-Protection 1; mode=block 
>Content-Length 227 
>Server GSE 
>Alternate-Protocol 80:quic 
>HTMLUnit(Request)
>
>
> GET /recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 HTTP/1.1 
>Host www.google.com 
>User-Agent Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0 
>Accept-Language en-US,en;q=0.5 
>Cookie NID=67=cA1Qro6B-PbS07zWxNtD3iJ0hHfuc0itJYr1CVnyDCL4PbJNELomp3zFICR6NGdC0C3rQ440_1Y743dl87W5cTtWbAadzMGOZ0-1hbYTqYOSqZrSeN922y0XQr5IcKGE 
>Accept-Encoding gzip, deflate 
>Referer http://www.patcurtis.com/captcha/recaptcha_demo.php 
>Connection keep-alive 
>Accept */* 
>Cache-Control max-age=0 
>WebServer(Response)
>
>
> HTTP/1.1 200 OK 
>Cache-Control no-cache, no-store, max-age=0, must-revalidate 
>Pragma no-cache 
>Expires Fri, 01 Jan 1990 00:00:00 GMT 
>Date Wed, 20 Aug 2014 18:22:58 GMT 
>Content-Type text/javascript 
>Content-Encoding gzip 
>X-Content-Type-Options nosniff 
>X-XSS-Protection 1; mode=block 
>Content-Length 6994 
>Server GSE 
>Alternate-Protocol 80:quic 
>*******************************************************************
>
>Kindly Let me know if you need any additional information. Please note that there are many request/response exchanges
>after the above steps till we get the actual CAPTCHA image
>
>Thanks for your help in advance.
>
>Regards
> Vasu
>
>
>
>------------------------------------------------------------------------------
>Slashdot TV.
>Video for Nerds.  Stuff that matters.
>http://tv.slashdot.org/
>_______________________________________________
>Htmlunit-user mailing list
>Htm...@li...
>https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>
>

Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google

[Vasudevan C. <vco...@gm...>]

Hi Ahmed,

   Copying the header exchange between HTMLUnit and Recaptcha server.

   HTMLUnit (First Request)

GET /challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 HTTP/1.1  Host
api.recaptcha.net  User-Agent Mozilla/5.0 (Windows NT 5.1; rv:31.0)
Gecko/20100101 Firefox/31.0  Accept-Language en-US,en;q=0.5
Accept-Encoding gzip,
deflate  Referer http://www.patcurtis.com/captcha/recaptcha_demo.php
Connection keep-alive  Accept */*
  WebServer (Response)

HTTP/1.1 302 Moved Temporarily  Cache-Control no-cache, no-store,
max-age=0, must-revalidate  Pragma no-cache  Expires Fri, 01 Jan 1990
00:00:00 GMT  Date Thu, 21 Aug 2014 09:39:10 GMT  Location
http://www.google.com/recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83
Content-Type text/html; charset=UTF-8  Content-Encoding gzip
X-Content-Type-Options nosniff  X-XSS-Protection 1; mode=block
Content-Length 227  Server GSE  Alternate-Protocol 80:quic
  HTMLUnit(Request)


GET /recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83
HTTP/1.1  Host www.google.com  User-Agent Mozilla/5.0 (Windows NT 5.1;
rv:31.0) Gecko/20100101 Firefox/31.0  Accept-Language en-US,en;q=0.5
Accept-Encoding gzip, deflate  Referer
http://www.patcurtis.com/captcha/recaptcha_demo.php  Connection keep-alive
Accept */*
  WebServer(Response)

HTTP/1.1 200 OK  Cache-Control no-cache, no-store, max-age=0,
must-revalidate  Pragma no-cache  Expires Fri, 01 Jan 1990 00:00:00 GMT
Date Thu, 21 Aug 2014 09:39:14 GMT  Set-Cookie
NID=67=Q5SsGkHHzte18yk1v8PHHpUlEMpWHhLKW1XJtIq9gql_ma_oLvAJ0W88OPBtn8JfDdLzeb7egYW1T8CyHiBuMviGYlRa7nh-wdFTVFCQqk0umJgym5WximGdBhfby_K5;Domain=.
google.com;Path=/;Expires=Fri, 20-Feb-2015 09:39:14 GMT;HttpOnly  P3P CP="This
is not a P3P policy! See
http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657
for more info."  Content-Type text/javascript  Content-Encoding gzip
X-Content-Type-Options nosniff  X-XSS-Protection 1; mode=block
Content-Length 7048  Server GSE  Alternate-Protocol 80:quic
  As you see the headers are identical. But Recaptcha service throws error
image after around 200 refreshes done from
  the browser.

  Let me know if you need any additional information.

   Thanks for your help in advance

Regards
 Vasu


On Thu, Aug 21, 2014 at 12:10 PM, Ahmed Ashour <asa...@ya...> wrote:

> Hi Vasu,
>
> There are two differences:
>     - Cookies in the second request of real browser, they seem to be
> stored from previous requests. Please clear cache and rerun the comparison.
> I also wonder from where did HtmlUnit send the cookies in the second
> request.
>     - 'Accept' header location: this could be a reason, but Cookie are the
> much more the possibility.
>
> Ahmed
>   ------------------------------
>  *From:* Vasudevan Comandur <vco...@gm...>
> *To:* Ahmed Ashour <asa...@ya...>; "
> htm...@li..." <htm...@li...>
>
> *Sent:* Wednesday, August 20, 2014 9:06 PM
> *Subject:* Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google
>
> Hi Ahmed,
>
>     I am giving you the request details for REAL Browser & HTMLUnit below.
>
> REal Browser (First Request)
>
>
> GET /challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 HTTP/1.1Host
> api.recaptcha.netUser-AgentMozilla/5.0 (Windows NT 5.1; rv:31.0)
> Gecko/20100101 Firefox/31.0Accept*/*Accept-Languageen-US,en;q=0.5
> Accept-Encodinggzip, deflateReferer
> http://www.patcurtis.com/captcha/recaptcha_demo.phpConnectionkeep-alive
> Cache-Controlmax-age=0
> WebServer(Response)
>
> HTTP/1.1 302 Moved TemporarilyCache-Controlno-cache, no-store, max-age=0,
> must-revalidatePragmano-cacheExpiresFri, 01 Jan 1990 00:00:00 GMTDateWed,
> 20 Aug 2014 11:40:18 GMTLocation
> http://www.google.com/recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83
> Content-Typetext/html; charset=UTF-8Content-Encodinggzip
> X-Content-Type-OptionsnosniffX-XSS-Protection1; mode=blockContent-Length
> 227ServerGSEAlternate-Protocol80:quic
> RB(Request)
>
> GET /recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83
> HTTP/1.1Hostwww.google.comUser-AgentMozilla/5.0 (Windows NT 5.1; rv:31.0)
> Gecko/20100101 Firefox/31.0Accept*/*Accept-Languageen-US,en;q=0.5
> Accept-Encodinggzip, deflateReferer
> http://www.patcurtis.com/captcha/recaptcha_demo.phpCookiePREF=ID=27454a65a243cd70:U=508d4e3f9fa20406:FF=0:LD=en:TM=1406649235:LM=1407436763:GM=1:S=aCHb3CgmGRRcedMi;
> _ga=GA1.1.1138220839.1407426218; OGP=-4061129:;
> NID=67=pLb98eXX7XshplSC25O1hUrM6VPe1bG9epaYJREF4EBGoV-TPIQ0MynwuZdb1ra4i_j8jfzeKgHyJRL917tqVvJ1drL2BTn2QE4HaRblzxd0lpITPiswWjYNWO3YMn8g0aV7bw3zFqdW9HrdT3h2iJxykpn7QTCJm7fFHZoSYVRR1zWL_jNpSHvs7u912M5K1skdqY-lGtU;
> SID=DQAAAMkAAAD1nK0mZZoRhWbUjiHM6nCXAWouJuka0s3NW3l9AO0Tp7vLrszmHqYEeOI2mbDn6Ew2DuOvg0DxTEDPsjQ_-lMPWoXn_8TQYCEpFyvdZUdqdYT1qb2jNjv7cb6192vCkFAjY4LPJo5tKDUvm8_UDsJGrrILWzwGEf9TMkyo8uMP_rZcL3JFbVWfsN7W8j2MZWRLlSoZ-wcpW8PiQc_se5XSVXoZKUdGd67OemZdbEBmX_VRZ66mJln6EtJeJdOWruXp_I72ldDaz9UZMvZJ7ySH;
> HSID=AbDchAUPI4DtlOcyV; APISID=xLgJV19gOJC9cCD_/ARvW0PuATO71HDPNc
> Connectionkeep-alive
> WebServer(Response)
>
> HTTP/1.1 200 OKCache-Controlno-cache, no-store, max-age=0, must-revalidate
> Pragmano-cacheExpiresFri, 01 Jan 1990 00:00:00 GMTDateWed, 20 Aug 2014
> 11:39:22 GMTContent-Typetext/javascriptContent-Encodinggzip
> X-Content-Type-OptionsnosniffX-XSS-Protection1; mode=blockContent-Length
> 7090ServerGSEAlternate-Protocol80:quic
> HTMLUnit (First Request)
>
> GET /challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 HTTP/1.1Host
> api.recaptcha.netUser-AgentMozilla/5.0 (Windows NT 5.1; rv:31.0)
> Gecko/20100101 Firefox/31.0Accept-Languageen-US,en;q=0.5Accept-Encodinggzip,
> deflateRefererhttp://www.patcurtis.com/captcha/recaptcha_demo.php
> Connectionkeep-aliveAccept*/*Cache-Controlmax-age=0
> WebServer(Response)
>
>
> HTTP/1.1 302 Moved TemporarilyCache-Controlno-cache, no-store, max-age=0,
> must-revalidatePragmano-cacheExpiresFri, 01 Jan 1990 00:00:00 GMTDateWed,
> 20 Aug 2014 18:22:58 GMTLocation
> http://www.google.com/recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83
> Content-Typetext/html; charset=UTF-8Content-Encodinggzip
> X-Content-Type-OptionsnosniffX-XSS-Protection1; mode=blockContent-Length
> 227ServerGSEAlternate-Protocol80:quic
> HTMLUnit(Request)
>
> GET /recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83
> HTTP/1.1Hostwww.google.comUser-AgentMozilla/5.0 (Windows NT 5.1; rv:31.0)
> Gecko/20100101 Firefox/31.0Accept-Languageen-US,en;q=0.5Cookie
> NID=67=cA1Qro6B-PbS07zWxNtD3iJ0hHfuc0itJYr1CVnyDCL4PbJNELomp3zFICR6NGdC0C3rQ440_1Y743dl87W5cTtWbAadzMGOZ0-1hbYTqYOSqZrSeN922y0XQr5IcKGE
> Accept-Encodinggzip, deflateReferer
> http://www.patcurtis.com/captcha/recaptcha_demo.phpConnectionkeep-alive
> Accept*/*Cache-Controlmax-age=0
> WebServer(Response)
>
> HTTP/1.1 200 OKCache-Controlno-cache, no-store, max-age=0, must-revalidate
> Pragmano-cacheExpiresFri, 01 Jan 1990 00:00:00 GMTDateWed, 20 Aug 2014
> 18:22:58 GMTContent-Typetext/javascriptContent-Encodinggzip
> X-Content-Type-OptionsnosniffX-XSS-Protection1; mode=blockContent-Length
> 6994ServerGSEAlternate-Protocol80:quic
> *******************************************************************
>
> Kindly Let me know if you need any additional information. Please note
> that there are many request/response exchanges
> after the above steps till we get the actual CAPTCHA image
>
> Thanks for your help in advance.
>
> Regards
>  Vasu
>
>
>
> ------------------------------------------------------------------------------
> Slashdot TV.
> Video for Nerds.  Stuff that matters.
> http://tv.slashdot.org/
> _______________________________________________
> Htmlunit-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>
>

Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google

[Vasudevan C. <vco...@gm...>]

Hi Ahmed,

   Copying below the HTTP header exchange with between Real Browser &
Recaptcha server of Google

   Browser (First Request)


GET /challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 HTTP/1.1  Host
api.recaptcha.net  User-Agent Mozilla/5.0 (Windows NT 5.1; rv:31.0)
Gecko/20100101 Firefox/31.0  Accept */*  Accept-Language en-US,en;q=0.5
Accept-Encoding gzip, deflate  Referer
http://www.patcurtis.com/captcha/recaptcha_demo.php  Connection keep-alive
  WebServer(Response)


HTTP/1.1 302 Moved Temporarily  Cache-Control no-cache, no-store,
max-age=0, must-revalidate  Pragma no-cache  Expires Fri, 01 Jan 1990
00:00:00 GMT  Date Thu, 21 Aug 2014 09:25:12 GMT  Location
http://www.google.com/recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83
Content-Type text/html; charset=UTF-8  Content-Encoding gzip
X-Content-Type-Options nosniff  X-XSS-Protection 1; mode=block
Content-Length 227  Server GSE  Alternate-Protocol 80:quic
Browser (Request)


GET /recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83
HTTP/1.1  Host www.google.com  User-Agent Mozilla/5.0 (Windows NT 5.1;
rv:31.0) Gecko/20100101 Firefox/31.0  Accept */*  Accept-Language
en-US,en;q=0.5  Accept-Encoding gzip, deflate  Referer
http://www.patcurtis.com/captcha/recaptcha_demo.php  Cookie
PREF=ID=27454a65a243cd70:U=508d4e3f9fa20406:FF=0:LD=en:TM=1406649235:LM=1407436763:GM=1:S=aCHb3CgmGRRcedMi;
_ga=GA1.1.1138220839.1407426218; OGP=-4061129:;
SID=DQAAAMkAAABEIRL4EB_u_34e9SPchv4-B1gv-ecyCxqE9yuqGcSCBx8fvByoEltivh8scqBkKd-g3_BS4QjJUAb54fr-Xl8SrIiSkJYVlD4dtJn2YB5JIgD7s_hCHT58eMV11jkFKy1NL3R91ibbv0inWCLlle59rF3VKaYOK1BhT_P0RFgbhttdSCq_sPccI_ejQe6Q4tem-CErtVlzQbnSq1idOr-XMoSHn6t0hzsRkrtFs7vzdShkayUlnnee5ruugolGON6gNN0gVQ2xY8XPdwYDBx_T;
HSID=AvFUJATSD2LR-GOTz; APISID=_q3L-5Jckjza4xOL/AM5CzbFXam4moeOPb
Connection keep-alive
  WebServer(Response)


HTTP/1.1 200 OK  Cache-Control no-cache, no-store, max-age=0,
must-revalidate  Pragma no-cache  Expires Fri, 01 Jan 1990 00:00:00 GMT
Date Thu, 21 Aug 2014 09:25:12 GMT  Set-Cookie
NID=67=cERW_-OHTpwGHJf-fiIRmBh2dLF6TRFLH2kPV6KndzHbzgI26c4jjkRV_FdtowrUp4cI44pCPlUXU5Ipsb3J985M7_cnzEr1-xZiMdZgXQTz4hmgzrfX9ugbd7IGDhed;Domain=.
google.com;Path=/;Expires=Fri, 20-Feb-2015 09:25:12 GMT;HttpOnly  P3P CP="This
is not a P3P policy! See
http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657
for more info."  Content-Type text/javascript  Content-Encoding gzip
X-Content-Type-Options nosniff  X-XSS-Protection 1; mode=block
Content-Length 7004  Server GSE  Alternate-Protocol 80:quic
  I will send you the HTTP Header exchange through HTMLUnit shortly.

Regards
 Vasu


On Thu, Aug 21, 2014 at 12:10 PM, Ahmed Ashour <asa...@ya...> wrote:

> Hi Vasu,
>
> There are two differences:
>     - Cookies in the second request of real browser, they seem to be
> stored from previous requests. Please clear cache and rerun the comparison.
> I also wonder from where did HtmlUnit send the cookies in the second
> request.
>     - 'Accept' header location: this could be a reason, but Cookie are the
> much more the possibility.
>
> Ahmed
>   ------------------------------
>  *From:* Vasudevan Comandur <vco...@gm...>
> *To:* Ahmed Ashour <asa...@ya...>; "
> htm...@li..." <htm...@li...>
>
> *Sent:* Wednesday, August 20, 2014 9:06 PM
> *Subject:* Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google
>
> Hi Ahmed,
>
>     I am giving you the request details for REAL Browser & HTMLUnit below.
>
> REal Browser (First Request)
>
>
> GET /challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 HTTP/1.1Host
> api.recaptcha.netUser-AgentMozilla/5.0 (Windows NT 5.1; rv:31.0)
> Gecko/20100101 Firefox/31.0 Accept*/*Accept-Languageen-US,en;q=0.5
> Accept-Encodinggzip, deflateReferer
> http://www.patcurtis.com/captcha/recaptcha_demo.php Connectionkeep-alive
> Cache-Controlmax-age=0
> WebServer(Response)
>
> HTTP/1.1 302 Moved TemporarilyCache-Controlno-cache, no-store, max-age=0,
> must-revalidate Pragmano-cacheExpiresFri, 01 Jan 1990 00:00:00 GMT DateWed,
> 20 Aug 2014 11:40:18 GMTLocation
> http://www.google.com/recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83
> Content-Typetext/html; charset=UTF-8Content-Encodinggzip
> X-Content-Type-OptionsnosniffX-XSS-Protection1; mode=blockContent-Length
> 227ServerGSEAlternate-Protocol80:quic
> RB(Request)
>
> GET /recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83
> HTTP/1.1Host www.google.comUser-AgentMozilla/5.0 (Windows NT 5.1;
> rv:31.0) Gecko/20100101 Firefox/31.0 Accept*/*Accept-Language
> en-US,en;q=0.5 Accept-Encodinggzip, deflateReferer
> http://www.patcurtis.com/captcha/recaptcha_demo.php CookiePREF=ID=27454a65a243cd70:U=508d4e3f9fa20406:FF=0:LD=en:TM=1406649235:LM=1407436763:GM=1:S=aCHb3CgmGRRcedMi;
> _ga=GA1.1.1138220839.1407426218; OGP=-4061129:;
> NID=67=pLb98eXX7XshplSC25O1hUrM6VPe1bG9epaYJREF4EBGoV-TPIQ0MynwuZdb1ra4i_j8jfzeKgHyJRL917tqVvJ1drL2BTn2QE4HaRblzxd0lpITPiswWjYNWO3YMn8g0aV7bw3zFqdW9HrdT3h2iJxykpn7QTCJm7fFHZoSYVRR1zWL_jNpSHvs7u912M5K1skdqY-lGtU;
> SID=DQAAAMkAAAD1nK0mZZoRhWbUjiHM6nCXAWouJuka0s3NW3l9AO0Tp7vLrszmHqYEeOI2mbDn6Ew2DuOvg0DxTEDPsjQ_-lMPWoXn_8TQYCEpFyvdZUdqdYT1qb2jNjv7cb6192vCkFAjY4LPJo5tKDUvm8_UDsJGrrILWzwGEf9TMkyo8uMP_rZcL3JFbVWfsN7W8j2MZWRLlSoZ-wcpW8PiQc_se5XSVXoZKUdGd67OemZdbEBmX_VRZ66mJln6EtJeJdOWruXp_I72ldDaz9UZMvZJ7ySH;
> HSID=AbDchAUPI4DtlOcyV; APISID=xLgJV19gOJC9cCD_/ARvW0PuATO71HDPNc
> Connectionkeep-alive
> WebServer(Response)
>
> HTTP/1.1 200 OKCache-Controlno-cache, no-store, max-age=0, must-revalidate
> Pragmano-cacheExpiresFri, 01 Jan 1990 00:00:00 GMT DateWed, 20 Aug 2014
> 11:39:22 GMTContent-Typetext/javascriptContent-Encoding gzip
> X-Content-Type-OptionsnosniffX-XSS-Protection 1; mode=blockContent-Length
> 7090ServerGSE Alternate-Protocol80:quic
> HTMLUnit (First Request)
>
> GET /challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 HTTP/1.1Host
> api.recaptcha.netUser-AgentMozilla/5.0 (Windows NT 5.1; rv:31.0)
> Gecko/20100101 Firefox/31.0 Accept-Languageen-US,en;q=0.5Accept-Encodinggzip,
> deflate Refererhttp://www.patcurtis.com/captcha/recaptcha_demo.php
> Connectionkeep-aliveAccept*/*Cache-Control max-age=0
> WebServer(Response)
>
>
> HTTP/1.1 302 Moved TemporarilyCache-Controlno-cache, no-store, max-age=0,
> must-revalidate Pragmano-cacheExpiresFri, 01 Jan 1990 00:00:00 GMT DateWed,
> 20 Aug 2014 18:22:58 GMTLocation
> http://www.google.com/recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83
> Content-Typetext/html; charset=UTF-8Content-Encodinggzip
> X-Content-Type-OptionsnosniffX-XSS-Protection1; mode=blockContent-Length
> 227ServerGSEAlternate-Protocol80:quic
> HTMLUnit(Request)
>
> GET /recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83
> HTTP/1.1Host www.google.comUser-AgentMozilla/5.0 (Windows NT 5.1;
> rv:31.0) Gecko/20100101 Firefox/31.0 Accept-Languageen-US,en;q=0.5Cookie
> NID=67=cA1Qro6B-PbS07zWxNtD3iJ0hHfuc0itJYr1CVnyDCL4PbJNELomp3zFICR6NGdC0C3rQ440_1Y743dl87W5cTtWbAadzMGOZ0-1hbYTqYOSqZrSeN922y0XQr5IcKGE
> Accept-Encodinggzip, deflateReferer
> http://www.patcurtis.com/captcha/recaptcha_demo.php Connectionkeep-alive
> Accept*/*Cache-Control max-age=0
> WebServer(Response)
>
> HTTP/1.1 200 OKCache-Controlno-cache, no-store, max-age=0, must-revalidate
> Pragmano-cacheExpiresFri, 01 Jan 1990 00:00:00 GMT DateWed, 20 Aug 2014
> 18:22:58 GMTContent-Typetext/javascriptContent-Encoding gzip
> X-Content-Type-OptionsnosniffX-XSS-Protection 1; mode=blockContent-Length
> 6994ServerGSE Alternate-Protocol80:quic
> *******************************************************************
>
> Kindly Let me know if you need any additional information. Please note
> that there are many request/response exchanges
> after the above steps till we get the actual CAPTCHA image
>
> Thanks for your help in advance.
>
> Regards
>  Vasu
>
>
>
> ------------------------------------------------------------------------------
> Slashdot TV.
> Video for Nerds.  Stuff that matters.
> http://tv.slashdot.org/
> _______________________________________________
> Htmlunit-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>
>

Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google

[Vasudevan C. <vco...@gm...>]

Hi Ahmed,

    I can rerun the application & the real browser after clearing the
cache. Accept header position was different which
    I also noticed. Does it have an impact?.

   My suggestion is if you can support the position in request header for
addRequestHeader method, then we can compare
    the behaviour exactly.

    I am storing the cookie value in my app for the designated browser and
sending the same to www.google.com for
    subsequent requests raised from the browser.

    I shall send you the header info after clearing the cache/cookies etc.

Regards
 Vasu



On Thu, Aug 21, 2014 at 12:10 PM, Ahmed Ashour <asa...@ya...> wrote:

> Hi Vasu,
>
> There are two differences:
>     - Cookies in the second request of real browser, they seem to be
> stored from previous requests. Please clear cache and rerun the comparison.
> I also wonder from where did HtmlUnit send the cookies in the second
> request.
>     - 'Accept' header location: this could be a reason, but Cookie are the
> much more the possibility.
>
> Ahmed
>   ------------------------------
>  *From:* Vasudevan Comandur <vco...@gm...>
> *To:* Ahmed Ashour <asa...@ya...>; "
> htm...@li..." <htm...@li...>
>
> *Sent:* Wednesday, August 20, 2014 9:06 PM
> *Subject:* Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google
>
> Hi Ahmed,
>
>     I am giving you the request details for REAL Browser & HTMLUnit below.
>
> REal Browser (First Request)
>
>
> GET /challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 HTTP/1.1Host
> api.recaptcha.netUser-AgentMozilla/5.0 (Windows NT 5.1; rv:31.0)
> Gecko/20100101 Firefox/31.0Accept*/*Accept-Languageen-US,en;q=0.5
> Accept-Encodinggzip, deflateReferer
> http://www.patcurtis.com/captcha/recaptcha_demo.phpConnectionkeep-alive
> Cache-Controlmax-age=0
> WebServer(Response)
>
> HTTP/1.1 302 Moved TemporarilyCache-Controlno-cache, no-store, max-age=0,
> must-revalidatePragmano-cacheExpiresFri, 01 Jan 1990 00:00:00 GMTDateWed,
> 20 Aug 2014 11:40:18 GMTLocation
> http://www.google.com/recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83
> Content-Typetext/html; charset=UTF-8Content-Encodinggzip
> X-Content-Type-OptionsnosniffX-XSS-Protection1; mode=blockContent-Length
> 227ServerGSEAlternate-Protocol80:quic
> RB(Request)
>
> GET /recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83
> HTTP/1.1Hostwww.google.comUser-AgentMozilla/5.0 (Windows NT 5.1; rv:31.0)
> Gecko/20100101 Firefox/31.0Accept*/*Accept-Languageen-US,en;q=0.5
> Accept-Encodinggzip, deflateReferer
> http://www.patcurtis.com/captcha/recaptcha_demo.phpCookiePREF=ID=27454a65a243cd70:U=508d4e3f9fa20406:FF=0:LD=en:TM=1406649235:LM=1407436763:GM=1:S=aCHb3CgmGRRcedMi;
> _ga=GA1.1.1138220839.1407426218; OGP=-4061129:;
> NID=67=pLb98eXX7XshplSC25O1hUrM6VPe1bG9epaYJREF4EBGoV-TPIQ0MynwuZdb1ra4i_j8jfzeKgHyJRL917tqVvJ1drL2BTn2QE4HaRblzxd0lpITPiswWjYNWO3YMn8g0aV7bw3zFqdW9HrdT3h2iJxykpn7QTCJm7fFHZoSYVRR1zWL_jNpSHvs7u912M5K1skdqY-lGtU;
> SID=DQAAAMkAAAD1nK0mZZoRhWbUjiHM6nCXAWouJuka0s3NW3l9AO0Tp7vLrszmHqYEeOI2mbDn6Ew2DuOvg0DxTEDPsjQ_-lMPWoXn_8TQYCEpFyvdZUdqdYT1qb2jNjv7cb6192vCkFAjY4LPJo5tKDUvm8_UDsJGrrILWzwGEf9TMkyo8uMP_rZcL3JFbVWfsN7W8j2MZWRLlSoZ-wcpW8PiQc_se5XSVXoZKUdGd67OemZdbEBmX_VRZ66mJln6EtJeJdOWruXp_I72ldDaz9UZMvZJ7ySH;
> HSID=AbDchAUPI4DtlOcyV; APISID=xLgJV19gOJC9cCD_/ARvW0PuATO71HDPNc
> Connectionkeep-alive
> WebServer(Response)
>
> HTTP/1.1 200 OKCache-Controlno-cache, no-store, max-age=0, must-revalidate
> Pragmano-cacheExpiresFri, 01 Jan 1990 00:00:00 GMTDateWed, 20 Aug 2014
> 11:39:22 GMTContent-Typetext/javascriptContent-Encodinggzip
> X-Content-Type-OptionsnosniffX-XSS-Protection1; mode=blockContent-Length
> 7090ServerGSEAlternate-Protocol80:quic
> HTMLUnit (First Request)
>
> GET /challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 HTTP/1.1Host
> api.recaptcha.netUser-AgentMozilla/5.0 (Windows NT 5.1; rv:31.0)
> Gecko/20100101 Firefox/31.0Accept-Languageen-US,en;q=0.5Accept-Encodinggzip,
> deflateRefererhttp://www.patcurtis.com/captcha/recaptcha_demo.php
> Connectionkeep-aliveAccept*/*Cache-Controlmax-age=0
> WebServer(Response)
>
>
> HTTP/1.1 302 Moved TemporarilyCache-Controlno-cache, no-store, max-age=0,
> must-revalidatePragmano-cacheExpiresFri, 01 Jan 1990 00:00:00 GMTDateWed,
> 20 Aug 2014 18:22:58 GMTLocation
> http://www.google.com/recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83
> Content-Typetext/html; charset=UTF-8Content-Encodinggzip
> X-Content-Type-OptionsnosniffX-XSS-Protection1; mode=blockContent-Length
> 227ServerGSEAlternate-Protocol80:quic
> HTMLUnit(Request)
>
> GET /recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83
> HTTP/1.1Hostwww.google.comUser-AgentMozilla/5.0 (Windows NT 5.1; rv:31.0)
> Gecko/20100101 Firefox/31.0Accept-Languageen-US,en;q=0.5Cookie
> NID=67=cA1Qro6B-PbS07zWxNtD3iJ0hHfuc0itJYr1CVnyDCL4PbJNELomp3zFICR6NGdC0C3rQ440_1Y743dl87W5cTtWbAadzMGOZ0-1hbYTqYOSqZrSeN922y0XQr5IcKGE
> Accept-Encodinggzip, deflateReferer
> http://www.patcurtis.com/captcha/recaptcha_demo.phpConnectionkeep-alive
> Accept*/*Cache-Controlmax-age=0
> WebServer(Response)
>
> HTTP/1.1 200 OKCache-Controlno-cache, no-store, max-age=0, must-revalidate
> Pragmano-cacheExpiresFri, 01 Jan 1990 00:00:00 GMTDateWed, 20 Aug 2014
> 18:22:58 GMTContent-Typetext/javascriptContent-Encodinggzip
> X-Content-Type-OptionsnosniffX-XSS-Protection1; mode=blockContent-Length
> 6994ServerGSEAlternate-Protocol80:quic
> *******************************************************************
>
> Kindly Let me know if you need any additional information. Please note
> that there are many request/response exchanges
> after the above steps till we get the actual CAPTCHA image
>
> Thanks for your help in advance.
>
> Regards
>  Vasu
>
>
>
> ------------------------------------------------------------------------------
> Slashdot TV.
> Video for Nerds.  Stuff that matters.
> http://tv.slashdot.org/
> _______________________________________________
> Htmlunit-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>
>

[Htmlunit-user] How to enable my (htmlunit) browser work as Flash enabled?

[dhashi88 <dha...@ho...>]

i was able to do in HtmlUnit 2.13 version by
private static BrowserVersion firefox17WithUptoDateFlash = new BrowserVersion(
BrowserVersion.FIREFOX_17.getApplicationName(),
BrowserVersion.FIREFOX_17.getApplicationVersion(),
BrowserVersion.FIREFOX_17.getUserAgent(),
BrowserVersion.FIREFOX_17.getBrowserVersionNumeric(),
new BrowserVersionFeatures[] {
BrowserVersionFeatures.JS_FRAME_RESOLVE_URL_WITH_PARENT_WINDOW,
BrowserVersionFeatures.STYLESHEET_HREF_EXPANDURL,
BrowserVersionFeatures.STYLESHEET_HREF_STYLE_NULL });
static {
PluginConfiguration plugin = new PluginConfiguration("Shockwave Flash",
"Shockwave Flash 14.0 r0", "NPSWF32_14_0_0_145.dll");
plugin.getMimeTypes().add(new PluginConfiguration.MimeType("application/x-shockwave-flash", "Adobe Flash movie","swf"));
//plugin.getMimeTypes().add(new PluginConfiguration.MimeType("application/futuresplash", "FutureSplash movie", "spl"));
firefox17WithUptoDateFlash.getPlugins().add(plugin);
}

but same is not working for HtmlUnit 2.15 . what is the reason. How can i achieve this?

Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google

[Ahmed A. <asa...@ya...>]

Hi Vasu,

There are two differences:
    - Cookies in the second request of real browser, they seem to be stored from previous requests. Please clear cache and rerun the comparison. I also wonder from where did HtmlUnit send the cookies in the second request.
    - 'Accept' header location: this could be a reason, but Cookie are the much more the possibility.
 
Ahmed

________________________________
 From: Vasudevan Comandur <vco...@gm...>
To: Ahmed Ashour <asa...@ya...>; "htm...@li..." <htm...@li...> 
Sent: Wednesday, August 20, 2014 9:06 PM
Subject: Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google
 


Hi Ahmed,

    I am giving you the request details for REAL Browser & HTMLUnit below.

REal Browser (First Request)



 GET /challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 HTTP/1.1 
Host api.recaptcha.net 
User-Agent Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0 
Accept */* 
Accept-Language en-US,en;q=0.5 
Accept-Encoding gzip, deflate 
Referer http://www.patcurtis.com/captcha/recaptcha_demo.php 
Connection keep-alive 
Cache-Control max-age=0 
WebServer(Response)


 HTTP/1.1 302 Moved Temporarily 
Cache-Control no-cache, no-store, max-age=0, must-revalidate 
Pragma no-cache 
Expires Fri, 01 Jan 1990 00:00:00 GMT 
Date Wed, 20 Aug 2014 11:40:18 GMT 
Location http://www.google.com/recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 
Content-Type text/html; charset=UTF-8 
Content-Encoding gzip 
X-Content-Type-Options nosniff 
X-XSS-Protection 1; mode=block 
Content-Length 227 
Server GSE 
Alternate-Protocol 80:quic 
RB(Request)


 GET /recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 HTTP/1.1 
Host www.google.com 
User-Agent Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0 
Accept */* 
Accept-Language en-US,en;q=0.5 
Accept-Encoding gzip, deflate 
Referer http://www.patcurtis.com/captcha/recaptcha_demo.php 
Cookie PREF=ID=27454a65a243cd70:U=508d4e3f9fa20406:FF=0:LD=en:TM=1406649235:LM=1407436763:GM=1:S=aCHb3CgmGRRcedMi; _ga=GA1.1.1138220839.1407426218; OGP=-4061129:; NID=67=pLb98eXX7XshplSC25O1hUrM6VPe1bG9epaYJREF4EBGoV-TPIQ0MynwuZdb1ra4i_j8jfzeKgHyJRL917tqVvJ1drL2BTn2QE4HaRblzxd0lpITPiswWjYNWO3YMn8g0aV7bw3zFqdW9HrdT3h2iJxykpn7QTCJm7fFHZoSYVRR1zWL_jNpSHvs7u912M5K1skdqY-lGtU; SID=DQAAAMkAAAD1nK0mZZoRhWbUjiHM6nCXAWouJuka0s3NW3l9AO0Tp7vLrszmHqYEeOI2mbDn6Ew2DuOvg0DxTEDPsjQ_-lMPWoXn_8TQYCEpFyvdZUdqdYT1qb2jNjv7cb6192vCkFAjY4LPJo5tKDUvm8_UDsJGrrILWzwGEf9TMkyo8uMP_rZcL3JFbVWfsN7W8j2MZWRLlSoZ-wcpW8PiQc_se5XSVXoZKUdGd67OemZdbEBmX_VRZ66mJln6EtJeJdOWruXp_I72ldDaz9UZMvZJ7ySH; HSID=AbDchAUPI4DtlOcyV; APISID=xLgJV19gOJC9cCD_/ARvW0PuATO71HDPNc 
Connection keep-alive 
WebServer(Response)


 HTTP/1.1 200 OK 
Cache-Control no-cache, no-store, max-age=0, must-revalidate 
Pragma no-cache 
Expires Fri, 01 Jan 1990 00:00:00 GMT 
Date Wed, 20 Aug 2014 11:39:22 GMT 
Content-Type text/javascript 
Content-Encoding gzip 
X-Content-Type-Options nosniff 
X-XSS-Protection 1; mode=block 
Content-Length 7090 
Server GSE 
Alternate-Protocol 80:quic 
HTMLUnit (First Request)


 GET /challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 HTTP/1.1 
Host api.recaptcha.net 
User-Agent Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0 
Accept-Language en-US,en;q=0.5 
Accept-Encoding gzip, deflate 
Referer http://www.patcurtis.com/captcha/recaptcha_demo.php 
Connection keep-alive 
Accept */* 
Cache-Control max-age=0 
WebServer(Response)



 HTTP/1.1 302 Moved Temporarily 
Cache-Control no-cache, no-store, max-age=0, must-revalidate 
Pragma no-cache 
Expires Fri, 01 Jan 1990 00:00:00 GMT 
Date Wed, 20 Aug 2014 18:22:58 GMT 
Location http://www.google.com/recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 
Content-Type text/html; charset=UTF-8 
Content-Encoding gzip 
X-Content-Type-Options nosniff 
X-XSS-Protection 1; mode=block 
Content-Length 227 
Server GSE 
Alternate-Protocol 80:quic 
HTMLUnit(Request)


 GET /recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 HTTP/1.1 
Host www.google.com 
User-Agent Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0 
Accept-Language en-US,en;q=0.5 
Cookie NID=67=cA1Qro6B-PbS07zWxNtD3iJ0hHfuc0itJYr1CVnyDCL4PbJNELomp3zFICR6NGdC0C3rQ440_1Y743dl87W5cTtWbAadzMGOZ0-1hbYTqYOSqZrSeN922y0XQr5IcKGE 
Accept-Encoding gzip, deflate 
Referer http://www.patcurtis.com/captcha/recaptcha_demo.php 
Connection keep-alive 
Accept */* 
Cache-Control max-age=0 
WebServer(Response)


 HTTP/1.1 200 OK 
Cache-Control no-cache, no-store, max-age=0, must-revalidate 
Pragma no-cache 
Expires Fri, 01 Jan 1990 00:00:00 GMT 
Date Wed, 20 Aug 2014 18:22:58 GMT 
Content-Type text/javascript 
Content-Encoding gzip 
X-Content-Type-Options nosniff 
X-XSS-Protection 1; mode=block 
Content-Length 6994 
Server GSE 
Alternate-Protocol 80:quic 
*******************************************************************

Kindly Let me know if you need any additional information. Please note that there are many request/response exchanges
after the above steps till we get the actual CAPTCHA image

Thanks for your help in advance.

Regards
 Vasu

Re: [Htmlunit-user] HTMLUnit with Recaptcha service of Google

[Vasudevan C. <vco...@gm...>]

Hi Ahmed,

    I am giving you the request details for REAL Browser & HTMLUnit below.

REal Browser (First Request)


GET /challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 HTTP/1.1  Host
api.recaptcha.net  User-Agent Mozilla/5.0 (Windows NT 5.1; rv:31.0)
Gecko/20100101 Firefox/31.0  Accept */*  Accept-Language en-US,en;q=0.5
Accept-Encoding gzip, deflate  Referer
http://www.patcurtis.com/captcha/recaptcha_demo.php  Connection keep-alive
Cache-Control max-age=0
WebServer(Response)

HTTP/1.1 302 Moved Temporarily  Cache-Control no-cache, no-store,
max-age=0, must-revalidate  Pragma no-cache  Expires Fri, 01 Jan 1990
00:00:00 GMT  Date Wed, 20 Aug 2014 11:40:18 GMT  Location
http://www.google.com/recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83
Content-Type text/html; charset=UTF-8  Content-Encoding gzip
X-Content-Type-Options nosniff  X-XSS-Protection 1; mode=block
Content-Length 227  Server GSE  Alternate-Protocol 80:quic
RB(Request)

GET /recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83
HTTP/1.1  Host www.google.com  User-Agent Mozilla/5.0 (Windows NT 5.1;
rv:31.0) Gecko/20100101 Firefox/31.0  Accept */*  Accept-Language
en-US,en;q=0.5  Accept-Encoding gzip, deflate  Referer
http://www.patcurtis.com/captcha/recaptcha_demo.php  Cookie
PREF=ID=27454a65a243cd70:U=508d4e3f9fa20406:FF=0:LD=en:TM=1406649235:LM=1407436763:GM=1:S=aCHb3CgmGRRcedMi;
_ga=GA1.1.1138220839.1407426218; OGP=-4061129:;
NID=67=pLb98eXX7XshplSC25O1hUrM6VPe1bG9epaYJREF4EBGoV-TPIQ0MynwuZdb1ra4i_j8jfzeKgHyJRL917tqVvJ1drL2BTn2QE4HaRblzxd0lpITPiswWjYNWO3YMn8g0aV7bw3zFqdW9HrdT3h2iJxykpn7QTCJm7fFHZoSYVRR1zWL_jNpSHvs7u912M5K1skdqY-lGtU;
SID=DQAAAMkAAAD1nK0mZZoRhWbUjiHM6nCXAWouJuka0s3NW3l9AO0Tp7vLrszmHqYEeOI2mbDn6Ew2DuOvg0DxTEDPsjQ_-lMPWoXn_8TQYCEpFyvdZUdqdYT1qb2jNjv7cb6192vCkFAjY4LPJo5tKDUvm8_UDsJGrrILWzwGEf9TMkyo8uMP_rZcL3JFbVWfsN7W8j2MZWRLlSoZ-wcpW8PiQc_se5XSVXoZKUdGd67OemZdbEBmX_VRZ66mJln6EtJeJdOWruXp_I72ldDaz9UZMvZJ7ySH;
HSID=AbDchAUPI4DtlOcyV; APISID=xLgJV19gOJC9cCD_/ARvW0PuATO71HDPNc
Connection keep-alive
WebServer(Response)

HTTP/1.1 200 OK  Cache-Control no-cache, no-store, max-age=0,
must-revalidate  Pragma no-cache  Expires Fri, 01 Jan 1990 00:00:00 GMT
Date Wed, 20 Aug 2014 11:39:22 GMT  Content-Type text/javascript
Content-Encoding gzip  X-Content-Type-Options nosniff  X-XSS-Protection 1;
mode=block  Content-Length 7090  Server GSE  Alternate-Protocol 80:quic
HTMLUnit (First Request)

GET /challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83 HTTP/1.1  Host
api.recaptcha.net  User-Agent Mozilla/5.0 (Windows NT 5.1; rv:31.0)
Gecko/20100101 Firefox/31.0  Accept-Language en-US,en;q=0.5
Accept-Encoding gzip,
deflate  Referer http://www.patcurtis.com/captcha/recaptcha_demo.php
Connection keep-alive  Accept */*  Cache-Control max-age=0
WebServer(Response)


HTTP/1.1 302 Moved Temporarily  Cache-Control no-cache, no-store,
max-age=0, must-revalidate  Pragma no-cache  Expires Fri, 01 Jan 1990
00:00:00 GMT  Date Wed, 20 Aug 2014 18:22:58 GMT  Location
http://www.google.com/recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83
Content-Type text/html; charset=UTF-8  Content-Encoding gzip
X-Content-Type-Options nosniff  X-XSS-Protection 1; mode=block
Content-Length 227  Server GSE  Alternate-Protocol 80:quic
HTMLUnit(Request)

GET /recaptcha/api/challenge?k=6Lf1-AQAAAAAAL9EskceOVja5Uh5_yo1FRtb1Q83
HTTP/1.1  Host www.google.com  User-Agent Mozilla/5.0 (Windows NT 5.1;
rv:31.0) Gecko/20100101 Firefox/31.0  Accept-Language en-US,en;q=0.5  Cookie
NID=67=cA1Qro6B-PbS07zWxNtD3iJ0hHfuc0itJYr1CVnyDCL4PbJNELomp3zFICR6NGdC0C3rQ440_1Y743dl87W5cTtWbAadzMGOZ0-1hbYTqYOSqZrSeN922y0XQr5IcKGE
Accept-Encoding gzip, deflate  Referer
http://www.patcurtis.com/captcha/recaptcha_demo.php  Connection keep-alive
Accept */*  Cache-Control max-age=0
WebServer(Response)

HTTP/1.1 200 OK  Cache-Control no-cache, no-store, max-age=0,
must-revalidate  Pragma no-cache  Expires Fri, 01 Jan 1990 00:00:00 GMT
Date Wed, 20 Aug 2014 18:22:58 GMT  Content-Type text/javascript
Content-Encoding gzip  X-Content-Type-Options nosniff  X-XSS-Protection 1;
mode=block  Content-Length 6994  Server GSE  Alternate-Protocol 80:quic
*******************************************************************

Kindly Let me know if you need any additional information. Please note that
there are many request/response exchanges
after the above steps till we get the actual CAPTCHA image

Thanks for your help in advance.

Regards
 Vasu




On Wed, Aug 20, 2014 at 9:45 PM, Ahmed Ashour <asa...@ya...> wrote:

> Hi Vasu,
>
> Interesting finding.
>
> A possibility is that HtmlUnit is faster than FF.
>
> The second thing, as you said, request difference, which you could provide
> more details about.
>
> How about testing with HtmlUnit but with IE/Chrome simluation?
>
> A remote question: is this case reproducible?
>
> Ahmed
>   ------------------------------
>  *From:* Vasudevan Comandur <vco...@gm...>
> *To:* "htm...@li..." <
> htm...@li...>
> *Sent:* Wednesday, August 20, 2014 4:38 PM
> *Subject:* [Htmlunit-user] HTMLUnit with Recaptcha service of Google
>
> Hi,
>
>     I am facing a problem connecting to Recaptcha service through HTMLUnit.
>
>    I did the following.
>
>    1. I used FF browser to connect to Recaptcha service using a demo site
> having
>        the link  http://www.patcurtis.com/captcha/recaptcha_demo.php
>
>    2. Captcha image will get displayed inside the recaptcha widget.
>
>     3. I was pressing the reload current page continuously after the
> captcha image
>         is displayed. I did not get any error image even after reloading
> 500 times.
>
>    I repeated the same exercise through HTMLUnit program.
>
>    I was getting the captcha image for about 200 times of reload. But
> after that
>    recaptcha service was throwing an error image. The link of the error
> image is
>    http://www.gstatic.com/recaptcha/api/img/dos_captcha.jpg. Reloading
> after
>    the error image throws back the same image page.
>
>   Is there any reason for such a different behaviour through HTMLUnit?. I
> was
>   expecting the same behaviour to happen with HTMLUnit as normal browser.
>
>   Does the order of HTTP request headers matter?. I see the order through
>   charles proxy and see the difference.
>
>   Any pointers to solve this issue would help me.
>
>   Looking forward to your reply.
>
> Regards
>  Vasu
>
>
>
> ------------------------------------------------------------------------------
> Slashdot TV.
> Video for Nerds.  Stuff that matters.
> http://tv.slashdot.org/
> _______________________________________________
> Htmlunit-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>
>