curl-loader-devel — curl-loader usage and development issues

Problem with cookies

[web l. a. p. t. t. <cur...@li...>]

Arthur,

Y are correct. This was broken in 0.32 version.

This is a shame on my side to make such a stupid error.
My test was not fetching that as well.

The matters have been corrected.
Thank you. Added you to our THANKS file.
If you can provide your surname, it will be added.

The patch  and commit  you can see below.

Sincerely,
Robert


---------- Forwarded message ----------
From: cor...@us... < cor...@us...>
Date: Jul 25, 2007 12:48 PM
Subject: SF.net SVN: curl-loader: [488] trunk/curl-loader
To: cur...@li...

Revision: 488

http://curl-loader.svn.sourceforge.net/curl-loader/?rev=488&view=rev
Author:   coroberti
Date:     2007-07-25 02:47:58 -0700 (Wed, 25 Jul 2007)

Log Message:
-----------
Bugfix for loading credentials from file.

Modified Paths:
--------------
    trunk/curl-loader/conf-examples/credentials.cred
    trunk/curl-loader/conf-examples/post-form-tokens-fr-file.conf
    trunk/curl-loader/doc/QUICK-START
    trunk/curl-loader/doc/THANKS
    trunk/curl-loader/parse_conf.c

Modified: trunk/curl-loader/conf-examples/credentials.cred
===================================================================
--- trunk/curl-loader/conf-examples/credentials.cred    2007-07-22 05:02:20
UTC (rev 487)
+++ trunk/curl-loader/conf-examples/credentials.cred    2007-07-25 09:47:58
UTC (rev 488)
@@ -1,4 +1,3 @@
-# Separator used here is ':' Use %20, when you need a white space
-david:meleh
-israel:hai
-hai:vkayam
+# Separator used here is ','
+david,meleh,israel
+hai,hai,vkayam

Modified: trunk/curl-loader/conf-examples/post-form-tokens-fr-file.conf
===================================================================
--- trunk/curl-loader/conf-examples/post-form-tokens-fr-file.conf
2007-07-22 05:02:20 UTC (rev 487)
+++ trunk/curl-loader/conf-examples/post-form-tokens-fr-file.conf
2007-07-25 09:47:58 UTC (rev 488)
@@ -1,6 +1,6 @@
########### GENERAL SECTION ################################
BATCH_NAME= post-form-tokens-fr-file
-CLIENTS_NUM_MAX=3
+CLIENTS_NUM_MAX=2
INTERFACE=eth0
NETMASK=24
IP_ADDR_MIN=194.90.71.215

Modified: trunk/curl-loader/doc/QUICK-START
===================================================================
--- trunk/curl-loader/doc/QUICK-START   2007-07-22 05:02:20 UTC (rev 487)
+++ trunk/curl-loader/doc/QUICK-START   2007-07-25 09:47:58 UTC (rev 488)
@@ -16,11 +16,10 @@
files (e.g. crypto.h), export environment variable OPENSSLDIR with
the value of that directory.
For example: $export OPENSSLDIR=the-full-path-to-the-directory
-
-Another known issue is libidn.so, which means, that some linux
distributions
-do have some libidn.so.11, but not libidn.so. Resolve it by creating a
softlink.
-In some cases, you may be required to remove -lidn from the linking line,
-whereas you need to comment a line in Makefile and uncomment another.
+
+We are building libcurl with --without-libidn option. Users, that would
like
+to resolve IDNA domain names with "international" letters can edit our
+Makefile and use instead --with-libidn=full-path-dir.

Run the following commands from your bash linux shell:
$tar zxfv curl-loader-<version>.tar.gz

Modified: trunk/curl-loader/doc/THANKS
===================================================================
--- trunk/curl-loader/doc/THANKS        2007-07-22 05:02:20 UTC (rev 487)
+++ trunk/curl-loader/doc/THANKS        2007-07-25 09:47:58 UTC (rev 488)
@@ -11,3 +11,4 @@
Aleksandar Lazic <al-...@no...>
Jeremy Hicks <je...@no...>
John Gatewood Ham < bur...@gm...>
+Artur B <ar...@gm...>

Modified: trunk/curl-loader/parse_conf.c
===================================================================
--- trunk/curl-loader/parse_conf.c      2007-07-22 05:02:20 UTC (rev 487)
+++ trunk/curl-loader/parse_conf.c      2007-07-25 09:47:58 UTC (rev 488)
@@ -213,7 +213,7 @@
                                     size_t input_length,
                                     form_records_cdata* form_record,
                                     size_t record_num,
-                                    char* separator);
+                                    char** separator);

static int add_param_to_batch (char*const input,
                                size_t input_length,
@@ -388,17 +388,17 @@
                                     size_t input_len,
                                     form_records_cdata* form_record,
                                     size_t record_num,
-                                    char* separator)
+                                    char** separator)
{
-  const char separators_supported [] =
+  const char* separators_supported [] =
     {
-      ',',
-      ':',
-      ';',
-      ' ',
-      '@',
-      '/',
-      '\0'
+      ",",
+      ":",
+      ";",
+      " ",
+      "@",
+      "/",
+      0
     };
   char* sp = NULL;
   int i;
@@ -416,9 +416,9 @@
     {
       for (i = 0; separators_supported [i]; i++)
         {
-          if ((sp = strchr (input, separators_supported [i])))
+          if ((sp = strchr (input, *separators_supported [i])))
             {
-              *separator = *sp; /* Remember the separator */
+              *separator = (char *) separators_supported [i]; /* Remember
the separator */
               break;
             }
         }
@@ -429,9 +429,10 @@
                    "%s - failed to locate in the first string \"%s\" \n"
                    "any supported separator.\nThe supported separators
are:\n",
                __func__, input);
+
           for (i = 0; separators_supported [i]; i++)
             {
-              fprintf (stderr,"\"%c\"\n", separators_supported [i]);
+              fprintf (stderr,"\"%s\"\n", separators_supported [i]);
             }
           return -1;
         }
@@ -440,9 +441,9 @@
   char * token = 0, *strtokp = 0;
   size_t token_count  = 0;

-  for (token = strtok_r (input, separator, &strtokp);
+  for (token = strtok_r (input, *separator, &strtokp);
        token != 0;
-       token = strtok_r (0, separator, &strtokp))
+       token = strtok_r (0, *separator, &strtokp))
     {
       size_t token_len = strlen (token);

@@ -2687,7 +2688,7 @@
{
   char fgets_buff[512];
   FILE* fp;
-  char sep;
+  char* sep = 0; // strtok_r requires a string with '\0'

   /*
      Open the file with form records
@@ -2737,6 +2738,16 @@
               continue;
             }

+          if (fgets_buff[string_len - 2] == '\r')
+            {
+              fgets_buff[string_len - 2] = '\0';
+            }
+
+          if (fgets_buff[string_len -1] == '\n')
+            {
+              fgets_buff[string_len -1] = '\0';
+            }
+
           if ((int)url->form_records_num >= bctx->client_num_max)
             {
               fprintf (stderr,


This was sent by the SourceForge.net collaborative development platform, the
world's largest Open Source development site.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>   http://get.splunk.com/
_______________________________________________
curl-loader-tracker mailing list
cur...@li...
https://lists.sourceforge.net/lists/listinfo/curl-loader-tracker


-- 
Sincerely,
Robert Iakobashvili,
coroberti %x40 gmail %x2e com
...........................................................
http://curl-loader.sourceforge.net
A web testing and traffic generation tool.

Re: Problem with cookies

[web l. a. p. t. t. <cur...@li...>]

Robert,

1) Well, basically the bug was in wrong assumption that
strtok_r() expects merely a pointer to a char as delimiter,
when it actually expects null-terminated string. There is
no way that code could work, unless, by some lucky
coincidence, the byte next to the delimeter char happened
to be null. See the patch below:

Index: parse_conf.c
===================================================================
--- parse_conf.c        (revision 487)
+++ parse_conf.c        (working copy)
@@ -419,6 +419,7 @@
           if ((sp = strchr (input, separators_supported [i])))
             {
               *separator = *sp; /* Remember the separator */
+              separator[1] = '\0';
               break;
             }
         }
@@ -441,8 +442,8 @@
   size_t token_count  = 0;

   for (token = strtok_r (input, separator, &strtokp);
-       token != 0;
-       token = strtok_r (0, separator, &strtokp))
+       token != NULL;
+       token = strtok_r (NULL, separator, &strtokp))
     {
       size_t token_len = strlen (token);

@@ -2687,8 +2688,16 @@
 {
   char fgets_buff[512];
   FILE* fp;
-  char sep;
+  char *sep = malloc(sizeof(char) * 2);

+  if (NULL == sep)
+  {
+      fprintf (stderr,
+               "%s - failed to allocate memory for separator with errno 
%d.\n",
+               __func__, errno);
+      return -1;
+  }
+
   /*
      Open the file with form records
   */
@@ -2751,7 +2760,7 @@
                                        string_len,
                                        &url->form_records_array[url->form_records_num],
                                        url->form_records_num,
-                                       &sep) == -1)
+                                       sep) == -1)
           {
               fprintf (stderr,
                        "%s - error: load_client_credentials_buffers () 
failed "


---------------------------------------------------------------------------

2) Yes, I did update the repo to version 487 today, and it still
fails unless I add -lidn to LIBS in Makefile.


Artur

----- Original Message ----- 
From: "Robert Iakobashvili" <cor...@gm...>
To: <cur...@li...>
Sent: Tuesday, July 24, 2007 8:52 PM
Subject: Re: Problem with cookies


> Artur,
>
> On 7/25/07, Artur B <ar...@gm...> wrote:
>>I've done some investigation today and found a bug in credentials file 
>>parsing
>>code (load_form_records_file() and load_form_record_string() in
> parse_config.c).
>>Apparently nobody tested this feature before.
>
> We do tested, but "errare humanum est" ...
>
>>I've fixed it in my repo and now it works
>>just fine reading email:password pairs correctly from the credentials
> file. I need write
>>access to SVN to commit it.
>
> Great!
> Could you, please, either post the patch to the mailing list, or
> explain what to fix?
> Thanks.
>
>> Also, build fails for me unless I manually
>> add "-lidn" to LIBS variable in Makefile.
>> Shouldn't we fix it too?
>
> It was supposed to be fixed by a commit 2-3 days ago by
> adding an option --without-libidn to the Makefile. Could you, please,
> run
> $svn update
> with a further
> $make cleanall; make
> to see, if it works for you? Thanks.
>
>
> -- 
> Sincerely,
> Robert Iakobashvili,
> coroberti %x40 gmail %x2e com
> ...........................................................
> http://curl-loader.sourceforge.net
> A web testing and traffic generation tool.
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems?  Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >>  http://get.splunk.com/
> _______________________________________________
> curl-loader-devel mailing list
> cur...@li...
> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel 

Re: Problem with cookies

[Robert I. <cor...@gm...>]

Artur,

On 7/25/07, Artur B <ar...@gm...> wrote:
>I've done some investigation today and found a bug in credentials file parsing
>code (load_form_records_file() and load_form_record_string() in
parse_config.c).
>Apparently nobody tested this feature before.

We do tested, but "errare humanum est" ...

>I've fixed it in my repo and now it works
>just fine reading email:password pairs correctly from the credentials
file. I need write
>access to SVN to commit it.

Great!
Could you, please, either post the patch to the mailing list, or
explain what to fix?
Thanks.

> Also, build fails for me unless I manually
> add "-lidn" to LIBS variable in Makefile.
> Shouldn't we fix it too?

It was supposed to be fixed by a commit 2-3 days ago by
adding an option --without-libidn to the Makefile. Could you, please,
run
$svn update
with a further
$make cleanall; make
to see, if it works for you? Thanks.


-- 
Sincerely,
Robert Iakobashvili,
coroberti %x40 gmail %x2e com
...........................................................
http://curl-loader.sourceforge.net
A web testing and traffic generation tool.

Re: Problem with cookies

[Artur B <ar...@gm...>]

Also, build fails for me unless I manually
add "-lidn" to LIBS variable in Makefile.
Shouldn't we fix it too?

Artur


On 7/24/07, Artur B <ar...@gm...> wrote:
>
> Robert,
>
> I've done some investigation today and found a bug in credentials file
> parsing
> code (load_form_records_file() and load_form_record_string() in
> parse_config.c).
> Apparently nobody tested this feature before. I've fixed it in my repo and
> now it works
> just fine reading email:password pairs correctly from the credentials
> file. I need write
> access to SVN to commit it.
>
> Artur
>
>
>
> On 7/20/07, Robert Iakobashvili <cor...@gm...> wrote:
> >
> > Artur,
> >
> > Please, correct you e-mail agent to make replies
> > to the mailing list and not to me directly. Thanks.
> >
> > On 7/21/07, Artur B <ar...@gm...> wrote:
> > > FORM_STRING of the following form doesn't seem to work:
> > >
> > > USERNAME= "user"
> > > PASSWORD= "testuser"
> > > FORM_USAGE_TYPE= UNIQUE_USERS_SAME_PASSWORD
> > > FORM_STRING=
> > > "action=login&email=%s%d...@lt...&pword=%s"
> >
> > Are you doing GET?
> > Unclear.
> >
> > > And I have bunch of email addresses with index number in the middle.
> > Reading
> > > email:password
> > > from the file doesn't work either, I've tried all possible
> > combinations
> > > already.
> >
> > All these features are known to be working with POST, and the code
> > should be working with GET (that you are trying - guessing) as well,
> > but not tested.
> >
> > Please, look at init_client_formed_buffer () in loader.c file. Please,
> > make a run with
> > debugger and see, if you are coming there.
> >
> > However, if you are trying
> > REQUEST_TYPE=POST with
> > MULTIPART_FORM_DATA
> > this is not a bug but rather absence of the feature support, which
> > works currently
> > only as AS_IS strings.
> >
> > The best thing, if you find a bug, is to provide us patches.
> > Another option is to fill properly PROBLEM_REPORTING form, add it
> > to bugzilla, as well as post it to the list.
> >
> > Thank you.
> >
> >
> > Sincerely,
> > Robert Iakobashvili,
> > coroberti %x40 gmail %x2e com
> > ...........................................................
> > http://curl-loader.sourceforge.net
> > A web testing and traffic generation tool.
> >
> >
> > -------------------------------------------------------------------------
> > This SF.net email is sponsored by: Microsoft
> > Defy all challenges. Microsoft(R) Visual Studio 2005.
> > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> > _______________________________________________
> > curl-loader-devel mailing list
> > cur...@li...
> > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
> >
>
>

Re: Problem with cookies

[Artur B <ar...@gm...>]

Robert,

I've done some investigation today and found a bug in credentials file
parsing
code (load_form_records_file() and load_form_record_string() in
parse_config.c).
Apparently nobody tested this feature before. I've fixed it in my repo and
now it works
just fine reading email:password pairs correctly from the credentials file.
I need write
access to SVN to commit it.

Artur



On 7/20/07, Robert Iakobashvili <cor...@gm...> wrote:
>
> Artur,
>
> Please, correct you e-mail agent to make replies
> to the mailing list and not to me directly. Thanks.
>
> On 7/21/07, Artur B <ar...@gm...> wrote:
> > FORM_STRING of the following form doesn't seem to work:
> >
> > USERNAME= "user"
> > PASSWORD= "testuser"
> > FORM_USAGE_TYPE= UNIQUE_USERS_SAME_PASSWORD
> > FORM_STRING=
> > "action=login&email=%s%d...@lt...&pword=%s"
>
> Are you doing GET?
> Unclear.
>
> > And I have bunch of email addresses with index number in the middle.
> Reading
> > email:password
> > from the file doesn't work either, I've tried all possible combinations
> > already.
>
> All these features are known to be working with POST, and the code
> should be working with GET (that you are trying - guessing) as well,
> but not tested.
>
> Please, look at init_client_formed_buffer () in loader.c file. Please,
> make a run with
> debugger and see, if you are coming there.
>
> However, if you are trying
> REQUEST_TYPE=POST with
> MULTIPART_FORM_DATA
> this is not a bug but rather absence of the feature support, which
> works currently
> only as AS_IS strings.
>
> The best thing, if you find a bug, is to provide us patches.
> Another option is to fill properly PROBLEM_REPORTING form, add it
> to bugzilla, as well as post it to the list.
>
> Thank you.
>
>
> Sincerely,
> Robert Iakobashvili,
> coroberti %x40 gmail %x2e com
> ...........................................................
> http://curl-loader.sourceforge.net
> A web testing and traffic generation tool.
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2005.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> curl-loader-devel mailing list
> cur...@li...
> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>

Re: Problem with cookies

[Robert I. <cor...@gm...>]

Artur,

Please, correct you e-mail agent to make replies
to the mailing list and not to me directly. Thanks.

On 7/21/07, Artur B <ar...@gm...> wrote:
> FORM_STRING of the following form doesn't seem to work:
>
> USERNAME= "user"
> PASSWORD= "testuser"
> FORM_USAGE_TYPE= UNIQUE_USERS_SAME_PASSWORD
> FORM_STRING=
> "action=login&email=%s%d...@lt...&pword=%s"

Are you doing GET?
Unclear.

> And I have bunch of email addresses with index number in the middle. Reading
> email:password
> from the file doesn't work either, I've tried all possible combinations
> already.

All these features are known to be working with POST, and the code
should be working with GET (that you are trying - guessing) as well,
but not tested.

Please, look at init_client_formed_buffer () in loader.c file. Please,
make a run with
debugger and see, if you are coming there.

However, if you are trying
REQUEST_TYPE=POST with
MULTIPART_FORM_DATA
this is not a bug but rather absence of the feature support, which
works currently
only as AS_IS strings.

The best thing, if you find a bug, is to provide us patches.
Another option is to fill properly PROBLEM_REPORTING form, add it
to bugzilla, as well as post it to the list.

Thank you.


Sincerely,
Robert Iakobashvili,
coroberti %x40 gmail %x2e com
...........................................................
http://curl-loader.sourceforge.net
A web testing and traffic generation tool.

Re: Problem with cookies

[Robert I. <cor...@gm...>]

Artur,

On 7/20/07, Artur B <ar...@gm...> wrote:
> Ok, sorry for flow of emails. The following also works:
>
> REQUEST_TYPE=POST
> MULTIPART_FORM_DATA="action=login"
> MULTIPART_FORM_DATA="email=us...@lt..."
> MULTIPART_FORM_DATA="pword=testuser"


We have not added to this option tokens from file, unique users/passwords
etc and actually it is  currently supported as AS_IS only option.

If you know a bit C-language, this is an easy staff to add.


-- 
Sincerely,
Robert Iakobashvili,
coroberti %x40 gmail %x2e com
...........................................................
http://curl-loader.sourceforge.net
A web testing and traffic generation tool.

Re: Problem with cookies

[Robert I. <cor...@gm...>]

Artur,

please, post to the mail-list.

On 7/20/07, Artur B <ar...@gm...> wrote:
> Ok, problem solved, apparently it's not POST but GET login,
> so the following lines in config file handle everything:
> REQUEST_TYPE=GET

Great! Nice to learn it.

> Although in the login page form it says 'method="post"', it doesn't work
> with POST
> but works with GET just fine.
> The question is now: how do I simulate multiple users' logins with GET, not
> just user1?

It should work in a usual fashion.
Y can see as an example:
curl-loader-0.40/conf-examples/get-forms.conf


Sincerely,
Robert Iakobashvili,
coroberti %x40 gmail %x2e com
...........................................................
http://curl-loader.sourceforge.net
A web testing and traffic generation tool.

Re: Problem with cookies

[Robert I. <cor...@gm...>]

On 7/20/07, Artur B <ar...@gm...> wrote:
> 1) Server's logs are not helpful, I was looking in them from the start,
> no usefull information there, just 302 and 200 codes returned.

Your web-application seems to be rejecting authentication.
Not apache, but your php, perl, java, etc application should
contain a log and report all failed authentications.

> 2) This site is HTTPS-only so capturing the traffic will do nothing.

I see.

> 3) The email/password are correct, and I tried all combinations of input
> already
> (reading from the file and using USERNAME and PASSWORD tags). What's
> interesting, when I deliberatly specify incorrect email/password then
> nothing changes,

One more arg in favor of the failed authentication.

> 4) I can't find any option in curl-loader for logging everything it sends
> over HTTP(S),
> such option would be very helpful in debugging the program. It shouldn't be
> hard to implement and I think that's where we should start from.

Correct. However, we have really a zero bandwidth till September.
I will try my best, but cannot promise any date. Adding it to our
highest priority list TODO.

My filing is that you application requires multi-part form.
Y may try using it.

Another option is that you will use a Firefox extention for logging/capture of
the header files. Many your ago it was Mozilla and it was working with HTTPS
good. Interesting, which Content-Type header browser sends in POST:
"application/x-www-form-urlencoded"  or "multipart/form-data"
If the second, thus, clearly multipart/form is necessary.


Sincerely,
Robert Iakobashvili,
coroberti %x40 gmail %x2e com
...........................................................
http://curl-loader.sourceforge.net
A web testing and traffic generation tool.

Re: Problem with cookies

[Robert I. <cor...@gm...>]

Arthur,

On 7/20/07, Robert Iakobashvili <cor...@gm...> wrote:
> > > I'm thinking, has it something to do with how username:password are posted?
> > > On url1 (2nd URL in config file) I'm basically getting original login page
> > > returned with
> > > email address, which just was sent with POST, already in the text field. But
> > > no messages about incorrect login/password or anything like that. This is
> > > strange.
>
> I need to see the capture (ethereal/wireshark/tcpdump)

Sorry, for the flow of e-mails.

After some thinking, it could be that the problem is solely
in authentication, and the non-existing cookies are just the consequences.
The bugfix we did on our side seems to be optional and not a must.

Some servers/portals are not rejecting any failed authentications
by 4xx messages (like bad passwords, e-mails, etc),
but ether:
a) sending 3xx redirect to some problem describing page with subsequent
200 OK without setting to browser cookies;
b) sending 200 OK, but not setting to browser cookies;
c) etc methods;

One more place of assistance could be the web-server logs.
There are several logs in the case of apache, like access log,
authentication log, etc

Could it be that there are more parameters required by the POST form?
Could it be, that the e-mail used for authentication is not in database or
misspelled in the token/credentials file?

One more option to try is "multi-part form data POST-ing as in RFC1867".
This is a fresh feature in curl-loader without support for tokens loaded from
a file yet.

In any case packet captures (if there is an option for the
non-encrypted mode) and
curl-loader logs of headers and bodies could be of assistance.

Sincerely,
Robert Iakobashvili,
coroberti %x40 gmail %x2e com
...........................................................
http://curl-loader.sourceforge.net
A web testing and traffic generation tool.

Re: Problem with cookies

[Robert I. <cor...@gm...>]

On 7/20/07, Robert Iakobashvili <cor...@gm...> wrote:
> Hi Artur,
>
> On 7/20/07, Artur B <ar...@gm...> wrote:
> > Hi Robert,
>
> > I'm thinking, has it something to do with how username:password are posted?
> > On url1 (2nd URL in config file) I'm basically getting original login page
> > returned with
> > email address, which just was sent with POST, already in the text field. But
> > no messages about incorrect login/password or anything like that. This is
> > strange.

I need to see the capture (ethereal/wireshark/tcpdump)


> > Also note that I'm using "email=%s&pword=%s" as a FORM_STRING, not a
> > (supposedly)
> > default "username=%s&password=%s", is it supported?
>
This is supported.

-- 
Sincerely,
Robert Iakobashvili,
coroberti %x40 gmail %x2e com
...........................................................
http://curl-loader.sourceforge.net
A web testing and traffic generation tool.

Re: Problem with cookies

[Robert I. <cor...@gm...>]

On 7/20/07, Artur B <ar...@gm...> wrote:
> Hi Robert,
>
> Well, first I've just replaced loader.c file with one from rev. 485 but
> build
> failed (I have stable 0.40 version). Then I manually edited loader.c file
> to reflect changes you have made to it (version 485-476 diff), but the
> problem
> with cookies didn't go away. Finally, I've installed svn, checked out
> complete
> version 485 and built it, problem is still there.

Even better, if you can made an ethereal/wireshark capture
of this particular transaction, if it can be done in without TLS/SSL
encryption with http and not https url.

The non-encrypted capture can be mostly helpful.
Y can send it to me directly.

Sincerely,
Robert Iakobashvili,
coroberti %x40 gmail %x2e com
...........................................................
http://curl-loader.sourceforge.net
A web testing and traffic generation tool.

Re: Problem with cookies

[Robert I. <cor...@gm...>]

On 7/20/07, Artur B <ar...@gm...> wrote:
> Robert,
>
> One more detail which might help in debugging this issue.
> The site I'm testing has SSL serificate with is not issued by
> trusted CA. When accessing this site with IE7 I'm having
> warning page first. When pulling it with curl utility I'm getting
> the following error:
>
> error:14090086:SSL
> routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify
> failed
>
> Using curl's -k option takes care of this though. But I can't see any
> options
> in curl-loader for dealing with SSL/TLS related things. It seems to pull
> pages from this site not issuing any SSL errors though.

It's OK, we are placing the -k option by default (being a loader),
whereas we can consider in future to make it configurable as well.

-- 
Sincerely,
Robert Iakobashvili,
coroberti %x40 gmail %x2e com
...........................................................
http://curl-loader.sourceforge.net
A web testing and traffic generation tool.

Re: Problem with cookies

[Robert I. <cor...@gm...>]

Hi Artur,

On 7/20/07, Artur B <ar...@gm...> wrote:
> Hi Robert,

> I'm thinking, has it something to do with how username:password are posted?
> On url1 (2nd URL in config file) I'm basically getting original login page
> returned with
> email address, which just was sent with POST, already in the text field. But
> no messages about incorrect login/password or anything like that. This is
> strange.
>
> Also note that I'm using "email=%s&pword=%s" as a FORM_STRING, not a
> (supposedly)
> default "username=%s&password=%s", is it supported?

It's OK.

Could you, please, send to me (you might wish directly) a gzipped
tarball of all you headers and bodies?

tar zcfv myservice.tar.gz ./myservice

Thanks

Sincerely,
Robert Iakobashvili,
coroberti %x40 gmail %x2e com
...........................................................
http://curl-loader.sourceforge.net
A web testing and traffic generation tool.

Re: Error in Make

[Robert I. <cor...@gm...>]

Hi Yanto,

On 7/19/07, yanto t <ind...@gm...> wrote:
> I need to use a load testing tool for a project and I found out
> that the curl-loader is very suitable for my purpose, but i still
> can't use it because :
>  I always encounter error when I do make.

Thank you very much for your PROBLEM_REPORTING form,
provided in due course.

The error comes from libcurl build, which does not support
gcc 2.95. curl-loader - our layer framework does not support
the compiler as well.

Please, note, that the kernel 2.4 can work or not with the
latest curl-loader. We are testing it mainly on 2.6 series.

My suggestion is, please, install a newer linux distribution
with gcc of 3.* or 4.* series. Better to use something, that is
running kernel 2.6.

Take care,

Sincerely,
Robert Iakobashvili,
coroberti %x40 gmail %x2e com
...........................................................
http://curl-loader.sourceforge.net
A web testing and traffic generation tool.

Fwd: Error in Make

[yanto t <ind...@gm...>]

ICAgICAgICB0b3RhbDogICAgdXNlZDogICAgZnJlZTogIHNoYXJlZDogYnVmZmVyczogIGNhY2hl
ZDoKTWVtOiAgNTE3NzcxMjY0IDE1MjQ0NDkyOCAzNjUzMjYzMzYgICAgICAgIDAgIDM4MTc0NzIg
ODgyMDczNjAKU3dhcDogICAgICAgIDAgICAgICAgIDAgICAgICAgIDAKTWVtVG90YWw6ICAgICAg
IDUwNTYzNiBrQgpNZW1GcmVlOiAgICAgICAgMzU2NzY0IGtCCk1lbVNoYXJlZDogICAgICAgICAg
IDAga0IKQnVmZmVyczogICAgICAgICAgMzcyOCBrQgpDYWNoZWQ6ICAgICAgICAgIDg2MTQwIGtC
ClN3YXBDYWNoZWQ6ICAgICAgICAgIDAga0IKQWN0aXZlOiAgICAgICAgICAzNzEzNiBrQgpJbmFj
dGl2ZTogICAgICAgIDUyODU2IGtCCkhpZ2hUb3RhbDogICAgICAgICAgIDAga0IKSGlnaEZyZWU6
ICAgICAgICAgICAgMCBrQgpMb3dUb3RhbDogICAgICAgNTA1NjM2IGtCCkxvd0ZyZWU6ICAgICAg
ICAzNTY3NjQga0IKU3dhcFRvdGFsOiAgICAgICAgICAgMCBrQgpTd2FwRnJlZTogICAgICAgICAg
ICAwIGtCCg==

Re: Problem with cookies

[Robert I. <cor...@gm...>]

Hi Artur,

On 7/19/07, Artur B <ar...@gm...> wrote:
> I like curl-loader, it's a nice tool.
> But I'm constanly having the following message
> returned when trying to login with curl-loader:
>
> "You have either timed out or disabled cookies on your browser. Please make
> sure cookies are enabled"
>
> That page returned on url2 (3rd URL in below config file)
> Please see corresponding headers returned and my config file below.
> Thanks in advance for your help.

Thanks for your detailed analyzes.
This looks like a bug in the tool. Cookies should be enabled only once
at the very first client handle initialization.

Y can try the fix from svn, using the instructions:
http://sourceforge.net/svn/?group_id=179599

If you have any problems of working with svn, please,
use the recent stable version 0.40 and download
the most recent version of file loader.c from here
http://curl-loader.svn.sourceforge.net/viewvc/curl-loader/trunk/curl-loader/loader.c?view=log
Substitute the file in the distro and run, e.g.
make clean; make

Please, report any further problems on your side, if any.

-- 
Sincerely,
Robert Iakobashvili,
coroberti %x40 gmail %x2e com
...........................................................
http://curl-loader.sourceforge.net
A web testing and traffic generation tool.

Problem with cookies

[Artur B <ar...@gm...>]

 Hi,

I like curl-loader, it's a nice tool.
But I'm constanly having the following message
returned when trying to login with curl-loader:

"You have either timed out or disabled cookies on your browser. Please make
sure cookies are enabled"

That page returned on url2 (3rd URL in below config file)
Please see corresponding headers returned and my config file below.
Thanks in advance for your help.

Art

-----------------------------------------------------------------------

HTTP/1.1 302 Found
Date: Thu, 19 Jul 2007 00:15:40 GMT
Server: Apache
P3P: CP="STP ADM CON CUR DEV OUR INT NAV", policyref="/w3c/p3p.xml"
Location: /app/logout
Transfer-Encoding: chunked
Content-Type: text/plain

HTTP/1.1 302 Found
Date: Thu, 19 Jul 2007 00:15:40 GMT
Server: Apache
Set-Cookie: websession=; path=/app; expires=Thu, 19-Jul-2007 00:15:39 GMT
P3P: CP="STP ADM CON CUR DEV OUR INT NAV", policyref="/w3c/p3p.xml"
Location: /exec/logout
Transfer-Encoding: chunked
Content-Type: text/plain

HTTP/1.1 302 Found
Date: Thu, 19 Jul 2007 00:15:40 GMT
Server: Apache
P3P: CP="STP ADM CON CUR DEV OUR INT NAV", policyref="/w3c/p3p.xml"
Cache-Control: private
Pragma: no-cache
Location: https://mc-s100.test.company.com/exec/nocookie
Transfer-Encoding: chunked
Content-Type: text/plain

HTTP/1.1 200 OK
Date: Thu, 19 Jul 2007 00:15:40 GMT
Server: Apache
P3P: CP="STP ADM CON CUR DEV OUR INT NAV", policyref="/w3c/p3p.xml"
Cache-Control: private
Pragma: no-cache
Transfer-Encoding: chunked
Content-Type: text/html

########### GENERAL SECTION ###################
#
BATCH_NAME=web-service
CLIENTS_NUM_MAX=1
CLIENTS_NUM_START=1
CLIENTS_RAMPUP_INC=0
INTERFACE=eth0
NETMASK=24
IP_ADDR_MIN= 172.16.0.181
IP_ADDR_MAX=172.16.0.181
URLS_NUM=8
CYCLES_NUM= 200

########### URLs SECTION #######################

### Login URL - cycling

# GET-part
#
URL=https://login.test.company.com/exec/login
URL_SHORT_NAME="Login-GET"
REQUEST_TYPE=GET
TIMER_URL_COMPLETION = 3000
TIMER_AFTER_URL_SLEEP =1000
LOG_RESP_HEADERS=1
LOG_RESP_BODIES=1

# POST-part
#
URL=""
URL_SHORT_NAME="Login-POST"
URL_USE_CURRENT=1
#USERNAME= "us...@lt..."
#PASSWORD= "testuser"
REQUEST_TYPE=POST
FORM_USAGE_TYPE= RECORDS_FROM_FILE
FORM_RECORDS_FILE= /work/curl-loader/credentials.txt
FORM_STRING= "email=%s&pword=%s"
TIMER_URL_COMPLETION = 3000
TIMER_AFTER_URL_SLEEP =1000
LOG_RESP_HEADERS=1
LOG_RESP_BODIES=1

### Cycling URL
#
URL=https://mc-s100.test.company.com/app/msgctr/junk_quarantine
URL_SHORT_NAME="MC link selection"
REQUEST_TYPE=GET
TIMER_URL_COMPLETION = 3000
TIMER_AFTER_URL_SLEEP =1000
LOG_RESP_HEADERS=1
LOG_RESP_BODIES=1

#
URL=https://mc-s100.test.company.com/app/msgctr/virus_quarantine
URL_SHORT_NAME="Junk"
REQUEST_TYPE=GET
TIMER_URL_COMPLETION = 3000
TIMER_AFTER_URL_SLEEP =1000
LOG_RESP_HEADERS=1
LOG_RESP_BODIES=1

#
URL=https://mc-s100.test.company.com/app/msgctr/delivered_quarantine
URL_SHORT_NAME="Delivered"
REQUEST_TYPE=GET
TIMER_URL_COMPLETION = 3000
TIMER_AFTER_URL_SLEEP =1000
LOG_RESP_HEADERS=1
LOG_RESP_BODIES=1

#
URL=https://mc-s100.test.company.com/app/msgctr/trash
URL_SHORT_NAME="Trash"
REQUEST_TYPE=GET
TIMER_URL_COMPLETION = 3000
TIMER_AFTER_URL_SLEEP =1000
LOG_RESP_HEADERS=1
LOG_RESP_BODIES=1

#
URL=https://mc-s100.test.company.com/app/msgctr/account
URL_SHORT_NAME="Account"
REQUEST_TYPE=GET
TIMER_URL_COMPLETION = 3000
TIMER_AFTER_URL_SLEEP =1000
LOG_RESP_HEADERS=1
LOG_RESP_BODIES=1

### Logoff URL - cycling, uses GET and cookies to logoff.
#
URL=https://mc-s100.test.company.com/app/logout?redir=/app/logout
URL_SHORT_NAME="Logoff"
REQUEST_TYPE=GET
TIMER_URL_COMPLETION = 3000
TIMER_AFTER_URL_SLEEP =0
LOG_RESP_HEADERS=1
LOG_RESP_BODIES=1

Re: URL cycling with staggered URLs

[Robert I. <cor...@gm...>]

Hi John,

On 6/24/07, Robert Iakobashvili <cor...@gm...> wrote:

> > QUESTION/ SUGGESTION/ PATCH:
> >
> > What I want
> >     If I have N URLs and many clients, I would like curl-loader to
> >     if (process % N) == 0 then start on URL 0
> >     if (process % N) == 1 then start on URL 1
> >     if (process % N) == 2 then start on URL 2
> >    (and then if I have more processes than URLs, wrap back to URL 0
> >    when I reach URL N-1)
> >
> > Why do I want this?
> > This means that if I have a large set of URLs (too big for server file cache),
> > I can force the server to work hard at loading files from disk and get
> > a more realisitic load for my server

> We have the two features in our RoadMap/TODO list:
> 2. An option to download a url not only once a cycle, but according to its
>    "Weight", probability.
> 11. Usage of random time intervals, e.g 100-200 (from 100 to 200 msec);

The options are inside the latest version 0.40.
See for the tags FETCH_PROBABILITY and FETCH_PROBABILITY_ONCE
for the first feature.

Random timers taken from an interval are another feature.

Best wishes.

-- 
Sincerely,
Robert Iakobashvili,
coroberti %x40 gmail %x2e com
...........................................................
http://curl-loader.sourceforge.net
A web testing and traffic generation tool.

Re: compile fails on Fedora

[Robert I. <cor...@gm...>]

Jari and Michael,

On 7/11/07, Robert Iakobashvili <cor...@gm...> wrote:
> On 7/11/07, jari pietila <jar...@ho...> wrote:
> > My build attempts on Fedora 5 and 6 failed with the following error below.
> > Is there a way to get around this?
> >
> > -jari
> >
> > # curl-loader-0.40]# make
> > gcc  -g  -o curl-loader  obj/batch.o  obj/cl_alloc.o  obj/client.o
> > obj/conf.o  obj/environment.o  obj/heap.o  obj/ip_secondary.o  obj/loader.o
> > obj/loader_fsm.o  obj/loader_hyper.o  obj/loader_smooth.o  obj/mpool.o
> > obj/parse_conf.o  obj/screen.o  obj/ssl_thr_lock.o  obj/statistics.o
> > obj/timer_queue.o  obj/url.o -L./lib -L/usr//lib -ldl -lpthread -lrt -lcurl
> > -levent -lz -lssl -lcrypto
> > ./lib/libcurl.a(strerror.o): In function `Curl_idn_strerror':
> > ../../../packages/curl/lib/strerror.c:682: undefined
> > reference to `idna_strerror'

Another option to workaround about this is to build libcurl
with a configuration option:
--without-libidn \

Please, find the Makefile for the latest 0.40 version attached.
For other versions you'll need to edit the Makefile manually
and embedd the string above to libcurl target.
Please, substitute or edit the Makefile in curl-loader-0.40
directory and run:
make cleanall; make

Michael, could you, please, verify, that the fix works for your linux
computers? Thanks.

Sincerely,
Robert Iakobashvili,
coroberti %x40 gmail %x2e com
...........................................................
http://curl-loader.sourceforge.net
A web testing and traffic generation tool.

Re: random users on threads

[Robert I. <cor...@gm...>]

Hi Jeremy,

On 7/11/07, Jeremy Brown <jb...@no...> wrote:
> The problem I have is that I need to test a large numbers of users

Great, I like it!
Y can see some recent staff devoted to it here:
http://curl-loader.sourceforge.net/high-load-hw/index.html

> with fewer IP's(threads) then users.

Not understanding this.
Current implementation enables either to use a unique IP-address
for user/client or to use a single common IP-address (which limits
the theoretical number of users/clients to 64K - number of ports to bind()).

All users are doing in a single thread now, unless we are not
separating them to some N number by using -t <N> option.
Thus, lets say when 20 000 users/clients are run with -t 4 option
are separated to 4 sub-batches, each of 5000 users/clients, and
each sub-batch runs independently in a separate thread, where
the first (0) thread collects all statistics.

> I would like to use a random user each time in the thread runs. ie 100,000 users and 400 threads

Threads are quit another concept, and normally lower numbers, e.g. equal to
to the numbers of logical CPUs are recommended.

>I already have a file with the users in the FORM_RECORDS_FILE and
would like to just >grab a new one each time the thread runs.

Would you like to run each cycle a one new user or several new users?
Lets say from a pool of 100 000 users to run each cycle a 1000 users?

> Or I could use >UNIQUE_USERS_SAME_PASSWORD with the userX with the same password and just >enable random on the users for each iteration/cycle.

I got it. There is no currently support for running some random users,
but patches are welcomed!
In any case, we will add your proposal to our TODO list.

-- 
Sincerely,
Robert Iakobashvili,
coroberti %x40 gmail %x2e com
...........................................................
http://curl-loader.sourceforge.net
A web testing and traffic generation tool.

Re: compile fails on Fedora

[Robert I. <cor...@gm...>]

Hi Jari,

On 7/11/07, jari pietila <jar...@ho...> wrote:
> My build attempts on Fedora 5 and 6 failed with the following error below.
> Is there a way to get around this?
>
> -jari
>
> # curl-loader-0.40]# make
> gcc  -g  -o curl-loader  obj/batch.o  obj/cl_alloc.o  obj/client.o
> obj/conf.o  obj/environment.o  obj/heap.o  obj/ip_secondary.o  obj/loader.o
> obj/loader_fsm.o  obj/loader_hyper.o  obj/loader_smooth.o  obj/mpool.o
> obj/parse_conf.o  obj/screen.o  obj/ssl_thr_lock.o  obj/statistics.o
> obj/timer_queue.o  obj/url.o -L./lib -L/usr//lib -ldl -lpthread -lrt -lcurl
> -levent -lz -lssl -lcrypto
> ./lib/libcurl.a(strerror.o): In function `Curl_idn_strerror':
> ../../../packages/curl/lib/strerror.c:682: undefined
> reference to `idna_strerror'

This is an issue described in FAQs.
Taking it from there:

"Another known issue is libidn.so, which means, that some linux
distributions do have some libidn.so.11, but not libidn.so. Resolve it
by creating a softlink.
Sometimes you may need to edit the Makefile and
to remove -lidn from the linking string."

In your case, there is no -lidn in your linking line, and this is not helpful.
Please;

a) return -lidn to the linking line, where it was;
b) run a search for libidn.so.*, and, if not found, install the appropriate
package for your distribution.
c) If some libidn.so.* found or installed, create a softlink, like:
#cd to the directory, where some libidn.so.10 (or 11, etc) is located
#ln -s libidn.so.10 libidn.so
d) remake it.

Please, next time use the filled PROBLEM-REPORTING form

Sincerely,
Robert Iakobashvili,
coroberti %x40 gmail %x2e com
...........................................................
http://curl-loader.sourceforge.net
A web testing and traffic generation tool.

compile fails on Fedora

[jari p. <jar...@ho...>]

Hi,My build attempts on Fedora 5 and 6 failed with the following error belo=
w.Is there a way to get around this?-jari# curl-loader-0.40]# makegcc  -g  =
-o curl-loader  obj/batch.o  obj/cl_alloc.o  obj/client.o  obj/conf.o  obj/=
environment.o  obj/heap.o  obj/ip_secondary.o  obj/loader.o  obj/loader_fsm=
.o  obj/loader_hyper.o  obj/loader_smooth.o  obj/mpool.o  obj/parse_conf.o =
 obj/screen.o  obj/ssl_thr_lock.o  obj/statistics.o  obj/timer_queue.o  obj=
/url.o -L./lib -L/usr//lib -ldl -lpthread -lrt -lcurl -levent -lz -lssl -lc=
rypto./lib/libcurl.a(strerror.o): In function `Curl_idn_strerror':../../../=
packages/curl/lib/strerror.c:682: undefined reference to `idna_strerror'./l=
ib/libcurl.a(url.o): In function `Curl_disconnect':../../../packages/curl/l=
ib/url.c:1872: undefined reference to `idn_free'../../../packages/curl/lib/=
url.c:1876: undefined reference to `idn_free'./lib/libcurl.a(url.o): In fun=
ction `tld_check_name':../../../packages/curl/lib/url.c:2516: undefined ref=
erence to `idna_to_unicode_lzlz'../../../packages/curl/lib/url.c:2520: unde=
fined reference to `tld_check_lz'../../../packages/curl/lib/url.c:2524: und=
efined reference to `tld_strerror'../../../packages/curl/lib/url.c:2535: un=
defined reference to `idn_free'./lib/libcurl.a(url.o): In function `fix_hos=
tname':../../../packages/curl/lib/url.c:2557: undefined reference to `strin=
gprep_check_version'../../../packages/curl/lib/url.c:2560: undefined refere=
nce to `idna_to_ascii_lz'../../../packages/curl/lib/url.c:2561: undefined r=
eference to `stringprep_locale_charset'collect2: ld returned 1 exit statusm=
ake: *** [curl-loader] Error 1[root@tornio curl-loader-0.40]#[root@tornio c=
url-loader-0.40]#[root@tornio curl-loader-0.40]# uname -aLinux tornio 2.6.1=
8-1.2798.fc6 #1 SMP Mon Oct 16 14:37:32 EDT 2006 i686 i686 i386 GNU/Linux[r=
oot@tornio curl-loader-0.40]# gcc -vUsing built-in specs.Target: i386-redha=
t-linuxConfigured with: ../configure --prefix=3D/usr --mandir=3D/usr/share/=
man --infodir=3D/usr/share/info --enable-shared --enable-threads=3Dposix --=
enable-checking=3Drelease --with-system-zlib --enable-__cxa_atexit --disabl=
e-libunwind-exceptions --enable-libgcj-multifile --enable-languages=3Dc,c++=
,objc,obj-c++,java,fortran,ada --enable-java-awt=3Dgtk --disable-dssi --ena=
ble-plugin --with-java-home=3D/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre --wit=
h-cpu=3Dgeneric --host=3Di386-redhat-linuxThread model: posixgcc version 4.=
1.1 20061011 (Red Hat 4.1.1-30)See what you=92re getting into=85before you =
go there. Check it out!
_________________________________________________________________
Local listings, incredible imagery, and driving directions - all in one pla=
ce! Find it!
http://maps.live.com/?wip=3D69&FORM=3DMGAC01=

random users on threads

[Jeremy B. <jb...@no...>]

Hi,

I have been searching through the docs and archives and don't really see =
an answer to this question.=20

The problem I have is that I need to test a large numbers of users with =
fewer IP's(threads) then users. I would like to use a random user each =
time in the thread runs. ie 100,000 users and 400 threads I already have a =
file with the users in the FORM_RECORDS_FILE and would like to just grab a =
new one each time the thread runs. Or I could use UNIQUE_USERS_SAME_PASSWOR=
D with the userX with the same password and just enable random on the =
users for each iteration/cycle. I don't mind if they overlap users at this =
time but it would be nice to have unique users running against product.  =
Instead of tying a user to a thread and having it run for the duration of =
the test. Is this possible with the current options in the configuration?=
=20


Thanks
Jeremy Brown=20

Re: WebLoad HTTP performance numbers

[Vlad W. <wvl...@gm...>]

On 7/10/07, Robert Iakobashvili <cor...@gm...> wrote:
>
> On 7/10/07, Vlad W. <wvl...@gm...> wrote:
> > > For example, I do not know, but Vlad seems to be in position to
> > > comment about Mercury LoadRunner numbers.
>
> > It was long time ago and I don't remember the numbers...
>
> Y are kidding ...


Sometimes.

To conclude, LoadRunner is also not in business of performance and stress
> load testing, but rather in business process testing.


Not exactly. Its place is "application performace testing", including
business process simulation script creating, load testing of the application
this script and multiple virual users, and analysis of obtained statistics.

The business process testing itself, i.e. functional testing, is the area of
WinRunner/QuickTest.

On the high-end we will be competing with IXIA IxLoad and Spirent Avalanche,
> and by adding more context analysis features will fill the gap between
> the Big-HW tools and business process testing (WebLoad, LoadRunner,
> OpenSTA),


That's would be good position.

Vlad