RE: Regarding the randomization of source IP and content fetched

[SAJAL B. <s.b...@qu...>]

Hi,

Thanks for your reply.

The last question was more like if the current version of Curl Loader has ability to download recursively, or even just download everything that is referred to from a remote resource, be it a HTML page or a FTP directory listing, similar to -r parameter in wget.

Thanks and Regards
----
Sajal Bhatia
Research Masters Student
QUT, Brisbane
AUSTRALIA

________________________________
From: Robert Iakobashvili [cor...@gm...]
Sent: Tuesday, 11 May 2010 9:23 PM
To: curl-loader-devel
Subject: Re: Regarding the randomization of source IP and content fetched

Hi Sajal,

On Tue, May 11, 2010 at 8:29 AM, SAJAL BHATIA <s.b...@qu...<mailto:s.b...@qu...>> wrote:


 *
Can we randomize the source IP within a given range. I mean if we specify the IP address range to be from 172.168.1.1/16<http://172.168.1.1/16> to 172.168.255.254/16<http://172.168.255.254/16>, and have maximum number of clients to be 10 K, then can we assign random IP addresses to these 5 K client from this range

Not in the current version, but it is a rather small development.


 *
 *
Another question is related to randomizing the fetched content. Can we make each of these clients to fetch some random pages from a web site?

If you'll take the version from the subversion,
it has URL probability feature restored.

Look into the URL-probability.

 *
 *
Can we ask any 'n' number of clients from these 10 K clients to recursively fetch a given web page 'x' number of times?

There is no support for sub-batches (groups).

Y can either arrange another client computer with such purpose,
or to play with URL-probability tag.


 *

The main motive is to create a web based data set as close as possible to genuine traffic.



Sure, good questions.
Patches are welcomed!


Thanks and Regards

----
Sajal Bhatia
Research Masters Student
QUT, Brisbane
AUSTRALIA

------------------------------------------------------------------------------


_______________________________________________
curl-loader-devel mailing list
cur...@li...<mailto:cur...@li...>
https://lists.sourceforge.net/lists/listinfo/curl-loader-devel




--
Truly,
Robert Iakobashvili, Ph.D.
......................................................................
www.ghotit.com<http://www.ghotit.com>
Assistive technology that understands you
......................................................................