Re: FRESH_CONNECT=1 and TIMER_AFTER_URL_SLEEP =5000

[Robert I. <cor...@gm...>]

On 8/5/07, BuraphaLinux Server <bur...@gm...> wrote:
>
> Hello,
>
>     In testing I did not see any difference.  In the logs I get blocks
> like this when curl-loader is cycling too fast (sorry gmail wraps
> badly):


The difference is when FRESH_CONNECT=1 and sleep time positive
curl-loader is supposed to close its connection to server, go to sleep
for the sleep timeout, and re-establish the connection after sleeping.

If you will increase the sleeping time, you can hopefully observe the
behavior in sniffer.


: eff-url:
> ftp://anonymous:joe%040@10.16.68.213/apache/httpd/httpd-2.0.59.tar.bz2,
> url:
> 121 20 (10.16.68.197)  <= Recv header: eff-url:
> ftp://anonymous:joe%040@10.16.68.213/apache/httpd/httpd-2.0.59.tar.bz2,
> url:
> 121 20 (10.16.68.197) :<= WARNING: parsing error: wrong response code
> (FTP?) 0 .
> 121 20 (10.16.68.197)  !! ERROR: This doesn't seem like a nice
> ftp-server response
> : eff-url:
> ftp://anonymous:joe%040@10.16.68.213/apache/httpd/httpd-2.0.59.tar.bz2,
> url:
> 121 20 (10.16.68.197)  :== Info: Expire cleared
> : eff-url:
> ftp://anonymous:joe%040@10.16.68.213/apache/httpd/httpd-2.0.59.tar.bz2,
> url:
> 121 20 (10.16.68.197)  :== Info: Closing connection #117
>
> I worry that 'parsing error' means it doesn't do the delay, but I
> don't really know.


it's not the issue.

The server side of this will say this in the vsftpd log file:
>
> Sun Aug  5 18:17:58 2007 [pid 10839] CONNECT: Client "10.16.68.197",
> "Connection refused: too many sessions."
>
> Using ss -tan (or netstat) I can verify there are too many connections
> on the server.
> The vsftpd server has been set for 100 clients with these entries in
> vsftpd.conf:
>
> #
> # Maximum number of simultaneous clients
> #
> max_clients=100
> #
> # Maximum number of connections from 1 ip
> #
> max_per_ip=100
>
> (and I painfully tested by hand that this limit works as expected)
>
> My configuration is ftp.small for curl-loader and is this:
>
> ########### GENERAL SECTION ################################
>
> BATCH_NAME= ftpsmall
> CLIENTS_NUM_MAX=50 # Same as CLIENTS_NUM
> CLIENTS_NUM_START=5
> CLIENTS_RAMPUP_INC=5
> INTERFACE=eth0
> NETMASK=32
> IP_ADDR_MIN=10.16.68.197
> IP_ADDR_MAX=10.16.68.197
> CYCLES_NUM=-1
> URLS_NUM=1
>
> ########### URL SECTION ####################################
>
> URL=ftp://anonymous:joe%040@10.16.68.213/apache/httpd/httpd-2.0.59.tar.bz2
> FRESH_CONNECT=1
> URL_SHORT_NAME="small"
> TIMER_URL_COMPLETION = 0 # In msec. When positive, Now it is enforced
> by cancelling url fetch on timeout
> TIMER_AFTER_URL_SLEEP =5000
> TIMER_TCP_CONN_SETUP=50


Looks good, but I would  make TIMER_TCP_CONN_SETUP not larger than 10
seconds.

and was invoked with this script (as root user):
>
> #! /sbin/bash
> rm -fr ftpsmall.*
> ulimit -d unlimited
> ulimit -f unlimited
> ulimit -m unlimited
> ulimit -n 19999
> ulimit -t unlimited
> ulimit -u unlimited
> ulimit -v unlimited
> ulimit -x unlimited
> echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle
> echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse
> echo 1 > /proc/sys/net/ipv4/tcp_ecn
> echo 1 > /proc/sys/net/ipv4/tcp_window_scaling
> curl-loader -f ftp.small -v -u -w
>
> With the limit set to 50 on the client and the delays,



OK.

Do you see connections closed by client (curl-loader)
in  network  (sniffer wireshark/ethereal capture with a single curl-loader
client could be of
assistance) and re-established after 5 seconds,
or the connections are remaining stalled?

We are calling libcurl function, which is supposed to release the
connection,
but if this is buggy, thus, we can dig into the issue.

Another option is that the release by FTP could take some time.
Could you try a larger timeout, like 10000, 15 000?

Sincerely,
Robert Iakobashvili,
coroberti %x40 gmail %x2e com
...........................................................
http://curl-loader.sourceforge.net
A web testing and traffic generation tool.