[curl-loader-devel] [PROF RFC 1/2] profiling of libcurl by curl-loader, steady state

[Robert I. <cor...@gm...>]

CURL VERSION:
snapshot 7.16.2-20070326
with a
recently submitted patch for optimization
of curl_multi_socket () by passing bitmask,
named here curl_multi_socket_noselect
(renamed to curl_multi_socket_action)

HOST MACHINE and OPERATING SYSTEM:
 linux, debian with vanilla kernel 2.6.20.6

COMPILER:
gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)

CURL CONFIGURATION:
configure --prefix $(CURL_BUILD) \
        --enable-thread \
        --enable-ipv6 \
        --with-random=/dev/urandom \
        --with-ssl=/usr/include/openssl \
        --enable-shared=no \
        CFLAGS="-pg -g  -O3 -ffast-math -finline-functions -funroll-all-loops \
	-finline-limit=1000 -mmmx -msse -foptimize-sibling-calls -mtune=pentium4 \
        -mcpu=pentium4"

COMPILATION OPTIONS:
are the same for the application, libcurl and libevent:

-pg -g  -O3 -ffast-math -finline-functions -funroll-all-loops \
-finline-limit=1000 -mmmx -msse -foptimize-sibling-calls -mtune=pentium4 \
-mcpu=pentium4



    CURL_EASY_SETOPT:

The curl_easy_setopt used are:
CURLOPT_IPRESOLVE
CURLOPT_INTERFACE
CURLOPT_NOSIGNAL
CURLOPT_URL with http://localhost/ACE-INSTALL.html (104K file)
CURLOPT_CONNECTTIMEOUT 5
CURLOPT_FRESHCONNECT 0
CURLOPT_DNS_CACHE_TIMEOUT with -1
CURLOPT_VERBOSE with 1
CURLOPT_DEBUGFUNCTION with client_tracing_function
CURLOPT_DEBUGDATA, cctx;
CURLOPT_WRITEFUNCTION with do_nothing_write_function
CURLOPT_SSL_VERIFYPEER 0 /* not used, http-url */
CURLOPT_SSL_VERIFYHOST 0 /* not used, http-url */

CURLOPT_PRIVATE, cctx
CURLOPT_WRITEDATA, cctx - for hyper
CURLOPT_ERRORBUFFER, bctx->error_buffer

CURLOPT_FOLLOWLOCATION, 1 /* not used, no 3xx */
CURLOPT_UNRESTRICTED_AUTH, 1 /* not used, no auth */
CURLOPT_MAXREDIRS, -1 /* not used, no 3xx */
CURLOPT_USERAGENT, bctx->user_agent /*MSIE-6 like string */
CURLOPT_COOKIEFILE, "" /* not used, no cookies*/


SERVER:
lighttpd-1.4.13 (ssl) from debian,
the stock configuration file with the more strings
added:
server.event-handler = "linux-sysepoll"
server.max-fds = 32000

FILE FETCHED:
104K static file

CLIENT APPLICATION:
curl-loader

CLIENT LOADING MODES:
Smooth mode - using curl_multi_perform based on poll ()
                          demultiplexing;
Hyper mode - based on hipev.c example and using epoll ()
based demultiplexing with further curl_multi_socket_noselect ()

CLIENT LOAD:
For each loading mode (smooth and hyper) two loads have been
run each for the time 200 seconds:
- 200 libcurl clients loading localhost server with 0 msec between
the fetched file and the new request;
- 200 libcurl clients localhost server with 1000 msec between
the fetched file and the new request

The clients have been started simulteneously without any gradual
increase, however, due to the 200 seconds of the run, the impact
of connection establishment is rather low.

PROFILING METHODS:
gprof - provides application level information - collected
to files suffixed *.prof;
oprofile- whole system information, including glibc
and kernel functions, filtered for the relevant curl-loader
application to files suffixed *.op;


RESULTING FILES:
Smooth mode experiments:
smooth-mode.profiling.tar.bz2:

smooth-0.prof - gprof res of smooth mode, 0 msec delay between requests;
smooth-0.op - oprofile res of smooth mode, 0 msec delay between requests;
smooth-1000.prof - gprof res of smooth mode, 1000 msec delay b requests;
smooth-1000.op - oprofile res of smooth mode, 1000 msec delay b requests;


Hyper mode experiments:
hyper-mode.profiling.tar.bz2:
hyper-0.prof - gprof res of hyper mode, 0 msec delay between requests;
hyper-0.op - oprofile res of hyper mode, 0 msec delay between requests;
hyper-1000.prof - gprof res of hyper mode, 1000 msec delay b requests;
hyper-1000.op - oprofile res of hyper mode, 1000 msec delay b requests;


The recommendations and analyses of curl gurus would be very much
appreciated. Any your feedback would be highly valued.

Profiling of the loads with high number of connection establishements
/ new clients
per second is out of the mail and will be a subject of some next
e-mail (hopefully). In meanwhile, very premature results (without engagement)
50 new clients per second are totally burning out my CPU with
Curl_connect () mainly is the CPU eater.

Sincerely,
Robert Iakobashvili,
coroberti %x40 gmail %x2e com
...................................................................
Navigare necesse est, vivere non est necesse
...................................................................
http://curl-loader.sourceforge.net
An open-source HTTP/S, FTP/S traffic
generating, and web testing tool.