Re: curl-loader https: low performance due to DNS query?

[Robert I. <cor...@gm...>]

Sorry, Fred.

I'm poised to think that thread-locking with openssl
is not the issue here but rather libcurl interaction with openssl is to blame.

The fix could be done only in libcurl or even in openssl.

If you could manage to compile openssl with -pg -O2 and
to pass these openssl binaries to linking,
perhaps, you could see more in your profiling.

Sincerely,
Robert

On Wed, Mar 7, 2012 at 2:43 AM, Fred Huang <di...@gm...> wrote:
> yes, I use -t 8 option because the test box has 8 logic CPUs (Dell 990,
> Intel Core i7 2600, 16GB memory).
> if I do not use -t option, the CPU used by curl-loader will be 100% on one
> of 8 logic CPU (12% of total CPU), and the CAPS drops to 100:
> # gprof /usr/bin/curl-loader gmon.out -p | head -20
> Flat profile:
> Each sample counts as 0.01 seconds.
>   %   cumulative   self              self     total
>  time   seconds   seconds    calls   s/call   s/call  name
>  34.12      2.74     2.74    17385     0.00     0.00  create_conn
>  11.21      3.64     0.90 30998583     0.00     0.00
>  hostcache_timestamp_remove
>   9.84      4.43     0.79    13433     0.00     0.00
>  Curl_hash_clean_with_criterium
>   7.97      5.07     0.64   381818     0.00     0.00  Curl_hash_pick
>   5.85      5.54     0.47    17385     0.00     0.00  ConnectionStore
>   5.11      5.95     0.41    16385     0.00     0.00
>  curl_multi_remove_handle
>   4.11      6.28     0.33 16509889     0.00     0.00  Curl_str_key_compare
>   3.36      6.55     0.27 23959914     0.00     0.00  Curl_socket_check
>   3.24      6.81     0.26    56703     0.00     0.00  Curl_hash_add
>   2.62      7.02     0.21   635750     0.00     0.00  dprintf_formatf
>   1.25      7.12     0.10 92421190     0.00     0.00  ares__is_list_empty
>   0.75      7.18     0.06    97957     0.00     0.00  Curl_readwrite
>   0.62      7.23     0.05 23422260     0.00     0.00  addbyter
>   0.62      7.28     0.05  1136775     0.00     0.00
>  client_tracing_function
>   0.50      7.32     0.04  3091375     0.00     0.00  ares__init_list_head
>
> I also tried to use
>  curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 20);
> but get no improvement on CAPS.
>
>
> Regards,
> Fred
>
> 2012/3/7 Robert Iakobashvili <cor...@gm...>
>>
>> Dear Fred,
>>
>> If I understand correctly your observations, the issue is not in:
>>
>> 1. DNS resolving;
>> 2. Version of cares or libcurl;
>>
>> but rather in
>>
>> Less effective handling of HTTPS versus HTTP in libcurl.
>>
>> Please, confirm that my understanding is correct.
>>
>>
>> HTTPS URLs are using an extra locking in openssl with -t option
>> that could be making the issue.
>> If you are using -t option, try to load without it and let's say with
>> 300  clients.
>>
>> If you are not using -t, thus the only way to proceed is to share
>> your observations with libcurl people at
>>
>> libcurl development <cur...@co...>
>>
>> Thank you.
>>
>> Best wishes.
>> Robert
>>
>> On Tue, Mar 6, 2012 at 6:17 PM, Fred Huang <di...@gm...> wrote:
>> > 1. the major CPU consumer is curl-loader. here is the top:
>> > top - 07:36:36 up 23:31,  5 users,  load average: 4.32, 2.26, 0.91
>> > Tasks: 237 total,   2 running, 235 sleeping,   0 stopped,   0 zombie
>> > Cpu0  : 76.6%us,  3.6%sy,  0.0%ni, 19.4%id,  0.0%wa,  0.0%hi,  0.3%si,
>> >  0.0%
>> > Cpu1  : 72.4%us,  4.0%sy,  0.0%ni, 22.9%id,  0.0%wa,  0.0%hi,  0.7%si,
>> >  0.0%
>> > Cpu2  : 65.8%us,  3.3%sy,  0.0%ni, 30.6%id,  0.0%wa,  0.0%hi,  0.3%si,
>> >  0.0%
>> > Cpu3  : 75.2%us,  4.3%sy,  0.0%ni, 19.5%id,  0.0%wa,  0.0%hi,  1.0%si,
>> >  0.0%
>> > Cpu4  : 79.1%us,  4.0%sy,  0.0%ni, 16.8%id,  0.0%wa,  0.0%hi,  0.0%si,
>> >  0.0%
>> > Cpu5  : 75.8%us,  4.3%sy,  0.0%ni, 19.2%id,  0.0%wa,  0.0%hi,  0.7%si,
>> >  0.0%
>> > Cpu6  : 74.1%us,  5.0%sy,  0.0%ni, 20.3%id,  0.0%wa,  0.0%hi,  0.7%si,
>> >  0.0%
>> > Cpu7  : 78.7%us,  4.0%sy,  0.0%ni, 16.6%id,  0.0%wa,  0.0%hi,  0.7%si,
>> >  0.0%
>> > Mem:  16209932k total,  3829500k used, 12380432k free,    99112k buffers
>> > Swap: 18448376k total,        0k used, 18448376k free,   320284k cached
>> >
>> >   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>> > 30795 root      20   0 3200m 2.8g 2296 S 79.2 18.4  21:26.84 curl-loader
>> >  1832 nobody    20   0 13944 1736  580 S  0.1  0.0   0:32.65 dnsmasq
>> >     9 root      20   0     0    0    0 S  0.0  0.0   0:00.53 ksoftirqd/1
>> > 30825 root      20   0 15220 1360  960 R  0.0  0.0   0:00.42 top
>> >     1 root      20   0 19400 1568 1256 S  0.0  0.0   0:00.75 init
>> >     2 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kthreadd
>> >     3 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/0
>> >     4 root      20   0     0    0    0 S  0.0  0.0   0:00.06 ksoftirqd/0
>> >     5 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/0
>> >
>> >
>> > 2. 1000 loading clients
>> >
>> > 3. yes, tried 0.52 but no lucky -- CAPS 380, throughput 200Mbps, CPU 90%
>> > top - 08:17:10 up 1 day, 12 min,  8 users,  load average: 4.66, 3.39,
>> > 1.88
>> > Tasks: 245 total,   2 running, 243 sleeping,   0 stopped,   0 zombie
>> > Cpu0  : 82.2%us,  4.0%sy,  0.0%ni, 13.9%id,  0.0%wa,  0.0%hi,  0.0%si,
>> >  0.0%
>> > Cpu1  : 86.1%us,  4.3%sy,  0.0%ni,  7.9%id,  0.0%wa,  0.0%hi,  1.7%si,
>> >  0.0%
>> > Cpu2  : 85.3%us,  4.0%sy,  0.0%ni, 10.0%id,  0.0%wa,  0.0%hi,  0.7%si,
>> >  0.0%
>> > Cpu3  : 81.8%us,  3.0%sy,  0.0%ni, 14.2%id,  0.0%wa,  0.0%hi,  1.0%si,
>> >  0.0%
>> > Cpu4  : 83.7%us,  4.0%sy,  0.0%ni, 12.3%id,  0.0%wa,  0.0%hi,  0.0%si,
>> >  0.0%
>> > Cpu5  : 92.7%us,  3.3%sy,  0.0%ni,  3.3%id,  0.0%wa,  0.0%hi,  0.7%si,
>> >  0.0%
>> > Cpu6  : 89.7%us,  2.0%sy,  0.0%ni,  8.0%id,  0.0%wa,  0.0%hi,  0.3%si,
>> >  0.0%
>> > Cpu7  : 88.1%us,  3.6%sy,  0.0%ni,  6.6%id,  0.0%wa,  0.0%hi,  1.7%si,
>> >  0.0%
>> > Mem:  16209932k total,  1795748k used, 14414184k free,   102824k buffers
>> > Swap: 18448376k total,        0k used, 18448376k free,   382988k cached
>> >
>> >   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>> > 12236 root      20   0 1150m 894m 2572 S 90.7  5.7  51:11.91 curl-loader
>> >  1832 nobody    20   0 13944 1736  580 S  0.1  0.0   0:47.04 dnsmasq
>> >
>> > ]# gprof /usr/bin/curl-loader gmon.out -p | head -20
>> > Flat profile:
>> > Each sample counts as 0.01 seconds.
>> >   %   cumulative   self              self     total
>> >  time   seconds   seconds    calls   s/call   s/call  name
>> >  43.92    151.95   151.95   157295     0.00     0.00
>> >  Curl_hash_clean_with_criterium
>> >  22.90    231.17    79.22 1712190971     0.00     0.00
>> >  hostcache_timestamp_remove
>> >  13.15    276.68    45.51  3260321     0.00     0.00  Curl_hash_pick
>> >   7.51    302.65    25.97   394171     0.00     0.00  Curl_hash_add
>> >   5.98    323.34    20.68 567437518     0.00     0.00
>> >  Curl_str_key_compare
>> >   1.00    326.80     3.46  5380220     0.00     0.00  dprintf_formatf
>> >   0.41    328.21     1.41                             locking_function
>> >   0.32    329.32     1.11 469222138     0.00     0.00
>> >  ares__is_list_empty
>> >   0.31    330.40     1.08  3284508     0.00     0.00  multi_runsingle
>> >   0.28    331.38     0.98 10006806     0.00     0.00
>> >  client_tracing_function
>> >   0.24    332.21     0.83   151951     0.00     0.00  Curl_num_addresses
>> >   0.22    332.97     0.77 188075672     0.00     0.00  addbyter
>> >   0.19    333.64     0.67   841840     0.00     0.00  Curl_readwrite
>> >   0.19    334.29     0.65                             id_function
>> >   0.18    334.90     0.61    39843     0.00     0.01
>> >  curl_multi_socket_all
>> >
>> >
>> >
>> > 4. yes, it is correct for 0.56. If use same URL list by replacing https
>> > to
>> > http, I can get CAPS 1500 and throughput 870Mbps with CPU% < 60%
>> >
>> >
>> > 2012/3/6 Robert Iakobashvili <cor...@gm...>
>> >>
>> >> Several questions:
>> >>
>> >> 1. Could you please run top and see if there are other processes
>> >> burning
>> >> CPU?
>> >> 2. What is the number of loading clients that you use?
>> >> 3. Could you try curl-loader-0.52 if it compiles for you?
>> >> If not, please try to place curl-loader-7.19.7.tar.gz and
>> >> c-ares-16.0.tar.gz,
>> >> update their versions in Makefile and rebuild by make cleanall; make
>> >>
>> >> 4. Is it correct to say that for HTTP URLs (not HTTPS) you are getting
>> >> a good throughput
>> >>    and CAPS with curl-loader-0.56?
>> >>
>> >> Thanks,
>> >> Robert
>> >>
>> >> On Tue, Mar 6, 2012 at 3:52 AM, Fred Huang <di...@gm...> wrote:
>> >> > TIMEOUT -1 -> 60 works a little better but no performance improvement
>> >> > on
>> >> > CAPS or throughput.
>> >> >
>> >> > test 5:
>> >> > enable c-ares;
>> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60);
>> >> >
>> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve
>> >> > *.com
>> >> > to
>> >> > one IP address
>> >> > number of domain names in URLs: 780,000
>> >> > number of client: 1000
>> >> > CPU usage: 74%
>> >> >     cpu%  irq% sirq%  sys% iowt%  mem_used  buf&cached
>> >> >     74.0   0.0   0.7   4.2   0.0  3530.6Mb     250.8Mb
>> >> > SSL TPS: 360
>> >> > SSL throughput: 220Mbps
>> >> >
>> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -50
>> >> > Flat profile:
>> >> > Each sample counts as 0.01 seconds.
>> >> >   %   cumulative   self              self     total
>> >> >  time   seconds   seconds    calls   s/call   s/call  name
>> >> >  26.63     23.13    23.13   119226     0.00     0.00
>> >> >  Curl_hash_clean_with_criterium
>> >> >  20.34     40.80    17.67 317792146     0.00     0.00
>> >> >  hostcache_timestamp_remove
>> >> >  11.27     50.59     9.79  2579903     0.00     0.00  Curl_hash_pick
>> >> >   7.84     57.40     6.81   120606     0.00     0.00  create_conn
>> >> >   5.65     62.31     4.91 119117401     0.00     0.00
>> >> >  Curl_str_key_compare
>> >> >   4.98     66.64     4.33   329829     0.00     0.00  Curl_hash_add
>> >> >   2.80     69.08     2.44  4476606     0.00     0.00  dprintf_formatf
>> >> >   2.57     71.31     2.23   124019     0.00     0.00  ConnectionStore
>> >> >   1.50     72.61     1.30 740997609     0.00     0.00
>> >> >  ares__is_list_empty
>> >> >   1.16     73.61     1.01
>> >> > locking_function
>> >> >   0.76     74.27     0.66   238744     0.00     0.00  ares_cancel
>> >> >   0.70     74.88     0.61   457564     0.00     0.00
>> >> >  curl_multi_socket_action
>> >> >   0.67     75.46     0.58   118616     0.00     0.00
>> >> >  curl_multi_remove_handle
>> >> >   0.58     75.97     0.51  8211502     0.00     0.00
>> >> >  client_tracing_function
>> >> >   0.53     76.43     0.46   733561     0.00     0.00  Curl_readwrite
>> >> >   0.51     76.87     0.44  2183179     0.00     0.00  multi_runsingle
>> >> >   0.51     77.31     0.44 166585567     0.00     0.00  addbyter
>> >> >   0.44     77.69     0.38  2175805     0.00     0.00  Curl_raw_equal
>> >> >   0.43     78.06     0.38                             id_function
>> >> >   0.40     78.41     0.35 29789346     0.00     0.00
>> >> >  Curl_socket_check
>> >> >   0.40     78.76     0.35   479671     0.00     0.00
>> >> >  ossl_connect_common
>> >> >   0.40     79.11     0.35   490639     0.00     0.00
>> >> >  ares__init_list_node
>> >> >   0.33     79.40     0.29 94197670     0.00     0.00
>> >> >  Curl_raw_toupper
>> >> >   0.33     79.69     0.29  3268031     0.00     0.00  Curl_pgrsUpdate
>> >> >   0.28     79.93     0.24     8797     0.00     0.00
>> >> >  curl_multi_perform
>> >> >   0.27     80.17     0.24  2233641     0.00     0.00  Curl_infof
>> >> >   0.26     80.40     0.23  1839837     0.00     0.00  Curl_splay
>> >> >   0.24     80.61     0.21 121713028     0.00     0.00  curl_strequal
>> >> >   0.21     80.79     0.18   243333     0.00     0.00
>> >> >  Curl_ssl_getsessionid
>> >> >   0.18     80.95     0.16   122180     0.00     0.00
>> >> >  Curl_num_addresses
>> >> >   0.17     81.09     0.15  8271666     0.00     0.00  Curl_debug
>> >> >   0.16     81.23     0.14 18902452     0.00     0.00  alloc_addbyter
>> >> >   0.16     81.37     0.14     8851     0.00     0.00
>> >> >  curl_multi_socket_all
>> >> >   0.14     81.50     0.13  2236769     0.00     0.00  Curl_read
>> >> >   0.14     81.62     0.12  2271700     0.00     0.00  fd_key_compare
>> >> >   0.14     81.74     0.12   323167     0.00     0.00  event_del
>> >> >   0.14     81.86     0.12   205436     0.00     0.00
>> >> >  read_udp_packets
>> >> >   0.14     81.98     0.12   200979     0.00     0.00  processfds
>> >> >   0.13     82.09     0.11   123886     0.00     0.00
>> >> >  Curl_if_is_interface_name
>> >> >   0.13     82.20     0.11   120350     0.00     0.00  Curl_http
>> >> >   0.13     82.31     0.11                             ssl_tls_trace
>> >> >   0.12     82.41     0.10   119229     0.00     0.00
>> >> >  Curl_splayremovebyaddr
>> >> >   0.12     82.51     0.10  7798717     0.00     0.00  Curl_raw_nequal
>> >> >   0.12     82.61     0.10   784485     0.00     0.00  event_add
>> >> >   0.10     82.70     0.09   643482     0.00     0.00  Curl_expire
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > test 6:
>> >> > enable c-ares;
>> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60);
>> >> > CURL_VER:=7.24.0-20120109 ->  CURL_VER:=7.23.0
>> >> >
>> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve
>> >> > *.com
>> >> > to
>> >> > one IP address
>> >> > number of domain names in URLs: 780,000
>> >> > number of client: 1000
>> >> > CPU usage: 86%
>> >> >     cpu%  irq% sirq%  sys% iowt%  mem_used  buf&cached
>> >> >      85.9   0.0   0.2   2.1   0.0  3537.2Mb     261.4Mb
>> >> > SSL TPS: 360
>> >> > SSL throughput: 200Mbps
>> >> >
>> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20
>> >> > Flat profile:
>> >> > Each sample counts as 0.01 seconds.
>> >> >   %   cumulative   self              self     total
>> >> >  time   seconds   seconds    calls   s/call   s/call  name
>> >> >  27.19     19.96    19.96   103472     0.00     0.00
>> >> >  Curl_hash_clean_with_criterium
>> >> >  21.73     35.91    15.95 274168981     0.00     0.00
>> >> >  hostcache_timestamp_remove
>> >> >  10.33     43.49     7.58  2256441     0.00     0.00  Curl_hash_pick
>> >> >   7.21     48.78     5.29   104540     0.00     0.00  create_conn
>> >> >   6.20     53.33     4.55 103413405     0.00     0.00
>> >> >  Curl_str_key_compare
>> >> >   4.88     56.91     3.58   287109     0.00     0.00  Curl_hash_add
>> >> >   3.02     59.13     2.22  3797855     0.00     0.00  dprintf_formatf
>> >> >   2.10     60.67     1.54   107711     0.00     0.00  ConnectionStore
>> >> >   1.65     61.88     1.21 642725327     0.00     0.00
>> >> >  ares__is_list_empty
>> >> >   1.19     62.76     0.88
>> >> > locking_function
>> >> >   0.87     63.40     0.64   207817     0.00     0.00  ares_cancel
>> >> >   0.56     63.81     0.41   404108     0.00     0.00
>> >> >  curl_multi_socket_action
>> >> >   0.54     64.21     0.40   425603     0.00     0.00
>> >> >  ares__init_list_node
>> >> >   0.53     64.60     0.39  7068112     0.00     0.00
>> >> >  client_tracing_function
>> >> >   0.49     64.96     0.36  1908587     0.00     0.00  multi_runsingle
>> >> >
>> >> >
>> >> >
>> >> > test 7:
>> >> > enable c-ares;
>> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1);
>> >> > CURL_VER:=7.24.0-20120109 ->  CURL_VER:=7.23.0
>> >> >
>> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve
>> >> > *.com
>> >> > to
>> >> > one IP address
>> >> > number of domain names in URLs: 780,000
>> >> > number of client: 1000
>> >> > CPU usage: 86%
>> >> >     cpu%  irq% sirq%  sys% iowt%  mem_used  buf&cached
>> >> >     86.6   0.0   0.6   3.3   0.0  3624.8Mb     263.0Mb
>> >> > SSL TPS: 300
>> >> > SSL throughput: 200Mbps
>> >> >
>> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20
>> >> > Flat profile:
>> >> > Each sample counts as 0.01 seconds.
>> >> >   %   cumulative   self              self     total
>> >> >  time   seconds   seconds    calls   s/call   s/call  name
>> >> >  39.83    369.71   369.71   207173     0.00     0.00
>> >> >  Curl_hash_clean_with_criterium
>> >> >  24.27    594.98   225.27 3198337733     0.00     0.00
>> >> >  hostcache_timestamp_remove
>> >> >  13.75    722.59   127.61  4312864     0.00     0.00  Curl_hash_pick
>> >> >   9.19    807.87    85.28   546507     0.00     0.00  Curl_hash_add
>> >> >   6.14    864.88    57.01 1022419684     0.00     0.00
>> >> >  Curl_str_key_compare
>> >> >   2.06    884.00    19.12   203749     0.00     0.00  create_conn
>> >> >   0.54    888.99     4.99   213306     0.00     0.00  ConnectionStore
>> >> >   0.49    893.50     4.51  7331251     0.00     0.00  dprintf_formatf
>> >> >   0.29    896.16     2.66 1287182329     0.00     0.00
>> >> >  ares__is_list_empty
>> >> >   0.23    898.28     2.12
>> >> > locking_function
>> >> >   0.20    900.10     1.82   212210     0.00     0.00
>> >> >  Curl_num_addresses
>> >> >   0.17    901.66     1.56   198385     0.00     0.00
>> >> >  curl_multi_remove_handle
>> >> >   0.12    902.81     1.15   411114     0.00     0.00  ares_cancel
>> >> >   0.12    903.95     1.14  3683448     0.00     0.00  multi_runsingle
>> >> >   0.11    904.97     1.03 13664120     0.00     0.00
>> >> >  client_tracing_function
>> >> >
>> >> >
>> >> > test 8:
>> >> > enable c-ares;
>> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60);
>> >> > CURL_VER:=7.24.0-20120109 ->  CURL_VER:=7.24.0
>> >> >
>> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve
>> >> > *.com
>> >> > to
>> >> > one IP address
>> >> > number of domain names in URLs: 780,000
>> >> > number of client: 1000
>> >> > CPU usage: 75%
>> >> >     cpu%  irq% sirq%  sys% iowt%  mem_used  buf&cached
>> >> >      75.2   0.0   0.2   2.1   0.0  3537.2Mb     261.4Mb
>> >> > SSL TPS: 380
>> >> > SSL throughput: 220Mbps
>> >> >
>> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20
>> >> > Flat profile:
>> >> > Each sample counts as 0.01 seconds.
>> >> >   %   cumulative   self              self     total
>> >> >  time   seconds   seconds    calls   s/call   s/call  name
>> >> >  27.58     12.31    12.31    71140     0.00     0.00
>> >> >  Curl_hash_clean_with_criterium
>> >> >  19.09     20.83     8.52 176729692     0.00     0.00
>> >> >  hostcache_timestamp_remove
>> >> >  10.44     25.49     4.66  1589309     0.00     0.00  Curl_hash_pick
>> >> >   6.85     28.55     3.06    73204     0.00     0.00  create_conn
>> >> >   5.56     31.03     2.48 67620530     0.00     0.00
>> >> >  Curl_str_key_compare
>> >> >   4.41     33.00     1.97   200684     0.00     0.00  Curl_hash_add
>> >> >   3.56     34.59     1.59  2717956     0.00     0.00  dprintf_formatf
>> >> >   1.72     35.36     0.77    75247     0.00     0.00  ConnectionStore
>> >> >   1.70     36.12     0.76 444342256     0.00     0.00
>> >> >  ares__is_list_empty
>> >> >   1.23     36.67     0.55
>> >> > locking_function
>> >> >   1.16     37.19     0.52   143342     0.00     0.00  ares_cancel
>> >> >   0.99     37.63     0.44    72211     0.00     0.00
>> >> >  curl_multi_remove_handle
>> >> >   0.74     37.96     0.33  1333371     0.00     0.00  multi_runsingle
>> >> >   0.68     38.27     0.31 101147798     0.00     0.00  addbyter
>> >> >   0.58     38.53     0.26   288737     0.00     0.00
>> >> >  curl_multi_socket_action
>> >> >
>> >> >
>> >> >
>> >> > test 9:
>> >> > enable c-ares;
>> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1);
>> >> > CURL_VER:=7.24.0-20120109 ->  CURL_VER:=7.24.0
>> >> >
>> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve
>> >> > *.com
>> >> > to
>> >> > one IP address
>> >> > number of domain names in URLs: 780,000
>> >> > number of client: 1000
>> >> > CPU usage: 75%
>> >> >     cpu%  irq% sirq%  sys% iowt%  mem_used  buf&cached
>> >> >     75.1   0.0   0.6   4.1   0.0  3545.2Mb     260.1Mb
>> >> > SSL TPS: 380
>> >> > SSL throughput: 200Mbps
>> >> >
>> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20
>> >> > Flat profile:
>> >> > Each sample counts as 0.01 seconds.
>> >> >   %   cumulative   self              self     total
>> >> >  time   seconds   seconds    calls   s/call   s/call  name
>> >> >  34.00     31.13    31.13    72495     0.00     0.00
>> >> >  Curl_hash_clean_with_criterium
>> >> >  23.64     52.78    21.65 386070412     0.00     0.00
>> >> >  hostcache_timestamp_remove
>> >> >  13.28     64.94    12.16  1619440     0.00     0.00  Curl_hash_pick
>> >> >   5.92     70.36     5.42 137653129     0.00     0.00
>> >> >  Curl_str_key_compare
>> >> >   5.50     75.40     5.04   203018     0.00     0.00  Curl_hash_add
>> >> >   4.55     79.57     4.17    74618     0.00     0.00  create_conn
>> >> >   1.70     81.13     1.56  2757148     0.00     0.00  dprintf_formatf
>> >> >   0.98     82.03     0.90    76945     0.00     0.00  ConnectionStore
>> >> >   0.95     82.90     0.87 455775100     0.00     0.00
>> >> >  ares__is_list_empty
>> >> >   0.57     83.42     0.53
>> >> > locking_function
>> >> >   0.39     83.78     0.36   146742     0.00     0.00  ares_cancel
>> >> >   0.35     84.10     0.32 102968806     0.00     0.00  addbyter
>> >> >   0.35     84.42     0.32   456587     0.00     0.00  Curl_readwrite
>> >> >   0.35     84.74     0.32    73133     0.00     0.00
>> >> >  curl_multi_remove_handle
>> >> >   0.34     85.05     0.31  1360545     0.00     0.00  multi_runsingle
>> >> >
>> >> >
>> >> > 2012/3/6 Robert Iakobashvili <cor...@gm...>
>> >> >>
>> >> >> If it will not be helpful this means that recent
>> >> >> changes in libcurl library have broken DNS interface.
>> >> >>
>> >> >> You can try a previos version and the latest libcurl, like:
>> >> >>
>> >> >> http://curl.haxx.se/download/curl-7.23.0.tar.gz
>> >> >> and
>> >> >> http://curl.haxx.se/downloadcurl-7.24.0.tar.gz
>> >> >>
>> >> >> by:
>> >> >>
>> >> >> 1. downloading a libcurl version and placing it to
>> >> >> curl-loader-0.56/packages ;
>> >> >> 2. correcting the Makefile in curl-loader-0.56
>> >> >> CURL_VER:=7.24.0-20120109 ->  CURL_VER:=7.24 or CURL_VER:=7.23
>> >> >> 3. make cleanall; make
>> >> >>
>> >> >> Please, send us your observations.
>> >> >> Thanks!
>> >> >> Robert
>> >> >>
>> >> >> On Mon, Mar 5, 2012 at 6:57 PM, Robert Iakobashvili
>> >> >> <cor...@gm...>
>> >> >> wrote:
>> >> >> > If so, please try to change
>> >> >> >
>> >> >> > loader.c, line 578 which looks like:
>> >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1);
>> >> >> >
>> >> >> > 1. instead of -1 try to set some value like 60
>> >> >> > 2. make clean; make
>> >> >> >
>> >> >> > Thanks,
>> >> >> > Robert
>> >> >> >
>> >> >> > On Mon, Mar 5, 2012 at 6:19 PM, Fred Huang <di...@gm...>
>> >> >> > wrote:
>> >> >> >>
>> >> >> >> Looks like the bottleneck comes from libcurl other than c-ares...
>> >> >> >> test 3 (no ares patch applied):
>> >> >> >> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve
>> >> >> >> *.com
>> >> >> >> to one IP address
>> >> >> >> number of domain names in URLs: 780,000
>> >> >> >> number of client: 1000
>> >> >> >> CPU usage: 85%
>> >> >> >>     cpu%  irq% sirq%  sys% iowt%  mem_used  buf&cached
>> >> >> >>     85.6   0.0   0.8   5.9   0.0  3334.3Mb     112.0Mb
>> >> >> >> SSL TPS: 320
>> >> >> >> SSL throughput: 200Mbps
>> >> >> >>
>> >> >> >> # gprof /usr/bin/curl-loader gmon.out -p | head -50
>> >> >> >> Flat profile:
>> >> >> >> Each sample counts as 0.01 seconds.
>> >> >> >>   %   cumulative   self              self     total
>> >> >> >>  time   seconds   seconds    calls   s/call   s/call  name
>> >> >> >>  40.80    120.16   120.16   116079     0.00     0.00
>> >> >> >>  Curl_hash_clean_with_criterium
>> >> >> >>  24.46    192.18    72.02 983166323     0.00     0.00
>> >> >> >>  hostcache_timestamp_remove
>> >> >> >>  13.30    231.35    39.17  2059341     0.00     0.00
>> >> >> >>  Curl_hash_pick
>> >> >> >>   6.45    250.35    19.00   240087     0.00     0.00
>> >> >> >>  Curl_hash_add
>> >> >> >>   4.81    264.51    14.16 334257102     0.00     0.00
>> >> >> >>  Curl_str_key_compare
>> >> >> >>   3.40    274.52    10.01   119514     0.00     0.00  create_conn
>> >> >> >>   0.93    277.25     2.73  4478910     0.00     0.00
>> >> >> >>  dprintf_formatf
>> >> >> >>   0.90    279.91     2.66   122209     0.00     0.00
>> >> >> >>  ConnectionStore
>> >> >> >>   0.40    281.09     1.18
>> >> >> >> locking_function
>> >> >> >>   0.27    281.88     0.79   114759     0.00     0.00
>> >> >> >>  curl_multi_remove_handle
>> >> >> >>   0.21    282.51     0.63  7993664     0.00     0.00
>> >> >> >>  client_tracing_function
>> >> >> >>   0.21    283.12     0.61   122685     0.00     0.00
>> >> >> >>  Curl_num_addresses
>> >> >> >>   0.19    283.68     0.56   397028     0.00     0.00
>> >> >> >>  curl_multi_socket_action
>> >> >> >>   0.19    284.23     0.55  2036116     0.00     0.00
>> >> >> >>  multi_runsingle
>> >> >> >>   0.17    284.72     0.49   703687     0.00     0.00
>> >> >> >>  Curl_readwrite
>> >> >> >>   0.16    285.19     0.47                             id_function
>> >> >> >>   0.15    285.62     0.43 29692029     0.00     0.00
>> >> >> >>  Curl_socket_check
>> >> >> >>   0.14    286.02     0.40 91973856     0.00     0.00
>> >> >> >>  Curl_raw_toupper
>> >> >> >>   0.13    286.40     0.38 164664144     0.00     0.00  addbyter
>> >> >> >>   0.13    286.77     0.37  2180797     0.00     0.00  Curl_infof
>> >> >> >>   0.11    287.10     0.33   122675     0.00     0.00
>> >> >> >>  Curl_if_is_interface_name
>> >> >> >>   0.10    287.39     0.29   471318     0.00     0.00
>> >> >> >>  ossl_connect_common
>> >> >> >>   0.10    287.67     0.28  2113528     0.00     0.00
>> >> >> >>  Curl_raw_equal
>> >> >> >>   0.09    287.93     0.26     8676     0.00     0.02
>> >> >> >>  curl_multi_socket_all
>> >> >> >>   0.08    288.17     0.24  1663262     0.00     0.00  Curl_splay
>> >> >> >>   0.08    288.41     0.24     8505     0.00     0.01
>> >> >> >>  curl_multi_perform
>> >> >> >>   0.07    288.63     0.22  3114576     0.00     0.00
>> >> >> >>  Curl_pgrsUpdate
>> >> >> >>   0.06    288.80     0.17   119637     0.00     0.00
>> >> >> >>  singleipconnect
>> >> >> >>   0.06    288.97     0.17  7994818     0.00     0.00  Curl_debug
>> >> >> >>   0.05    289.13     0.16   235820     0.00     0.00  event_del
>> >> >> >>   0.05    289.29     0.16 119393071     0.00     0.00
>> >> >> >>  curl_strequal
>> >> >> >>   0.05    289.45     0.16 18535473     0.00     0.00
>> >> >> >>  alloc_addbyter
>> >> >> >>   0.05    289.61     0.16   356074     0.00     0.00
>> >> >> >>  Curl_hash_str
>> >> >> >>   0.05    289.76     0.15  1797233     0.00     0.00
>> >> >> >>  fd_key_compare
>> >> >> >>   0.05    289.91     0.15   234966     0.00     0.00
>> >> >> >>  Curl_ssl_getsessionid
>> >> >> >>   0.04    290.04     0.13   585853     0.00     0.00
>> >> >> >>  Curl_llist_destroy
>> >> >> >>   0.04    290.16     0.12   240826     0.00     0.00  Curl_resolv
>> >> >> >>   0.03    290.26     0.10  7514328     0.00     0.00
>> >> >> >>  Curl_raw_nequal
>> >> >> >>   0.03    290.36     0.10  2214436     0.00     0.00  Curl_read
>> >> >> >>   0.03    290.46     0.10   116872     0.00     0.00
>> >> >> >>  Curl_http_readwrite_headers
>> >> >> >>   0.03    290.56     0.10
>> >> >> >> ssl_tls_trace
>> >> >> >>   0.03    290.66     0.10  8013097     0.00     0.00
>> >> >> >>  scan_response
>> >> >> >>   0.03    290.75     0.09 29488555     0.00     0.00
>> >> >> >>  curlx_tvdiff
>> >> >> >>   0.03    290.84     0.09  2142468     0.00     0.00  Curl_setopt
>> >> >> >>   0.03    290.93     0.09   476299     0.00     0.00  Curl_expire
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> 2012/3/5 Robert Iakobashvili <cor...@gm...>
>> >> >> >>>
>> >> >> >>> Dear Fred Huang,
>> >> >> >>>
>> >> >> >>> Thanks for your posting.
>> >> >> >>>
>> >> >> >>> Could you please check if this is due to some badness at cares?
>> >> >> >>> Thanks.
>> >> >> >>>
>> >> >> >>> Let's exclude usage of cares based DNS resolving by applying the
>> >> >> >>> patch
>> >> >> >>> attached
>> >> >> >>> and use instead global DNS cache.
>> >> >> >>>
>> >> >> >>> cd ./curl-loader-0.56
>> >> >> >>> patch -p1 < path-to/disable-cares-use-global-dns.patch
>> >> >> >>> make cleanall; make
>> >> >> >>>
>> >> >> >>> See if it helps.
>> >> >> >>>
>> >> >> >>> Global cache, unfortunately, is not thread safe.
>> >> >> >>>
>> >> >> >>> If it helps, we'll try to restore the previous version of cares.
>> >> >> >>>
>> >> >> >>> Thanks,
>> >> >> >>> Robert
>> >> >> >>>
>> >> >> >>> On Mon, Mar 5, 2012 at 4:28 PM, Fred Huang <di...@gm...>
>> >> >> >>> wrote:
>> >> >> >>>>
>> >> >> >>>> test 1:
>> >> >> >>>>
>> >> >> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache,
>> >> >> >>>> resolve
>> >> >> >>>> *.com to one IP address
>> >> >> >>>> number of domain names in all https URLs: 780,000
>> >> >> >>>> number of client: 1000
>> >> >> >>>> CPU usage: 78%
>> >> >> >>>>     cpu%  irq% sirq%  sys% iowt%  mem_used  buf&cached
>> >> >> >>>>     78.8   0.0   0.6   3.8   0.0  3640.9Mb     282.1Mb
>> >> >> >>>> SSL TPS: 300
>> >> >> >>>> SSL throughput: 200Mbps
>> >> >> >>>>
>> >> >> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50
>> >> >> >>>> Flat profile:
>> >> >> >>>> Each sample counts as 0.01 seconds.
>> >> >> >>>>   %   cumulative   self              self     total
>> >> >> >>>>  time   seconds   seconds    calls   s/call   s/call  name
>> >> >> >>>>  38.55    127.10   127.10   130975     0.00     0.00
>> >> >> >>>>  Curl_hash_clean_with_criterium
>> >> >> >>>>  24.63    208.32    81.22 1247528407     0.00     0.00
>> >> >> >>>>  hostcache_timestamp_remove
>> >> >> >>>>  13.08    251.44    43.12  2791988     0.00     0.00
>> >> >> >>>>  Curl_hash_pick
>> >> >> >>>>   7.30    275.51    24.07   352490     0.00     0.00
>> >> >> >>>>  Curl_hash_add
>> >> >> >>>>   6.01    295.31    19.80 410760770     0.00     0.00
>> >> >> >>>>  Curl_str_key_compare
>> >> >> >>>>   2.95    305.04     9.73   131015     0.00     0.00
>> >> >> >>>>  create_conn
>> >> >> >>>>   0.88    307.94     2.90  4841390     0.00     0.00
>> >> >> >>>>  dprintf_formatf
>> >> >> >>>>   0.74    310.38     2.44   136036     0.00     0.00
>> >> >> >>>>  ConnectionStore
>> >> >> >>>>   0.48    311.96     1.58 814604911     0.00     0.00
>> >> >> >>>>  ares__is_list_empty
>> >> >> >>>>   0.36    313.16     1.20
>> >> >> >>>> locking_function
>> >> >> >>>>   0.27    314.06     0.90   261608     0.00     0.00
>> >> >> >>>>  ares_cancel
>> >> >> >>>>   0.22    314.79     0.73   127417     0.00     0.00
>> >> >> >>>>  curl_multi_remove_handle
>> >> >> >>>>   0.19    315.40     0.62  8906301     0.00     0.00
>> >> >> >>>>  client_tracing_function
>> >> >> >>>>   0.17    315.97     0.57   135305     0.00     0.00
>> >> >> >>>>  Curl_num_addresses
>> >> >> >>>>   0.17    316.54     0.57 181937907     0.00     0.00  addbyter
>> >> >> >>>>   0.17    317.09     0.55  2374363     0.00     0.00
>> >> >> >>>>  multi_runsingle
>> >> >> >>>>   0.16    317.62     0.53   541954     0.00     0.00
>> >> >> >>>>  ares__init_list_node
>> >> >> >>>>   0.16    318.14     0.52   494412     0.00     0.00
>> >> >> >>>>  curl_multi_socket_action
>> >> >> >>>>   0.14    318.60     0.46 33578897     0.00     0.00
>> >> >> >>>>  Curl_socket_check
>> >> >> >>>>   0.13    319.04     0.44 102951052     0.00     0.00
>> >> >> >>>>  Curl_raw_toupper
>> >> >> >>>>   0.13    319.48     0.44   787558     0.00     0.00
>> >> >> >>>>  Curl_readwrite
>> >> >> >>>>   0.13    319.91     0.44
>> >> >> >>>> id_function
>> >> >> >>>>   0.12    320.29     0.38   391390     0.00     0.00
>> >> >> >>>>  Curl_hash_str
>> >> >> >>>>   0.09    320.58     0.29     9216     0.00     0.00
>> >> >> >>>>  curl_multi_perform
>> >> >> >>>>   0.08    320.86     0.28  2350908     0.00     0.00
>> >> >> >>>>  Curl_raw_equal
>> >> >> >>>>   0.08    321.14     0.28  2019415     0.00     0.00
>> >> >> >>>>  Curl_splay
>> >> >> >>>>   0.08    321.40     0.26   520625     0.00     0.00
>> >> >> >>>>  ossl_connect_common
>> >> >> >>>>   0.08    321.65     0.25  3532992     0.00     0.00
>> >> >> >>>>  Curl_pgrsUpdate
>> >> >> >>>>   0.07    321.87     0.22 131778537     0.00     0.00
>> >> >> >>>>  curl_strequal
>> >> >> >>>>   0.05    322.05     0.18  8954118     0.00     0.00
>> >> >> >>>>  scan_response
>> >> >> >>>>   0.05    322.23     0.18   533644     0.00     0.00
>> >> >> >>>>  ares_expand_name
>> >> >> >>>>   0.05    322.40     0.17  2477345     0.00     0.00
>> >> >> >>>>  fd_key_compare
>> >> >> >>>>   0.05    322.57     0.17  2429135     0.00     0.00
>> >> >> >>>>  Curl_infof
>> >> >> >>>>   0.05    322.74     0.17   130966     0.00     0.00
>> >> >> >>>>  singleipconnect
>> >> >> >>>>   0.05    322.90     0.16     9292     0.00     0.00
>> >> >> >>>>  curl_multi_socket_all
>> >> >> >>>>   0.05    323.06     0.16   270137     0.00     0.00
>> >> >> >>>>  ares__get_hostent
>> >> >> >>>>   0.05    323.21     0.15  8943657     0.00     0.00
>> >> >> >>>>  Curl_debug
>> >> >> >>>>   0.04    323.35     0.14   262493     0.00     0.00
>> >> >> >>>>  Curl_ssl_getsessionid
>> >> >> >>>>   0.04    323.48     0.13   566006     0.00     0.00
>> >> >> >>>>  socket_callback
>> >> >> >>>>   0.04    323.61     0.13 33983864     0.00     0.00
>> >> >> >>>>  curlx_tvdiff
>> >> >> >>>>   0.04    323.74     0.13   132853     0.00     0.00
>> >> >> >>>>  Curl_http_readwrite_headers
>> >> >> >>>>   0.04    323.86     0.12   344032     0.00     0.00  epoll_del
>> >> >> >>>>   0.04    323.98     0.12   223477     0.00     0.00  Curl_poll
>> >> >> >>>>   0.04    324.10     0.12  8526280     0.00     0.00
>> >> >> >>>>  Curl_raw_nequal
>> >> >> >>>>   0.03    324.21     0.11   348839     0.00     0.00  event_del
>> >> >> >>>>
>> >> >> >>>>
>> >> >> >>>>
>> >> >> >>>>
>> >> >> >>>> test 2:
>> >> >> >>>>
>> >> >> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache,
>> >> >> >>>> resolve
>> >> >> >>>> *.com to one IP address
>> >> >> >>>> number of domain names in all https URLs: 1
>> >> >> >>>> number of client: 1000
>> >> >> >>>> CPU usage: 75%
>> >> >> >>>> SSL TPS: 1300
>> >> >> >>>> SSL throughput: 700Mbps
>> >> >> >>>>
>> >> >> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50
>> >> >> >>>> Flat profile:
>> >> >> >>>> Each sample counts as 0.01 seconds.
>> >> >> >>>>   %   cumulative   self              self     total
>> >> >> >>>>  time   seconds   seconds    calls   s/call   s/call  name
>> >> >> >>>>   8.98      2.45     2.45  5127312     0.00     0.00
>> >> >> >>>>  dprintf_formatf
>> >> >> >>>>   8.90      4.88     2.43 1292796503     0.00     0.00
>> >> >> >>>>  ares__is_list_empty
>> >> >> >>>>   6.12      6.55     1.67   322461     0.00     0.00
>> >> >> >>>>  create_conn
>> >> >> >>>>   4.58      7.80     1.25   419872     0.00     0.00
>> >> >> >>>>  ares_cancel
>> >> >> >>>>   4.10      8.92     1.12  2100162     0.00     0.00
>> >> >> >>>>  Curl_readwrite
>> >> >> >>>>   3.92      9.99     1.07  4527388     0.00     0.00
>> >> >> >>>>  Curl_hash_pick
>> >> >> >>>>   3.43     10.93     0.94
>> >> >> >>>> locking_function
>> >> >> >>>>   3.04     11.76     0.83   909759     0.00     0.00
>> >> >> >>>>  curl_multi_socket_action
>> >> >> >>>>   3.00     12.58     0.82    97252     0.00     0.00
>> >> >> >>>>  Curl_hash_clean_with_criterium
>> >> >> >>>>   2.71     13.32     0.74 179704016     0.00     0.00
>> >> >> >>>>  Curl_raw_toupper
>> >> >> >>>>   2.68     14.05     0.73  4646524     0.00     0.00
>> >> >> >>>>  multi_runsingle
>> >> >> >>>>   2.44     14.71     0.67 13862992     0.00     0.00
>> >> >> >>>>  client_tracing_function
>> >> >> >>>>   2.42     15.37     0.66   319735     0.00     0.00
>> >> >> >>>>  curl_multi_remove_handle
>> >> >> >>>>   2.42     16.03     0.66       22     0.03     0.03
>> >> >> >>>>  ares__init_list_node
>> >> >> >>>>   1.94     16.56     0.53  9930244     0.00     0.00
>> >> >> >>>>  hostcache_timestamp_remove
>> >> >> >>>>   1.94     17.09     0.53 169320088     0.00     0.00  addbyter
>> >> >> >>>>   1.80     17.58     0.49
>> >> >> >>>> id_function
>> >> >> >>>>   1.36     17.95     0.37  7057146     0.00     0.00
>> >> >> >>>>  Curl_pgrsUpdate
>> >> >> >>>>   1.36     18.32     0.37    16835     0.00     0.00
>> >> >> >>>>  curl_multi_socket_all
>> >> >> >>>>   1.25     18.66     0.34    16719     0.00     0.00
>> >> >> >>>>  curl_multi_perform
>> >> >> >>>>   1.21     18.99     0.33   329403     0.00     0.00
>> >> >> >>>>  Curl_http_readwrite_headers
>> >> >> >>>>   1.17     19.31     0.32  4036624     0.00     0.00
>> >> >> >>>>  Curl_splay
>> >> >> >>>>   1.14     19.62     0.31 20838902     0.00     0.00
>> >> >> >>>>  Curl_raw_nequal
>> >> >> >>>>   0.82     19.85     0.23  2764123     0.00     0.00
>> >> >> >>>>  Curl_infof
>> >> >> >>>>   0.81     20.07     0.22  5933857     0.00     0.00  ossl_recv
>> >> >> >>>>   0.77     20.28     0.21   323112     0.00     0.00
>> >> >> >>>>  Curl_splayremovebyaddr
>> >> >> >>>>   0.75     20.48     0.21  5943991     0.00     0.00  Curl_read
>> >> >> >>>>   0.70     20.67     0.19   425174     0.00     0.00  event_del
>> >> >> >>>>   0.70     20.86     0.19    98652     0.00     0.00
>> >> >> >>>>  Curl_if_is_interface_name
>> >> >> >>>>   0.70     21.05     0.19  8677733     0.00     0.00
>> >> >> >>>>  Curl_socket_check
>> >> >> >>>>   0.59     21.21     0.16  4129554     0.00     0.00
>> >> >> >>>>  fd_key_compare
>> >> >> >>>>   0.57     21.37     0.16 14017019     0.00     0.00
>> >> >> >>>>  Curl_debug
>> >> >> >>>>   0.55     21.52     0.15  9376861     0.00     0.00
>> >> >> >>>>  stat_data_in_add
>> >> >> >>>>   0.55     21.67     0.15  6840468     0.00     0.00
>> >> >> >>>>  Curl_timeleft
>> >> >> >>>>   0.55     21.82     0.15  2054602     0.00     0.00
>> >> >> >>>>  Curl_raw_equal
>> >> >> >>>>   0.55     21.97     0.15  1291611     0.00     0.00
>> >> >> >>>>  Curl_expire
>> >> >> >>>>   0.51     22.11     0.14 16570063     0.00     0.00
>> >> >> >>>>  curlx_tvnow
>> >> >> >>>>   0.51     22.25     0.14 13863284     0.00     0.00
>> >> >> >>>>  scan_response
>> >> >> >>>>   0.51     22.39     0.14  5498864     0.00     0.00
>> >> >> >>>>  Curl_setopt
>> >> >> >>>>   0.48     22.52     0.13 97490291     0.00     0.00
>> >> >> >>>>  curl_strequal
>> >> >> >>>>   0.44     22.64     0.12   324253     0.00     0.00  Curl_http
>> >> >> >>>>   0.44     22.76     0.12   394386     0.00     0.00
>> >> >> >>>>  ossl_connect_common
>> >> >> >>>>   0.40     22.87     0.11  3581335     0.00     0.00
>> >> >> >>>>  Curl_getinfo
>> >> >> >>>>   0.38     22.97     0.11 26983092     0.00     0.00
>> >> >> >>>>  alloc_addbyter
>> >> >> >>>>   0.37     23.07     0.10  8729044     0.00     0.00
>> >> >> >>>>  Curl_client_write
>> >> >> >>>>
>> >> >> >>>>
>> >> >> >>>>
>> >> >> >>>>
>> >> >> >>>>
>> >> >> >>>> ------------------------------------------------------------------------------
>> >> >> >>>> Try before you buy = See our experts in action!
>> >> >> >>>> The most comprehensive online learning library for Microsoft
>> >> >> >>>> developers
>> >> >> >>>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5,
>> >> >> >>>> CSS3,
>> >> >> >>>> MVC3,
>> >> >> >>>> Metro Style Apps, more. Free future releases when you subscribe
>> >> >> >>>> now!
>> >> >> >>>> http://p.sf.net/sfu/learndevnow-dev2
>> >> >> >>>> _______________________________________________
>> >> >> >>>> curl-loader-devel mailing list
>> >> >> >>>> cur...@li...
>> >> >> >>>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>> >> >> >>>>
>> >> >> >>>
>> >> >> >>>
>> >> >> >>>
>> >> >> >>> --
>> >> >> >>> Regards,
>> >> >> >>> Robert Iakobashvili, Ph.D.
>> >> >> >>>
>> >> >> >>> Home:       http://www.ghotit.com
>> >> >> >>> Blog:         http://www.ghotit.com/category/ghotit-blogs/
>> >> >> >>> Twitter:      http://twitter.com/ghotit
>> >> >> >>> Facebook: http://facebook.com/ghotit
>> >> >> >>>
>> >> >> >>>
>> >> >> >>> ......................................................................
>> >> >> >>>                    Ghotit Dyslexia
>> >> >> >>> Assistive technology that understands you
>> >> >> >>>
>> >> >> >>>
>> >> >> >>> ......................................................................
>> >> >> >>>
>> >> >> >>>
>> >> >> >>>
>> >> >> >>>
>> >> >> >>>
>> >> >> >>> ------------------------------------------------------------------------------
>> >> >> >>> Try before you buy = See our experts in action!
>> >> >> >>> The most comprehensive online learning library for Microsoft
>> >> >> >>> developers
>> >> >> >>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5,
>> >> >> >>> CSS3,
>> >> >> >>> MVC3,
>> >> >> >>> Metro Style Apps, more. Free future releases when you subscribe
>> >> >> >>> now!
>> >> >> >>> http://p.sf.net/sfu/learndevnow-dev2
>> >> >> >>> _______________________________________________
>> >> >> >>> curl-loader-devel mailing list
>> >> >> >>> cur...@li...
>> >> >> >>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>> >> >> >>>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> ------------------------------------------------------------------------------
>> >> >> >> Try before you buy = See our experts in action!
>> >> >> >> The most comprehensive online learning library for Microsoft
>> >> >> >> developers
>> >> >> >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5,
>> >> >> >> CSS3,
>> >> >> >> MVC3,
>> >> >> >> Metro Style Apps, more. Free future releases when you subscribe
>> >> >> >> now!
>> >> >> >> http://p.sf.net/sfu/learndevnow-dev2
>> >> >> >> _______________________________________________
>> >> >> >> curl-loader-devel mailing list
>> >> >> >> cur...@li...
>> >> >> >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>> >> >> >>
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > --
>> >> >> > Regards,
>> >> >> > Robert Iakobashvili, Ph.D.
>> >> >> >
>> >> >> > Home:       http://www.ghotit.com
>> >> >> > Blog:         http://www.ghotit.com/category/ghotit-blogs/
>> >> >> > Twitter:      http://twitter.com/ghotit
>> >> >> > Facebook: http://facebook.com/ghotit
>> >> >> >
>> >> >> >
>> >> >> > ......................................................................
>> >> >> >                    Ghotit Dyslexia
>> >> >> > Assistive technology that understands you
>> >> >> >
>> >> >> >
>> >> >> > ......................................................................
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> Regards,
>> >> >> Robert Iakobashvili, Ph.D.
>> >> >>
>> >> >> Home:       http://www.ghotit.com
>> >> >> Blog:         http://www.ghotit.com/category/ghotit-blogs/
>> >> >> Twitter:      http://twitter.com/ghotit
>> >> >> Facebook: http://facebook.com/ghotit
>> >> >>
>> >> >> ......................................................................
>> >> >>                    Ghotit Dyslexia
>> >> >> Assistive technology that understands you
>> >> >>
>> >> >> ......................................................................
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> ------------------------------------------------------------------------------
>> >> >> Try before you buy = See our experts in action!
>> >> >> The most comprehensive online learning library for Microsoft
>> >> >> developers
>> >> >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3,
>> >> >> MVC3,
>> >> >> Metro Style Apps, more. Free future releases when you subscribe now!
>> >> >> http://p.sf.net/sfu/learndevnow-dev2
>> >> >> _______________________________________________
>> >> >> curl-loader-devel mailing list
>> >> >> cur...@li...
>> >> >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > ------------------------------------------------------------------------------
>> >> > Keep Your Developer Skills Current with LearnDevNow!
>> >> > The most comprehensive online learning library for Microsoft
>> >> > developers
>> >> > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3,
>> >> > MVC3,
>> >> > Metro Style Apps, more. Free future releases when you subscribe now!
>> >> > http://p.sf.net/sfu/learndevnow-d2d
>> >> > _______________________________________________
>> >> > curl-loader-devel mailing list
>> >> > cur...@li...
>> >> > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Regards,
>> >> Robert Iakobashvili, Ph.D.
>> >>
>> >> Home:       http://www.ghotit.com
>> >> Blog:         http://www.ghotit.com/category/ghotit-blogs/
>> >> Twitter:      http://twitter.com/ghotit
>> >> Facebook: http://facebook.com/ghotit
>> >> ......................................................................
>> >>                    Ghotit Dyslexia
>> >> Assistive technology that understands you
>> >> ......................................................................
>> >>
>> >>
>> >>
>> >> ------------------------------------------------------------------------------
>> >> Keep Your Developer Skills Current with LearnDevNow!
>> >> The most comprehensive online learning library for Microsoft developers
>> >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3,
>> >> MVC3,
>> >> Metro Style Apps, more. Free future releases when you subscribe now!
>> >> http://p.sf.net/sfu/learndevnow-d2d
>> >> _______________________________________________
>> >> curl-loader-devel mailing list
>> >> cur...@li...
>> >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>> >
>> >
>> >
>> >
>> > ------------------------------------------------------------------------------
>> > Keep Your Developer Skills Current with LearnDevNow!
>> > The most comprehensive online learning library for Microsoft developers
>> > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
>> > Metro Style Apps, more. Free future releases when you subscribe now!
>> > http://p.sf.net/sfu/learndevnow-d2d
>> > _______________________________________________
>> > curl-loader-devel mailing list
>> > cur...@li...
>> > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>> >
>>
>>
>>
>> --
>> Regards,
>> Robert Iakobashvili, Ph.D.
>>
>> Home:       http://www.ghotit.com
>> Blog:         http://www.ghotit.com/category/ghotit-blogs/
>> Twitter:      http://twitter.com/ghotit
>> Facebook: http://facebook.com/ghotit
>> ......................................................................
>>                    Ghotit Dyslexia
>> Assistive technology that understands you
>> ......................................................................
>>
>>
>> ------------------------------------------------------------------------------
>> Keep Your Developer Skills Current with LearnDevNow!
>> The most comprehensive online learning library for Microsoft developers
>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
>> Metro Style Apps, more. Free future releases when you subscribe now!
>> http://p.sf.net/sfu/learndevnow-d2d
>> _______________________________________________
>> curl-loader-devel mailing list
>> cur...@li...
>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>
>
>
> ------------------------------------------------------------------------------
> Virtualization & Cloud Management Using Capacity Planning
> Cloud computing makes use of virtualization - but cloud computing
> also focuses on allowing computing to be delivered as a service.
> http://www.accelacomm.com/jaw/sfnl/114/51521223/
> _______________________________________________
> curl-loader-devel mailing list
> cur...@li...
> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>



-- 
Regards,
Robert Iakobashvili, Ph.D.

Home:       http://www.ghotit.com
Blog:         http://www.ghotit.com/category/ghotit-blogs/
Twitter:      http://twitter.com/ghotit
Facebook: http://facebook.com/ghotit
......................................................................
                   Ghotit Dyslexia
Assistive technology that understands you
......................................................................