Re: curl-loader https: low performance due to DNS query?
Status: Alpha
Brought to you by:
coroberti
Menu
▾
▴
Re: curl-loader https: low performance due to DNS query?
|
From: Fred H. <di...@gm...> - 2012-03-07 13:49:19
|
I tried -t 16 or -t 32, the performance drops to approx. 50 CAPS... 2012/3/7 Robert Iakobashvili <cor...@gm...> > Fred, > > On more suggestion, which is a general one. > > If N is the number of your logical CPUs, > usually, I'd recommend to run 2*N + B threads. > > Thus, you can try: > > -t 16 > > -t 32 > > and it MAY increased a bit CAPS. > > Thanks, > Robert > > On Wed, Mar 7, 2012 at 2:43 AM, Fred Huang <di...@gm...> wrote: > > yes, I use -t 8 option because the test box has 8 logic CPUs (Dell 990, > > Intel Core i7 2600, 16GB memory). > > if I do not use -t option, the CPU used by curl-loader will be 100% on > one > > of 8 logic CPU (12% of total CPU), and the CAPS drops to 100: > > # gprof /usr/bin/curl-loader gmon.out -p | head -20 > > Flat profile: > > Each sample counts as 0.01 seconds. > > % cumulative self self total > > time seconds seconds calls s/call s/call name > > 34.12 2.74 2.74 17385 0.00 0.00 create_conn > > 11.21 3.64 0.90 30998583 0.00 0.00 > > hostcache_timestamp_remove > > 9.84 4.43 0.79 13433 0.00 0.00 > > Curl_hash_clean_with_criterium > > 7.97 5.07 0.64 381818 0.00 0.00 Curl_hash_pick > > 5.85 5.54 0.47 17385 0.00 0.00 ConnectionStore > > 5.11 5.95 0.41 16385 0.00 0.00 > > curl_multi_remove_handle > > 4.11 6.28 0.33 16509889 0.00 0.00 > Curl_str_key_compare > > 3.36 6.55 0.27 23959914 0.00 0.00 Curl_socket_check > > 3.24 6.81 0.26 56703 0.00 0.00 Curl_hash_add > > 2.62 7.02 0.21 635750 0.00 0.00 dprintf_formatf > > 1.25 7.12 0.10 92421190 0.00 0.00 ares__is_list_empty > > 0.75 7.18 0.06 97957 0.00 0.00 Curl_readwrite > > 0.62 7.23 0.05 23422260 0.00 0.00 addbyter > > 0.62 7.28 0.05 1136775 0.00 0.00 > > client_tracing_function > > 0.50 7.32 0.04 3091375 0.00 0.00 > ares__init_list_head > > > > I also tried to use > > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 20); > > but get no improvement on CAPS. > > > > > > Regards, > > Fred > > > > 2012/3/7 Robert Iakobashvili <cor...@gm...> > >> > >> Dear Fred, > >> > >> If I understand correctly your observations, the issue is not in: > >> > >> 1. DNS resolving; > >> 2. Version of cares or libcurl; > >> > >> but rather in > >> > >> Less effective handling of HTTPS versus HTTP in libcurl. > >> > >> Please, confirm that my understanding is correct. > >> > >> > >> HTTPS URLs are using an extra locking in openssl with -t option > >> that could be making the issue. > >> If you are using -t option, try to load without it and let's say with > >> 300 clients. > >> > >> If you are not using -t, thus the only way to proceed is to share > >> your observations with libcurl people at > >> > >> libcurl development <cur...@co...> > >> > >> Thank you. > >> > >> Best wishes. > >> Robert > >> > >> On Tue, Mar 6, 2012 at 6:17 PM, Fred Huang <di...@gm...> wrote: > >> > 1. the major CPU consumer is curl-loader. here is the top: > >> > top - 07:36:36 up 23:31, 5 users, load average: 4.32, 2.26, 0.91 > >> > Tasks: 237 total, 2 running, 235 sleeping, 0 stopped, 0 zombie > >> > Cpu0 : 76.6%us, 3.6%sy, 0.0%ni, 19.4%id, 0.0%wa, 0.0%hi, 0.3%si, > >> > 0.0% > >> > Cpu1 : 72.4%us, 4.0%sy, 0.0%ni, 22.9%id, 0.0%wa, 0.0%hi, 0.7%si, > >> > 0.0% > >> > Cpu2 : 65.8%us, 3.3%sy, 0.0%ni, 30.6%id, 0.0%wa, 0.0%hi, 0.3%si, > >> > 0.0% > >> > Cpu3 : 75.2%us, 4.3%sy, 0.0%ni, 19.5%id, 0.0%wa, 0.0%hi, 1.0%si, > >> > 0.0% > >> > Cpu4 : 79.1%us, 4.0%sy, 0.0%ni, 16.8%id, 0.0%wa, 0.0%hi, 0.0%si, > >> > 0.0% > >> > Cpu5 : 75.8%us, 4.3%sy, 0.0%ni, 19.2%id, 0.0%wa, 0.0%hi, 0.7%si, > >> > 0.0% > >> > Cpu6 : 74.1%us, 5.0%sy, 0.0%ni, 20.3%id, 0.0%wa, 0.0%hi, 0.7%si, > >> > 0.0% > >> > Cpu7 : 78.7%us, 4.0%sy, 0.0%ni, 16.6%id, 0.0%wa, 0.0%hi, 0.7%si, > >> > 0.0% > >> > Mem: 16209932k total, 3829500k used, 12380432k free, 99112k > buffers > >> > Swap: 18448376k total, 0k used, 18448376k free, 320284k > cached > >> > > >> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > >> > 30795 root 20 0 3200m 2.8g 2296 S 79.2 18.4 21:26.84 > curl-loader > >> > 1832 nobody 20 0 13944 1736 580 S 0.1 0.0 0:32.65 dnsmasq > >> > 9 root 20 0 0 0 0 S 0.0 0.0 0:00.53 > ksoftirqd/1 > >> > 30825 root 20 0 15220 1360 960 R 0.0 0.0 0:00.42 top > >> > 1 root 20 0 19400 1568 1256 S 0.0 0.0 0:00.75 init > >> > 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd > >> > 3 root RT 0 0 0 0 S 0.0 0.0 0:00.00 > migration/0 > >> > 4 root 20 0 0 0 0 S 0.0 0.0 0:00.06 > ksoftirqd/0 > >> > 5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 > migration/0 > >> > > >> > > >> > 2. 1000 loading clients > >> > > >> > 3. yes, tried 0.52 but no lucky -- CAPS 380, throughput 200Mbps, CPU > 90% > >> > top - 08:17:10 up 1 day, 12 min, 8 users, load average: 4.66, 3.39, > >> > 1.88 > >> > Tasks: 245 total, 2 running, 243 sleeping, 0 stopped, 0 zombie > >> > Cpu0 : 82.2%us, 4.0%sy, 0.0%ni, 13.9%id, 0.0%wa, 0.0%hi, 0.0%si, > >> > 0.0% > >> > Cpu1 : 86.1%us, 4.3%sy, 0.0%ni, 7.9%id, 0.0%wa, 0.0%hi, 1.7%si, > >> > 0.0% > >> > Cpu2 : 85.3%us, 4.0%sy, 0.0%ni, 10.0%id, 0.0%wa, 0.0%hi, 0.7%si, > >> > 0.0% > >> > Cpu3 : 81.8%us, 3.0%sy, 0.0%ni, 14.2%id, 0.0%wa, 0.0%hi, 1.0%si, > >> > 0.0% > >> > Cpu4 : 83.7%us, 4.0%sy, 0.0%ni, 12.3%id, 0.0%wa, 0.0%hi, 0.0%si, > >> > 0.0% > >> > Cpu5 : 92.7%us, 3.3%sy, 0.0%ni, 3.3%id, 0.0%wa, 0.0%hi, 0.7%si, > >> > 0.0% > >> > Cpu6 : 89.7%us, 2.0%sy, 0.0%ni, 8.0%id, 0.0%wa, 0.0%hi, 0.3%si, > >> > 0.0% > >> > Cpu7 : 88.1%us, 3.6%sy, 0.0%ni, 6.6%id, 0.0%wa, 0.0%hi, 1.7%si, > >> > 0.0% > >> > Mem: 16209932k total, 1795748k used, 14414184k free, 102824k > buffers > >> > Swap: 18448376k total, 0k used, 18448376k free, 382988k > cached > >> > > >> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > >> > 12236 root 20 0 1150m 894m 2572 S 90.7 5.7 51:11.91 > curl-loader > >> > 1832 nobody 20 0 13944 1736 580 S 0.1 0.0 0:47.04 dnsmasq > >> > > >> > ]# gprof /usr/bin/curl-loader gmon.out -p | head -20 > >> > Flat profile: > >> > Each sample counts as 0.01 seconds. > >> > % cumulative self self total > >> > time seconds seconds calls s/call s/call name > >> > 43.92 151.95 151.95 157295 0.00 0.00 > >> > Curl_hash_clean_with_criterium > >> > 22.90 231.17 79.22 1712190971 0.00 0.00 > >> > hostcache_timestamp_remove > >> > 13.15 276.68 45.51 3260321 0.00 0.00 Curl_hash_pick > >> > 7.51 302.65 25.97 394171 0.00 0.00 Curl_hash_add > >> > 5.98 323.34 20.68 567437518 0.00 0.00 > >> > Curl_str_key_compare > >> > 1.00 326.80 3.46 5380220 0.00 0.00 dprintf_formatf > >> > 0.41 328.21 1.41 locking_function > >> > 0.32 329.32 1.11 469222138 0.00 0.00 > >> > ares__is_list_empty > >> > 0.31 330.40 1.08 3284508 0.00 0.00 multi_runsingle > >> > 0.28 331.38 0.98 10006806 0.00 0.00 > >> > client_tracing_function > >> > 0.24 332.21 0.83 151951 0.00 0.00 > Curl_num_addresses > >> > 0.22 332.97 0.77 188075672 0.00 0.00 addbyter > >> > 0.19 333.64 0.67 841840 0.00 0.00 Curl_readwrite > >> > 0.19 334.29 0.65 id_function > >> > 0.18 334.90 0.61 39843 0.00 0.01 > >> > curl_multi_socket_all > >> > > >> > > >> > > >> > 4. yes, it is correct for 0.56. If use same URL list by replacing > https > >> > to > >> > http, I can get CAPS 1500 and throughput 870Mbps with CPU% < 60% > >> > > >> > > >> > 2012/3/6 Robert Iakobashvili <cor...@gm...> > >> >> > >> >> Several questions: > >> >> > >> >> 1. Could you please run top and see if there are other processes > >> >> burning > >> >> CPU? > >> >> 2. What is the number of loading clients that you use? > >> >> 3. Could you try curl-loader-0.52 if it compiles for you? > >> >> If not, please try to place curl-loader-7.19.7.tar.gz and > >> >> c-ares-16.0.tar.gz, > >> >> update their versions in Makefile and rebuild by make cleanall; make > >> >> > >> >> 4. Is it correct to say that for HTTP URLs (not HTTPS) you are > getting > >> >> a good throughput > >> >> and CAPS with curl-loader-0.56? > >> >> > >> >> Thanks, > >> >> Robert > >> >> > >> >> On Tue, Mar 6, 2012 at 3:52 AM, Fred Huang <di...@gm...> wrote: > >> >> > TIMEOUT -1 -> 60 works a little better but no performance > improvement > >> >> > on > >> >> > CAPS or throughput. > >> >> > > >> >> > test 5: > >> >> > enable c-ares; > >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); > >> >> > > >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve > >> >> > *.com > >> >> > to > >> >> > one IP address > >> >> > number of domain names in URLs: 780,000 > >> >> > number of client: 1000 > >> >> > CPU usage: 74% > >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached > >> >> > 74.0 0.0 0.7 4.2 0.0 3530.6Mb 250.8Mb > >> >> > SSL TPS: 360 > >> >> > SSL throughput: 220Mbps > >> >> > > >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -50 > >> >> > Flat profile: > >> >> > Each sample counts as 0.01 seconds. > >> >> > % cumulative self self total > >> >> > time seconds seconds calls s/call s/call name > >> >> > 26.63 23.13 23.13 119226 0.00 0.00 > >> >> > Curl_hash_clean_with_criterium > >> >> > 20.34 40.80 17.67 317792146 0.00 0.00 > >> >> > hostcache_timestamp_remove > >> >> > 11.27 50.59 9.79 2579903 0.00 0.00 > Curl_hash_pick > >> >> > 7.84 57.40 6.81 120606 0.00 0.00 create_conn > >> >> > 5.65 62.31 4.91 119117401 0.00 0.00 > >> >> > Curl_str_key_compare > >> >> > 4.98 66.64 4.33 329829 0.00 0.00 Curl_hash_add > >> >> > 2.80 69.08 2.44 4476606 0.00 0.00 > dprintf_formatf > >> >> > 2.57 71.31 2.23 124019 0.00 0.00 > ConnectionStore > >> >> > 1.50 72.61 1.30 740997609 0.00 0.00 > >> >> > ares__is_list_empty > >> >> > 1.16 73.61 1.01 > >> >> > locking_function > >> >> > 0.76 74.27 0.66 238744 0.00 0.00 ares_cancel > >> >> > 0.70 74.88 0.61 457564 0.00 0.00 > >> >> > curl_multi_socket_action > >> >> > 0.67 75.46 0.58 118616 0.00 0.00 > >> >> > curl_multi_remove_handle > >> >> > 0.58 75.97 0.51 8211502 0.00 0.00 > >> >> > client_tracing_function > >> >> > 0.53 76.43 0.46 733561 0.00 0.00 > Curl_readwrite > >> >> > 0.51 76.87 0.44 2183179 0.00 0.00 > multi_runsingle > >> >> > 0.51 77.31 0.44 166585567 0.00 0.00 addbyter > >> >> > 0.44 77.69 0.38 2175805 0.00 0.00 > Curl_raw_equal > >> >> > 0.43 78.06 0.38 id_function > >> >> > 0.40 78.41 0.35 29789346 0.00 0.00 > >> >> > Curl_socket_check > >> >> > 0.40 78.76 0.35 479671 0.00 0.00 > >> >> > ossl_connect_common > >> >> > 0.40 79.11 0.35 490639 0.00 0.00 > >> >> > ares__init_list_node > >> >> > 0.33 79.40 0.29 94197670 0.00 0.00 > >> >> > Curl_raw_toupper > >> >> > 0.33 79.69 0.29 3268031 0.00 0.00 > Curl_pgrsUpdate > >> >> > 0.28 79.93 0.24 8797 0.00 0.00 > >> >> > curl_multi_perform > >> >> > 0.27 80.17 0.24 2233641 0.00 0.00 Curl_infof > >> >> > 0.26 80.40 0.23 1839837 0.00 0.00 Curl_splay > >> >> > 0.24 80.61 0.21 121713028 0.00 0.00 > curl_strequal > >> >> > 0.21 80.79 0.18 243333 0.00 0.00 > >> >> > Curl_ssl_getsessionid > >> >> > 0.18 80.95 0.16 122180 0.00 0.00 > >> >> > Curl_num_addresses > >> >> > 0.17 81.09 0.15 8271666 0.00 0.00 Curl_debug > >> >> > 0.16 81.23 0.14 18902452 0.00 0.00 > alloc_addbyter > >> >> > 0.16 81.37 0.14 8851 0.00 0.00 > >> >> > curl_multi_socket_all > >> >> > 0.14 81.50 0.13 2236769 0.00 0.00 Curl_read > >> >> > 0.14 81.62 0.12 2271700 0.00 0.00 > fd_key_compare > >> >> > 0.14 81.74 0.12 323167 0.00 0.00 event_del > >> >> > 0.14 81.86 0.12 205436 0.00 0.00 > >> >> > read_udp_packets > >> >> > 0.14 81.98 0.12 200979 0.00 0.00 processfds > >> >> > 0.13 82.09 0.11 123886 0.00 0.00 > >> >> > Curl_if_is_interface_name > >> >> > 0.13 82.20 0.11 120350 0.00 0.00 Curl_http > >> >> > 0.13 82.31 0.11 ssl_tls_trace > >> >> > 0.12 82.41 0.10 119229 0.00 0.00 > >> >> > Curl_splayremovebyaddr > >> >> > 0.12 82.51 0.10 7798717 0.00 0.00 > Curl_raw_nequal > >> >> > 0.12 82.61 0.10 784485 0.00 0.00 event_add > >> >> > 0.10 82.70 0.09 643482 0.00 0.00 Curl_expire > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > test 6: > >> >> > enable c-ares; > >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); > >> >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0 > >> >> > > >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve > >> >> > *.com > >> >> > to > >> >> > one IP address > >> >> > number of domain names in URLs: 780,000 > >> >> > number of client: 1000 > >> >> > CPU usage: 86% > >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached > >> >> > 85.9 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb > >> >> > SSL TPS: 360 > >> >> > SSL throughput: 200Mbps > >> >> > > >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 > >> >> > Flat profile: > >> >> > Each sample counts as 0.01 seconds. > >> >> > % cumulative self self total > >> >> > time seconds seconds calls s/call s/call name > >> >> > 27.19 19.96 19.96 103472 0.00 0.00 > >> >> > Curl_hash_clean_with_criterium > >> >> > 21.73 35.91 15.95 274168981 0.00 0.00 > >> >> > hostcache_timestamp_remove > >> >> > 10.33 43.49 7.58 2256441 0.00 0.00 > Curl_hash_pick > >> >> > 7.21 48.78 5.29 104540 0.00 0.00 create_conn > >> >> > 6.20 53.33 4.55 103413405 0.00 0.00 > >> >> > Curl_str_key_compare > >> >> > 4.88 56.91 3.58 287109 0.00 0.00 Curl_hash_add > >> >> > 3.02 59.13 2.22 3797855 0.00 0.00 > dprintf_formatf > >> >> > 2.10 60.67 1.54 107711 0.00 0.00 > ConnectionStore > >> >> > 1.65 61.88 1.21 642725327 0.00 0.00 > >> >> > ares__is_list_empty > >> >> > 1.19 62.76 0.88 > >> >> > locking_function > >> >> > 0.87 63.40 0.64 207817 0.00 0.00 ares_cancel > >> >> > 0.56 63.81 0.41 404108 0.00 0.00 > >> >> > curl_multi_socket_action > >> >> > 0.54 64.21 0.40 425603 0.00 0.00 > >> >> > ares__init_list_node > >> >> > 0.53 64.60 0.39 7068112 0.00 0.00 > >> >> > client_tracing_function > >> >> > 0.49 64.96 0.36 1908587 0.00 0.00 > multi_runsingle > >> >> > > >> >> > > >> >> > > >> >> > test 7: > >> >> > enable c-ares; > >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); > >> >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0 > >> >> > > >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve > >> >> > *.com > >> >> > to > >> >> > one IP address > >> >> > number of domain names in URLs: 780,000 > >> >> > number of client: 1000 > >> >> > CPU usage: 86% > >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached > >> >> > 86.6 0.0 0.6 3.3 0.0 3624.8Mb 263.0Mb > >> >> > SSL TPS: 300 > >> >> > SSL throughput: 200Mbps > >> >> > > >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 > >> >> > Flat profile: > >> >> > Each sample counts as 0.01 seconds. > >> >> > % cumulative self self total > >> >> > time seconds seconds calls s/call s/call name > >> >> > 39.83 369.71 369.71 207173 0.00 0.00 > >> >> > Curl_hash_clean_with_criterium > >> >> > 24.27 594.98 225.27 3198337733 0.00 0.00 > >> >> > hostcache_timestamp_remove > >> >> > 13.75 722.59 127.61 4312864 0.00 0.00 > Curl_hash_pick > >> >> > 9.19 807.87 85.28 546507 0.00 0.00 Curl_hash_add > >> >> > 6.14 864.88 57.01 1022419684 0.00 0.00 > >> >> > Curl_str_key_compare > >> >> > 2.06 884.00 19.12 203749 0.00 0.00 create_conn > >> >> > 0.54 888.99 4.99 213306 0.00 0.00 > ConnectionStore > >> >> > 0.49 893.50 4.51 7331251 0.00 0.00 > dprintf_formatf > >> >> > 0.29 896.16 2.66 1287182329 0.00 0.00 > >> >> > ares__is_list_empty > >> >> > 0.23 898.28 2.12 > >> >> > locking_function > >> >> > 0.20 900.10 1.82 212210 0.00 0.00 > >> >> > Curl_num_addresses > >> >> > 0.17 901.66 1.56 198385 0.00 0.00 > >> >> > curl_multi_remove_handle > >> >> > 0.12 902.81 1.15 411114 0.00 0.00 ares_cancel > >> >> > 0.12 903.95 1.14 3683448 0.00 0.00 > multi_runsingle > >> >> > 0.11 904.97 1.03 13664120 0.00 0.00 > >> >> > client_tracing_function > >> >> > > >> >> > > >> >> > test 8: > >> >> > enable c-ares; > >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); > >> >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0 > >> >> > > >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve > >> >> > *.com > >> >> > to > >> >> > one IP address > >> >> > number of domain names in URLs: 780,000 > >> >> > number of client: 1000 > >> >> > CPU usage: 75% > >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached > >> >> > 75.2 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb > >> >> > SSL TPS: 380 > >> >> > SSL throughput: 220Mbps > >> >> > > >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 > >> >> > Flat profile: > >> >> > Each sample counts as 0.01 seconds. > >> >> > % cumulative self self total > >> >> > time seconds seconds calls s/call s/call name > >> >> > 27.58 12.31 12.31 71140 0.00 0.00 > >> >> > Curl_hash_clean_with_criterium > >> >> > 19.09 20.83 8.52 176729692 0.00 0.00 > >> >> > hostcache_timestamp_remove > >> >> > 10.44 25.49 4.66 1589309 0.00 0.00 > Curl_hash_pick > >> >> > 6.85 28.55 3.06 73204 0.00 0.00 create_conn > >> >> > 5.56 31.03 2.48 67620530 0.00 0.00 > >> >> > Curl_str_key_compare > >> >> > 4.41 33.00 1.97 200684 0.00 0.00 Curl_hash_add > >> >> > 3.56 34.59 1.59 2717956 0.00 0.00 > dprintf_formatf > >> >> > 1.72 35.36 0.77 75247 0.00 0.00 > ConnectionStore > >> >> > 1.70 36.12 0.76 444342256 0.00 0.00 > >> >> > ares__is_list_empty > >> >> > 1.23 36.67 0.55 > >> >> > locking_function > >> >> > 1.16 37.19 0.52 143342 0.00 0.00 ares_cancel > >> >> > 0.99 37.63 0.44 72211 0.00 0.00 > >> >> > curl_multi_remove_handle > >> >> > 0.74 37.96 0.33 1333371 0.00 0.00 > multi_runsingle > >> >> > 0.68 38.27 0.31 101147798 0.00 0.00 addbyter > >> >> > 0.58 38.53 0.26 288737 0.00 0.00 > >> >> > curl_multi_socket_action > >> >> > > >> >> > > >> >> > > >> >> > test 9: > >> >> > enable c-ares; > >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); > >> >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0 > >> >> > > >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve > >> >> > *.com > >> >> > to > >> >> > one IP address > >> >> > number of domain names in URLs: 780,000 > >> >> > number of client: 1000 > >> >> > CPU usage: 75% > >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached > >> >> > 75.1 0.0 0.6 4.1 0.0 3545.2Mb 260.1Mb > >> >> > SSL TPS: 380 > >> >> > SSL throughput: 200Mbps > >> >> > > >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 > >> >> > Flat profile: > >> >> > Each sample counts as 0.01 seconds. > >> >> > % cumulative self self total > >> >> > time seconds seconds calls s/call s/call name > >> >> > 34.00 31.13 31.13 72495 0.00 0.00 > >> >> > Curl_hash_clean_with_criterium > >> >> > 23.64 52.78 21.65 386070412 0.00 0.00 > >> >> > hostcache_timestamp_remove > >> >> > 13.28 64.94 12.16 1619440 0.00 0.00 > Curl_hash_pick > >> >> > 5.92 70.36 5.42 137653129 0.00 0.00 > >> >> > Curl_str_key_compare > >> >> > 5.50 75.40 5.04 203018 0.00 0.00 Curl_hash_add > >> >> > 4.55 79.57 4.17 74618 0.00 0.00 create_conn > >> >> > 1.70 81.13 1.56 2757148 0.00 0.00 > dprintf_formatf > >> >> > 0.98 82.03 0.90 76945 0.00 0.00 > ConnectionStore > >> >> > 0.95 82.90 0.87 455775100 0.00 0.00 > >> >> > ares__is_list_empty > >> >> > 0.57 83.42 0.53 > >> >> > locking_function > >> >> > 0.39 83.78 0.36 146742 0.00 0.00 ares_cancel > >> >> > 0.35 84.10 0.32 102968806 0.00 0.00 addbyter > >> >> > 0.35 84.42 0.32 456587 0.00 0.00 > Curl_readwrite > >> >> > 0.35 84.74 0.32 73133 0.00 0.00 > >> >> > curl_multi_remove_handle > >> >> > 0.34 85.05 0.31 1360545 0.00 0.00 > multi_runsingle > >> >> > > >> >> > > >> >> > 2012/3/6 Robert Iakobashvili <cor...@gm...> > >> >> >> > >> >> >> If it will not be helpful this means that recent > >> >> >> changes in libcurl library have broken DNS interface. > >> >> >> > >> >> >> You can try a previos version and the latest libcurl, like: > >> >> >> > >> >> >> http://curl.haxx.se/download/curl-7.23.0.tar.gz > >> >> >> and > >> >> >> http://curl.haxx.se/downloadcurl-7.24.0.tar.gz > >> >> >> > >> >> >> by: > >> >> >> > >> >> >> 1. downloading a libcurl version and placing it to > >> >> >> curl-loader-0.56/packages ; > >> >> >> 2. correcting the Makefile in curl-loader-0.56 > >> >> >> CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24 or CURL_VER:=7.23 > >> >> >> 3. make cleanall; make > >> >> >> > >> >> >> Please, send us your observations. > >> >> >> Thanks! > >> >> >> Robert > >> >> >> > >> >> >> On Mon, Mar 5, 2012 at 6:57 PM, Robert Iakobashvili > >> >> >> <cor...@gm...> > >> >> >> wrote: > >> >> >> > If so, please try to change > >> >> >> > > >> >> >> > loader.c, line 578 which looks like: > >> >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); > >> >> >> > > >> >> >> > 1. instead of -1 try to set some value like 60 > >> >> >> > 2. make clean; make > >> >> >> > > >> >> >> > Thanks, > >> >> >> > Robert > >> >> >> > > >> >> >> > On Mon, Mar 5, 2012 at 6:19 PM, Fred Huang <di...@gm...> > >> >> >> > wrote: > >> >> >> >> > >> >> >> >> Looks like the bottleneck comes from libcurl other than > c-ares... > >> >> >> >> test 3 (no ares patch applied): > >> >> >> >> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, > resolve > >> >> >> >> *.com > >> >> >> >> to one IP address > >> >> >> >> number of domain names in URLs: 780,000 > >> >> >> >> number of client: 1000 > >> >> >> >> CPU usage: 85% > >> >> >> >> cpu% irq% sirq% sys% iowt% mem_used buf&cached > >> >> >> >> 85.6 0.0 0.8 5.9 0.0 3334.3Mb 112.0Mb > >> >> >> >> SSL TPS: 320 > >> >> >> >> SSL throughput: 200Mbps > >> >> >> >> > >> >> >> >> # gprof /usr/bin/curl-loader gmon.out -p | head -50 > >> >> >> >> Flat profile: > >> >> >> >> Each sample counts as 0.01 seconds. > >> >> >> >> % cumulative self self total > >> >> >> >> time seconds seconds calls s/call s/call name > >> >> >> >> 40.80 120.16 120.16 116079 0.00 0.00 > >> >> >> >> Curl_hash_clean_with_criterium > >> >> >> >> 24.46 192.18 72.02 983166323 0.00 0.00 > >> >> >> >> hostcache_timestamp_remove > >> >> >> >> 13.30 231.35 39.17 2059341 0.00 0.00 > >> >> >> >> Curl_hash_pick > >> >> >> >> 6.45 250.35 19.00 240087 0.00 0.00 > >> >> >> >> Curl_hash_add > >> >> >> >> 4.81 264.51 14.16 334257102 0.00 0.00 > >> >> >> >> Curl_str_key_compare > >> >> >> >> 3.40 274.52 10.01 119514 0.00 0.00 > create_conn > >> >> >> >> 0.93 277.25 2.73 4478910 0.00 0.00 > >> >> >> >> dprintf_formatf > >> >> >> >> 0.90 279.91 2.66 122209 0.00 0.00 > >> >> >> >> ConnectionStore > >> >> >> >> 0.40 281.09 1.18 > >> >> >> >> locking_function > >> >> >> >> 0.27 281.88 0.79 114759 0.00 0.00 > >> >> >> >> curl_multi_remove_handle > >> >> >> >> 0.21 282.51 0.63 7993664 0.00 0.00 > >> >> >> >> client_tracing_function > >> >> >> >> 0.21 283.12 0.61 122685 0.00 0.00 > >> >> >> >> Curl_num_addresses > >> >> >> >> 0.19 283.68 0.56 397028 0.00 0.00 > >> >> >> >> curl_multi_socket_action > >> >> >> >> 0.19 284.23 0.55 2036116 0.00 0.00 > >> >> >> >> multi_runsingle > >> >> >> >> 0.17 284.72 0.49 703687 0.00 0.00 > >> >> >> >> Curl_readwrite > >> >> >> >> 0.16 285.19 0.47 > id_function > >> >> >> >> 0.15 285.62 0.43 29692029 0.00 0.00 > >> >> >> >> Curl_socket_check > >> >> >> >> 0.14 286.02 0.40 91973856 0.00 0.00 > >> >> >> >> Curl_raw_toupper > >> >> >> >> 0.13 286.40 0.38 164664144 0.00 0.00 addbyter > >> >> >> >> 0.13 286.77 0.37 2180797 0.00 0.00 > Curl_infof > >> >> >> >> 0.11 287.10 0.33 122675 0.00 0.00 > >> >> >> >> Curl_if_is_interface_name > >> >> >> >> 0.10 287.39 0.29 471318 0.00 0.00 > >> >> >> >> ossl_connect_common > >> >> >> >> 0.10 287.67 0.28 2113528 0.00 0.00 > >> >> >> >> Curl_raw_equal > >> >> >> >> 0.09 287.93 0.26 8676 0.00 0.02 > >> >> >> >> curl_multi_socket_all > >> >> >> >> 0.08 288.17 0.24 1663262 0.00 0.00 > Curl_splay > >> >> >> >> 0.08 288.41 0.24 8505 0.00 0.01 > >> >> >> >> curl_multi_perform > >> >> >> >> 0.07 288.63 0.22 3114576 0.00 0.00 > >> >> >> >> Curl_pgrsUpdate > >> >> >> >> 0.06 288.80 0.17 119637 0.00 0.00 > >> >> >> >> singleipconnect > >> >> >> >> 0.06 288.97 0.17 7994818 0.00 0.00 > Curl_debug > >> >> >> >> 0.05 289.13 0.16 235820 0.00 0.00 event_del > >> >> >> >> 0.05 289.29 0.16 119393071 0.00 0.00 > >> >> >> >> curl_strequal > >> >> >> >> 0.05 289.45 0.16 18535473 0.00 0.00 > >> >> >> >> alloc_addbyter > >> >> >> >> 0.05 289.61 0.16 356074 0.00 0.00 > >> >> >> >> Curl_hash_str > >> >> >> >> 0.05 289.76 0.15 1797233 0.00 0.00 > >> >> >> >> fd_key_compare > >> >> >> >> 0.05 289.91 0.15 234966 0.00 0.00 > >> >> >> >> Curl_ssl_getsessionid > >> >> >> >> 0.04 290.04 0.13 585853 0.00 0.00 > >> >> >> >> Curl_llist_destroy > >> >> >> >> 0.04 290.16 0.12 240826 0.00 0.00 > Curl_resolv > >> >> >> >> 0.03 290.26 0.10 7514328 0.00 0.00 > >> >> >> >> Curl_raw_nequal > >> >> >> >> 0.03 290.36 0.10 2214436 0.00 0.00 Curl_read > >> >> >> >> 0.03 290.46 0.10 116872 0.00 0.00 > >> >> >> >> Curl_http_readwrite_headers > >> >> >> >> 0.03 290.56 0.10 > >> >> >> >> ssl_tls_trace > >> >> >> >> 0.03 290.66 0.10 8013097 0.00 0.00 > >> >> >> >> scan_response > >> >> >> >> 0.03 290.75 0.09 29488555 0.00 0.00 > >> >> >> >> curlx_tvdiff > >> >> >> >> 0.03 290.84 0.09 2142468 0.00 0.00 > Curl_setopt > >> >> >> >> 0.03 290.93 0.09 476299 0.00 0.00 > Curl_expire > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> 2012/3/5 Robert Iakobashvili <cor...@gm...> > >> >> >> >>> > >> >> >> >>> Dear Fred Huang, > >> >> >> >>> > >> >> >> >>> Thanks for your posting. > >> >> >> >>> > >> >> >> >>> Could you please check if this is due to some badness at > cares? > >> >> >> >>> Thanks. > >> >> >> >>> > >> >> >> >>> Let's exclude usage of cares based DNS resolving by applying > the > >> >> >> >>> patch > >> >> >> >>> attached > >> >> >> >>> and use instead global DNS cache. > >> >> >> >>> > >> >> >> >>> cd ./curl-loader-0.56 > >> >> >> >>> patch -p1 < path-to/disable-cares-use-global-dns.patch > >> >> >> >>> make cleanall; make > >> >> >> >>> > >> >> >> >>> See if it helps. > >> >> >> >>> > >> >> >> >>> Global cache, unfortunately, is not thread safe. > >> >> >> >>> > >> >> >> >>> If it helps, we'll try to restore the previous version of > cares. > >> >> >> >>> > >> >> >> >>> Thanks, > >> >> >> >>> Robert > >> >> >> >>> > >> >> >> >>> On Mon, Mar 5, 2012 at 4:28 PM, Fred Huang <di...@gm...> > >> >> >> >>> wrote: > >> >> >> >>>> > >> >> >> >>>> test 1: > >> >> >> >>>> > >> >> >> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, > >> >> >> >>>> resolve > >> >> >> >>>> *.com to one IP address > >> >> >> >>>> number of domain names in all https URLs: 780,000 > >> >> >> >>>> number of client: 1000 > >> >> >> >>>> CPU usage: 78% > >> >> >> >>>> cpu% irq% sirq% sys% iowt% mem_used buf&cached > >> >> >> >>>> 78.8 0.0 0.6 3.8 0.0 3640.9Mb 282.1Mb > >> >> >> >>>> SSL TPS: 300 > >> >> >> >>>> SSL throughput: 200Mbps > >> >> >> >>>> > >> >> >> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50 > >> >> >> >>>> Flat profile: > >> >> >> >>>> Each sample counts as 0.01 seconds. > >> >> >> >>>> % cumulative self self total > >> >> >> >>>> time seconds seconds calls s/call s/call name > >> >> >> >>>> 38.55 127.10 127.10 130975 0.00 0.00 > >> >> >> >>>> Curl_hash_clean_with_criterium > >> >> >> >>>> 24.63 208.32 81.22 1247528407 0.00 0.00 > >> >> >> >>>> hostcache_timestamp_remove > >> >> >> >>>> 13.08 251.44 43.12 2791988 0.00 0.00 > >> >> >> >>>> Curl_hash_pick > >> >> >> >>>> 7.30 275.51 24.07 352490 0.00 0.00 > >> >> >> >>>> Curl_hash_add > >> >> >> >>>> 6.01 295.31 19.80 410760770 0.00 0.00 > >> >> >> >>>> Curl_str_key_compare > >> >> >> >>>> 2.95 305.04 9.73 131015 0.00 0.00 > >> >> >> >>>> create_conn > >> >> >> >>>> 0.88 307.94 2.90 4841390 0.00 0.00 > >> >> >> >>>> dprintf_formatf > >> >> >> >>>> 0.74 310.38 2.44 136036 0.00 0.00 > >> >> >> >>>> ConnectionStore > >> >> >> >>>> 0.48 311.96 1.58 814604911 0.00 0.00 > >> >> >> >>>> ares__is_list_empty > >> >> >> >>>> 0.36 313.16 1.20 > >> >> >> >>>> locking_function > >> >> >> >>>> 0.27 314.06 0.90 261608 0.00 0.00 > >> >> >> >>>> ares_cancel > >> >> >> >>>> 0.22 314.79 0.73 127417 0.00 0.00 > >> >> >> >>>> curl_multi_remove_handle > >> >> >> >>>> 0.19 315.40 0.62 8906301 0.00 0.00 > >> >> >> >>>> client_tracing_function > >> >> >> >>>> 0.17 315.97 0.57 135305 0.00 0.00 > >> >> >> >>>> Curl_num_addresses > >> >> >> >>>> 0.17 316.54 0.57 181937907 0.00 0.00 > addbyter > >> >> >> >>>> 0.17 317.09 0.55 2374363 0.00 0.00 > >> >> >> >>>> multi_runsingle > >> >> >> >>>> 0.16 317.62 0.53 541954 0.00 0.00 > >> >> >> >>>> ares__init_list_node > >> >> >> >>>> 0.16 318.14 0.52 494412 0.00 0.00 > >> >> >> >>>> curl_multi_socket_action > >> >> >> >>>> 0.14 318.60 0.46 33578897 0.00 0.00 > >> >> >> >>>> Curl_socket_check > >> >> >> >>>> 0.13 319.04 0.44 102951052 0.00 0.00 > >> >> >> >>>> Curl_raw_toupper > >> >> >> >>>> 0.13 319.48 0.44 787558 0.00 0.00 > >> >> >> >>>> Curl_readwrite > >> >> >> >>>> 0.13 319.91 0.44 > >> >> >> >>>> id_function > >> >> >> >>>> 0.12 320.29 0.38 391390 0.00 0.00 > >> >> >> >>>> Curl_hash_str > >> >> >> >>>> 0.09 320.58 0.29 9216 0.00 0.00 > >> >> >> >>>> curl_multi_perform > >> >> >> >>>> 0.08 320.86 0.28 2350908 0.00 0.00 > >> >> >> >>>> Curl_raw_equal > >> >> >> >>>> 0.08 321.14 0.28 2019415 0.00 0.00 > >> >> >> >>>> Curl_splay > >> >> >> >>>> 0.08 321.40 0.26 520625 0.00 0.00 > >> >> >> >>>> ossl_connect_common > >> >> >> >>>> 0.08 321.65 0.25 3532992 0.00 0.00 > >> >> >> >>>> Curl_pgrsUpdate > >> >> >> >>>> 0.07 321.87 0.22 131778537 0.00 0.00 > >> >> >> >>>> curl_strequal > >> >> >> >>>> 0.05 322.05 0.18 8954118 0.00 0.00 > >> >> >> >>>> scan_response > >> >> >> >>>> 0.05 322.23 0.18 533644 0.00 0.00 > >> >> >> >>>> ares_expand_name > >> >> >> >>>> 0.05 322.40 0.17 2477345 0.00 0.00 > >> >> >> >>>> fd_key_compare > >> >> >> >>>> 0.05 322.57 0.17 2429135 0.00 0.00 > >> >> >> >>>> Curl_infof > >> >> >> >>>> 0.05 322.74 0.17 130966 0.00 0.00 > >> >> >> >>>> singleipconnect > >> >> >> >>>> 0.05 322.90 0.16 9292 0.00 0.00 > >> >> >> >>>> curl_multi_socket_all > >> >> >> >>>> 0.05 323.06 0.16 270137 0.00 0.00 > >> >> >> >>>> ares__get_hostent > >> >> >> >>>> 0.05 323.21 0.15 8943657 0.00 0.00 > >> >> >> >>>> Curl_debug > >> >> >> >>>> 0.04 323.35 0.14 262493 0.00 0.00 > >> >> >> >>>> Curl_ssl_getsessionid > >> >> >> >>>> 0.04 323.48 0.13 566006 0.00 0.00 > >> >> >> >>>> socket_callback > >> >> >> >>>> 0.04 323.61 0.13 33983864 0.00 0.00 > >> >> >> >>>> curlx_tvdiff > >> >> >> >>>> 0.04 323.74 0.13 132853 0.00 0.00 > >> >> >> >>>> Curl_http_readwrite_headers > >> >> >> >>>> 0.04 323.86 0.12 344032 0.00 0.00 > epoll_del > >> >> >> >>>> 0.04 323.98 0.12 223477 0.00 0.00 > Curl_poll > >> >> >> >>>> 0.04 324.10 0.12 8526280 0.00 0.00 > >> >> >> >>>> Curl_raw_nequal > >> >> >> >>>> 0.03 324.21 0.11 348839 0.00 0.00 > event_del > >> >> >> >>>> > >> >> >> >>>> > >> >> >> >>>> > >> >> >> >>>> > >> >> >> >>>> test 2: > >> >> >> >>>> > >> >> >> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, > >> >> >> >>>> resolve > >> >> >> >>>> *.com to one IP address > >> >> >> >>>> number of domain names in all https URLs: 1 > >> >> >> >>>> number of client: 1000 > >> >> >> >>>> CPU usage: 75% > >> >> >> >>>> SSL TPS: 1300 > >> >> >> >>>> SSL throughput: 700Mbps > >> >> >> >>>> > >> >> >> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50 > >> >> >> >>>> Flat profile: > >> >> >> >>>> Each sample counts as 0.01 seconds. > >> >> >> >>>> % cumulative self self total > >> >> >> >>>> time seconds seconds calls s/call s/call name > >> >> >> >>>> 8.98 2.45 2.45 5127312 0.00 0.00 > >> >> >> >>>> dprintf_formatf > >> >> >> >>>> 8.90 4.88 2.43 1292796503 0.00 0.00 > >> >> >> >>>> ares__is_list_empty > >> >> >> >>>> 6.12 6.55 1.67 322461 0.00 0.00 > >> >> >> >>>> create_conn > >> >> >> >>>> 4.58 7.80 1.25 419872 0.00 0.00 > >> >> >> >>>> ares_cancel > >> >> >> >>>> 4.10 8.92 1.12 2100162 0.00 0.00 > >> >> >> >>>> Curl_readwrite > >> >> >> >>>> 3.92 9.99 1.07 4527388 0.00 0.00 > >> >> >> >>>> Curl_hash_pick > >> >> >> >>>> 3.43 10.93 0.94 > >> >> >> >>>> locking_function > >> >> >> >>>> 3.04 11.76 0.83 909759 0.00 0.00 > >> >> >> >>>> curl_multi_socket_action > >> >> >> >>>> 3.00 12.58 0.82 97252 0.00 0.00 > >> >> >> >>>> Curl_hash_clean_with_criterium > >> >> >> >>>> 2.71 13.32 0.74 179704016 0.00 0.00 > >> >> >> >>>> Curl_raw_toupper > >> >> >> >>>> 2.68 14.05 0.73 4646524 0.00 0.00 > >> >> >> >>>> multi_runsingle > >> >> >> >>>> 2.44 14.71 0.67 13862992 0.00 0.00 > >> >> >> >>>> client_tracing_function > >> >> >> >>>> 2.42 15.37 0.66 319735 0.00 0.00 > >> >> >> >>>> curl_multi_remove_handle > >> >> >> >>>> 2.42 16.03 0.66 22 0.03 0.03 > >> >> >> >>>> ares__init_list_node > >> >> >> >>>> 1.94 16.56 0.53 9930244 0.00 0.00 > >> >> >> >>>> hostcache_timestamp_remove > >> >> >> >>>> 1.94 17.09 0.53 169320088 0.00 0.00 > addbyter > >> >> >> >>>> 1.80 17.58 0.49 > >> >> >> >>>> id_function > >> >> >> >>>> 1.36 17.95 0.37 7057146 0.00 0.00 > >> >> >> >>>> Curl_pgrsUpdate > >> >> >> >>>> 1.36 18.32 0.37 16835 0.00 0.00 > >> >> >> >>>> curl_multi_socket_all > >> >> >> >>>> 1.25 18.66 0.34 16719 0.00 0.00 > >> >> >> >>>> curl_multi_perform > >> >> >> >>>> 1.21 18.99 0.33 329403 0.00 0.00 > >> >> >> >>>> Curl_http_readwrite_headers > >> >> >> >>>> 1.17 19.31 0.32 4036624 0.00 0.00 > >> >> >> >>>> Curl_splay > >> >> >> >>>> 1.14 19.62 0.31 20838902 0.00 0.00 > >> >> >> >>>> Curl_raw_nequal > >> >> >> >>>> 0.82 19.85 0.23 2764123 0.00 0.00 > >> >> >> >>>> Curl_infof > >> >> >> >>>> 0.81 20.07 0.22 5933857 0.00 0.00 > ossl_recv > >> >> >> >>>> 0.77 20.28 0.21 323112 0.00 0.00 > >> >> >> >>>> Curl_splayremovebyaddr > >> >> >> >>>> 0.75 20.48 0.21 5943991 0.00 0.00 > Curl_read > >> >> >> >>>> 0.70 20.67 0.19 425174 0.00 0.00 > event_del > >> >> >> >>>> 0.70 20.86 0.19 98652 0.00 0.00 > >> >> >> >>>> Curl_if_is_interface_name > >> >> >> >>>> 0.70 21.05 0.19 8677733 0.00 0.00 > >> >> >> >>>> Curl_socket_check > >> >> >> >>>> 0.59 21.21 0.16 4129554 0.00 0.00 > >> >> >> >>>> fd_key_compare > >> >> >> >>>> 0.57 21.37 0.16 14017019 0.00 0.00 > >> >> >> >>>> Curl_debug > >> >> >> >>>> 0.55 21.52 0.15 9376861 0.00 0.00 > >> >> >> >>>> stat_data_in_add > >> >> >> >>>> 0.55 21.67 0.15 6840468 0.00 0.00 > >> >> >> >>>> Curl_timeleft > >> >> >> >>>> 0.55 21.82 0.15 2054602 0.00 0.00 > >> >> >> >>>> Curl_raw_equal > >> >> >> >>>> 0.55 21.97 0.15 1291611 0.00 0.00 > >> >> >> >>>> Curl_expire > >> >> >> >>>> 0.51 22.11 0.14 16570063 0.00 0.00 > >> >> >> >>>> curlx_tvnow > >> >> >> >>>> 0.51 22.25 0.14 13863284 0.00 0.00 > >> >> >> >>>> scan_response > >> >> >> >>>> 0.51 22.39 0.14 5498864 0.00 0.00 > >> >> >> >>>> Curl_setopt > >> >> >> >>>> 0.48 22.52 0.13 97490291 0.00 0.00 > >> >> >> >>>> curl_strequal > >> >> >> >>>> 0.44 22.64 0.12 324253 0.00 0.00 > Curl_http > >> >> >> >>>> 0.44 22.76 0.12 394386 0.00 0.00 > >> >> >> >>>> ossl_connect_common > >> >> >> >>>> 0.40 22.87 0.11 3581335 0.00 0.00 > >> >> >> >>>> Curl_getinfo > >> >> >> >>>> 0.38 22.97 0.11 26983092 0.00 0.00 > >> >> >> >>>> alloc_addbyter > >> >> >> >>>> 0.37 23.07 0.10 8729044 0.00 0.00 > >> >> >> >>>> Curl_client_write > >> >> >> >>>> > >> >> >> >>>> > >> >> >> >>>> > >> >> >> >>>> > >> >> >> >>>> > >> >> >> >>>> > ------------------------------------------------------------------------------ > >> >> >> >>>> Try before you buy = See our experts in action! > >> >> >> >>>> The most comprehensive online learning library for Microsoft > >> >> >> >>>> developers > >> >> >> >>>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, > >> >> >> >>>> CSS3, > >> >> >> >>>> MVC3, > >> >> >> >>>> Metro Style Apps, more. Free future releases when you > subscribe > >> >> >> >>>> now! > >> >> >> >>>> http://p.sf.net/sfu/learndevnow-dev2 > >> >> >> >>>> _______________________________________________ > >> >> >> >>>> curl-loader-devel mailing list > >> >> >> >>>> cur...@li... > >> >> >> >>>> > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > >> >> >> >>>> > >> >> >> >>> > >> >> >> >>> > >> >> >> >>> > >> >> >> >>> -- > >> >> >> >>> Regards, > >> >> >> >>> Robert Iakobashvili, Ph.D. > >> >> >> >>> > >> >> >> >>> Home: http://www.ghotit.com > >> >> >> >>> Blog: http://www.ghotit.com/category/ghotit-blogs/ > >> >> >> >>> Twitter: http://twitter.com/ghotit > >> >> >> >>> Facebook: http://facebook.com/ghotit > >> >> >> >>> > >> >> >> >>> > >> >> >> >>> > ...................................................................... > >> >> >> >>> Ghotit Dyslexia > >> >> >> >>> Assistive technology that understands you > >> >> >> >>> > >> >> >> >>> > >> >> >> >>> > ...................................................................... > >> >> >> >>> > >> >> >> >>> > >> >> >> >>> > >> >> >> >>> > >> >> >> >>> > >> >> >> >>> > ------------------------------------------------------------------------------ > >> >> >> >>> Try before you buy = See our experts in action! > >> >> >> >>> The most comprehensive online learning library for Microsoft > >> >> >> >>> developers > >> >> >> >>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, > >> >> >> >>> CSS3, > >> >> >> >>> MVC3, > >> >> >> >>> Metro Style Apps, more. Free future releases when you > subscribe > >> >> >> >>> now! > >> >> >> >>> http://p.sf.net/sfu/learndevnow-dev2 > >> >> >> >>> _______________________________________________ > >> >> >> >>> curl-loader-devel mailing list > >> >> >> >>> cur...@li... > >> >> >> >>> > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > >> >> >> >>> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > ------------------------------------------------------------------------------ > >> >> >> >> Try before you buy = See our experts in action! > >> >> >> >> The most comprehensive online learning library for Microsoft > >> >> >> >> developers > >> >> >> >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, > >> >> >> >> CSS3, > >> >> >> >> MVC3, > >> >> >> >> Metro Style Apps, more. Free future releases when you subscribe > >> >> >> >> now! > >> >> >> >> http://p.sf.net/sfu/learndevnow-dev2 > >> >> >> >> _______________________________________________ > >> >> >> >> curl-loader-devel mailing list > >> >> >> >> cur...@li... > >> >> >> >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > >> >> >> >> > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > -- > >> >> >> > Regards, > >> >> >> > Robert Iakobashvili, Ph.D. > >> >> >> > > >> >> >> > Home: http://www.ghotit.com > >> >> >> > Blog: http://www.ghotit.com/category/ghotit-blogs/ > >> >> >> > Twitter: http://twitter.com/ghotit > >> >> >> > Facebook: http://facebook.com/ghotit > >> >> >> > > >> >> >> > > >> >> >> > > ...................................................................... > >> >> >> > Ghotit Dyslexia > >> >> >> > Assistive technology that understands you > >> >> >> > > >> >> >> > > >> >> >> > > ...................................................................... > >> >> >> > >> >> >> > >> >> >> > >> >> >> -- > >> >> >> Regards, > >> >> >> Robert Iakobashvili, Ph.D. > >> >> >> > >> >> >> Home: http://www.ghotit.com > >> >> >> Blog: http://www.ghotit.com/category/ghotit-blogs/ > >> >> >> Twitter: http://twitter.com/ghotit > >> >> >> Facebook: http://facebook.com/ghotit > >> >> >> > >> >> >> > ...................................................................... > >> >> >> Ghotit Dyslexia > >> >> >> Assistive technology that understands you > >> >> >> > >> >> >> > ...................................................................... > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > ------------------------------------------------------------------------------ > >> >> >> Try before you buy = See our experts in action! > >> >> >> The most comprehensive online learning library for Microsoft > >> >> >> developers > >> >> >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, > >> >> >> MVC3, > >> >> >> Metro Style Apps, more. Free future releases when you subscribe > now! > >> >> >> http://p.sf.net/sfu/learndevnow-dev2 > >> >> >> _______________________________________________ > >> >> >> curl-loader-devel mailing list > >> >> >> cur...@li... > >> >> >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > ------------------------------------------------------------------------------ > >> >> > Keep Your Developer Skills Current with LearnDevNow! > >> >> > The most comprehensive online learning library for Microsoft > >> >> > developers > >> >> > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, > >> >> > MVC3, > >> >> > Metro Style Apps, more. Free future releases when you subscribe > now! > >> >> > http://p.sf.net/sfu/learndevnow-d2d > >> >> > _______________________________________________ > >> >> > curl-loader-devel mailing list > >> >> > cur...@li... > >> >> > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > >> >> > > >> >> > >> >> > >> >> > >> >> -- > >> >> Regards, > >> >> Robert Iakobashvili, Ph.D. > >> >> > >> >> Home: http://www.ghotit.com > >> >> Blog: http://www.ghotit.com/category/ghotit-blogs/ > >> >> Twitter: http://twitter.com/ghotit > >> >> Facebook: http://facebook.com/ghotit > >> >> > ...................................................................... > >> >> Ghotit Dyslexia > >> >> Assistive technology that understands you > >> >> > ...................................................................... > >> >> > >> >> > >> >> > >> >> > ------------------------------------------------------------------------------ > >> >> Keep Your Developer Skills Current with LearnDevNow! > >> >> The most comprehensive online learning library for Microsoft > developers > >> >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, > >> >> MVC3, > >> >> Metro Style Apps, more. Free future releases when you subscribe now! > >> >> http://p.sf.net/sfu/learndevnow-d2d > >> >> _______________________________________________ > >> >> curl-loader-devel mailing list > >> >> cur...@li... > >> >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > >> > > >> > > >> > > >> > > >> > > ------------------------------------------------------------------------------ > >> > Keep Your Developer Skills Current with LearnDevNow! > >> > The most comprehensive online learning library for Microsoft > developers > >> > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, > MVC3, > >> > Metro Style Apps, more. Free future releases when you subscribe now! > >> > http://p.sf.net/sfu/learndevnow-d2d > >> > _______________________________________________ > >> > curl-loader-devel mailing list > >> > cur...@li... > >> > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > >> > > >> > >> > >> > >> -- > >> Regards, > >> Robert Iakobashvili, Ph.D. > >> > >> Home: http://www.ghotit.com > >> Blog: http://www.ghotit.com/category/ghotit-blogs/ > >> Twitter: http://twitter.com/ghotit > >> Facebook: http://facebook.com/ghotit > >> ...................................................................... > >> Ghotit Dyslexia > >> Assistive technology that understands you > >> ...................................................................... > >> > >> > >> > ------------------------------------------------------------------------------ > >> Keep Your Developer Skills Current with LearnDevNow! > >> The most comprehensive online learning library for Microsoft developers > >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, > >> Metro Style Apps, more. Free future releases when you subscribe now! > >> http://p.sf.net/sfu/learndevnow-d2d > >> _______________________________________________ > >> curl-loader-devel mailing list > >> cur...@li... > >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > > > > > > > > > ------------------------------------------------------------------------------ > > Virtualization & Cloud Management Using Capacity Planning > > Cloud computing makes use of virtualization - but cloud computing > > also focuses on allowing computing to be delivered as a service. > > http://www.accelacomm.com/jaw/sfnl/114/51521223/ > > _______________________________________________ > > curl-loader-devel mailing list > > cur...@li... > > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > > > > > > -- > Regards, > Robert Iakobashvili, Ph.D. > > Home: http://www.ghotit.com > Blog: http://www.ghotit.com/category/ghotit-blogs/ > Twitter: http://twitter.com/ghotit > Facebook: http://facebook.com/ghotit > ...................................................................... > Ghotit Dyslexia > Assistive technology that understands you > ...................................................................... > > > ------------------------------------------------------------------------------ > Virtualization & Cloud Management Using Capacity Planning > Cloud computing makes use of virtualization - but cloud computing > also focuses on allowing computing to be delivered as a service. > http://www.accelacomm.com/jaw/sfnl/114/51521223/ > _______________________________________________ > curl-loader-devel mailing list > cur...@li... > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > |