Thread: curl-loader https: low performance due to DNS query?
Status: Alpha
Brought to you by:
coroberti
Menu
▾
▴
curl-loader-devel
|
From: Fred H. <di...@gm...> - 2012-03-05 14:28:49
|
test 1:
dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com to
one IP address
number of domain names in all https URLs: 780,000
number of client: 1000
CPU usage: 78%
cpu% irq% sirq% sys% iowt% mem_used buf&cached
78.8 0.0 0.6 3.8 0.0 3640.9Mb 282.1Mb
SSL TPS: 300
SSL throughput: 200Mbps
# gprof /usr/bin/curl-loader gmon.out -p | head -50
Flat profile:
Each sample counts as 0.01 seconds.
% cumulative self self total
time seconds seconds calls s/call s/call name
* 38.55 127.10 127.10 130975 0.00 0.00
Curl_hash_clean_with_criterium*
* 24.63 208.32 81.22 1247528407 0.00 0.00
hostcache_timestamp_remove*
* 13.08 251.44 43.12 2791988 0.00 0.00 Curl_hash_pick*
7.30 275.51 24.07 352490 0.00 0.00 Curl_hash_add
6.01 295.31 19.80 410760770 0.00 0.00 Curl_str_key_compare
2.95 305.04 9.73 131015 0.00 0.00 create_conn
0.88 307.94 2.90 4841390 0.00 0.00 dprintf_formatf
0.74 310.38 2.44 136036 0.00 0.00 ConnectionStore
0.48 311.96 1.58 814604911 0.00 0.00 ares__is_list_empty
0.36 313.16 1.20 locking_function
0.27 314.06 0.90 261608 0.00 0.00 ares_cancel
0.22 314.79 0.73 127417 0.00 0.00
curl_multi_remove_handle
0.19 315.40 0.62 8906301 0.00 0.00
client_tracing_function
0.17 315.97 0.57 135305 0.00 0.00 Curl_num_addresses
0.17 316.54 0.57 181937907 0.00 0.00 addbyter
0.17 317.09 0.55 2374363 0.00 0.00 multi_runsingle
0.16 317.62 0.53 541954 0.00 0.00 ares__init_list_node
0.16 318.14 0.52 494412 0.00 0.00
curl_multi_socket_action
0.14 318.60 0.46 33578897 0.00 0.00 Curl_socket_check
0.13 319.04 0.44 102951052 0.00 0.00 Curl_raw_toupper
0.13 319.48 0.44 787558 0.00 0.00 Curl_readwrite
0.13 319.91 0.44 id_function
0.12 320.29 0.38 391390 0.00 0.00 Curl_hash_str
0.09 320.58 0.29 9216 0.00 0.00 curl_multi_perform
0.08 320.86 0.28 2350908 0.00 0.00 Curl_raw_equal
0.08 321.14 0.28 2019415 0.00 0.00 Curl_splay
0.08 321.40 0.26 520625 0.00 0.00 ossl_connect_common
0.08 321.65 0.25 3532992 0.00 0.00 Curl_pgrsUpdate
0.07 321.87 0.22 131778537 0.00 0.00 curl_strequal
0.05 322.05 0.18 8954118 0.00 0.00 scan_response
0.05 322.23 0.18 533644 0.00 0.00 ares_expand_name
0.05 322.40 0.17 2477345 0.00 0.00 fd_key_compare
0.05 322.57 0.17 2429135 0.00 0.00 Curl_infof
0.05 322.74 0.17 130966 0.00 0.00 singleipconnect
0.05 322.90 0.16 9292 0.00 0.00 curl_multi_socket_all
0.05 323.06 0.16 270137 0.00 0.00 ares__get_hostent
0.05 323.21 0.15 8943657 0.00 0.00 Curl_debug
0.04 323.35 0.14 262493 0.00 0.00 Curl_ssl_getsessionid
0.04 323.48 0.13 566006 0.00 0.00 socket_callback
0.04 323.61 0.13 33983864 0.00 0.00 curlx_tvdiff
0.04 323.74 0.13 132853 0.00 0.00
Curl_http_readwrite_headers
0.04 323.86 0.12 344032 0.00 0.00 epoll_del
0.04 323.98 0.12 223477 0.00 0.00 Curl_poll
0.04 324.10 0.12 8526280 0.00 0.00 Curl_raw_nequal
0.03 324.21 0.11 348839 0.00 0.00 event_del
test 2:
dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com to
one IP address
number of domain names in all https URLs: 1
number of client: 1000
CPU usage: 75%
SSL TPS: 1300
SSL throughput: 700Mbps
# gprof /usr/bin/curl-loader gmon.out -p | head -50
Flat profile:
Each sample counts as 0.01 seconds.
% cumulative self self total
time seconds seconds calls s/call s/call name
8.98 2.45 2.45 5127312 0.00 0.00 dprintf_formatf
8.90 4.88 2.43 1292796503 0.00 0.00 ares__is_list_empty
6.12 6.55 1.67 322461 0.00 0.00 create_conn
4.58 7.80 1.25 419872 0.00 0.00 ares_cancel
4.10 8.92 1.12 2100162 0.00 0.00 Curl_readwrite
3.92 9.99 1.07 4527388 0.00 0.00 Curl_hash_pick
3.43 10.93 0.94 locking_function
3.04 11.76 0.83 909759 0.00 0.00
curl_multi_socket_action
3.00 12.58 0.82 97252 0.00 0.00
Curl_hash_clean_with_criterium
2.71 13.32 0.74 179704016 0.00 0.00 Curl_raw_toupper
2.68 14.05 0.73 4646524 0.00 0.00 multi_runsingle
2.44 14.71 0.67 13862992 0.00 0.00
client_tracing_function
2.42 15.37 0.66 319735 0.00 0.00
curl_multi_remove_handle
2.42 16.03 0.66 22 0.03 0.03 ares__init_list_node
1.94 16.56 0.53 9930244 0.00 0.00
hostcache_timestamp_remove
1.94 17.09 0.53 169320088 0.00 0.00 addbyter
1.80 17.58 0.49 id_function
1.36 17.95 0.37 7057146 0.00 0.00 Curl_pgrsUpdate
1.36 18.32 0.37 16835 0.00 0.00 curl_multi_socket_all
1.25 18.66 0.34 16719 0.00 0.00 curl_multi_perform
1.21 18.99 0.33 329403 0.00 0.00
Curl_http_readwrite_headers
1.17 19.31 0.32 4036624 0.00 0.00 Curl_splay
1.14 19.62 0.31 20838902 0.00 0.00 Curl_raw_nequal
0.82 19.85 0.23 2764123 0.00 0.00 Curl_infof
0.81 20.07 0.22 5933857 0.00 0.00 ossl_recv
0.77 20.28 0.21 323112 0.00 0.00 Curl_splayremovebyaddr
0.75 20.48 0.21 5943991 0.00 0.00 Curl_read
0.70 20.67 0.19 425174 0.00 0.00 event_del
0.70 20.86 0.19 98652 0.00 0.00
Curl_if_is_interface_name
0.70 21.05 0.19 8677733 0.00 0.00 Curl_socket_check
0.59 21.21 0.16 4129554 0.00 0.00 fd_key_compare
0.57 21.37 0.16 14017019 0.00 0.00 Curl_debug
0.55 21.52 0.15 9376861 0.00 0.00 stat_data_in_add
0.55 21.67 0.15 6840468 0.00 0.00 Curl_timeleft
0.55 21.82 0.15 2054602 0.00 0.00 Curl_raw_equal
0.55 21.97 0.15 1291611 0.00 0.00 Curl_expire
0.51 22.11 0.14 16570063 0.00 0.00 curlx_tvnow
0.51 22.25 0.14 13863284 0.00 0.00 scan_response
0.51 22.39 0.14 5498864 0.00 0.00 Curl_setopt
0.48 22.52 0.13 97490291 0.00 0.00 curl_strequal
0.44 22.64 0.12 324253 0.00 0.00 Curl_http
0.44 22.76 0.12 394386 0.00 0.00 ossl_connect_common
0.40 22.87 0.11 3581335 0.00 0.00 Curl_getinfo
0.38 22.97 0.11 26983092 0.00 0.00 alloc_addbyter
0.37 23.07 0.10 8729044 0.00 0.00 Curl_client_write
|
|
From: Robert I. <cor...@gm...> - 2012-03-05 15:15:40
Attachments:
disable-cares-use-global-dns.patch
|
Dear Fred Huang, Thanks for your posting. Could you please check if this is due to some badness at cares? Thanks. Let's exclude usage of cares based DNS resolving by applying the patch attached and use instead global DNS cache. cd ./curl-loader-0.56 patch -p1 < path-to/disable-cares-use-global-dns.patch make cleanall; make See if it helps. Global cache, unfortunately, is not thread safe. If it helps, we'll try to restore the previous version of cares. Thanks, Robert On Mon, Mar 5, 2012 at 4:28 PM, Fred Huang <di...@gm...> wrote: > test 1: > > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com > to one IP address > number of domain names in all https URLs: 780,000 > number of client: 1000 > CPU usage: 78% > cpu% irq% sirq% sys% iowt% mem_used buf&cached > 78.8 0.0 0.6 3.8 0.0 3640.9Mb 282.1Mb > SSL TPS: 300 > SSL throughput: 200Mbps > > # gprof /usr/bin/curl-loader gmon.out -p | head -50 > Flat profile: > Each sample counts as 0.01 seconds. > % cumulative self self total > time seconds seconds calls s/call s/call name > * 38.55 127.10 127.10 130975 0.00 0.00 > Curl_hash_clean_with_criterium* > * 24.63 208.32 81.22 1247528407 0.00 0.00 > hostcache_timestamp_remove* > * 13.08 251.44 43.12 2791988 0.00 0.00 Curl_hash_pick* > 7.30 275.51 24.07 352490 0.00 0.00 Curl_hash_add > 6.01 295.31 19.80 410760770 0.00 0.00 Curl_str_key_compare > 2.95 305.04 9.73 131015 0.00 0.00 create_conn > 0.88 307.94 2.90 4841390 0.00 0.00 dprintf_formatf > 0.74 310.38 2.44 136036 0.00 0.00 ConnectionStore > 0.48 311.96 1.58 814604911 0.00 0.00 ares__is_list_empty > 0.36 313.16 1.20 locking_function > 0.27 314.06 0.90 261608 0.00 0.00 ares_cancel > 0.22 314.79 0.73 127417 0.00 0.00 > curl_multi_remove_handle > 0.19 315.40 0.62 8906301 0.00 0.00 > client_tracing_function > 0.17 315.97 0.57 135305 0.00 0.00 Curl_num_addresses > 0.17 316.54 0.57 181937907 0.00 0.00 addbyter > 0.17 317.09 0.55 2374363 0.00 0.00 multi_runsingle > 0.16 317.62 0.53 541954 0.00 0.00 ares__init_list_node > 0.16 318.14 0.52 494412 0.00 0.00 > curl_multi_socket_action > 0.14 318.60 0.46 33578897 0.00 0.00 Curl_socket_check > 0.13 319.04 0.44 102951052 0.00 0.00 Curl_raw_toupper > 0.13 319.48 0.44 787558 0.00 0.00 Curl_readwrite > 0.13 319.91 0.44 id_function > 0.12 320.29 0.38 391390 0.00 0.00 Curl_hash_str > 0.09 320.58 0.29 9216 0.00 0.00 curl_multi_perform > 0.08 320.86 0.28 2350908 0.00 0.00 Curl_raw_equal > 0.08 321.14 0.28 2019415 0.00 0.00 Curl_splay > 0.08 321.40 0.26 520625 0.00 0.00 ossl_connect_common > 0.08 321.65 0.25 3532992 0.00 0.00 Curl_pgrsUpdate > 0.07 321.87 0.22 131778537 0.00 0.00 curl_strequal > 0.05 322.05 0.18 8954118 0.00 0.00 scan_response > 0.05 322.23 0.18 533644 0.00 0.00 ares_expand_name > 0.05 322.40 0.17 2477345 0.00 0.00 fd_key_compare > 0.05 322.57 0.17 2429135 0.00 0.00 Curl_infof > 0.05 322.74 0.17 130966 0.00 0.00 singleipconnect > 0.05 322.90 0.16 9292 0.00 0.00 curl_multi_socket_all > 0.05 323.06 0.16 270137 0.00 0.00 ares__get_hostent > 0.05 323.21 0.15 8943657 0.00 0.00 Curl_debug > 0.04 323.35 0.14 262493 0.00 0.00 Curl_ssl_getsessionid > 0.04 323.48 0.13 566006 0.00 0.00 socket_callback > 0.04 323.61 0.13 33983864 0.00 0.00 curlx_tvdiff > 0.04 323.74 0.13 132853 0.00 0.00 > Curl_http_readwrite_headers > 0.04 323.86 0.12 344032 0.00 0.00 epoll_del > 0.04 323.98 0.12 223477 0.00 0.00 Curl_poll > 0.04 324.10 0.12 8526280 0.00 0.00 Curl_raw_nequal > 0.03 324.21 0.11 348839 0.00 0.00 event_del > > > > > test 2: > > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com > to one IP address > number of domain names in all https URLs: 1 > number of client: 1000 > CPU usage: 75% > SSL TPS: 1300 > SSL throughput: 700Mbps > > # gprof /usr/bin/curl-loader gmon.out -p | head -50 > Flat profile: > Each sample counts as 0.01 seconds. > % cumulative self self total > time seconds seconds calls s/call s/call name > 8.98 2.45 2.45 5127312 0.00 0.00 dprintf_formatf > 8.90 4.88 2.43 1292796503 0.00 0.00 ares__is_list_empty > 6.12 6.55 1.67 322461 0.00 0.00 create_conn > 4.58 7.80 1.25 419872 0.00 0.00 ares_cancel > 4.10 8.92 1.12 2100162 0.00 0.00 Curl_readwrite > 3.92 9.99 1.07 4527388 0.00 0.00 Curl_hash_pick > 3.43 10.93 0.94 locking_function > 3.04 11.76 0.83 909759 0.00 0.00 > curl_multi_socket_action > 3.00 12.58 0.82 97252 0.00 0.00 > Curl_hash_clean_with_criterium > 2.71 13.32 0.74 179704016 0.00 0.00 Curl_raw_toupper > 2.68 14.05 0.73 4646524 0.00 0.00 multi_runsingle > 2.44 14.71 0.67 13862992 0.00 0.00 > client_tracing_function > 2.42 15.37 0.66 319735 0.00 0.00 > curl_multi_remove_handle > 2.42 16.03 0.66 22 0.03 0.03 ares__init_list_node > 1.94 16.56 0.53 9930244 0.00 0.00 > hostcache_timestamp_remove > 1.94 17.09 0.53 169320088 0.00 0.00 addbyter > 1.80 17.58 0.49 id_function > 1.36 17.95 0.37 7057146 0.00 0.00 Curl_pgrsUpdate > 1.36 18.32 0.37 16835 0.00 0.00 curl_multi_socket_all > 1.25 18.66 0.34 16719 0.00 0.00 curl_multi_perform > 1.21 18.99 0.33 329403 0.00 0.00 > Curl_http_readwrite_headers > 1.17 19.31 0.32 4036624 0.00 0.00 Curl_splay > 1.14 19.62 0.31 20838902 0.00 0.00 Curl_raw_nequal > 0.82 19.85 0.23 2764123 0.00 0.00 Curl_infof > 0.81 20.07 0.22 5933857 0.00 0.00 ossl_recv > 0.77 20.28 0.21 323112 0.00 0.00 > Curl_splayremovebyaddr > 0.75 20.48 0.21 5943991 0.00 0.00 Curl_read > 0.70 20.67 0.19 425174 0.00 0.00 event_del > 0.70 20.86 0.19 98652 0.00 0.00 > Curl_if_is_interface_name > 0.70 21.05 0.19 8677733 0.00 0.00 Curl_socket_check > 0.59 21.21 0.16 4129554 0.00 0.00 fd_key_compare > 0.57 21.37 0.16 14017019 0.00 0.00 Curl_debug > 0.55 21.52 0.15 9376861 0.00 0.00 stat_data_in_add > 0.55 21.67 0.15 6840468 0.00 0.00 Curl_timeleft > 0.55 21.82 0.15 2054602 0.00 0.00 Curl_raw_equal > 0.55 21.97 0.15 1291611 0.00 0.00 Curl_expire > 0.51 22.11 0.14 16570063 0.00 0.00 curlx_tvnow > 0.51 22.25 0.14 13863284 0.00 0.00 scan_response > 0.51 22.39 0.14 5498864 0.00 0.00 Curl_setopt > 0.48 22.52 0.13 97490291 0.00 0.00 curl_strequal > 0.44 22.64 0.12 324253 0.00 0.00 Curl_http > 0.44 22.76 0.12 394386 0.00 0.00 ossl_connect_common > 0.40 22.87 0.11 3581335 0.00 0.00 Curl_getinfo > 0.38 22.97 0.11 26983092 0.00 0.00 alloc_addbyter > 0.37 23.07 0.10 8729044 0.00 0.00 Curl_client_write > > > > ------------------------------------------------------------------------------ > Try before you buy = See our experts in action! > The most comprehensive online learning library for Microsoft developers > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, > Metro Style Apps, more. Free future releases when you subscribe now! > http://p.sf.net/sfu/learndevnow-dev2 > _______________________________________________ > curl-loader-devel mailing list > cur...@li... > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > > -- Regards, Robert Iakobashvili, Ph.D. Home: http://www.ghotit.com Blog: http://www.ghotit.com/category/ghotit-blogs/ Twitter: http://twitter.com/ghotit Facebook: http://facebook.com/ghotit ...................................................................... Ghotit Dyslexia Assistive technology that understands you ...................................................................... |
|
From: Fred H. <di...@gm...> - 2012-03-05 16:19:23
|
Looks like the bottleneck comes from libcurl other than c-ares...
test 3 (no ares patch applied):
dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com to
one IP address
number of domain names in URLs: 780,000
number of client: 1000
CPU usage: 85%
cpu% irq% sirq% sys% iowt% mem_used buf&cached
85.6 0.0 0.8 5.9 0.0 3334.3Mb 112.0Mb
SSL TPS: *320*
SSL throughput: 200Mbps
# gprof /usr/bin/curl-loader gmon.out -p | head -50
Flat profile:
Each sample counts as 0.01 seconds.
% cumulative self self total
time seconds seconds calls s/call s/call name
* 40.80 120.16 120.16 116079 0.00 0.00
Curl_hash_clean_with_criterium*
* 24.46 192.18 72.02 983166323 0.00 0.00
hostcache_timestamp_remove*
* 13.30 231.35 39.17 2059341 0.00 0.00 Curl_hash_pick*
6.45 250.35 19.00 240087 0.00 0.00 Curl_hash_add
4.81 264.51 14.16 334257102 0.00 0.00 Curl_str_key_compare
3.40 274.52 10.01 119514 0.00 0.00 create_conn
0.93 277.25 2.73 4478910 0.00 0.00 dprintf_formatf
0.90 279.91 2.66 122209 0.00 0.00 ConnectionStore
0.40 281.09 1.18 locking_function
0.27 281.88 0.79 114759 0.00 0.00
curl_multi_remove_handle
0.21 282.51 0.63 7993664 0.00 0.00
client_tracing_function
0.21 283.12 0.61 122685 0.00 0.00 Curl_num_addresses
0.19 283.68 0.56 397028 0.00 0.00
curl_multi_socket_action
0.19 284.23 0.55 2036116 0.00 0.00 multi_runsingle
0.17 284.72 0.49 703687 0.00 0.00 Curl_readwrite
0.16 285.19 0.47 id_function
0.15 285.62 0.43 29692029 0.00 0.00 Curl_socket_check
0.14 286.02 0.40 91973856 0.00 0.00 Curl_raw_toupper
0.13 286.40 0.38 164664144 0.00 0.00 addbyter
0.13 286.77 0.37 2180797 0.00 0.00 Curl_infof
0.11 287.10 0.33 122675 0.00 0.00
Curl_if_is_interface_name
0.10 287.39 0.29 471318 0.00 0.00 ossl_connect_common
0.10 287.67 0.28 2113528 0.00 0.00 Curl_raw_equal
0.09 287.93 0.26 8676 0.00 0.02 curl_multi_socket_all
0.08 288.17 0.24 1663262 0.00 0.00 Curl_splay
0.08 288.41 0.24 8505 0.00 0.01 curl_multi_perform
0.07 288.63 0.22 3114576 0.00 0.00 Curl_pgrsUpdate
0.06 288.80 0.17 119637 0.00 0.00 singleipconnect
0.06 288.97 0.17 7994818 0.00 0.00 Curl_debug
0.05 289.13 0.16 235820 0.00 0.00 event_del
0.05 289.29 0.16 119393071 0.00 0.00 curl_strequal
0.05 289.45 0.16 18535473 0.00 0.00 alloc_addbyter
0.05 289.61 0.16 356074 0.00 0.00 Curl_hash_str
0.05 289.76 0.15 1797233 0.00 0.00 fd_key_compare
0.05 289.91 0.15 234966 0.00 0.00 Curl_ssl_getsessionid
0.04 290.04 0.13 585853 0.00 0.00 Curl_llist_destroy
0.04 290.16 0.12 240826 0.00 0.00 Curl_resolv
0.03 290.26 0.10 7514328 0.00 0.00 Curl_raw_nequal
0.03 290.36 0.10 2214436 0.00 0.00 Curl_read
0.03 290.46 0.10 116872 0.00 0.00
Curl_http_readwrite_headers
0.03 290.56 0.10 ssl_tls_trace
0.03 290.66 0.10 8013097 0.00 0.00 scan_response
0.03 290.75 0.09 29488555 0.00 0.00 curlx_tvdiff
0.03 290.84 0.09 2142468 0.00 0.00 Curl_setopt
0.03 290.93 0.09 476299 0.00 0.00 Curl_expire
2012/3/5 Robert Iakobashvili <cor...@gm...>
> Dear Fred Huang,
>
> Thanks for your posting.
>
> Could you please check if this is due to some badness at cares?
> Thanks.
>
> Let's exclude usage of cares based DNS resolving by applying the patch
> attached
> and use instead global DNS cache.
>
> cd ./curl-loader-0.56
> patch -p1 < path-to/disable-cares-use-global-dns.patch
> make cleanall; make
>
> See if it helps.
>
> Global cache, unfortunately, is not thread safe.
>
> If it helps, we'll try to restore the previous version of cares.
>
> Thanks,
> Robert
>
> On Mon, Mar 5, 2012 at 4:28 PM, Fred Huang <di...@gm...> wrote:
>
>> test 1:
>>
>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com
>> to one IP address
>> number of domain names in all https URLs: 780,000
>> number of client: 1000
>> CPU usage: 78%
>> cpu% irq% sirq% sys% iowt% mem_used buf&cached
>> 78.8 0.0 0.6 3.8 0.0 3640.9Mb 282.1Mb
>> SSL TPS: 300
>> SSL throughput: 200Mbps
>>
>> # gprof /usr/bin/curl-loader gmon.out -p | head -50
>> Flat profile:
>> Each sample counts as 0.01 seconds.
>> % cumulative self self total
>> time seconds seconds calls s/call s/call name
>> * 38.55 127.10 127.10 130975 0.00 0.00
>> Curl_hash_clean_with_criterium*
>> * 24.63 208.32 81.22 1247528407 0.00 0.00
>> hostcache_timestamp_remove*
>> * 13.08 251.44 43.12 2791988 0.00 0.00 Curl_hash_pick*
>> 7.30 275.51 24.07 352490 0.00 0.00 Curl_hash_add
>> 6.01 295.31 19.80 410760770 0.00 0.00
>> Curl_str_key_compare
>> 2.95 305.04 9.73 131015 0.00 0.00 create_conn
>> 0.88 307.94 2.90 4841390 0.00 0.00 dprintf_formatf
>> 0.74 310.38 2.44 136036 0.00 0.00 ConnectionStore
>> 0.48 311.96 1.58 814604911 0.00 0.00 ares__is_list_empty
>> 0.36 313.16 1.20 locking_function
>> 0.27 314.06 0.90 261608 0.00 0.00 ares_cancel
>> 0.22 314.79 0.73 127417 0.00 0.00
>> curl_multi_remove_handle
>> 0.19 315.40 0.62 8906301 0.00 0.00
>> client_tracing_function
>> 0.17 315.97 0.57 135305 0.00 0.00 Curl_num_addresses
>> 0.17 316.54 0.57 181937907 0.00 0.00 addbyter
>> 0.17 317.09 0.55 2374363 0.00 0.00 multi_runsingle
>> 0.16 317.62 0.53 541954 0.00 0.00 ares__init_list_node
>> 0.16 318.14 0.52 494412 0.00 0.00
>> curl_multi_socket_action
>> 0.14 318.60 0.46 33578897 0.00 0.00 Curl_socket_check
>> 0.13 319.04 0.44 102951052 0.00 0.00 Curl_raw_toupper
>> 0.13 319.48 0.44 787558 0.00 0.00 Curl_readwrite
>> 0.13 319.91 0.44 id_function
>> 0.12 320.29 0.38 391390 0.00 0.00 Curl_hash_str
>> 0.09 320.58 0.29 9216 0.00 0.00 curl_multi_perform
>> 0.08 320.86 0.28 2350908 0.00 0.00 Curl_raw_equal
>> 0.08 321.14 0.28 2019415 0.00 0.00 Curl_splay
>> 0.08 321.40 0.26 520625 0.00 0.00 ossl_connect_common
>> 0.08 321.65 0.25 3532992 0.00 0.00 Curl_pgrsUpdate
>> 0.07 321.87 0.22 131778537 0.00 0.00 curl_strequal
>> 0.05 322.05 0.18 8954118 0.00 0.00 scan_response
>> 0.05 322.23 0.18 533644 0.00 0.00 ares_expand_name
>> 0.05 322.40 0.17 2477345 0.00 0.00 fd_key_compare
>> 0.05 322.57 0.17 2429135 0.00 0.00 Curl_infof
>> 0.05 322.74 0.17 130966 0.00 0.00 singleipconnect
>> 0.05 322.90 0.16 9292 0.00 0.00
>> curl_multi_socket_all
>> 0.05 323.06 0.16 270137 0.00 0.00 ares__get_hostent
>> 0.05 323.21 0.15 8943657 0.00 0.00 Curl_debug
>> 0.04 323.35 0.14 262493 0.00 0.00
>> Curl_ssl_getsessionid
>> 0.04 323.48 0.13 566006 0.00 0.00 socket_callback
>> 0.04 323.61 0.13 33983864 0.00 0.00 curlx_tvdiff
>> 0.04 323.74 0.13 132853 0.00 0.00
>> Curl_http_readwrite_headers
>> 0.04 323.86 0.12 344032 0.00 0.00 epoll_del
>> 0.04 323.98 0.12 223477 0.00 0.00 Curl_poll
>> 0.04 324.10 0.12 8526280 0.00 0.00 Curl_raw_nequal
>> 0.03 324.21 0.11 348839 0.00 0.00 event_del
>>
>>
>>
>>
>> test 2:
>>
>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com
>> to one IP address
>> number of domain names in all https URLs: 1
>> number of client: 1000
>> CPU usage: 75%
>> SSL TPS: 1300
>> SSL throughput: 700Mbps
>>
>> # gprof /usr/bin/curl-loader gmon.out -p | head -50
>> Flat profile:
>> Each sample counts as 0.01 seconds.
>> % cumulative self self total
>> time seconds seconds calls s/call s/call name
>> 8.98 2.45 2.45 5127312 0.00 0.00 dprintf_formatf
>> 8.90 4.88 2.43 1292796503 0.00 0.00
>> ares__is_list_empty
>> 6.12 6.55 1.67 322461 0.00 0.00 create_conn
>> 4.58 7.80 1.25 419872 0.00 0.00 ares_cancel
>> 4.10 8.92 1.12 2100162 0.00 0.00 Curl_readwrite
>> 3.92 9.99 1.07 4527388 0.00 0.00 Curl_hash_pick
>> 3.43 10.93 0.94 locking_function
>> 3.04 11.76 0.83 909759 0.00 0.00
>> curl_multi_socket_action
>> 3.00 12.58 0.82 97252 0.00 0.00
>> Curl_hash_clean_with_criterium
>> 2.71 13.32 0.74 179704016 0.00 0.00 Curl_raw_toupper
>> 2.68 14.05 0.73 4646524 0.00 0.00 multi_runsingle
>> 2.44 14.71 0.67 13862992 0.00 0.00
>> client_tracing_function
>> 2.42 15.37 0.66 319735 0.00 0.00
>> curl_multi_remove_handle
>> 2.42 16.03 0.66 22 0.03 0.03 ares__init_list_node
>> 1.94 16.56 0.53 9930244 0.00 0.00
>> hostcache_timestamp_remove
>> 1.94 17.09 0.53 169320088 0.00 0.00 addbyter
>> 1.80 17.58 0.49 id_function
>> 1.36 17.95 0.37 7057146 0.00 0.00 Curl_pgrsUpdate
>> 1.36 18.32 0.37 16835 0.00 0.00
>> curl_multi_socket_all
>> 1.25 18.66 0.34 16719 0.00 0.00 curl_multi_perform
>> 1.21 18.99 0.33 329403 0.00 0.00
>> Curl_http_readwrite_headers
>> 1.17 19.31 0.32 4036624 0.00 0.00 Curl_splay
>> 1.14 19.62 0.31 20838902 0.00 0.00 Curl_raw_nequal
>> 0.82 19.85 0.23 2764123 0.00 0.00 Curl_infof
>> 0.81 20.07 0.22 5933857 0.00 0.00 ossl_recv
>> 0.77 20.28 0.21 323112 0.00 0.00
>> Curl_splayremovebyaddr
>> 0.75 20.48 0.21 5943991 0.00 0.00 Curl_read
>> 0.70 20.67 0.19 425174 0.00 0.00 event_del
>> 0.70 20.86 0.19 98652 0.00 0.00
>> Curl_if_is_interface_name
>> 0.70 21.05 0.19 8677733 0.00 0.00 Curl_socket_check
>> 0.59 21.21 0.16 4129554 0.00 0.00 fd_key_compare
>> 0.57 21.37 0.16 14017019 0.00 0.00 Curl_debug
>> 0.55 21.52 0.15 9376861 0.00 0.00 stat_data_in_add
>> 0.55 21.67 0.15 6840468 0.00 0.00 Curl_timeleft
>> 0.55 21.82 0.15 2054602 0.00 0.00 Curl_raw_equal
>> 0.55 21.97 0.15 1291611 0.00 0.00 Curl_expire
>> 0.51 22.11 0.14 16570063 0.00 0.00 curlx_tvnow
>> 0.51 22.25 0.14 13863284 0.00 0.00 scan_response
>> 0.51 22.39 0.14 5498864 0.00 0.00 Curl_setopt
>> 0.48 22.52 0.13 97490291 0.00 0.00 curl_strequal
>> 0.44 22.64 0.12 324253 0.00 0.00 Curl_http
>> 0.44 22.76 0.12 394386 0.00 0.00 ossl_connect_common
>> 0.40 22.87 0.11 3581335 0.00 0.00 Curl_getinfo
>> 0.38 22.97 0.11 26983092 0.00 0.00 alloc_addbyter
>> 0.37 23.07 0.10 8729044 0.00 0.00 Curl_client_write
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Try before you buy = See our experts in action!
>> The most comprehensive online learning library for Microsoft developers
>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
>> Metro Style Apps, more. Free future releases when you subscribe now!
>> http://p.sf.net/sfu/learndevnow-dev2
>> _______________________________________________
>> curl-loader-devel mailing list
>> cur...@li...
>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>>
>>
>
>
> --
> Regards,
> Robert Iakobashvili, Ph.D.
>
> Home: http://www.ghotit.com
> Blog: http://www.ghotit.com/category/ghotit-blogs/
> Twitter: http://twitter.com/ghotit
> Facebook: http://facebook.com/ghotit
> ......................................................................
> Ghotit Dyslexia
> Assistive technology that understands you
> ......................................................................
>
>
>
> ------------------------------------------------------------------------------
> Try before you buy = See our experts in action!
> The most comprehensive online learning library for Microsoft developers
> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
> Metro Style Apps, more. Free future releases when you subscribe now!
> http://p.sf.net/sfu/learndevnow-dev2
> _______________________________________________
> curl-loader-devel mailing list
> cur...@li...
> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>
>
|
|
From: Robert I. <cor...@gm...> - 2012-03-05 16:57:54
|
If so, please try to change loader.c, line 578 which looks like: curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); 1. instead of -1 try to set some value like 60 2. make clean; make Thanks, Robert On Mon, Mar 5, 2012 at 6:19 PM, Fred Huang <di...@gm...> wrote: > > Looks like the bottleneck comes from libcurl other than c-ares... > test 3 (no ares patch applied): > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com to one IP address > number of domain names in URLs: 780,000 > number of client: 1000 > CPU usage: 85% > cpu% irq% sirq% sys% iowt% mem_used buf&cached > 85.6 0.0 0.8 5.9 0.0 3334.3Mb 112.0Mb > SSL TPS: 320 > SSL throughput: 200Mbps > > # gprof /usr/bin/curl-loader gmon.out -p | head -50 > Flat profile: > Each sample counts as 0.01 seconds. > % cumulative self self total > time seconds seconds calls s/call s/call name > 40.80 120.16 120.16 116079 0.00 0.00 Curl_hash_clean_with_criterium > 24.46 192.18 72.02 983166323 0.00 0.00 hostcache_timestamp_remove > 13.30 231.35 39.17 2059341 0.00 0.00 Curl_hash_pick > 6.45 250.35 19.00 240087 0.00 0.00 Curl_hash_add > 4.81 264.51 14.16 334257102 0.00 0.00 Curl_str_key_compare > 3.40 274.52 10.01 119514 0.00 0.00 create_conn > 0.93 277.25 2.73 4478910 0.00 0.00 dprintf_formatf > 0.90 279.91 2.66 122209 0.00 0.00 ConnectionStore > 0.40 281.09 1.18 locking_function > 0.27 281.88 0.79 114759 0.00 0.00 curl_multi_remove_handle > 0.21 282.51 0.63 7993664 0.00 0.00 client_tracing_function > 0.21 283.12 0.61 122685 0.00 0.00 Curl_num_addresses > 0.19 283.68 0.56 397028 0.00 0.00 curl_multi_socket_action > 0.19 284.23 0.55 2036116 0.00 0.00 multi_runsingle > 0.17 284.72 0.49 703687 0.00 0.00 Curl_readwrite > 0.16 285.19 0.47 id_function > 0.15 285.62 0.43 29692029 0.00 0.00 Curl_socket_check > 0.14 286.02 0.40 91973856 0.00 0.00 Curl_raw_toupper > 0.13 286.40 0.38 164664144 0.00 0.00 addbyter > 0.13 286.77 0.37 2180797 0.00 0.00 Curl_infof > 0.11 287.10 0.33 122675 0.00 0.00 Curl_if_is_interface_name > 0.10 287.39 0.29 471318 0.00 0.00 ossl_connect_common > 0.10 287.67 0.28 2113528 0.00 0.00 Curl_raw_equal > 0.09 287.93 0.26 8676 0.00 0.02 curl_multi_socket_all > 0.08 288.17 0.24 1663262 0.00 0.00 Curl_splay > 0.08 288.41 0.24 8505 0.00 0.01 curl_multi_perform > 0.07 288.63 0.22 3114576 0.00 0.00 Curl_pgrsUpdate > 0.06 288.80 0.17 119637 0.00 0.00 singleipconnect > 0.06 288.97 0.17 7994818 0.00 0.00 Curl_debug > 0.05 289.13 0.16 235820 0.00 0.00 event_del > 0.05 289.29 0.16 119393071 0.00 0.00 curl_strequal > 0.05 289.45 0.16 18535473 0.00 0.00 alloc_addbyter > 0.05 289.61 0.16 356074 0.00 0.00 Curl_hash_str > 0.05 289.76 0.15 1797233 0.00 0.00 fd_key_compare > 0.05 289.91 0.15 234966 0.00 0.00 Curl_ssl_getsessionid > 0.04 290.04 0.13 585853 0.00 0.00 Curl_llist_destroy > 0.04 290.16 0.12 240826 0.00 0.00 Curl_resolv > 0.03 290.26 0.10 7514328 0.00 0.00 Curl_raw_nequal > 0.03 290.36 0.10 2214436 0.00 0.00 Curl_read > 0.03 290.46 0.10 116872 0.00 0.00 Curl_http_readwrite_headers > 0.03 290.56 0.10 ssl_tls_trace > 0.03 290.66 0.10 8013097 0.00 0.00 scan_response > 0.03 290.75 0.09 29488555 0.00 0.00 curlx_tvdiff > 0.03 290.84 0.09 2142468 0.00 0.00 Curl_setopt > 0.03 290.93 0.09 476299 0.00 0.00 Curl_expire > > > > > > > > > 2012/3/5 Robert Iakobashvili <cor...@gm...> >> >> Dear Fred Huang, >> >> Thanks for your posting. >> >> Could you please check if this is due to some badness at cares? >> Thanks. >> >> Let's exclude usage of cares based DNS resolving by applying the patch attached >> and use instead global DNS cache. >> >> cd ./curl-loader-0.56 >> patch -p1 < path-to/disable-cares-use-global-dns.patch >> make cleanall; make >> >> See if it helps. >> >> Global cache, unfortunately, is not thread safe. >> >> If it helps, we'll try to restore the previous version of cares. >> >> Thanks, >> Robert >> >> On Mon, Mar 5, 2012 at 4:28 PM, Fred Huang <di...@gm...> wrote: >>> >>> test 1: >>> >>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com to one IP address >>> number of domain names in all https URLs: 780,000 >>> number of client: 1000 >>> CPU usage: 78% >>> cpu% irq% sirq% sys% iowt% mem_used buf&cached >>> 78.8 0.0 0.6 3.8 0.0 3640.9Mb 282.1Mb >>> SSL TPS: 300 >>> SSL throughput: 200Mbps >>> >>> # gprof /usr/bin/curl-loader gmon.out -p | head -50 >>> Flat profile: >>> Each sample counts as 0.01 seconds. >>> % cumulative self self total >>> time seconds seconds calls s/call s/call name >>> 38.55 127.10 127.10 130975 0.00 0.00 Curl_hash_clean_with_criterium >>> 24.63 208.32 81.22 1247528407 0.00 0.00 hostcache_timestamp_remove >>> 13.08 251.44 43.12 2791988 0.00 0.00 Curl_hash_pick >>> 7.30 275.51 24.07 352490 0.00 0.00 Curl_hash_add >>> 6.01 295.31 19.80 410760770 0.00 0.00 Curl_str_key_compare >>> 2.95 305.04 9.73 131015 0.00 0.00 create_conn >>> 0.88 307.94 2.90 4841390 0.00 0.00 dprintf_formatf >>> 0.74 310.38 2.44 136036 0.00 0.00 ConnectionStore >>> 0.48 311.96 1.58 814604911 0.00 0.00 ares__is_list_empty >>> 0.36 313.16 1.20 locking_function >>> 0.27 314.06 0.90 261608 0.00 0.00 ares_cancel >>> 0.22 314.79 0.73 127417 0.00 0.00 curl_multi_remove_handle >>> 0.19 315.40 0.62 8906301 0.00 0.00 client_tracing_function >>> 0.17 315.97 0.57 135305 0.00 0.00 Curl_num_addresses >>> 0.17 316.54 0.57 181937907 0.00 0.00 addbyter >>> 0.17 317.09 0.55 2374363 0.00 0.00 multi_runsingle >>> 0.16 317.62 0.53 541954 0.00 0.00 ares__init_list_node >>> 0.16 318.14 0.52 494412 0.00 0.00 curl_multi_socket_action >>> 0.14 318.60 0.46 33578897 0.00 0.00 Curl_socket_check >>> 0.13 319.04 0.44 102951052 0.00 0.00 Curl_raw_toupper >>> 0.13 319.48 0.44 787558 0.00 0.00 Curl_readwrite >>> 0.13 319.91 0.44 id_function >>> 0.12 320.29 0.38 391390 0.00 0.00 Curl_hash_str >>> 0.09 320.58 0.29 9216 0.00 0.00 curl_multi_perform >>> 0.08 320.86 0.28 2350908 0.00 0.00 Curl_raw_equal >>> 0.08 321.14 0.28 2019415 0.00 0.00 Curl_splay >>> 0.08 321.40 0.26 520625 0.00 0.00 ossl_connect_common >>> 0.08 321.65 0.25 3532992 0.00 0.00 Curl_pgrsUpdate >>> 0.07 321.87 0.22 131778537 0.00 0.00 curl_strequal >>> 0.05 322.05 0.18 8954118 0.00 0.00 scan_response >>> 0.05 322.23 0.18 533644 0.00 0.00 ares_expand_name >>> 0.05 322.40 0.17 2477345 0.00 0.00 fd_key_compare >>> 0.05 322.57 0.17 2429135 0.00 0.00 Curl_infof >>> 0.05 322.74 0.17 130966 0.00 0.00 singleipconnect >>> 0.05 322.90 0.16 9292 0.00 0.00 curl_multi_socket_all >>> 0.05 323.06 0.16 270137 0.00 0.00 ares__get_hostent >>> 0.05 323.21 0.15 8943657 0.00 0.00 Curl_debug >>> 0.04 323.35 0.14 262493 0.00 0.00 Curl_ssl_getsessionid >>> 0.04 323.48 0.13 566006 0.00 0.00 socket_callback >>> 0.04 323.61 0.13 33983864 0.00 0.00 curlx_tvdiff >>> 0.04 323.74 0.13 132853 0.00 0.00 Curl_http_readwrite_headers >>> 0.04 323.86 0.12 344032 0.00 0.00 epoll_del >>> 0.04 323.98 0.12 223477 0.00 0.00 Curl_poll >>> 0.04 324.10 0.12 8526280 0.00 0.00 Curl_raw_nequal >>> 0.03 324.21 0.11 348839 0.00 0.00 event_del >>> >>> >>> >>> >>> test 2: >>> >>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com to one IP address >>> number of domain names in all https URLs: 1 >>> number of client: 1000 >>> CPU usage: 75% >>> SSL TPS: 1300 >>> SSL throughput: 700Mbps >>> >>> # gprof /usr/bin/curl-loader gmon.out -p | head -50 >>> Flat profile: >>> Each sample counts as 0.01 seconds. >>> % cumulative self self total >>> time seconds seconds calls s/call s/call name >>> 8.98 2.45 2.45 5127312 0.00 0.00 dprintf_formatf >>> 8.90 4.88 2.43 1292796503 0.00 0.00 ares__is_list_empty >>> 6.12 6.55 1.67 322461 0.00 0.00 create_conn >>> 4.58 7.80 1.25 419872 0.00 0.00 ares_cancel >>> 4.10 8.92 1.12 2100162 0.00 0.00 Curl_readwrite >>> 3.92 9.99 1.07 4527388 0.00 0.00 Curl_hash_pick >>> 3.43 10.93 0.94 locking_function >>> 3.04 11.76 0.83 909759 0.00 0.00 curl_multi_socket_action >>> 3.00 12.58 0.82 97252 0.00 0.00 Curl_hash_clean_with_criterium >>> 2.71 13.32 0.74 179704016 0.00 0.00 Curl_raw_toupper >>> 2.68 14.05 0.73 4646524 0.00 0.00 multi_runsingle >>> 2.44 14.71 0.67 13862992 0.00 0.00 client_tracing_function >>> 2.42 15.37 0.66 319735 0.00 0.00 curl_multi_remove_handle >>> 2.42 16.03 0.66 22 0.03 0.03 ares__init_list_node >>> 1.94 16.56 0.53 9930244 0.00 0.00 hostcache_timestamp_remove >>> 1.94 17.09 0.53 169320088 0.00 0.00 addbyter >>> 1.80 17.58 0.49 id_function >>> 1.36 17.95 0.37 7057146 0.00 0.00 Curl_pgrsUpdate >>> 1.36 18.32 0.37 16835 0.00 0.00 curl_multi_socket_all >>> 1.25 18.66 0.34 16719 0.00 0.00 curl_multi_perform >>> 1.21 18.99 0.33 329403 0.00 0.00 Curl_http_readwrite_headers >>> 1.17 19.31 0.32 4036624 0.00 0.00 Curl_splay >>> 1.14 19.62 0.31 20838902 0.00 0.00 Curl_raw_nequal >>> 0.82 19.85 0.23 2764123 0.00 0.00 Curl_infof >>> 0.81 20.07 0.22 5933857 0.00 0.00 ossl_recv >>> 0.77 20.28 0.21 323112 0.00 0.00 Curl_splayremovebyaddr >>> 0.75 20.48 0.21 5943991 0.00 0.00 Curl_read >>> 0.70 20.67 0.19 425174 0.00 0.00 event_del >>> 0.70 20.86 0.19 98652 0.00 0.00 Curl_if_is_interface_name >>> 0.70 21.05 0.19 8677733 0.00 0.00 Curl_socket_check >>> 0.59 21.21 0.16 4129554 0.00 0.00 fd_key_compare >>> 0.57 21.37 0.16 14017019 0.00 0.00 Curl_debug >>> 0.55 21.52 0.15 9376861 0.00 0.00 stat_data_in_add >>> 0.55 21.67 0.15 6840468 0.00 0.00 Curl_timeleft >>> 0.55 21.82 0.15 2054602 0.00 0.00 Curl_raw_equal >>> 0.55 21.97 0.15 1291611 0.00 0.00 Curl_expire >>> 0.51 22.11 0.14 16570063 0.00 0.00 curlx_tvnow >>> 0.51 22.25 0.14 13863284 0.00 0.00 scan_response >>> 0.51 22.39 0.14 5498864 0.00 0.00 Curl_setopt >>> 0.48 22.52 0.13 97490291 0.00 0.00 curl_strequal >>> 0.44 22.64 0.12 324253 0.00 0.00 Curl_http >>> 0.44 22.76 0.12 394386 0.00 0.00 ossl_connect_common >>> 0.40 22.87 0.11 3581335 0.00 0.00 Curl_getinfo >>> 0.38 22.97 0.11 26983092 0.00 0.00 alloc_addbyter >>> 0.37 23.07 0.10 8729044 0.00 0.00 Curl_client_write >>> >>> >>> ------------------------------------------------------------------------------ >>> Try before you buy = See our experts in action! >>> The most comprehensive online learning library for Microsoft developers >>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, >>> Metro Style Apps, more. Free future releases when you subscribe now! >>> http://p.sf.net/sfu/learndevnow-dev2 >>> _______________________________________________ >>> curl-loader-devel mailing list >>> cur...@li... >>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >>> >> >> >> >> -- >> Regards, >> Robert Iakobashvili, Ph.D. >> >> Home: http://www.ghotit.com >> Blog: http://www.ghotit.com/category/ghotit-blogs/ >> Twitter: http://twitter.com/ghotit >> Facebook: http://facebook.com/ghotit >> ...................................................................... >> Ghotit Dyslexia >> Assistive technology that understands you >> ...................................................................... >> >> >> ------------------------------------------------------------------------------ >> Try before you buy = See our experts in action! >> The most comprehensive online learning library for Microsoft developers >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, >> Metro Style Apps, more. Free future releases when you subscribe now! >> http://p.sf.net/sfu/learndevnow-dev2 >> _______________________________________________ >> curl-loader-devel mailing list >> cur...@li... >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >> > > > ------------------------------------------------------------------------------ > Try before you buy = See our experts in action! > The most comprehensive online learning library for Microsoft developers > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, > Metro Style Apps, more. Free future releases when you subscribe now! > http://p.sf.net/sfu/learndevnow-dev2 > _______________________________________________ > curl-loader-devel mailing list > cur...@li... > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > -- Regards, Robert Iakobashvili, Ph.D. Home: http://www.ghotit.com Blog: http://www.ghotit.com/category/ghotit-blogs/ Twitter: http://twitter.com/ghotit Facebook: http://facebook.com/ghotit ...................................................................... Ghotit Dyslexia Assistive technology that understands you ...................................................................... |
|
From: Robert I. <cor...@gm...> - 2012-03-05 17:12:12
|
If it will not be helpful this means that recent changes in libcurl library have broken DNS interface. You can try a previos version and the latest libcurl, like: http://curl.haxx.se/download/curl-7.23.0.tar.gz and http://curl.haxx.se/downloadcurl-7.24.0.tar.gz by: 1. downloading a libcurl version and placing it to curl-loader-0.56/packages ; 2. correcting the Makefile in curl-loader-0.56 CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24 or CURL_VER:=7.23 3. make cleanall; make Please, send us your observations. Thanks! Robert On Mon, Mar 5, 2012 at 6:57 PM, Robert Iakobashvili <cor...@gm...> wrote: > If so, please try to change > > loader.c, line 578 which looks like: > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); > > 1. instead of -1 try to set some value like 60 > 2. make clean; make > > Thanks, > Robert > > On Mon, Mar 5, 2012 at 6:19 PM, Fred Huang <di...@gm...> wrote: >> >> Looks like the bottleneck comes from libcurl other than c-ares... >> test 3 (no ares patch applied): >> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com to one IP address >> number of domain names in URLs: 780,000 >> number of client: 1000 >> CPU usage: 85% >> cpu% irq% sirq% sys% iowt% mem_used buf&cached >> 85.6 0.0 0.8 5.9 0.0 3334.3Mb 112.0Mb >> SSL TPS: 320 >> SSL throughput: 200Mbps >> >> # gprof /usr/bin/curl-loader gmon.out -p | head -50 >> Flat profile: >> Each sample counts as 0.01 seconds. >> % cumulative self self total >> time seconds seconds calls s/call s/call name >> 40.80 120.16 120.16 116079 0.00 0.00 Curl_hash_clean_with_criterium >> 24.46 192.18 72.02 983166323 0.00 0.00 hostcache_timestamp_remove >> 13.30 231.35 39.17 2059341 0.00 0.00 Curl_hash_pick >> 6.45 250.35 19.00 240087 0.00 0.00 Curl_hash_add >> 4.81 264.51 14.16 334257102 0.00 0.00 Curl_str_key_compare >> 3.40 274.52 10.01 119514 0.00 0.00 create_conn >> 0.93 277.25 2.73 4478910 0.00 0.00 dprintf_formatf >> 0.90 279.91 2.66 122209 0.00 0.00 ConnectionStore >> 0.40 281.09 1.18 locking_function >> 0.27 281.88 0.79 114759 0.00 0.00 curl_multi_remove_handle >> 0.21 282.51 0.63 7993664 0.00 0.00 client_tracing_function >> 0.21 283.12 0.61 122685 0.00 0.00 Curl_num_addresses >> 0.19 283.68 0.56 397028 0.00 0.00 curl_multi_socket_action >> 0.19 284.23 0.55 2036116 0.00 0.00 multi_runsingle >> 0.17 284.72 0.49 703687 0.00 0.00 Curl_readwrite >> 0.16 285.19 0.47 id_function >> 0.15 285.62 0.43 29692029 0.00 0.00 Curl_socket_check >> 0.14 286.02 0.40 91973856 0.00 0.00 Curl_raw_toupper >> 0.13 286.40 0.38 164664144 0.00 0.00 addbyter >> 0.13 286.77 0.37 2180797 0.00 0.00 Curl_infof >> 0.11 287.10 0.33 122675 0.00 0.00 Curl_if_is_interface_name >> 0.10 287.39 0.29 471318 0.00 0.00 ossl_connect_common >> 0.10 287.67 0.28 2113528 0.00 0.00 Curl_raw_equal >> 0.09 287.93 0.26 8676 0.00 0.02 curl_multi_socket_all >> 0.08 288.17 0.24 1663262 0.00 0.00 Curl_splay >> 0.08 288.41 0.24 8505 0.00 0.01 curl_multi_perform >> 0.07 288.63 0.22 3114576 0.00 0.00 Curl_pgrsUpdate >> 0.06 288.80 0.17 119637 0.00 0.00 singleipconnect >> 0.06 288.97 0.17 7994818 0.00 0.00 Curl_debug >> 0.05 289.13 0.16 235820 0.00 0.00 event_del >> 0.05 289.29 0.16 119393071 0.00 0.00 curl_strequal >> 0.05 289.45 0.16 18535473 0.00 0.00 alloc_addbyter >> 0.05 289.61 0.16 356074 0.00 0.00 Curl_hash_str >> 0.05 289.76 0.15 1797233 0.00 0.00 fd_key_compare >> 0.05 289.91 0.15 234966 0.00 0.00 Curl_ssl_getsessionid >> 0.04 290.04 0.13 585853 0.00 0.00 Curl_llist_destroy >> 0.04 290.16 0.12 240826 0.00 0.00 Curl_resolv >> 0.03 290.26 0.10 7514328 0.00 0.00 Curl_raw_nequal >> 0.03 290.36 0.10 2214436 0.00 0.00 Curl_read >> 0.03 290.46 0.10 116872 0.00 0.00 Curl_http_readwrite_headers >> 0.03 290.56 0.10 ssl_tls_trace >> 0.03 290.66 0.10 8013097 0.00 0.00 scan_response >> 0.03 290.75 0.09 29488555 0.00 0.00 curlx_tvdiff >> 0.03 290.84 0.09 2142468 0.00 0.00 Curl_setopt >> 0.03 290.93 0.09 476299 0.00 0.00 Curl_expire >> >> >> >> >> >> >> >> >> 2012/3/5 Robert Iakobashvili <cor...@gm...> >>> >>> Dear Fred Huang, >>> >>> Thanks for your posting. >>> >>> Could you please check if this is due to some badness at cares? >>> Thanks. >>> >>> Let's exclude usage of cares based DNS resolving by applying the patch attached >>> and use instead global DNS cache. >>> >>> cd ./curl-loader-0.56 >>> patch -p1 < path-to/disable-cares-use-global-dns.patch >>> make cleanall; make >>> >>> See if it helps. >>> >>> Global cache, unfortunately, is not thread safe. >>> >>> If it helps, we'll try to restore the previous version of cares. >>> >>> Thanks, >>> Robert >>> >>> On Mon, Mar 5, 2012 at 4:28 PM, Fred Huang <di...@gm...> wrote: >>>> >>>> test 1: >>>> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com to one IP address >>>> number of domain names in all https URLs: 780,000 >>>> number of client: 1000 >>>> CPU usage: 78% >>>> cpu% irq% sirq% sys% iowt% mem_used buf&cached >>>> 78.8 0.0 0.6 3.8 0.0 3640.9Mb 282.1Mb >>>> SSL TPS: 300 >>>> SSL throughput: 200Mbps >>>> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50 >>>> Flat profile: >>>> Each sample counts as 0.01 seconds. >>>> % cumulative self self total >>>> time seconds seconds calls s/call s/call name >>>> 38.55 127.10 127.10 130975 0.00 0.00 Curl_hash_clean_with_criterium >>>> 24.63 208.32 81.22 1247528407 0.00 0.00 hostcache_timestamp_remove >>>> 13.08 251.44 43.12 2791988 0.00 0.00 Curl_hash_pick >>>> 7.30 275.51 24.07 352490 0.00 0.00 Curl_hash_add >>>> 6.01 295.31 19.80 410760770 0.00 0.00 Curl_str_key_compare >>>> 2.95 305.04 9.73 131015 0.00 0.00 create_conn >>>> 0.88 307.94 2.90 4841390 0.00 0.00 dprintf_formatf >>>> 0.74 310.38 2.44 136036 0.00 0.00 ConnectionStore >>>> 0.48 311.96 1.58 814604911 0.00 0.00 ares__is_list_empty >>>> 0.36 313.16 1.20 locking_function >>>> 0.27 314.06 0.90 261608 0.00 0.00 ares_cancel >>>> 0.22 314.79 0.73 127417 0.00 0.00 curl_multi_remove_handle >>>> 0.19 315.40 0.62 8906301 0.00 0.00 client_tracing_function >>>> 0.17 315.97 0.57 135305 0.00 0.00 Curl_num_addresses >>>> 0.17 316.54 0.57 181937907 0.00 0.00 addbyter >>>> 0.17 317.09 0.55 2374363 0.00 0.00 multi_runsingle >>>> 0.16 317.62 0.53 541954 0.00 0.00 ares__init_list_node >>>> 0.16 318.14 0.52 494412 0.00 0.00 curl_multi_socket_action >>>> 0.14 318.60 0.46 33578897 0.00 0.00 Curl_socket_check >>>> 0.13 319.04 0.44 102951052 0.00 0.00 Curl_raw_toupper >>>> 0.13 319.48 0.44 787558 0.00 0.00 Curl_readwrite >>>> 0.13 319.91 0.44 id_function >>>> 0.12 320.29 0.38 391390 0.00 0.00 Curl_hash_str >>>> 0.09 320.58 0.29 9216 0.00 0.00 curl_multi_perform >>>> 0.08 320.86 0.28 2350908 0.00 0.00 Curl_raw_equal >>>> 0.08 321.14 0.28 2019415 0.00 0.00 Curl_splay >>>> 0.08 321.40 0.26 520625 0.00 0.00 ossl_connect_common >>>> 0.08 321.65 0.25 3532992 0.00 0.00 Curl_pgrsUpdate >>>> 0.07 321.87 0.22 131778537 0.00 0.00 curl_strequal >>>> 0.05 322.05 0.18 8954118 0.00 0.00 scan_response >>>> 0.05 322.23 0.18 533644 0.00 0.00 ares_expand_name >>>> 0.05 322.40 0.17 2477345 0.00 0.00 fd_key_compare >>>> 0.05 322.57 0.17 2429135 0.00 0.00 Curl_infof >>>> 0.05 322.74 0.17 130966 0.00 0.00 singleipconnect >>>> 0.05 322.90 0.16 9292 0.00 0.00 curl_multi_socket_all >>>> 0.05 323.06 0.16 270137 0.00 0.00 ares__get_hostent >>>> 0.05 323.21 0.15 8943657 0.00 0.00 Curl_debug >>>> 0.04 323.35 0.14 262493 0.00 0.00 Curl_ssl_getsessionid >>>> 0.04 323.48 0.13 566006 0.00 0.00 socket_callback >>>> 0.04 323.61 0.13 33983864 0.00 0.00 curlx_tvdiff >>>> 0.04 323.74 0.13 132853 0.00 0.00 Curl_http_readwrite_headers >>>> 0.04 323.86 0.12 344032 0.00 0.00 epoll_del >>>> 0.04 323.98 0.12 223477 0.00 0.00 Curl_poll >>>> 0.04 324.10 0.12 8526280 0.00 0.00 Curl_raw_nequal >>>> 0.03 324.21 0.11 348839 0.00 0.00 event_del >>>> >>>> >>>> >>>> >>>> test 2: >>>> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com to one IP address >>>> number of domain names in all https URLs: 1 >>>> number of client: 1000 >>>> CPU usage: 75% >>>> SSL TPS: 1300 >>>> SSL throughput: 700Mbps >>>> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50 >>>> Flat profile: >>>> Each sample counts as 0.01 seconds. >>>> % cumulative self self total >>>> time seconds seconds calls s/call s/call name >>>> 8.98 2.45 2.45 5127312 0.00 0.00 dprintf_formatf >>>> 8.90 4.88 2.43 1292796503 0.00 0.00 ares__is_list_empty >>>> 6.12 6.55 1.67 322461 0.00 0.00 create_conn >>>> 4.58 7.80 1.25 419872 0.00 0.00 ares_cancel >>>> 4.10 8.92 1.12 2100162 0.00 0.00 Curl_readwrite >>>> 3.92 9.99 1.07 4527388 0.00 0.00 Curl_hash_pick >>>> 3.43 10.93 0.94 locking_function >>>> 3.04 11.76 0.83 909759 0.00 0.00 curl_multi_socket_action >>>> 3.00 12.58 0.82 97252 0.00 0.00 Curl_hash_clean_with_criterium >>>> 2.71 13.32 0.74 179704016 0.00 0.00 Curl_raw_toupper >>>> 2.68 14.05 0.73 4646524 0.00 0.00 multi_runsingle >>>> 2.44 14.71 0.67 13862992 0.00 0.00 client_tracing_function >>>> 2.42 15.37 0.66 319735 0.00 0.00 curl_multi_remove_handle >>>> 2.42 16.03 0.66 22 0.03 0.03 ares__init_list_node >>>> 1.94 16.56 0.53 9930244 0.00 0.00 hostcache_timestamp_remove >>>> 1.94 17.09 0.53 169320088 0.00 0.00 addbyter >>>> 1.80 17.58 0.49 id_function >>>> 1.36 17.95 0.37 7057146 0.00 0.00 Curl_pgrsUpdate >>>> 1.36 18.32 0.37 16835 0.00 0.00 curl_multi_socket_all >>>> 1.25 18.66 0.34 16719 0.00 0.00 curl_multi_perform >>>> 1.21 18.99 0.33 329403 0.00 0.00 Curl_http_readwrite_headers >>>> 1.17 19.31 0.32 4036624 0.00 0.00 Curl_splay >>>> 1.14 19.62 0.31 20838902 0.00 0.00 Curl_raw_nequal >>>> 0.82 19.85 0.23 2764123 0.00 0.00 Curl_infof >>>> 0.81 20.07 0.22 5933857 0.00 0.00 ossl_recv >>>> 0.77 20.28 0.21 323112 0.00 0.00 Curl_splayremovebyaddr >>>> 0.75 20.48 0.21 5943991 0.00 0.00 Curl_read >>>> 0.70 20.67 0.19 425174 0.00 0.00 event_del >>>> 0.70 20.86 0.19 98652 0.00 0.00 Curl_if_is_interface_name >>>> 0.70 21.05 0.19 8677733 0.00 0.00 Curl_socket_check >>>> 0.59 21.21 0.16 4129554 0.00 0.00 fd_key_compare >>>> 0.57 21.37 0.16 14017019 0.00 0.00 Curl_debug >>>> 0.55 21.52 0.15 9376861 0.00 0.00 stat_data_in_add >>>> 0.55 21.67 0.15 6840468 0.00 0.00 Curl_timeleft >>>> 0.55 21.82 0.15 2054602 0.00 0.00 Curl_raw_equal >>>> 0.55 21.97 0.15 1291611 0.00 0.00 Curl_expire >>>> 0.51 22.11 0.14 16570063 0.00 0.00 curlx_tvnow >>>> 0.51 22.25 0.14 13863284 0.00 0.00 scan_response >>>> 0.51 22.39 0.14 5498864 0.00 0.00 Curl_setopt >>>> 0.48 22.52 0.13 97490291 0.00 0.00 curl_strequal >>>> 0.44 22.64 0.12 324253 0.00 0.00 Curl_http >>>> 0.44 22.76 0.12 394386 0.00 0.00 ossl_connect_common >>>> 0.40 22.87 0.11 3581335 0.00 0.00 Curl_getinfo >>>> 0.38 22.97 0.11 26983092 0.00 0.00 alloc_addbyter >>>> 0.37 23.07 0.10 8729044 0.00 0.00 Curl_client_write >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> Try before you buy = See our experts in action! >>>> The most comprehensive online learning library for Microsoft developers >>>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, >>>> Metro Style Apps, more. Free future releases when you subscribe now! >>>> http://p.sf.net/sfu/learndevnow-dev2 >>>> _______________________________________________ >>>> curl-loader-devel mailing list >>>> cur...@li... >>>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >>>> >>> >>> >>> >>> -- >>> Regards, >>> Robert Iakobashvili, Ph.D. >>> >>> Home: http://www.ghotit.com >>> Blog: http://www.ghotit.com/category/ghotit-blogs/ >>> Twitter: http://twitter.com/ghotit >>> Facebook: http://facebook.com/ghotit >>> ...................................................................... >>> Ghotit Dyslexia >>> Assistive technology that understands you >>> ...................................................................... >>> >>> >>> ------------------------------------------------------------------------------ >>> Try before you buy = See our experts in action! >>> The most comprehensive online learning library for Microsoft developers >>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, >>> Metro Style Apps, more. Free future releases when you subscribe now! >>> http://p.sf.net/sfu/learndevnow-dev2 >>> _______________________________________________ >>> curl-loader-devel mailing list >>> cur...@li... >>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >>> >> >> >> ------------------------------------------------------------------------------ >> Try before you buy = See our experts in action! >> The most comprehensive online learning library for Microsoft developers >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, >> Metro Style Apps, more. Free future releases when you subscribe now! >> http://p.sf.net/sfu/learndevnow-dev2 >> _______________________________________________ >> curl-loader-devel mailing list >> cur...@li... >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >> > > > > -- > Regards, > Robert Iakobashvili, Ph.D. > > Home: http://www.ghotit.com > Blog: http://www.ghotit.com/category/ghotit-blogs/ > Twitter: http://twitter.com/ghotit > Facebook: http://facebook.com/ghotit > ...................................................................... > Ghotit Dyslexia > Assistive technology that understands you > ...................................................................... -- Regards, Robert Iakobashvili, Ph.D. Home: http://www.ghotit.com Blog: http://www.ghotit.com/category/ghotit-blogs/ Twitter: http://twitter.com/ghotit Facebook: http://facebook.com/ghotit ...................................................................... Ghotit Dyslexia Assistive technology that understands you ...................................................................... |
|
From: Fred H. <di...@gm...> - 2012-03-06 01:52:45
|
TIMEOUT -1 -> 60 works a little better but no performance improvement on
CAPS or throughput.
*test 5:*
enable c-ares;
curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60);
dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com to
one IP address
number of domain names in URLs: 780,000
number of client: 1000
CPU usage: 74%
cpu% irq% sirq% sys% iowt% mem_used buf&cached
74.0 0.0 0.7 4.2 0.0 3530.6Mb 250.8Mb
SSL TPS: 360
SSL throughput: 220Mbps
# gprof /usr/bin/curl-loader gmon.out -p | head -50
Flat profile:
Each sample counts as 0.01 seconds.
% cumulative self self total
time seconds seconds calls s/call s/call name
26.63 23.13 23.13 119226 0.00 0.00
Curl_hash_clean_with_criterium
20.34 40.80 17.67 317792146 0.00 0.00
hostcache_timestamp_remove
11.27 50.59 9.79 2579903 0.00 0.00 Curl_hash_pick
7.84 57.40 6.81 120606 0.00 0.00 create_conn
5.65 62.31 4.91 119117401 0.00 0.00 Curl_str_key_compare
4.98 66.64 4.33 329829 0.00 0.00 Curl_hash_add
2.80 69.08 2.44 4476606 0.00 0.00 dprintf_formatf
2.57 71.31 2.23 124019 0.00 0.00 ConnectionStore
1.50 72.61 1.30 740997609 0.00 0.00 ares__is_list_empty
1.16 73.61 1.01 locking_function
0.76 74.27 0.66 238744 0.00 0.00 ares_cancel
0.70 74.88 0.61 457564 0.00 0.00
curl_multi_socket_action
0.67 75.46 0.58 118616 0.00 0.00
curl_multi_remove_handle
0.58 75.97 0.51 8211502 0.00 0.00
client_tracing_function
0.53 76.43 0.46 733561 0.00 0.00 Curl_readwrite
0.51 76.87 0.44 2183179 0.00 0.00 multi_runsingle
0.51 77.31 0.44 166585567 0.00 0.00 addbyter
0.44 77.69 0.38 2175805 0.00 0.00 Curl_raw_equal
0.43 78.06 0.38 id_function
0.40 78.41 0.35 29789346 0.00 0.00 Curl_socket_check
0.40 78.76 0.35 479671 0.00 0.00 ossl_connect_common
0.40 79.11 0.35 490639 0.00 0.00 ares__init_list_node
0.33 79.40 0.29 94197670 0.00 0.00 Curl_raw_toupper
0.33 79.69 0.29 3268031 0.00 0.00 Curl_pgrsUpdate
0.28 79.93 0.24 8797 0.00 0.00 curl_multi_perform
0.27 80.17 0.24 2233641 0.00 0.00 Curl_infof
0.26 80.40 0.23 1839837 0.00 0.00 Curl_splay
0.24 80.61 0.21 121713028 0.00 0.00 curl_strequal
0.21 80.79 0.18 243333 0.00 0.00 Curl_ssl_getsessionid
0.18 80.95 0.16 122180 0.00 0.00 Curl_num_addresses
0.17 81.09 0.15 8271666 0.00 0.00 Curl_debug
0.16 81.23 0.14 18902452 0.00 0.00 alloc_addbyter
0.16 81.37 0.14 8851 0.00 0.00 curl_multi_socket_all
0.14 81.50 0.13 2236769 0.00 0.00 Curl_read
0.14 81.62 0.12 2271700 0.00 0.00 fd_key_compare
0.14 81.74 0.12 323167 0.00 0.00 event_del
0.14 81.86 0.12 205436 0.00 0.00 read_udp_packets
0.14 81.98 0.12 200979 0.00 0.00 processfds
0.13 82.09 0.11 123886 0.00 0.00
Curl_if_is_interface_name
0.13 82.20 0.11 120350 0.00 0.00 Curl_http
0.13 82.31 0.11 ssl_tls_trace
0.12 82.41 0.10 119229 0.00 0.00 Curl_splayremovebyaddr
0.12 82.51 0.10 7798717 0.00 0.00 Curl_raw_nequal
0.12 82.61 0.10 784485 0.00 0.00 event_add
0.10 82.70 0.09 643482 0.00 0.00 Curl_expire
*test 6:*
enable c-ares;
curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60);
CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0
dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com to
one IP address
number of domain names in URLs: 780,000
number of client: 1000
CPU usage: 86%
cpu% irq% sirq% sys% iowt% mem_used buf&cached
85.9 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb
SSL TPS: 360
SSL throughput: 200Mbps
# gprof /usr/bin/curl-loader gmon.out -p | head -20
Flat profile:
Each sample counts as 0.01 seconds.
% cumulative self self total
time seconds seconds calls s/call s/call name
27.19 19.96 19.96 103472 0.00 0.00
Curl_hash_clean_with_criterium
21.73 35.91 15.95 274168981 0.00 0.00
hostcache_timestamp_remove
10.33 43.49 7.58 2256441 0.00 0.00 Curl_hash_pick
7.21 48.78 5.29 104540 0.00 0.00 create_conn
6.20 53.33 4.55 103413405 0.00 0.00 Curl_str_key_compare
4.88 56.91 3.58 287109 0.00 0.00 Curl_hash_add
3.02 59.13 2.22 3797855 0.00 0.00 dprintf_formatf
2.10 60.67 1.54 107711 0.00 0.00 ConnectionStore
1.65 61.88 1.21 642725327 0.00 0.00 ares__is_list_empty
1.19 62.76 0.88 locking_function
0.87 63.40 0.64 207817 0.00 0.00 ares_cancel
0.56 63.81 0.41 404108 0.00 0.00
curl_multi_socket_action
0.54 64.21 0.40 425603 0.00 0.00 ares__init_list_node
0.53 64.60 0.39 7068112 0.00 0.00
client_tracing_function
0.49 64.96 0.36 1908587 0.00 0.00 multi_runsingle
*test 7:*
enable c-ares;
curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1);
CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0
dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com to
one IP address
number of domain names in URLs: 780,000
number of client: 1000
CPU usage: 86%
cpu% irq% sirq% sys% iowt% mem_used buf&cached
86.6 0.0 0.6 3.3 0.0 3624.8Mb 263.0Mb
SSL TPS: 300
SSL throughput: 200Mbps
# gprof /usr/bin/curl-loader gmon.out -p | head -20
Flat profile:
Each sample counts as 0.01 seconds.
% cumulative self self total
time seconds seconds calls s/call s/call name
39.83 369.71 369.71 207173 0.00 0.00
Curl_hash_clean_with_criterium
24.27 594.98 225.27 3198337733 0.00 0.00
hostcache_timestamp_remove
13.75 722.59 127.61 4312864 0.00 0.00 Curl_hash_pick
9.19 807.87 85.28 546507 0.00 0.00 Curl_hash_add
6.14 864.88 57.01 1022419684 0.00 0.00 Curl_str_key_compare
2.06 884.00 19.12 203749 0.00 0.00 create_conn
0.54 888.99 4.99 213306 0.00 0.00 ConnectionStore
0.49 893.50 4.51 7331251 0.00 0.00 dprintf_formatf
0.29 896.16 2.66 1287182329 0.00 0.00 ares__is_list_empty
0.23 898.28 2.12 locking_function
0.20 900.10 1.82 212210 0.00 0.00 Curl_num_addresses
0.17 901.66 1.56 198385 0.00 0.00
curl_multi_remove_handle
0.12 902.81 1.15 411114 0.00 0.00 ares_cancel
0.12 903.95 1.14 3683448 0.00 0.00 multi_runsingle
0.11 904.97 1.03 13664120 0.00 0.00
client_tracing_function
*test 8:*
enable c-ares;
curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60);
CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0
dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com to
one IP address
number of domain names in URLs: 780,000
number of client: 1000
CPU usage: 75%
cpu% irq% sirq% sys% iowt% mem_used buf&cached
75.2 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb
SSL TPS: 380
SSL throughput: 220Mbps
# gprof /usr/bin/curl-loader gmon.out -p | head -20
Flat profile:
Each sample counts as 0.01 seconds.
% cumulative self self total
time seconds seconds calls s/call s/call name
27.58 12.31 12.31 71140 0.00 0.00
Curl_hash_clean_with_criterium
19.09 20.83 8.52 176729692 0.00 0.00
hostcache_timestamp_remove
10.44 25.49 4.66 1589309 0.00 0.00 Curl_hash_pick
6.85 28.55 3.06 73204 0.00 0.00 create_conn
5.56 31.03 2.48 67620530 0.00 0.00 Curl_str_key_compare
4.41 33.00 1.97 200684 0.00 0.00 Curl_hash_add
3.56 34.59 1.59 2717956 0.00 0.00 dprintf_formatf
1.72 35.36 0.77 75247 0.00 0.00 ConnectionStore
1.70 36.12 0.76 444342256 0.00 0.00 ares__is_list_empty
1.23 36.67 0.55 locking_function
1.16 37.19 0.52 143342 0.00 0.00 ares_cancel
0.99 37.63 0.44 72211 0.00 0.00
curl_multi_remove_handle
0.74 37.96 0.33 1333371 0.00 0.00 multi_runsingle
0.68 38.27 0.31 101147798 0.00 0.00 addbyter
0.58 38.53 0.26 288737 0.00 0.00
curl_multi_socket_action
*test 9:*
enable c-ares;
curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1);
CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0
dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com to
one IP address
number of domain names in URLs: 780,000
number of client: 1000
CPU usage: 75%
cpu% irq% sirq% sys% iowt% mem_used buf&cached
75.1 0.0 0.6 4.1 0.0 3545.2Mb 260.1Mb
SSL TPS: 380
SSL throughput: 200Mbps
# gprof /usr/bin/curl-loader gmon.out -p | head -20
Flat profile:
Each sample counts as 0.01 seconds.
% cumulative self self total
time seconds seconds calls s/call s/call name
34.00 31.13 31.13 72495 0.00 0.00
Curl_hash_clean_with_criterium
23.64 52.78 21.65 386070412 0.00 0.00
hostcache_timestamp_remove
13.28 64.94 12.16 1619440 0.00 0.00 Curl_hash_pick
5.92 70.36 5.42 137653129 0.00 0.00 Curl_str_key_compare
5.50 75.40 5.04 203018 0.00 0.00 Curl_hash_add
4.55 79.57 4.17 74618 0.00 0.00 create_conn
1.70 81.13 1.56 2757148 0.00 0.00 dprintf_formatf
0.98 82.03 0.90 76945 0.00 0.00 ConnectionStore
0.95 82.90 0.87 455775100 0.00 0.00 ares__is_list_empty
0.57 83.42 0.53 locking_function
0.39 83.78 0.36 146742 0.00 0.00 ares_cancel
0.35 84.10 0.32 102968806 0.00 0.00 addbyter
0.35 84.42 0.32 456587 0.00 0.00 Curl_readwrite
0.35 84.74 0.32 73133 0.00 0.00
curl_multi_remove_handle
0.34 85.05 0.31 1360545 0.00 0.00 multi_runsingle
2012/3/6 Robert Iakobashvili <cor...@gm...>
> If it will not be helpful this means that recent
> changes in libcurl library have broken DNS interface.
>
> You can try a previos version and the latest libcurl, like:
>
> http://curl.haxx.se/download/curl-7.23.0.tar.gz
> and
> http://curl.haxx.se/downloadcurl-7.24.0.tar.gz
>
> by:
>
> 1. downloading a libcurl version and placing it to
> curl-loader-0.56/packages ;
> 2. correcting the Makefile in curl-loader-0.56
> CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24 or CURL_VER:=7.23
> 3. make cleanall; make
>
> Please, send us your observations.
> Thanks!
> Robert
>
> On Mon, Mar 5, 2012 at 6:57 PM, Robert Iakobashvili <cor...@gm...>
> wrote:
> > If so, please try to change
> >
> > loader.c, line 578 which looks like:
> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1);
> >
> > 1. instead of -1 try to set some value like 60
> > 2. make clean; make
> >
> > Thanks,
> > Robert
> >
> > On Mon, Mar 5, 2012 at 6:19 PM, Fred Huang <di...@gm...> wrote:
> >>
> >> Looks like the bottleneck comes from libcurl other than c-ares...
> >> test 3 (no ares patch applied):
> >> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve
> *.com to one IP address
> >> number of domain names in URLs: 780,000
> >> number of client: 1000
> >> CPU usage: 85%
> >> cpu% irq% sirq% sys% iowt% mem_used buf&cached
> >> 85.6 0.0 0.8 5.9 0.0 3334.3Mb 112.0Mb
> >> SSL TPS: 320
> >> SSL throughput: 200Mbps
> >>
> >> # gprof /usr/bin/curl-loader gmon.out -p | head -50
> >> Flat profile:
> >> Each sample counts as 0.01 seconds.
> >> % cumulative self self total
> >> time seconds seconds calls s/call s/call name
> >> 40.80 120.16 120.16 116079 0.00 0.00
> Curl_hash_clean_with_criterium
> >> 24.46 192.18 72.02 983166323 0.00 0.00
> hostcache_timestamp_remove
> >> 13.30 231.35 39.17 2059341 0.00 0.00 Curl_hash_pick
> >> 6.45 250.35 19.00 240087 0.00 0.00 Curl_hash_add
> >> 4.81 264.51 14.16 334257102 0.00 0.00
> Curl_str_key_compare
> >> 3.40 274.52 10.01 119514 0.00 0.00 create_conn
> >> 0.93 277.25 2.73 4478910 0.00 0.00 dprintf_formatf
> >> 0.90 279.91 2.66 122209 0.00 0.00 ConnectionStore
> >> 0.40 281.09 1.18 locking_function
> >> 0.27 281.88 0.79 114759 0.00 0.00
> curl_multi_remove_handle
> >> 0.21 282.51 0.63 7993664 0.00 0.00
> client_tracing_function
> >> 0.21 283.12 0.61 122685 0.00 0.00 Curl_num_addresses
> >> 0.19 283.68 0.56 397028 0.00 0.00
> curl_multi_socket_action
> >> 0.19 284.23 0.55 2036116 0.00 0.00 multi_runsingle
> >> 0.17 284.72 0.49 703687 0.00 0.00 Curl_readwrite
> >> 0.16 285.19 0.47 id_function
> >> 0.15 285.62 0.43 29692029 0.00 0.00 Curl_socket_check
> >> 0.14 286.02 0.40 91973856 0.00 0.00 Curl_raw_toupper
> >> 0.13 286.40 0.38 164664144 0.00 0.00 addbyter
> >> 0.13 286.77 0.37 2180797 0.00 0.00 Curl_infof
> >> 0.11 287.10 0.33 122675 0.00 0.00
> Curl_if_is_interface_name
> >> 0.10 287.39 0.29 471318 0.00 0.00
> ossl_connect_common
> >> 0.10 287.67 0.28 2113528 0.00 0.00 Curl_raw_equal
> >> 0.09 287.93 0.26 8676 0.00 0.02
> curl_multi_socket_all
> >> 0.08 288.17 0.24 1663262 0.00 0.00 Curl_splay
> >> 0.08 288.41 0.24 8505 0.00 0.01 curl_multi_perform
> >> 0.07 288.63 0.22 3114576 0.00 0.00 Curl_pgrsUpdate
> >> 0.06 288.80 0.17 119637 0.00 0.00 singleipconnect
> >> 0.06 288.97 0.17 7994818 0.00 0.00 Curl_debug
> >> 0.05 289.13 0.16 235820 0.00 0.00 event_del
> >> 0.05 289.29 0.16 119393071 0.00 0.00 curl_strequal
> >> 0.05 289.45 0.16 18535473 0.00 0.00 alloc_addbyter
> >> 0.05 289.61 0.16 356074 0.00 0.00 Curl_hash_str
> >> 0.05 289.76 0.15 1797233 0.00 0.00 fd_key_compare
> >> 0.05 289.91 0.15 234966 0.00 0.00
> Curl_ssl_getsessionid
> >> 0.04 290.04 0.13 585853 0.00 0.00 Curl_llist_destroy
> >> 0.04 290.16 0.12 240826 0.00 0.00 Curl_resolv
> >> 0.03 290.26 0.10 7514328 0.00 0.00 Curl_raw_nequal
> >> 0.03 290.36 0.10 2214436 0.00 0.00 Curl_read
> >> 0.03 290.46 0.10 116872 0.00 0.00
> Curl_http_readwrite_headers
> >> 0.03 290.56 0.10 ssl_tls_trace
> >> 0.03 290.66 0.10 8013097 0.00 0.00 scan_response
> >> 0.03 290.75 0.09 29488555 0.00 0.00 curlx_tvdiff
> >> 0.03 290.84 0.09 2142468 0.00 0.00 Curl_setopt
> >> 0.03 290.93 0.09 476299 0.00 0.00 Curl_expire
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> 2012/3/5 Robert Iakobashvili <cor...@gm...>
> >>>
> >>> Dear Fred Huang,
> >>>
> >>> Thanks for your posting.
> >>>
> >>> Could you please check if this is due to some badness at cares?
> >>> Thanks.
> >>>
> >>> Let's exclude usage of cares based DNS resolving by applying the patch
> attached
> >>> and use instead global DNS cache.
> >>>
> >>> cd ./curl-loader-0.56
> >>> patch -p1 < path-to/disable-cares-use-global-dns.patch
> >>> make cleanall; make
> >>>
> >>> See if it helps.
> >>>
> >>> Global cache, unfortunately, is not thread safe.
> >>>
> >>> If it helps, we'll try to restore the previous version of cares.
> >>>
> >>> Thanks,
> >>> Robert
> >>>
> >>> On Mon, Mar 5, 2012 at 4:28 PM, Fred Huang <di...@gm...> wrote:
> >>>>
> >>>> test 1:
> >>>>
> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve
> *.com to one IP address
> >>>> number of domain names in all https URLs: 780,000
> >>>> number of client: 1000
> >>>> CPU usage: 78%
> >>>> cpu% irq% sirq% sys% iowt% mem_used buf&cached
> >>>> 78.8 0.0 0.6 3.8 0.0 3640.9Mb 282.1Mb
> >>>> SSL TPS: 300
> >>>> SSL throughput: 200Mbps
> >>>>
> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50
> >>>> Flat profile:
> >>>> Each sample counts as 0.01 seconds.
> >>>> % cumulative self self total
> >>>> time seconds seconds calls s/call s/call name
> >>>> 38.55 127.10 127.10 130975 0.00 0.00
> Curl_hash_clean_with_criterium
> >>>> 24.63 208.32 81.22 1247528407 0.00 0.00
> hostcache_timestamp_remove
> >>>> 13.08 251.44 43.12 2791988 0.00 0.00 Curl_hash_pick
> >>>> 7.30 275.51 24.07 352490 0.00 0.00 Curl_hash_add
> >>>> 6.01 295.31 19.80 410760770 0.00 0.00
> Curl_str_key_compare
> >>>> 2.95 305.04 9.73 131015 0.00 0.00 create_conn
> >>>> 0.88 307.94 2.90 4841390 0.00 0.00 dprintf_formatf
> >>>> 0.74 310.38 2.44 136036 0.00 0.00 ConnectionStore
> >>>> 0.48 311.96 1.58 814604911 0.00 0.00
> ares__is_list_empty
> >>>> 0.36 313.16 1.20 locking_function
> >>>> 0.27 314.06 0.90 261608 0.00 0.00 ares_cancel
> >>>> 0.22 314.79 0.73 127417 0.00 0.00
> curl_multi_remove_handle
> >>>> 0.19 315.40 0.62 8906301 0.00 0.00
> client_tracing_function
> >>>> 0.17 315.97 0.57 135305 0.00 0.00
> Curl_num_addresses
> >>>> 0.17 316.54 0.57 181937907 0.00 0.00 addbyter
> >>>> 0.17 317.09 0.55 2374363 0.00 0.00 multi_runsingle
> >>>> 0.16 317.62 0.53 541954 0.00 0.00
> ares__init_list_node
> >>>> 0.16 318.14 0.52 494412 0.00 0.00
> curl_multi_socket_action
> >>>> 0.14 318.60 0.46 33578897 0.00 0.00
> Curl_socket_check
> >>>> 0.13 319.04 0.44 102951052 0.00 0.00
> Curl_raw_toupper
> >>>> 0.13 319.48 0.44 787558 0.00 0.00 Curl_readwrite
> >>>> 0.13 319.91 0.44 id_function
> >>>> 0.12 320.29 0.38 391390 0.00 0.00 Curl_hash_str
> >>>> 0.09 320.58 0.29 9216 0.00 0.00
> curl_multi_perform
> >>>> 0.08 320.86 0.28 2350908 0.00 0.00 Curl_raw_equal
> >>>> 0.08 321.14 0.28 2019415 0.00 0.00 Curl_splay
> >>>> 0.08 321.40 0.26 520625 0.00 0.00
> ossl_connect_common
> >>>> 0.08 321.65 0.25 3532992 0.00 0.00 Curl_pgrsUpdate
> >>>> 0.07 321.87 0.22 131778537 0.00 0.00 curl_strequal
> >>>> 0.05 322.05 0.18 8954118 0.00 0.00 scan_response
> >>>> 0.05 322.23 0.18 533644 0.00 0.00 ares_expand_name
> >>>> 0.05 322.40 0.17 2477345 0.00 0.00 fd_key_compare
> >>>> 0.05 322.57 0.17 2429135 0.00 0.00 Curl_infof
> >>>> 0.05 322.74 0.17 130966 0.00 0.00 singleipconnect
> >>>> 0.05 322.90 0.16 9292 0.00 0.00
> curl_multi_socket_all
> >>>> 0.05 323.06 0.16 270137 0.00 0.00
> ares__get_hostent
> >>>> 0.05 323.21 0.15 8943657 0.00 0.00 Curl_debug
> >>>> 0.04 323.35 0.14 262493 0.00 0.00
> Curl_ssl_getsessionid
> >>>> 0.04 323.48 0.13 566006 0.00 0.00 socket_callback
> >>>> 0.04 323.61 0.13 33983864 0.00 0.00 curlx_tvdiff
> >>>> 0.04 323.74 0.13 132853 0.00 0.00
> Curl_http_readwrite_headers
> >>>> 0.04 323.86 0.12 344032 0.00 0.00 epoll_del
> >>>> 0.04 323.98 0.12 223477 0.00 0.00 Curl_poll
> >>>> 0.04 324.10 0.12 8526280 0.00 0.00 Curl_raw_nequal
> >>>> 0.03 324.21 0.11 348839 0.00 0.00 event_del
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> test 2:
> >>>>
> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve
> *.com to one IP address
> >>>> number of domain names in all https URLs: 1
> >>>> number of client: 1000
> >>>> CPU usage: 75%
> >>>> SSL TPS: 1300
> >>>> SSL throughput: 700Mbps
> >>>>
> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50
> >>>> Flat profile:
> >>>> Each sample counts as 0.01 seconds.
> >>>> % cumulative self self total
> >>>> time seconds seconds calls s/call s/call name
> >>>> 8.98 2.45 2.45 5127312 0.00 0.00 dprintf_formatf
> >>>> 8.90 4.88 2.43 1292796503 0.00 0.00
> ares__is_list_empty
> >>>> 6.12 6.55 1.67 322461 0.00 0.00 create_conn
> >>>> 4.58 7.80 1.25 419872 0.00 0.00 ares_cancel
> >>>> 4.10 8.92 1.12 2100162 0.00 0.00 Curl_readwrite
> >>>> 3.92 9.99 1.07 4527388 0.00 0.00 Curl_hash_pick
> >>>> 3.43 10.93 0.94 locking_function
> >>>> 3.04 11.76 0.83 909759 0.00 0.00
> curl_multi_socket_action
> >>>> 3.00 12.58 0.82 97252 0.00 0.00
> Curl_hash_clean_with_criterium
> >>>> 2.71 13.32 0.74 179704016 0.00 0.00
> Curl_raw_toupper
> >>>> 2.68 14.05 0.73 4646524 0.00 0.00 multi_runsingle
> >>>> 2.44 14.71 0.67 13862992 0.00 0.00
> client_tracing_function
> >>>> 2.42 15.37 0.66 319735 0.00 0.00
> curl_multi_remove_handle
> >>>> 2.42 16.03 0.66 22 0.03 0.03
> ares__init_list_node
> >>>> 1.94 16.56 0.53 9930244 0.00 0.00
> hostcache_timestamp_remove
> >>>> 1.94 17.09 0.53 169320088 0.00 0.00 addbyter
> >>>> 1.80 17.58 0.49 id_function
> >>>> 1.36 17.95 0.37 7057146 0.00 0.00 Curl_pgrsUpdate
> >>>> 1.36 18.32 0.37 16835 0.00 0.00
> curl_multi_socket_all
> >>>> 1.25 18.66 0.34 16719 0.00 0.00
> curl_multi_perform
> >>>> 1.21 18.99 0.33 329403 0.00 0.00
> Curl_http_readwrite_headers
> >>>> 1.17 19.31 0.32 4036624 0.00 0.00 Curl_splay
> >>>> 1.14 19.62 0.31 20838902 0.00 0.00 Curl_raw_nequal
> >>>> 0.82 19.85 0.23 2764123 0.00 0.00 Curl_infof
> >>>> 0.81 20.07 0.22 5933857 0.00 0.00 ossl_recv
> >>>> 0.77 20.28 0.21 323112 0.00 0.00
> Curl_splayremovebyaddr
> >>>> 0.75 20.48 0.21 5943991 0.00 0.00 Curl_read
> >>>> 0.70 20.67 0.19 425174 0.00 0.00 event_del
> >>>> 0.70 20.86 0.19 98652 0.00 0.00
> Curl_if_is_interface_name
> >>>> 0.70 21.05 0.19 8677733 0.00 0.00
> Curl_socket_check
> >>>> 0.59 21.21 0.16 4129554 0.00 0.00 fd_key_compare
> >>>> 0.57 21.37 0.16 14017019 0.00 0.00 Curl_debug
> >>>> 0.55 21.52 0.15 9376861 0.00 0.00 stat_data_in_add
> >>>> 0.55 21.67 0.15 6840468 0.00 0.00 Curl_timeleft
> >>>> 0.55 21.82 0.15 2054602 0.00 0.00 Curl_raw_equal
> >>>> 0.55 21.97 0.15 1291611 0.00 0.00 Curl_expire
> >>>> 0.51 22.11 0.14 16570063 0.00 0.00 curlx_tvnow
> >>>> 0.51 22.25 0.14 13863284 0.00 0.00 scan_response
> >>>> 0.51 22.39 0.14 5498864 0.00 0.00 Curl_setopt
> >>>> 0.48 22.52 0.13 97490291 0.00 0.00 curl_strequal
> >>>> 0.44 22.64 0.12 324253 0.00 0.00 Curl_http
> >>>> 0.44 22.76 0.12 394386 0.00 0.00
> ossl_connect_common
> >>>> 0.40 22.87 0.11 3581335 0.00 0.00 Curl_getinfo
> >>>> 0.38 22.97 0.11 26983092 0.00 0.00 alloc_addbyter
> >>>> 0.37 23.07 0.10 8729044 0.00 0.00
> Curl_client_write
> >>>>
> >>>>
> >>>>
> ------------------------------------------------------------------------------
> >>>> Try before you buy = See our experts in action!
> >>>> The most comprehensive online learning library for Microsoft
> developers
> >>>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3,
> MVC3,
> >>>> Metro Style Apps, more. Free future releases when you subscribe now!
> >>>> http://p.sf.net/sfu/learndevnow-dev2
> >>>> _______________________________________________
> >>>> curl-loader-devel mailing list
> >>>> cur...@li...
> >>>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Regards,
> >>> Robert Iakobashvili, Ph.D.
> >>>
> >>> Home: http://www.ghotit.com
> >>> Blog: http://www.ghotit.com/category/ghotit-blogs/
> >>> Twitter: http://twitter.com/ghotit
> >>> Facebook: http://facebook.com/ghotit
> >>> ......................................................................
> >>> Ghotit Dyslexia
> >>> Assistive technology that understands you
> >>> ......................................................................
> >>>
> >>>
> >>>
> ------------------------------------------------------------------------------
> >>> Try before you buy = See our experts in action!
> >>> The most comprehensive online learning library for Microsoft developers
> >>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3,
> MVC3,
> >>> Metro Style Apps, more. Free future releases when you subscribe now!
> >>> http://p.sf.net/sfu/learndevnow-dev2
> >>> _______________________________________________
> >>> curl-loader-devel mailing list
> >>> cur...@li...
> >>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
> >>>
> >>
> >>
> >>
> ------------------------------------------------------------------------------
> >> Try before you buy = See our experts in action!
> >> The most comprehensive online learning library for Microsoft developers
> >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
> >> Metro Style Apps, more. Free future releases when you subscribe now!
> >> http://p.sf.net/sfu/learndevnow-dev2
> >> _______________________________________________
> >> curl-loader-devel mailing list
> >> cur...@li...
> >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
> >>
> >
> >
> >
> > --
> > Regards,
> > Robert Iakobashvili, Ph.D.
> >
> > Home: http://www.ghotit.com
> > Blog: http://www.ghotit.com/category/ghotit-blogs/
> > Twitter: http://twitter.com/ghotit
> > Facebook: http://facebook.com/ghotit
> > ......................................................................
> > Ghotit Dyslexia
> > Assistive technology that understands you
> > ......................................................................
>
>
>
> --
> Regards,
> Robert Iakobashvili, Ph.D.
>
> Home: http://www.ghotit.com
> Blog: http://www.ghotit.com/category/ghotit-blogs/
> Twitter: http://twitter.com/ghotit
> Facebook: http://facebook.com/ghotit
> ......................................................................
> Ghotit Dyslexia
> Assistive technology that understands you
> ......................................................................
>
>
> ------------------------------------------------------------------------------
> Try before you buy = See our experts in action!
> The most comprehensive online learning library for Microsoft developers
> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
> Metro Style Apps, more. Free future releases when you subscribe now!
> http://p.sf.net/sfu/learndevnow-dev2
> _______________________________________________
> curl-loader-devel mailing list
> cur...@li...
> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>
|
|
From: Robert I. <cor...@gm...> - 2012-03-06 05:28:39
|
Several questions:
1. Could you please run top and see if there are other processes burning CPU?
2. What is the number of loading clients that you use?
3. Could you try curl-loader-0.52 if it compiles for you?
If not, please try to place curl-loader-7.19.7.tar.gz and c-ares-16.0.tar.gz,
update their versions in Makefile and rebuild by make cleanall; make
4. Is it correct to say that for HTTP URLs (not HTTPS) you are getting
a good throughput
and CAPS with curl-loader-0.56?
Thanks,
Robert
On Tue, Mar 6, 2012 at 3:52 AM, Fred Huang <di...@gm...> wrote:
> TIMEOUT -1 -> 60 works a little better but no performance improvement on
> CAPS or throughput.
>
> test 5:
> enable c-ares;
> curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60);
>
> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com to
> one IP address
> number of domain names in URLs: 780,000
> number of client: 1000
> CPU usage: 74%
> cpu% irq% sirq% sys% iowt% mem_used buf&cached
> 74.0 0.0 0.7 4.2 0.0 3530.6Mb 250.8Mb
> SSL TPS: 360
> SSL throughput: 220Mbps
>
> # gprof /usr/bin/curl-loader gmon.out -p | head -50
> Flat profile:
> Each sample counts as 0.01 seconds.
> % cumulative self self total
> time seconds seconds calls s/call s/call name
> 26.63 23.13 23.13 119226 0.00 0.00
> Curl_hash_clean_with_criterium
> 20.34 40.80 17.67 317792146 0.00 0.00
> hostcache_timestamp_remove
> 11.27 50.59 9.79 2579903 0.00 0.00 Curl_hash_pick
> 7.84 57.40 6.81 120606 0.00 0.00 create_conn
> 5.65 62.31 4.91 119117401 0.00 0.00 Curl_str_key_compare
> 4.98 66.64 4.33 329829 0.00 0.00 Curl_hash_add
> 2.80 69.08 2.44 4476606 0.00 0.00 dprintf_formatf
> 2.57 71.31 2.23 124019 0.00 0.00 ConnectionStore
> 1.50 72.61 1.30 740997609 0.00 0.00 ares__is_list_empty
> 1.16 73.61 1.01 locking_function
> 0.76 74.27 0.66 238744 0.00 0.00 ares_cancel
> 0.70 74.88 0.61 457564 0.00 0.00
> curl_multi_socket_action
> 0.67 75.46 0.58 118616 0.00 0.00
> curl_multi_remove_handle
> 0.58 75.97 0.51 8211502 0.00 0.00
> client_tracing_function
> 0.53 76.43 0.46 733561 0.00 0.00 Curl_readwrite
> 0.51 76.87 0.44 2183179 0.00 0.00 multi_runsingle
> 0.51 77.31 0.44 166585567 0.00 0.00 addbyter
> 0.44 77.69 0.38 2175805 0.00 0.00 Curl_raw_equal
> 0.43 78.06 0.38 id_function
> 0.40 78.41 0.35 29789346 0.00 0.00 Curl_socket_check
> 0.40 78.76 0.35 479671 0.00 0.00 ossl_connect_common
> 0.40 79.11 0.35 490639 0.00 0.00 ares__init_list_node
> 0.33 79.40 0.29 94197670 0.00 0.00 Curl_raw_toupper
> 0.33 79.69 0.29 3268031 0.00 0.00 Curl_pgrsUpdate
> 0.28 79.93 0.24 8797 0.00 0.00 curl_multi_perform
> 0.27 80.17 0.24 2233641 0.00 0.00 Curl_infof
> 0.26 80.40 0.23 1839837 0.00 0.00 Curl_splay
> 0.24 80.61 0.21 121713028 0.00 0.00 curl_strequal
> 0.21 80.79 0.18 243333 0.00 0.00 Curl_ssl_getsessionid
> 0.18 80.95 0.16 122180 0.00 0.00 Curl_num_addresses
> 0.17 81.09 0.15 8271666 0.00 0.00 Curl_debug
> 0.16 81.23 0.14 18902452 0.00 0.00 alloc_addbyter
> 0.16 81.37 0.14 8851 0.00 0.00 curl_multi_socket_all
> 0.14 81.50 0.13 2236769 0.00 0.00 Curl_read
> 0.14 81.62 0.12 2271700 0.00 0.00 fd_key_compare
> 0.14 81.74 0.12 323167 0.00 0.00 event_del
> 0.14 81.86 0.12 205436 0.00 0.00 read_udp_packets
> 0.14 81.98 0.12 200979 0.00 0.00 processfds
> 0.13 82.09 0.11 123886 0.00 0.00
> Curl_if_is_interface_name
> 0.13 82.20 0.11 120350 0.00 0.00 Curl_http
> 0.13 82.31 0.11 ssl_tls_trace
> 0.12 82.41 0.10 119229 0.00 0.00 Curl_splayremovebyaddr
> 0.12 82.51 0.10 7798717 0.00 0.00 Curl_raw_nequal
> 0.12 82.61 0.10 784485 0.00 0.00 event_add
> 0.10 82.70 0.09 643482 0.00 0.00 Curl_expire
>
>
>
>
>
> test 6:
> enable c-ares;
> curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60);
> CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0
>
> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com to
> one IP address
> number of domain names in URLs: 780,000
> number of client: 1000
> CPU usage: 86%
> cpu% irq% sirq% sys% iowt% mem_used buf&cached
> 85.9 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb
> SSL TPS: 360
> SSL throughput: 200Mbps
>
> # gprof /usr/bin/curl-loader gmon.out -p | head -20
> Flat profile:
> Each sample counts as 0.01 seconds.
> % cumulative self self total
> time seconds seconds calls s/call s/call name
> 27.19 19.96 19.96 103472 0.00 0.00
> Curl_hash_clean_with_criterium
> 21.73 35.91 15.95 274168981 0.00 0.00
> hostcache_timestamp_remove
> 10.33 43.49 7.58 2256441 0.00 0.00 Curl_hash_pick
> 7.21 48.78 5.29 104540 0.00 0.00 create_conn
> 6.20 53.33 4.55 103413405 0.00 0.00 Curl_str_key_compare
> 4.88 56.91 3.58 287109 0.00 0.00 Curl_hash_add
> 3.02 59.13 2.22 3797855 0.00 0.00 dprintf_formatf
> 2.10 60.67 1.54 107711 0.00 0.00 ConnectionStore
> 1.65 61.88 1.21 642725327 0.00 0.00 ares__is_list_empty
> 1.19 62.76 0.88 locking_function
> 0.87 63.40 0.64 207817 0.00 0.00 ares_cancel
> 0.56 63.81 0.41 404108 0.00 0.00
> curl_multi_socket_action
> 0.54 64.21 0.40 425603 0.00 0.00 ares__init_list_node
> 0.53 64.60 0.39 7068112 0.00 0.00
> client_tracing_function
> 0.49 64.96 0.36 1908587 0.00 0.00 multi_runsingle
>
>
>
> test 7:
> enable c-ares;
> curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1);
> CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0
>
> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com to
> one IP address
> number of domain names in URLs: 780,000
> number of client: 1000
> CPU usage: 86%
> cpu% irq% sirq% sys% iowt% mem_used buf&cached
> 86.6 0.0 0.6 3.3 0.0 3624.8Mb 263.0Mb
> SSL TPS: 300
> SSL throughput: 200Mbps
>
> # gprof /usr/bin/curl-loader gmon.out -p | head -20
> Flat profile:
> Each sample counts as 0.01 seconds.
> % cumulative self self total
> time seconds seconds calls s/call s/call name
> 39.83 369.71 369.71 207173 0.00 0.00
> Curl_hash_clean_with_criterium
> 24.27 594.98 225.27 3198337733 0.00 0.00
> hostcache_timestamp_remove
> 13.75 722.59 127.61 4312864 0.00 0.00 Curl_hash_pick
> 9.19 807.87 85.28 546507 0.00 0.00 Curl_hash_add
> 6.14 864.88 57.01 1022419684 0.00 0.00 Curl_str_key_compare
> 2.06 884.00 19.12 203749 0.00 0.00 create_conn
> 0.54 888.99 4.99 213306 0.00 0.00 ConnectionStore
> 0.49 893.50 4.51 7331251 0.00 0.00 dprintf_formatf
> 0.29 896.16 2.66 1287182329 0.00 0.00 ares__is_list_empty
> 0.23 898.28 2.12 locking_function
> 0.20 900.10 1.82 212210 0.00 0.00 Curl_num_addresses
> 0.17 901.66 1.56 198385 0.00 0.00
> curl_multi_remove_handle
> 0.12 902.81 1.15 411114 0.00 0.00 ares_cancel
> 0.12 903.95 1.14 3683448 0.00 0.00 multi_runsingle
> 0.11 904.97 1.03 13664120 0.00 0.00
> client_tracing_function
>
>
> test 8:
> enable c-ares;
> curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60);
> CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0
>
> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com to
> one IP address
> number of domain names in URLs: 780,000
> number of client: 1000
> CPU usage: 75%
> cpu% irq% sirq% sys% iowt% mem_used buf&cached
> 75.2 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb
> SSL TPS: 380
> SSL throughput: 220Mbps
>
> # gprof /usr/bin/curl-loader gmon.out -p | head -20
> Flat profile:
> Each sample counts as 0.01 seconds.
> % cumulative self self total
> time seconds seconds calls s/call s/call name
> 27.58 12.31 12.31 71140 0.00 0.00
> Curl_hash_clean_with_criterium
> 19.09 20.83 8.52 176729692 0.00 0.00
> hostcache_timestamp_remove
> 10.44 25.49 4.66 1589309 0.00 0.00 Curl_hash_pick
> 6.85 28.55 3.06 73204 0.00 0.00 create_conn
> 5.56 31.03 2.48 67620530 0.00 0.00 Curl_str_key_compare
> 4.41 33.00 1.97 200684 0.00 0.00 Curl_hash_add
> 3.56 34.59 1.59 2717956 0.00 0.00 dprintf_formatf
> 1.72 35.36 0.77 75247 0.00 0.00 ConnectionStore
> 1.70 36.12 0.76 444342256 0.00 0.00 ares__is_list_empty
> 1.23 36.67 0.55 locking_function
> 1.16 37.19 0.52 143342 0.00 0.00 ares_cancel
> 0.99 37.63 0.44 72211 0.00 0.00
> curl_multi_remove_handle
> 0.74 37.96 0.33 1333371 0.00 0.00 multi_runsingle
> 0.68 38.27 0.31 101147798 0.00 0.00 addbyter
> 0.58 38.53 0.26 288737 0.00 0.00
> curl_multi_socket_action
>
>
>
> test 9:
> enable c-ares;
> curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1);
> CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0
>
> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com to
> one IP address
> number of domain names in URLs: 780,000
> number of client: 1000
> CPU usage: 75%
> cpu% irq% sirq% sys% iowt% mem_used buf&cached
> 75.1 0.0 0.6 4.1 0.0 3545.2Mb 260.1Mb
> SSL TPS: 380
> SSL throughput: 200Mbps
>
> # gprof /usr/bin/curl-loader gmon.out -p | head -20
> Flat profile:
> Each sample counts as 0.01 seconds.
> % cumulative self self total
> time seconds seconds calls s/call s/call name
> 34.00 31.13 31.13 72495 0.00 0.00
> Curl_hash_clean_with_criterium
> 23.64 52.78 21.65 386070412 0.00 0.00
> hostcache_timestamp_remove
> 13.28 64.94 12.16 1619440 0.00 0.00 Curl_hash_pick
> 5.92 70.36 5.42 137653129 0.00 0.00 Curl_str_key_compare
> 5.50 75.40 5.04 203018 0.00 0.00 Curl_hash_add
> 4.55 79.57 4.17 74618 0.00 0.00 create_conn
> 1.70 81.13 1.56 2757148 0.00 0.00 dprintf_formatf
> 0.98 82.03 0.90 76945 0.00 0.00 ConnectionStore
> 0.95 82.90 0.87 455775100 0.00 0.00 ares__is_list_empty
> 0.57 83.42 0.53 locking_function
> 0.39 83.78 0.36 146742 0.00 0.00 ares_cancel
> 0.35 84.10 0.32 102968806 0.00 0.00 addbyter
> 0.35 84.42 0.32 456587 0.00 0.00 Curl_readwrite
> 0.35 84.74 0.32 73133 0.00 0.00
> curl_multi_remove_handle
> 0.34 85.05 0.31 1360545 0.00 0.00 multi_runsingle
>
>
> 2012/3/6 Robert Iakobashvili <cor...@gm...>
>>
>> If it will not be helpful this means that recent
>> changes in libcurl library have broken DNS interface.
>>
>> You can try a previos version and the latest libcurl, like:
>>
>> http://curl.haxx.se/download/curl-7.23.0.tar.gz
>> and
>> http://curl.haxx.se/downloadcurl-7.24.0.tar.gz
>>
>> by:
>>
>> 1. downloading a libcurl version and placing it to
>> curl-loader-0.56/packages ;
>> 2. correcting the Makefile in curl-loader-0.56
>> CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24 or CURL_VER:=7.23
>> 3. make cleanall; make
>>
>> Please, send us your observations.
>> Thanks!
>> Robert
>>
>> On Mon, Mar 5, 2012 at 6:57 PM, Robert Iakobashvili <cor...@gm...>
>> wrote:
>> > If so, please try to change
>> >
>> > loader.c, line 578 which looks like:
>> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1);
>> >
>> > 1. instead of -1 try to set some value like 60
>> > 2. make clean; make
>> >
>> > Thanks,
>> > Robert
>> >
>> > On Mon, Mar 5, 2012 at 6:19 PM, Fred Huang <di...@gm...> wrote:
>> >>
>> >> Looks like the bottleneck comes from libcurl other than c-ares...
>> >> test 3 (no ares patch applied):
>> >> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com
>> >> to one IP address
>> >> number of domain names in URLs: 780,000
>> >> number of client: 1000
>> >> CPU usage: 85%
>> >> cpu% irq% sirq% sys% iowt% mem_used buf&cached
>> >> 85.6 0.0 0.8 5.9 0.0 3334.3Mb 112.0Mb
>> >> SSL TPS: 320
>> >> SSL throughput: 200Mbps
>> >>
>> >> # gprof /usr/bin/curl-loader gmon.out -p | head -50
>> >> Flat profile:
>> >> Each sample counts as 0.01 seconds.
>> >> % cumulative self self total
>> >> time seconds seconds calls s/call s/call name
>> >> 40.80 120.16 120.16 116079 0.00 0.00
>> >> Curl_hash_clean_with_criterium
>> >> 24.46 192.18 72.02 983166323 0.00 0.00
>> >> hostcache_timestamp_remove
>> >> 13.30 231.35 39.17 2059341 0.00 0.00 Curl_hash_pick
>> >> 6.45 250.35 19.00 240087 0.00 0.00 Curl_hash_add
>> >> 4.81 264.51 14.16 334257102 0.00 0.00
>> >> Curl_str_key_compare
>> >> 3.40 274.52 10.01 119514 0.00 0.00 create_conn
>> >> 0.93 277.25 2.73 4478910 0.00 0.00 dprintf_formatf
>> >> 0.90 279.91 2.66 122209 0.00 0.00 ConnectionStore
>> >> 0.40 281.09 1.18 locking_function
>> >> 0.27 281.88 0.79 114759 0.00 0.00
>> >> curl_multi_remove_handle
>> >> 0.21 282.51 0.63 7993664 0.00 0.00
>> >> client_tracing_function
>> >> 0.21 283.12 0.61 122685 0.00 0.00
>> >> Curl_num_addresses
>> >> 0.19 283.68 0.56 397028 0.00 0.00
>> >> curl_multi_socket_action
>> >> 0.19 284.23 0.55 2036116 0.00 0.00 multi_runsingle
>> >> 0.17 284.72 0.49 703687 0.00 0.00 Curl_readwrite
>> >> 0.16 285.19 0.47 id_function
>> >> 0.15 285.62 0.43 29692029 0.00 0.00 Curl_socket_check
>> >> 0.14 286.02 0.40 91973856 0.00 0.00 Curl_raw_toupper
>> >> 0.13 286.40 0.38 164664144 0.00 0.00 addbyter
>> >> 0.13 286.77 0.37 2180797 0.00 0.00 Curl_infof
>> >> 0.11 287.10 0.33 122675 0.00 0.00
>> >> Curl_if_is_interface_name
>> >> 0.10 287.39 0.29 471318 0.00 0.00
>> >> ossl_connect_common
>> >> 0.10 287.67 0.28 2113528 0.00 0.00 Curl_raw_equal
>> >> 0.09 287.93 0.26 8676 0.00 0.02
>> >> curl_multi_socket_all
>> >> 0.08 288.17 0.24 1663262 0.00 0.00 Curl_splay
>> >> 0.08 288.41 0.24 8505 0.00 0.01
>> >> curl_multi_perform
>> >> 0.07 288.63 0.22 3114576 0.00 0.00 Curl_pgrsUpdate
>> >> 0.06 288.80 0.17 119637 0.00 0.00 singleipconnect
>> >> 0.06 288.97 0.17 7994818 0.00 0.00 Curl_debug
>> >> 0.05 289.13 0.16 235820 0.00 0.00 event_del
>> >> 0.05 289.29 0.16 119393071 0.00 0.00 curl_strequal
>> >> 0.05 289.45 0.16 18535473 0.00 0.00 alloc_addbyter
>> >> 0.05 289.61 0.16 356074 0.00 0.00 Curl_hash_str
>> >> 0.05 289.76 0.15 1797233 0.00 0.00 fd_key_compare
>> >> 0.05 289.91 0.15 234966 0.00 0.00
>> >> Curl_ssl_getsessionid
>> >> 0.04 290.04 0.13 585853 0.00 0.00
>> >> Curl_llist_destroy
>> >> 0.04 290.16 0.12 240826 0.00 0.00 Curl_resolv
>> >> 0.03 290.26 0.10 7514328 0.00 0.00 Curl_raw_nequal
>> >> 0.03 290.36 0.10 2214436 0.00 0.00 Curl_read
>> >> 0.03 290.46 0.10 116872 0.00 0.00
>> >> Curl_http_readwrite_headers
>> >> 0.03 290.56 0.10 ssl_tls_trace
>> >> 0.03 290.66 0.10 8013097 0.00 0.00 scan_response
>> >> 0.03 290.75 0.09 29488555 0.00 0.00 curlx_tvdiff
>> >> 0.03 290.84 0.09 2142468 0.00 0.00 Curl_setopt
>> >> 0.03 290.93 0.09 476299 0.00 0.00 Curl_expire
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> 2012/3/5 Robert Iakobashvili <cor...@gm...>
>> >>>
>> >>> Dear Fred Huang,
>> >>>
>> >>> Thanks for your posting.
>> >>>
>> >>> Could you please check if this is due to some badness at cares?
>> >>> Thanks.
>> >>>
>> >>> Let's exclude usage of cares based DNS resolving by applying the patch
>> >>> attached
>> >>> and use instead global DNS cache.
>> >>>
>> >>> cd ./curl-loader-0.56
>> >>> patch -p1 < path-to/disable-cares-use-global-dns.patch
>> >>> make cleanall; make
>> >>>
>> >>> See if it helps.
>> >>>
>> >>> Global cache, unfortunately, is not thread safe.
>> >>>
>> >>> If it helps, we'll try to restore the previous version of cares.
>> >>>
>> >>> Thanks,
>> >>> Robert
>> >>>
>> >>> On Mon, Mar 5, 2012 at 4:28 PM, Fred Huang <di...@gm...> wrote:
>> >>>>
>> >>>> test 1:
>> >>>>
>> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve
>> >>>> *.com to one IP address
>> >>>> number of domain names in all https URLs: 780,000
>> >>>> number of client: 1000
>> >>>> CPU usage: 78%
>> >>>> cpu% irq% sirq% sys% iowt% mem_used buf&cached
>> >>>> 78.8 0.0 0.6 3.8 0.0 3640.9Mb 282.1Mb
>> >>>> SSL TPS: 300
>> >>>> SSL throughput: 200Mbps
>> >>>>
>> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50
>> >>>> Flat profile:
>> >>>> Each sample counts as 0.01 seconds.
>> >>>> % cumulative self self total
>> >>>> time seconds seconds calls s/call s/call name
>> >>>> 38.55 127.10 127.10 130975 0.00 0.00
>> >>>> Curl_hash_clean_with_criterium
>> >>>> 24.63 208.32 81.22 1247528407 0.00 0.00
>> >>>> hostcache_timestamp_remove
>> >>>> 13.08 251.44 43.12 2791988 0.00 0.00 Curl_hash_pick
>> >>>> 7.30 275.51 24.07 352490 0.00 0.00 Curl_hash_add
>> >>>> 6.01 295.31 19.80 410760770 0.00 0.00
>> >>>> Curl_str_key_compare
>> >>>> 2.95 305.04 9.73 131015 0.00 0.00 create_conn
>> >>>> 0.88 307.94 2.90 4841390 0.00 0.00 dprintf_formatf
>> >>>> 0.74 310.38 2.44 136036 0.00 0.00 ConnectionStore
>> >>>> 0.48 311.96 1.58 814604911 0.00 0.00
>> >>>> ares__is_list_empty
>> >>>> 0.36 313.16 1.20
>> >>>> locking_function
>> >>>> 0.27 314.06 0.90 261608 0.00 0.00 ares_cancel
>> >>>> 0.22 314.79 0.73 127417 0.00 0.00
>> >>>> curl_multi_remove_handle
>> >>>> 0.19 315.40 0.62 8906301 0.00 0.00
>> >>>> client_tracing_function
>> >>>> 0.17 315.97 0.57 135305 0.00 0.00
>> >>>> Curl_num_addresses
>> >>>> 0.17 316.54 0.57 181937907 0.00 0.00 addbyter
>> >>>> 0.17 317.09 0.55 2374363 0.00 0.00 multi_runsingle
>> >>>> 0.16 317.62 0.53 541954 0.00 0.00
>> >>>> ares__init_list_node
>> >>>> 0.16 318.14 0.52 494412 0.00 0.00
>> >>>> curl_multi_socket_action
>> >>>> 0.14 318.60 0.46 33578897 0.00 0.00
>> >>>> Curl_socket_check
>> >>>> 0.13 319.04 0.44 102951052 0.00 0.00
>> >>>> Curl_raw_toupper
>> >>>> 0.13 319.48 0.44 787558 0.00 0.00 Curl_readwrite
>> >>>> 0.13 319.91 0.44 id_function
>> >>>> 0.12 320.29 0.38 391390 0.00 0.00 Curl_hash_str
>> >>>> 0.09 320.58 0.29 9216 0.00 0.00
>> >>>> curl_multi_perform
>> >>>> 0.08 320.86 0.28 2350908 0.00 0.00 Curl_raw_equal
>> >>>> 0.08 321.14 0.28 2019415 0.00 0.00 Curl_splay
>> >>>> 0.08 321.40 0.26 520625 0.00 0.00
>> >>>> ossl_connect_common
>> >>>> 0.08 321.65 0.25 3532992 0.00 0.00 Curl_pgrsUpdate
>> >>>> 0.07 321.87 0.22 131778537 0.00 0.00 curl_strequal
>> >>>> 0.05 322.05 0.18 8954118 0.00 0.00 scan_response
>> >>>> 0.05 322.23 0.18 533644 0.00 0.00
>> >>>> ares_expand_name
>> >>>> 0.05 322.40 0.17 2477345 0.00 0.00 fd_key_compare
>> >>>> 0.05 322.57 0.17 2429135 0.00 0.00 Curl_infof
>> >>>> 0.05 322.74 0.17 130966 0.00 0.00 singleipconnect
>> >>>> 0.05 322.90 0.16 9292 0.00 0.00
>> >>>> curl_multi_socket_all
>> >>>> 0.05 323.06 0.16 270137 0.00 0.00
>> >>>> ares__get_hostent
>> >>>> 0.05 323.21 0.15 8943657 0.00 0.00 Curl_debug
>> >>>> 0.04 323.35 0.14 262493 0.00 0.00
>> >>>> Curl_ssl_getsessionid
>> >>>> 0.04 323.48 0.13 566006 0.00 0.00 socket_callback
>> >>>> 0.04 323.61 0.13 33983864 0.00 0.00 curlx_tvdiff
>> >>>> 0.04 323.74 0.13 132853 0.00 0.00
>> >>>> Curl_http_readwrite_headers
>> >>>> 0.04 323.86 0.12 344032 0.00 0.00 epoll_del
>> >>>> 0.04 323.98 0.12 223477 0.00 0.00 Curl_poll
>> >>>> 0.04 324.10 0.12 8526280 0.00 0.00 Curl_raw_nequal
>> >>>> 0.03 324.21 0.11 348839 0.00 0.00 event_del
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> test 2:
>> >>>>
>> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve
>> >>>> *.com to one IP address
>> >>>> number of domain names in all https URLs: 1
>> >>>> number of client: 1000
>> >>>> CPU usage: 75%
>> >>>> SSL TPS: 1300
>> >>>> SSL throughput: 700Mbps
>> >>>>
>> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50
>> >>>> Flat profile:
>> >>>> Each sample counts as 0.01 seconds.
>> >>>> % cumulative self self total
>> >>>> time seconds seconds calls s/call s/call name
>> >>>> 8.98 2.45 2.45 5127312 0.00 0.00 dprintf_formatf
>> >>>> 8.90 4.88 2.43 1292796503 0.00 0.00
>> >>>> ares__is_list_empty
>> >>>> 6.12 6.55 1.67 322461 0.00 0.00 create_conn
>> >>>> 4.58 7.80 1.25 419872 0.00 0.00 ares_cancel
>> >>>> 4.10 8.92 1.12 2100162 0.00 0.00 Curl_readwrite
>> >>>> 3.92 9.99 1.07 4527388 0.00 0.00 Curl_hash_pick
>> >>>> 3.43 10.93 0.94
>> >>>> locking_function
>> >>>> 3.04 11.76 0.83 909759 0.00 0.00
>> >>>> curl_multi_socket_action
>> >>>> 3.00 12.58 0.82 97252 0.00 0.00
>> >>>> Curl_hash_clean_with_criterium
>> >>>> 2.71 13.32 0.74 179704016 0.00 0.00
>> >>>> Curl_raw_toupper
>> >>>> 2.68 14.05 0.73 4646524 0.00 0.00 multi_runsingle
>> >>>> 2.44 14.71 0.67 13862992 0.00 0.00
>> >>>> client_tracing_function
>> >>>> 2.42 15.37 0.66 319735 0.00 0.00
>> >>>> curl_multi_remove_handle
>> >>>> 2.42 16.03 0.66 22 0.03 0.03
>> >>>> ares__init_list_node
>> >>>> 1.94 16.56 0.53 9930244 0.00 0.00
>> >>>> hostcache_timestamp_remove
>> >>>> 1.94 17.09 0.53 169320088 0.00 0.00 addbyter
>> >>>> 1.80 17.58 0.49 id_function
>> >>>> 1.36 17.95 0.37 7057146 0.00 0.00 Curl_pgrsUpdate
>> >>>> 1.36 18.32 0.37 16835 0.00 0.00
>> >>>> curl_multi_socket_all
>> >>>> 1.25 18.66 0.34 16719 0.00 0.00
>> >>>> curl_multi_perform
>> >>>> 1.21 18.99 0.33 329403 0.00 0.00
>> >>>> Curl_http_readwrite_headers
>> >>>> 1.17 19.31 0.32 4036624 0.00 0.00 Curl_splay
>> >>>> 1.14 19.62 0.31 20838902 0.00 0.00 Curl_raw_nequal
>> >>>> 0.82 19.85 0.23 2764123 0.00 0.00 Curl_infof
>> >>>> 0.81 20.07 0.22 5933857 0.00 0.00 ossl_recv
>> >>>> 0.77 20.28 0.21 323112 0.00 0.00
>> >>>> Curl_splayremovebyaddr
>> >>>> 0.75 20.48 0.21 5943991 0.00 0.00 Curl_read
>> >>>> 0.70 20.67 0.19 425174 0.00 0.00 event_del
>> >>>> 0.70 20.86 0.19 98652 0.00 0.00
>> >>>> Curl_if_is_interface_name
>> >>>> 0.70 21.05 0.19 8677733 0.00 0.00
>> >>>> Curl_socket_check
>> >>>> 0.59 21.21 0.16 4129554 0.00 0.00 fd_key_compare
>> >>>> 0.57 21.37 0.16 14017019 0.00 0.00 Curl_debug
>> >>>> 0.55 21.52 0.15 9376861 0.00 0.00
>> >>>> stat_data_in_add
>> >>>> 0.55 21.67 0.15 6840468 0.00 0.00 Curl_timeleft
>> >>>> 0.55 21.82 0.15 2054602 0.00 0.00 Curl_raw_equal
>> >>>> 0.55 21.97 0.15 1291611 0.00 0.00 Curl_expire
>> >>>> 0.51 22.11 0.14 16570063 0.00 0.00 curlx_tvnow
>> >>>> 0.51 22.25 0.14 13863284 0.00 0.00 scan_response
>> >>>> 0.51 22.39 0.14 5498864 0.00 0.00 Curl_setopt
>> >>>> 0.48 22.52 0.13 97490291 0.00 0.00 curl_strequal
>> >>>> 0.44 22.64 0.12 324253 0.00 0.00 Curl_http
>> >>>> 0.44 22.76 0.12 394386 0.00 0.00
>> >>>> ossl_connect_common
>> >>>> 0.40 22.87 0.11 3581335 0.00 0.00 Curl_getinfo
>> >>>> 0.38 22.97 0.11 26983092 0.00 0.00 alloc_addbyter
>> >>>> 0.37 23.07 0.10 8729044 0.00 0.00
>> >>>> Curl_client_write
>> >>>>
>> >>>>
>> >>>>
>> >>>> ------------------------------------------------------------------------------
>> >>>> Try before you buy = See our experts in action!
>> >>>> The most comprehensive online learning library for Microsoft
>> >>>> developers
>> >>>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3,
>> >>>> MVC3,
>> >>>> Metro Style Apps, more. Free future releases when you subscribe now!
>> >>>> http://p.sf.net/sfu/learndevnow-dev2
>> >>>> _______________________________________________
>> >>>> curl-loader-devel mailing list
>> >>>> cur...@li...
>> >>>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Regards,
>> >>> Robert Iakobashvili, Ph.D.
>> >>>
>> >>> Home: http://www.ghotit.com
>> >>> Blog: http://www.ghotit.com/category/ghotit-blogs/
>> >>> Twitter: http://twitter.com/ghotit
>> >>> Facebook: http://facebook.com/ghotit
>> >>> ......................................................................
>> >>> Ghotit Dyslexia
>> >>> Assistive technology that understands you
>> >>> ......................................................................
>> >>>
>> >>>
>> >>>
>> >>> ------------------------------------------------------------------------------
>> >>> Try before you buy = See our experts in action!
>> >>> The most comprehensive online learning library for Microsoft
>> >>> developers
>> >>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3,
>> >>> MVC3,
>> >>> Metro Style Apps, more. Free future releases when you subscribe now!
>> >>> http://p.sf.net/sfu/learndevnow-dev2
>> >>> _______________________________________________
>> >>> curl-loader-devel mailing list
>> >>> cur...@li...
>> >>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>> >>>
>> >>
>> >>
>> >>
>> >> ------------------------------------------------------------------------------
>> >> Try before you buy = See our experts in action!
>> >> The most comprehensive online learning library for Microsoft developers
>> >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3,
>> >> MVC3,
>> >> Metro Style Apps, more. Free future releases when you subscribe now!
>> >> http://p.sf.net/sfu/learndevnow-dev2
>> >> _______________________________________________
>> >> curl-loader-devel mailing list
>> >> cur...@li...
>> >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>> >>
>> >
>> >
>> >
>> > --
>> > Regards,
>> > Robert Iakobashvili, Ph.D.
>> >
>> > Home: http://www.ghotit.com
>> > Blog: http://www.ghotit.com/category/ghotit-blogs/
>> > Twitter: http://twitter.com/ghotit
>> > Facebook: http://facebook.com/ghotit
>> > ......................................................................
>> > Ghotit Dyslexia
>> > Assistive technology that understands you
>> > ......................................................................
>>
>>
>>
>> --
>> Regards,
>> Robert Iakobashvili, Ph.D.
>>
>> Home: http://www.ghotit.com
>> Blog: http://www.ghotit.com/category/ghotit-blogs/
>> Twitter: http://twitter.com/ghotit
>> Facebook: http://facebook.com/ghotit
>> ......................................................................
>> Ghotit Dyslexia
>> Assistive technology that understands you
>> ......................................................................
>>
>>
>> ------------------------------------------------------------------------------
>> Try before you buy = See o...
[truncated message content] |
|
From: Fred H. <di...@gm...> - 2012-03-06 16:17:26
|
1. the major CPU consumer is curl-loader. here is the top:
top - 07:36:36 up 23:31, 5 users, load average: 4.32, 2.26, 0.91
Tasks: 237 total, 2 running, 235 sleeping, 0 stopped, 0 zombie
Cpu0 : 76.6%us, 3.6%sy, 0.0%ni, 19.4%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%
Cpu1 : 72.4%us, 4.0%sy, 0.0%ni, 22.9%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0%
Cpu2 : 65.8%us, 3.3%sy, 0.0%ni, 30.6%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%
Cpu3 : 75.2%us, 4.3%sy, 0.0%ni, 19.5%id, 0.0%wa, 0.0%hi, 1.0%si, 0.0%
Cpu4 : 79.1%us, 4.0%sy, 0.0%ni, 16.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%
Cpu5 : 75.8%us, 4.3%sy, 0.0%ni, 19.2%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0%
Cpu6 : 74.1%us, 5.0%sy, 0.0%ni, 20.3%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0%
Cpu7 : 78.7%us, 4.0%sy, 0.0%ni, 16.6%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0%
Mem: 16209932k total, 3829500k used, 12380432k free, 99112k buffers
Swap: 18448376k total, 0k used, 18448376k free, 320284k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
30795 root 20 0 3200m 2.8g 2296 S *79.2* 18.4 21:26.84 curl-loader
1832 nobody 20 0 13944 1736 580 S 0.1 0.0 0:32.65 dnsmasq
9 root 20 0 0 0 0 S 0.0 0.0 0:00.53 ksoftirqd/1
30825 root 20 0 15220 1360 960 R 0.0 0.0 0:00.42 top
1 root 20 0 19400 1568 1256 S 0.0 0.0 0:00.75 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
4 root 20 0 0 0 0 S 0.0 0.0 0:00.06 ksoftirqd/0
5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
2. 1000 loading clients
3. yes, tried 0.52 but no lucky -- CAPS 380, throughput 200Mbps, CPU 90%
top - 08:17:10 up 1 day, 12 min, 8 users, load average: 4.66, 3.39, 1.88
Tasks: 245 total, 2 running, 243 sleeping, 0 stopped, 0 zombie
Cpu0 : 82.2%us, 4.0%sy, 0.0%ni, 13.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%
Cpu1 : 86.1%us, 4.3%sy, 0.0%ni, 7.9%id, 0.0%wa, 0.0%hi, 1.7%si, 0.0%
Cpu2 : 85.3%us, 4.0%sy, 0.0%ni, 10.0%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0%
Cpu3 : 81.8%us, 3.0%sy, 0.0%ni, 14.2%id, 0.0%wa, 0.0%hi, 1.0%si, 0.0%
Cpu4 : 83.7%us, 4.0%sy, 0.0%ni, 12.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%
Cpu5 : 92.7%us, 3.3%sy, 0.0%ni, 3.3%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0%
Cpu6 : 89.7%us, 2.0%sy, 0.0%ni, 8.0%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%
Cpu7 : 88.1%us, 3.6%sy, 0.0%ni, 6.6%id, 0.0%wa, 0.0%hi, 1.7%si, 0.0%
Mem: 16209932k total, 1795748k used, 14414184k free, 102824k buffers
Swap: 18448376k total, 0k used, 18448376k free, 382988k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12236 root 20 0 1150m 894m 2572 S 90.7 5.7 51:11.91 curl-loader
1832 nobody 20 0 13944 1736 580 S 0.1 0.0 0:47.04 dnsmasq
]# gprof /usr/bin/curl-loader gmon.out -p | head -20
Flat profile:
Each sample counts as 0.01 seconds.
% cumulative self self total
time seconds seconds calls s/call s/call name
43.92 151.95 151.95 157295 0.00 0.00
Curl_hash_clean_with_criterium
22.90 231.17 79.22 1712190971 0.00 0.00
hostcache_timestamp_remove
13.15 276.68 45.51 3260321 0.00 0.00 Curl_hash_pick
7.51 302.65 25.97 394171 0.00 0.00 Curl_hash_add
5.98 323.34 20.68 567437518 0.00 0.00 Curl_str_key_compare
1.00 326.80 3.46 5380220 0.00 0.00 dprintf_formatf
0.41 328.21 1.41 locking_function
0.32 329.32 1.11 469222138 0.00 0.00 ares__is_list_empty
0.31 330.40 1.08 3284508 0.00 0.00 multi_runsingle
0.28 331.38 0.98 10006806 0.00 0.00
client_tracing_function
0.24 332.21 0.83 151951 0.00 0.00 Curl_num_addresses
0.22 332.97 0.77 188075672 0.00 0.00 addbyter
0.19 333.64 0.67 841840 0.00 0.00 Curl_readwrite
0.19 334.29 0.65 id_function
0.18 334.90 0.61 39843 0.00 0.01 curl_multi_socket_all
4. yes, it is correct for 0.56. If use same URL list by replacing https to
http, I can get CAPS 1500 and throughput 870Mbps with CPU% < 60%
2012/3/6 Robert Iakobashvili <cor...@gm...>
> Several questions:
>
> 1. Could you please run top and see if there are other processes burning
> CPU?
> 2. What is the number of loading clients that you use?
> 3. Could you try curl-loader-0.52 if it compiles for you?
> If not, please try to place curl-loader-7.19.7.tar.gz and
> c-ares-16.0.tar.gz,
> update their versions in Makefile and rebuild by make cleanall; make
>
> 4. Is it correct to say that for HTTP URLs (not HTTPS) you are getting
> a good throughput
> and CAPS with curl-loader-0.56?
>
> Thanks,
> Robert
>
> On Tue, Mar 6, 2012 at 3:52 AM, Fred Huang <di...@gm...> wrote:
> > TIMEOUT -1 -> 60 works a little better but no performance improvement on
> > CAPS or throughput.
> >
> > test 5:
> > enable c-ares;
> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60);
> >
> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com
> to
> > one IP address
> > number of domain names in URLs: 780,000
> > number of client: 1000
> > CPU usage: 74%
> > cpu% irq% sirq% sys% iowt% mem_used buf&cached
> > 74.0 0.0 0.7 4.2 0.0 3530.6Mb 250.8Mb
> > SSL TPS: 360
> > SSL throughput: 220Mbps
> >
> > # gprof /usr/bin/curl-loader gmon.out -p | head -50
> > Flat profile:
> > Each sample counts as 0.01 seconds.
> > % cumulative self self total
> > time seconds seconds calls s/call s/call name
> > 26.63 23.13 23.13 119226 0.00 0.00
> > Curl_hash_clean_with_criterium
> > 20.34 40.80 17.67 317792146 0.00 0.00
> > hostcache_timestamp_remove
> > 11.27 50.59 9.79 2579903 0.00 0.00 Curl_hash_pick
> > 7.84 57.40 6.81 120606 0.00 0.00 create_conn
> > 5.65 62.31 4.91 119117401 0.00 0.00
> Curl_str_key_compare
> > 4.98 66.64 4.33 329829 0.00 0.00 Curl_hash_add
> > 2.80 69.08 2.44 4476606 0.00 0.00 dprintf_formatf
> > 2.57 71.31 2.23 124019 0.00 0.00 ConnectionStore
> > 1.50 72.61 1.30 740997609 0.00 0.00
> ares__is_list_empty
> > 1.16 73.61 1.01 locking_function
> > 0.76 74.27 0.66 238744 0.00 0.00 ares_cancel
> > 0.70 74.88 0.61 457564 0.00 0.00
> > curl_multi_socket_action
> > 0.67 75.46 0.58 118616 0.00 0.00
> > curl_multi_remove_handle
> > 0.58 75.97 0.51 8211502 0.00 0.00
> > client_tracing_function
> > 0.53 76.43 0.46 733561 0.00 0.00 Curl_readwrite
> > 0.51 76.87 0.44 2183179 0.00 0.00 multi_runsingle
> > 0.51 77.31 0.44 166585567 0.00 0.00 addbyter
> > 0.44 77.69 0.38 2175805 0.00 0.00 Curl_raw_equal
> > 0.43 78.06 0.38 id_function
> > 0.40 78.41 0.35 29789346 0.00 0.00 Curl_socket_check
> > 0.40 78.76 0.35 479671 0.00 0.00 ossl_connect_common
> > 0.40 79.11 0.35 490639 0.00 0.00
> ares__init_list_node
> > 0.33 79.40 0.29 94197670 0.00 0.00 Curl_raw_toupper
> > 0.33 79.69 0.29 3268031 0.00 0.00 Curl_pgrsUpdate
> > 0.28 79.93 0.24 8797 0.00 0.00 curl_multi_perform
> > 0.27 80.17 0.24 2233641 0.00 0.00 Curl_infof
> > 0.26 80.40 0.23 1839837 0.00 0.00 Curl_splay
> > 0.24 80.61 0.21 121713028 0.00 0.00 curl_strequal
> > 0.21 80.79 0.18 243333 0.00 0.00
> Curl_ssl_getsessionid
> > 0.18 80.95 0.16 122180 0.00 0.00 Curl_num_addresses
> > 0.17 81.09 0.15 8271666 0.00 0.00 Curl_debug
> > 0.16 81.23 0.14 18902452 0.00 0.00 alloc_addbyter
> > 0.16 81.37 0.14 8851 0.00 0.00
> curl_multi_socket_all
> > 0.14 81.50 0.13 2236769 0.00 0.00 Curl_read
> > 0.14 81.62 0.12 2271700 0.00 0.00 fd_key_compare
> > 0.14 81.74 0.12 323167 0.00 0.00 event_del
> > 0.14 81.86 0.12 205436 0.00 0.00 read_udp_packets
> > 0.14 81.98 0.12 200979 0.00 0.00 processfds
> > 0.13 82.09 0.11 123886 0.00 0.00
> > Curl_if_is_interface_name
> > 0.13 82.20 0.11 120350 0.00 0.00 Curl_http
> > 0.13 82.31 0.11 ssl_tls_trace
> > 0.12 82.41 0.10 119229 0.00 0.00
> Curl_splayremovebyaddr
> > 0.12 82.51 0.10 7798717 0.00 0.00 Curl_raw_nequal
> > 0.12 82.61 0.10 784485 0.00 0.00 event_add
> > 0.10 82.70 0.09 643482 0.00 0.00 Curl_expire
> >
> >
> >
> >
> >
> > test 6:
> > enable c-ares;
> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60);
> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0
> >
> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com
> to
> > one IP address
> > number of domain names in URLs: 780,000
> > number of client: 1000
> > CPU usage: 86%
> > cpu% irq% sirq% sys% iowt% mem_used buf&cached
> > 85.9 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb
> > SSL TPS: 360
> > SSL throughput: 200Mbps
> >
> > # gprof /usr/bin/curl-loader gmon.out -p | head -20
> > Flat profile:
> > Each sample counts as 0.01 seconds.
> > % cumulative self self total
> > time seconds seconds calls s/call s/call name
> > 27.19 19.96 19.96 103472 0.00 0.00
> > Curl_hash_clean_with_criterium
> > 21.73 35.91 15.95 274168981 0.00 0.00
> > hostcache_timestamp_remove
> > 10.33 43.49 7.58 2256441 0.00 0.00 Curl_hash_pick
> > 7.21 48.78 5.29 104540 0.00 0.00 create_conn
> > 6.20 53.33 4.55 103413405 0.00 0.00
> Curl_str_key_compare
> > 4.88 56.91 3.58 287109 0.00 0.00 Curl_hash_add
> > 3.02 59.13 2.22 3797855 0.00 0.00 dprintf_formatf
> > 2.10 60.67 1.54 107711 0.00 0.00 ConnectionStore
> > 1.65 61.88 1.21 642725327 0.00 0.00
> ares__is_list_empty
> > 1.19 62.76 0.88 locking_function
> > 0.87 63.40 0.64 207817 0.00 0.00 ares_cancel
> > 0.56 63.81 0.41 404108 0.00 0.00
> > curl_multi_socket_action
> > 0.54 64.21 0.40 425603 0.00 0.00
> ares__init_list_node
> > 0.53 64.60 0.39 7068112 0.00 0.00
> > client_tracing_function
> > 0.49 64.96 0.36 1908587 0.00 0.00 multi_runsingle
> >
> >
> >
> > test 7:
> > enable c-ares;
> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1);
> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0
> >
> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com
> to
> > one IP address
> > number of domain names in URLs: 780,000
> > number of client: 1000
> > CPU usage: 86%
> > cpu% irq% sirq% sys% iowt% mem_used buf&cached
> > 86.6 0.0 0.6 3.3 0.0 3624.8Mb 263.0Mb
> > SSL TPS: 300
> > SSL throughput: 200Mbps
> >
> > # gprof /usr/bin/curl-loader gmon.out -p | head -20
> > Flat profile:
> > Each sample counts as 0.01 seconds.
> > % cumulative self self total
> > time seconds seconds calls s/call s/call name
> > 39.83 369.71 369.71 207173 0.00 0.00
> > Curl_hash_clean_with_criterium
> > 24.27 594.98 225.27 3198337733 0.00 0.00
> > hostcache_timestamp_remove
> > 13.75 722.59 127.61 4312864 0.00 0.00 Curl_hash_pick
> > 9.19 807.87 85.28 546507 0.00 0.00 Curl_hash_add
> > 6.14 864.88 57.01 1022419684 0.00 0.00
> Curl_str_key_compare
> > 2.06 884.00 19.12 203749 0.00 0.00 create_conn
> > 0.54 888.99 4.99 213306 0.00 0.00 ConnectionStore
> > 0.49 893.50 4.51 7331251 0.00 0.00 dprintf_formatf
> > 0.29 896.16 2.66 1287182329 0.00 0.00
> ares__is_list_empty
> > 0.23 898.28 2.12 locking_function
> > 0.20 900.10 1.82 212210 0.00 0.00 Curl_num_addresses
> > 0.17 901.66 1.56 198385 0.00 0.00
> > curl_multi_remove_handle
> > 0.12 902.81 1.15 411114 0.00 0.00 ares_cancel
> > 0.12 903.95 1.14 3683448 0.00 0.00 multi_runsingle
> > 0.11 904.97 1.03 13664120 0.00 0.00
> > client_tracing_function
> >
> >
> > test 8:
> > enable c-ares;
> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60);
> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0
> >
> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com
> to
> > one IP address
> > number of domain names in URLs: 780,000
> > number of client: 1000
> > CPU usage: 75%
> > cpu% irq% sirq% sys% iowt% mem_used buf&cached
> > 75.2 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb
> > SSL TPS: 380
> > SSL throughput: 220Mbps
> >
> > # gprof /usr/bin/curl-loader gmon.out -p | head -20
> > Flat profile:
> > Each sample counts as 0.01 seconds.
> > % cumulative self self total
> > time seconds seconds calls s/call s/call name
> > 27.58 12.31 12.31 71140 0.00 0.00
> > Curl_hash_clean_with_criterium
> > 19.09 20.83 8.52 176729692 0.00 0.00
> > hostcache_timestamp_remove
> > 10.44 25.49 4.66 1589309 0.00 0.00 Curl_hash_pick
> > 6.85 28.55 3.06 73204 0.00 0.00 create_conn
> > 5.56 31.03 2.48 67620530 0.00 0.00
> Curl_str_key_compare
> > 4.41 33.00 1.97 200684 0.00 0.00 Curl_hash_add
> > 3.56 34.59 1.59 2717956 0.00 0.00 dprintf_formatf
> > 1.72 35.36 0.77 75247 0.00 0.00 ConnectionStore
> > 1.70 36.12 0.76 444342256 0.00 0.00
> ares__is_list_empty
> > 1.23 36.67 0.55 locking_function
> > 1.16 37.19 0.52 143342 0.00 0.00 ares_cancel
> > 0.99 37.63 0.44 72211 0.00 0.00
> > curl_multi_remove_handle
> > 0.74 37.96 0.33 1333371 0.00 0.00 multi_runsingle
> > 0.68 38.27 0.31 101147798 0.00 0.00 addbyter
> > 0.58 38.53 0.26 288737 0.00 0.00
> > curl_multi_socket_action
> >
> >
> >
> > test 9:
> > enable c-ares;
> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1);
> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0
> >
> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com
> to
> > one IP address
> > number of domain names in URLs: 780,000
> > number of client: 1000
> > CPU usage: 75%
> > cpu% irq% sirq% sys% iowt% mem_used buf&cached
> > 75.1 0.0 0.6 4.1 0.0 3545.2Mb 260.1Mb
> > SSL TPS: 380
> > SSL throughput: 200Mbps
> >
> > # gprof /usr/bin/curl-loader gmon.out -p | head -20
> > Flat profile:
> > Each sample counts as 0.01 seconds.
> > % cumulative self self total
> > time seconds seconds calls s/call s/call name
> > 34.00 31.13 31.13 72495 0.00 0.00
> > Curl_hash_clean_with_criterium
> > 23.64 52.78 21.65 386070412 0.00 0.00
> > hostcache_timestamp_remove
> > 13.28 64.94 12.16 1619440 0.00 0.00 Curl_hash_pick
> > 5.92 70.36 5.42 137653129 0.00 0.00
> Curl_str_key_compare
> > 5.50 75.40 5.04 203018 0.00 0.00 Curl_hash_add
> > 4.55 79.57 4.17 74618 0.00 0.00 create_conn
> > 1.70 81.13 1.56 2757148 0.00 0.00 dprintf_formatf
> > 0.98 82.03 0.90 76945 0.00 0.00 ConnectionStore
> > 0.95 82.90 0.87 455775100 0.00 0.00
> ares__is_list_empty
> > 0.57 83.42 0.53 locking_function
> > 0.39 83.78 0.36 146742 0.00 0.00 ares_cancel
> > 0.35 84.10 0.32 102968806 0.00 0.00 addbyter
> > 0.35 84.42 0.32 456587 0.00 0.00 Curl_readwrite
> > 0.35 84.74 0.32 73133 0.00 0.00
> > curl_multi_remove_handle
> > 0.34 85.05 0.31 1360545 0.00 0.00 multi_runsingle
> >
> >
> > 2012/3/6 Robert Iakobashvili <cor...@gm...>
> >>
> >> If it will not be helpful this means that recent
> >> changes in libcurl library have broken DNS interface.
> >>
> >> You can try a previos version and the latest libcurl, like:
> >>
> >> http://curl.haxx.se/download/curl-7.23.0.tar.gz
> >> and
> >> http://curl.haxx.se/downloadcurl-7.24.0.tar.gz
> >>
> >> by:
> >>
> >> 1. downloading a libcurl version and placing it to
> >> curl-loader-0.56/packages ;
> >> 2. correcting the Makefile in curl-loader-0.56
> >> CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24 or CURL_VER:=7.23
> >> 3. make cleanall; make
> >>
> >> Please, send us your observations.
> >> Thanks!
> >> Robert
> >>
> >> On Mon, Mar 5, 2012 at 6:57 PM, Robert Iakobashvili <
> cor...@gm...>
> >> wrote:
> >> > If so, please try to change
> >> >
> >> > loader.c, line 578 which looks like:
> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1);
> >> >
> >> > 1. instead of -1 try to set some value like 60
> >> > 2. make clean; make
> >> >
> >> > Thanks,
> >> > Robert
> >> >
> >> > On Mon, Mar 5, 2012 at 6:19 PM, Fred Huang <di...@gm...> wrote:
> >> >>
> >> >> Looks like the bottleneck comes from libcurl other than c-ares...
> >> >> test 3 (no ares patch applied):
> >> >> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve
> *.com
> >> >> to one IP address
> >> >> number of domain names in URLs: 780,000
> >> >> number of client: 1000
> >> >> CPU usage: 85%
> >> >> cpu% irq% sirq% sys% iowt% mem_used buf&cached
> >> >> 85.6 0.0 0.8 5.9 0.0 3334.3Mb 112.0Mb
> >> >> SSL TPS: 320
> >> >> SSL throughput: 200Mbps
> >> >>
> >> >> # gprof /usr/bin/curl-loader gmon.out -p | head -50
> >> >> Flat profile:
> >> >> Each sample counts as 0.01 seconds.
> >> >> % cumulative self self total
> >> >> time seconds seconds calls s/call s/call name
> >> >> 40.80 120.16 120.16 116079 0.00 0.00
> >> >> Curl_hash_clean_with_criterium
> >> >> 24.46 192.18 72.02 983166323 0.00 0.00
> >> >> hostcache_timestamp_remove
> >> >> 13.30 231.35 39.17 2059341 0.00 0.00 Curl_hash_pick
> >> >> 6.45 250.35 19.00 240087 0.00 0.00 Curl_hash_add
> >> >> 4.81 264.51 14.16 334257102 0.00 0.00
> >> >> Curl_str_key_compare
> >> >> 3.40 274.52 10.01 119514 0.00 0.00 create_conn
> >> >> 0.93 277.25 2.73 4478910 0.00 0.00 dprintf_formatf
> >> >> 0.90 279.91 2.66 122209 0.00 0.00 ConnectionStore
> >> >> 0.40 281.09 1.18
> locking_function
> >> >> 0.27 281.88 0.79 114759 0.00 0.00
> >> >> curl_multi_remove_handle
> >> >> 0.21 282.51 0.63 7993664 0.00 0.00
> >> >> client_tracing_function
> >> >> 0.21 283.12 0.61 122685 0.00 0.00
> >> >> Curl_num_addresses
> >> >> 0.19 283.68 0.56 397028 0.00 0.00
> >> >> curl_multi_socket_action
> >> >> 0.19 284.23 0.55 2036116 0.00 0.00 multi_runsingle
> >> >> 0.17 284.72 0.49 703687 0.00 0.00 Curl_readwrite
> >> >> 0.16 285.19 0.47 id_function
> >> >> 0.15 285.62 0.43 29692029 0.00 0.00
> Curl_socket_check
> >> >> 0.14 286.02 0.40 91973856 0.00 0.00
> Curl_raw_toupper
> >> >> 0.13 286.40 0.38 164664144 0.00 0.00 addbyter
> >> >> 0.13 286.77 0.37 2180797 0.00 0.00 Curl_infof
> >> >> 0.11 287.10 0.33 122675 0.00 0.00
> >> >> Curl_if_is_interface_name
> >> >> 0.10 287.39 0.29 471318 0.00 0.00
> >> >> ossl_connect_common
> >> >> 0.10 287.67 0.28 2113528 0.00 0.00 Curl_raw_equal
> >> >> 0.09 287.93 0.26 8676 0.00 0.02
> >> >> curl_multi_socket_all
> >> >> 0.08 288.17 0.24 1663262 0.00 0.00 Curl_splay
> >> >> 0.08 288.41 0.24 8505 0.00 0.01
> >> >> curl_multi_perform
> >> >> 0.07 288.63 0.22 3114576 0.00 0.00 Curl_pgrsUpdate
> >> >> 0.06 288.80 0.17 119637 0.00 0.00 singleipconnect
> >> >> 0.06 288.97 0.17 7994818 0.00 0.00 Curl_debug
> >> >> 0.05 289.13 0.16 235820 0.00 0.00 event_del
> >> >> 0.05 289.29 0.16 119393071 0.00 0.00 curl_strequal
> >> >> 0.05 289.45 0.16 18535473 0.00 0.00 alloc_addbyter
> >> >> 0.05 289.61 0.16 356074 0.00 0.00 Curl_hash_str
> >> >> 0.05 289.76 0.15 1797233 0.00 0.00 fd_key_compare
> >> >> 0.05 289.91 0.15 234966 0.00 0.00
> >> >> Curl_ssl_getsessionid
> >> >> 0.04 290.04 0.13 585853 0.00 0.00
> >> >> Curl_llist_destroy
> >> >> 0.04 290.16 0.12 240826 0.00 0.00 Curl_resolv
> >> >> 0.03 290.26 0.10 7514328 0.00 0.00 Curl_raw_nequal
> >> >> 0.03 290.36 0.10 2214436 0.00 0.00 Curl_read
> >> >> 0.03 290.46 0.10 116872 0.00 0.00
> >> >> Curl_http_readwrite_headers
> >> >> 0.03 290.56 0.10 ssl_tls_trace
> >> >> 0.03 290.66 0.10 8013097 0.00 0.00 scan_response
> >> >> 0.03 290.75 0.09 29488555 0.00 0.00 curlx_tvdiff
> >> >> 0.03 290.84 0.09 2142468 0.00 0.00 Curl_setopt
> >> >> 0.03 290.93 0.09 476299 0.00 0.00 Curl_expire
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> 2012/3/5 Robert Iakobashvili <cor...@gm...>
> >> >>>
> >> >>> Dear Fred Huang,
> >> >>>
> >> >>> Thanks for your posting.
> >> >>>
> >> >>> Could you please check if this is due to some badness at cares?
> >> >>> Thanks.
> >> >>>
> >> >>> Let's exclude usage of cares based DNS resolving by applying the
> patch
> >> >>> attached
> >> >>> and use instead global DNS cache.
> >> >>>
> >> >>> cd ./curl-loader-0.56
> >> >>> patch -p1 < path-to/disable-cares-use-global-dns.patch
> >> >>> make cleanall; make
> >> >>>
> >> >>> See if it helps.
> >> >>>
> >> >>> Global cache, unfortunately, is not thread safe.
> >> >>>
> >> >>> If it helps, we'll try to restore the previous version of cares.
> >> >>>
> >> >>> Thanks,
> >> >>> Robert
> >> >>>
> >> >>> On Mon, Mar 5, 2012 at 4:28 PM, Fred Huang <di...@gm...>
> wrote:
> >> >>>>
> >> >>>> test 1:
> >> >>>>
> >> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve
> >> >>>> *.com to one IP address
> >> >>>> number of domain names in all https URLs: 780,000
> >> >>>> number of client: 1000
> >> >>>> CPU usage: 78%
> >> >>>> cpu% irq% sirq% sys% iowt% mem_used buf&cached
> >> >>>> 78.8 0.0 0.6 3.8 0.0 3640.9Mb 282.1Mb
> >> >>>> SSL TPS: 300
> >> >>>> SSL throughput: 200Mbps
> >> >>>>
> >> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50
> >> >>>> Flat profile:
> >> >>>> Each sample counts as 0.01 seconds.
> >> >>>> % cumulative self self total
> >> >>>> time seconds seconds calls s/call s/call name
> >> >>>> 38.55 127.10 127.10 130975 0.00 0.00
> >> >>>> Curl_hash_clean_with_criterium
> >> >>>> 24.63 208.32 81.22 1247528407 0.00 0.00
> >> >>>> hostcache_timestamp_remove
> >> >>>> 13.08 251.44 43.12 2791988 0.00 0.00
> Curl_hash_pick
> >> >>>> 7.30 275.51 24.07 352490 0.00 0.00 Curl_hash_add
> >> >>>> 6.01 295.31 19.80 410760770 0.00 0.00
> >> >>>> Curl_str_key_compare
> >> >>>> 2.95 305.04 9.73 131015 0.00 0.00 create_conn
> >> >>>> 0.88 307.94 2.90 4841390 0.00 0.00
> dprintf_formatf
> >> >>>> 0.74 310.38 2.44 136036 0.00 0.00
> ConnectionStore
> >> >>>> 0.48 311.96 1.58 814604911 0.00 0.00
> >> >>>> ares__is_list_empty
> >> >>>> 0.36 313.16 1.20
> >> >>>> locking_function
> >> >>>> 0.27 314.06 0.90 261608 0.00 0.00 ares_cancel
> >> >>>> 0.22 314.79 0.73 127417 0.00 0.00
> >> >>>> curl_multi_remove_handle
> >> >>>> 0.19 315.40 0.62 8906301 0.00 0.00
> >> >>>> client_tracing_function
> >> >>>> 0.17 315.97 0.57 135305 0.00 0.00
> >> >>>> Curl_num_addresses
> >> >>>> 0.17 316.54 0.57 181937907 0.00 0.00 addbyter
> >> >>>> 0.17 317.09 0.55 2374363 0.00 0.00
> multi_runsingle
> >> >>>> 0.16 317.62 0.53 541954 0.00 0.00
> >> >>>> ares__init_list_node
> >> >>>> 0.16 318.14 0.52 494412 0.00 0.00
> >> >>>> curl_multi_socket_action
> >> >>>> 0.14 318.60 0.46 33578897 0.00 0.00
> >> >>>> Curl_socket_check
> >> >>>> 0.13 319.04 0.44 102951052 0.00 0.00
> >> >>>> Curl_raw_toupper
> >> >>>> 0.13 319.48 0.44 787558 0.00 0.00
> Curl_readwrite
> >> >>>> 0.13 319.91 0.44 id_function
> >> >>>> 0.12 320.29 0.38 391390 0.00 0.00 Curl_hash_str
> >> >>>> 0.09 320.58 0.29 9216 0.00 0.00
> >> >>>> curl_multi_perform
> >> >>>> 0.08 320.86 0.28 2350908 0.00 0.00
> Curl_raw_equal
> >> >>>> 0.08 321.14 0.28 2019415 0.00 0.00 Curl_splay
> >> >>>> 0.08 321.40 0.26 520625 0.00 0.00
> >> >>>> ossl_connect_common
> >> >>>> 0.08 321.65 0.25 3532992 0.00 0.00
> Curl_pgrsUpdate
> >> >>>> 0.07 321.87 0.22 131778537 0.00 0.00
> curl_strequal
> >> >>>> 0.05 322.05 0.18 8954118 0.00 0.00 scan_response
> >> >>>> 0.05 322.23 0.18 533644 0.00 0.00
> >> >>>> ares_expand_name
> >> >>>> 0.05 322.40 0.17 2477345 0.00 0.00
> fd_key_compare
> >> >>>> 0.05 322.57 0.17 2429135 0.00 0.00 Curl_infof
> >> >>>> 0.05 322.74 0.17 130966 0.00 0.00
> singleipconnect
> >> >>>> 0.05 322.90 0.16 9292 0.00 0.00
> >> >>>> curl_multi_socket_all
> >> >>>> 0.05 323.06 0.16 270137 0.00 0.00
> >> >>>> ares__get_hostent
> >> >>>> 0.05 323.21 0.15 8943657 0.00 0.00 Curl_debug
> >> >>>> 0.04 323.35 0.14 262493 0.00 0.00
> >> >>>> Curl_ssl_getsessionid
> >> >>>> 0.04 323.48 0.13 566006 0.00 0.00
> socket_callback
> >> >>>> 0.04 323.61 0.13 33983864 0.00 0.00 curlx_tvdiff
> >> >>>> 0.04 323.74 0.13 132853 0.00 0.00
> >> >>>> Curl_http_readwrite_headers
> >> >>>> 0.04 323.86 0.12 344032 0.00 0.00 epoll_del
> >> >>>> 0.04 323.98 0.12 223477 0.00 0.00 Curl_poll
> >> >>>> 0.04 324.10 0.12 8526280 0.00 0.00
> Curl_raw_nequal
> >> >>>> 0.03 324.21 0.11 348839 0.00 0.00 event_del
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> test 2:
> >> >>>>
> >> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve
> >> >>>> *.com to one IP address
> >> >>>> number of domain names in all https URLs: 1
> >> >>>> number of client: 1000
> >> >>>> CPU usage: 75%
> >> >>>> SSL TPS: 1300
> >> >>>> SSL throughput: 700Mbps
> >> >>>>
> >> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50
> >> >>>> Flat profile:
> >> >>>> Each sample counts as 0.01 seconds.
> >> >>>> % cumulative self self total
> >> >>>> time seconds seconds calls s/call s/call name
> >> >>>> 8.98 2.45 2.45 5127312 0.00 0.00
> dprintf_formatf
> >> >>>> 8.90 4.88 2.43 1292796503 0.00 0.00
> >> >>>> ares__is_list_empty
> >> >>>> 6.12 6.55 1.67 322461 0.00 0.00 create_conn
> >> >>>> 4.58 7.80 1.25 419872 0.00 0.00 ares_cancel
> >> >>>> 4.10 8.92 1.12 2100162 0.00 0.00
> Curl_readwrite
> >> >>>> 3.92 9.99 1.07 4527388 0.00 0.00
> Curl_hash_pick
> >> >>>> 3.43 10.93 0.94
> >> >>>> locking_function
> >> >>>> 3.04 11.76 0.83 909759 0.00 0.00
> >> >>>> curl_multi_socket_action
> >> >>>> 3.00 12.58 0.82 97252 0.00 0.00
> >> >>>> Curl_hash_clean_with_criterium
> >> >>>> 2.71 13.32 0.74 179704016 0.00 0.00
> >> >>>> Curl_raw_toupper
> >> >>>> 2.68 14.05 0.73 4646524 0.00 0.00
> multi_runsingle
> >> >>>> 2.44 14.71 0.67 13862992 0.00 0.00
> >> >>>> client_tracing_function
> >> >>>> 2.42 15.37 0.66 319735 0.00 0.00
> >> >>>> curl_multi_remove_handle
> >> >>>> 2.42 16.03 0.66 22 0.03 0.03
> >> >>>> ares__init_list_node
> >> >>>> 1.94 16.56 0.53 9930244 0.00 0.00
> >> >>>> hostcache_timestamp_remove
> >> >>>> 1.94 17.09 0.53 169320088 0.00 0.00 addbyter
> >> >>>> 1.80 17.58 0.49 id_function
> >> >>>> 1.36 17.95 0.37 7057146 0.00 0.00
> Curl_pgrsUpdate
> >> >>>> 1.36 18.32 0.37 16835 0.00 0.00
> >> >>>> curl_multi_socket_all
> >> >>>> 1.25 18.66 0.34 16719 0.00 0.00
> >> >>>> curl_multi_perform
> >> >>>> 1.21 18.99 0.33 329403 0.00 0.00
> >> >>>> Curl_http_readwrit...
[truncated message content] |
|
From: Fred H. <di...@gm...> - 2012-03-06 16:27:00
|
different order if use HTTP URLs: # gprof /usr/bin/curl-loader gmon.out -p | head -20 Flat profile: Each sample counts as 0.01 seconds. % cumulative self self total time seconds seconds calls s/call s/call name 39.57 154.26 154.26 92591 0.00 0.00 Curl_hash_clean_with_criterium 20.06 232.47 78.21 4286341 0.00 0.00 Curl_hash_pick 16.07 295.12 62.65 253498 0.00 0.00 Curl_hash_add 12.55 344.06 48.93 1073699488 0.00 0.00 * hostcache_timestamp_remove* 6.89 370.94 26.88 361026319 0.00 0.00 Curl_str_key_compare 0.38 372.43 1.49 1153627 0.00 0.00 dprintf_formatf 0.37 373.86 1.44 2007540 0.00 0.00 Curl_readwrite 0.31 375.08 1.22 85075 0.00 0.00 Curl_num_addresses 0.25 376.04 0.96 3092327 0.00 0.00 multi_runsingle 0.23 376.92 0.88 263320355 0.00 0.00 ares__is_list_empty 0.20 377.71 0.79 39752 0.00 0.01 curl_multi_socket_all 0.19 378.44 0.73 256897 0.00 0.00 Curl_hash_str 0.17 379.11 0.67 89266 0.00 0.00 ares_cancel 0.15 379.70 0.59 90755 0.00 0.00 Curl_if2ip 0.13 380.21 0.51 3191375 0.00 0.00 client_tracing_function 2012/3/7 Fred Huang <di...@gm...> > 1. the major CPU consumer is curl-loader. here is the top: > top - 07:36:36 up 23:31, 5 users, load average: 4.32, 2.26, 0.91 > Tasks: 237 total, 2 running, 235 sleeping, 0 stopped, 0 zombie > Cpu0 : 76.6%us, 3.6%sy, 0.0%ni, 19.4%id, 0.0%wa, 0.0%hi, 0.3%si, > 0.0% > Cpu1 : 72.4%us, 4.0%sy, 0.0%ni, 22.9%id, 0.0%wa, 0.0%hi, 0.7%si, > 0.0% > Cpu2 : 65.8%us, 3.3%sy, 0.0%ni, 30.6%id, 0.0%wa, 0.0%hi, 0.3%si, > 0.0% > Cpu3 : 75.2%us, 4.3%sy, 0.0%ni, 19.5%id, 0.0%wa, 0.0%hi, 1.0%si, > 0.0% > Cpu4 : 79.1%us, 4.0%sy, 0.0%ni, 16.8%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0% > Cpu5 : 75.8%us, 4.3%sy, 0.0%ni, 19.2%id, 0.0%wa, 0.0%hi, 0.7%si, > 0.0% > Cpu6 : 74.1%us, 5.0%sy, 0.0%ni, 20.3%id, 0.0%wa, 0.0%hi, 0.7%si, > 0.0% > Cpu7 : 78.7%us, 4.0%sy, 0.0%ni, 16.6%id, 0.0%wa, 0.0%hi, 0.7%si, > 0.0% > Mem: 16209932k total, 3829500k used, 12380432k free, 99112k buffers > Swap: 18448376k total, 0k used, 18448376k free, 320284k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 30795 root 20 0 3200m 2.8g 2296 S *79.2* 18.4 21:26.84 curl-loader > 1832 nobody 20 0 13944 1736 580 S 0.1 0.0 0:32.65 dnsmasq > 9 root 20 0 0 0 0 S 0.0 0.0 0:00.53 ksoftirqd/1 > 30825 root 20 0 15220 1360 960 R 0.0 0.0 0:00.42 top > 1 root 20 0 19400 1568 1256 S 0.0 0.0 0:00.75 init > 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd > 3 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0 > 4 root 20 0 0 0 0 S 0.0 0.0 0:00.06 ksoftirqd/0 > 5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0 > > > 2. 1000 loading clients > > 3. yes, tried 0.52 but no lucky -- CAPS 380, throughput 200Mbps, CPU 90% > top - 08:17:10 up 1 day, 12 min, 8 users, load average: 4.66, 3.39, 1.88 > Tasks: 245 total, 2 running, 243 sleeping, 0 stopped, 0 zombie > Cpu0 : 82.2%us, 4.0%sy, 0.0%ni, 13.9%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0% > Cpu1 : 86.1%us, 4.3%sy, 0.0%ni, 7.9%id, 0.0%wa, 0.0%hi, 1.7%si, > 0.0% > Cpu2 : 85.3%us, 4.0%sy, 0.0%ni, 10.0%id, 0.0%wa, 0.0%hi, 0.7%si, > 0.0% > Cpu3 : 81.8%us, 3.0%sy, 0.0%ni, 14.2%id, 0.0%wa, 0.0%hi, 1.0%si, > 0.0% > Cpu4 : 83.7%us, 4.0%sy, 0.0%ni, 12.3%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0% > Cpu5 : 92.7%us, 3.3%sy, 0.0%ni, 3.3%id, 0.0%wa, 0.0%hi, 0.7%si, > 0.0% > Cpu6 : 89.7%us, 2.0%sy, 0.0%ni, 8.0%id, 0.0%wa, 0.0%hi, 0.3%si, > 0.0% > Cpu7 : 88.1%us, 3.6%sy, 0.0%ni, 6.6%id, 0.0%wa, 0.0%hi, 1.7%si, > 0.0% > Mem: 16209932k total, 1795748k used, 14414184k free, 102824k buffers > Swap: 18448376k total, 0k used, 18448376k free, 382988k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 12236 root 20 0 1150m 894m 2572 S 90.7 5.7 51:11.91 curl-loader > 1832 nobody 20 0 13944 1736 580 S 0.1 0.0 0:47.04 dnsmasq > > ]# gprof /usr/bin/curl-loader gmon.out -p | head -20 > Flat profile: > Each sample counts as 0.01 seconds. > % cumulative self self total > time seconds seconds calls s/call s/call name > 43.92 151.95 151.95 157295 0.00 0.00 > Curl_hash_clean_with_criterium > 22.90 231.17 79.22 1712190971 0.00 0.00 > hostcache_timestamp_remove > 13.15 276.68 45.51 3260321 0.00 0.00 Curl_hash_pick > 7.51 302.65 25.97 394171 0.00 0.00 Curl_hash_add > 5.98 323.34 20.68 567437518 0.00 0.00 Curl_str_key_compare > 1.00 326.80 3.46 5380220 0.00 0.00 dprintf_formatf > 0.41 328.21 1.41 locking_function > 0.32 329.32 1.11 469222138 0.00 0.00 ares__is_list_empty > 0.31 330.40 1.08 3284508 0.00 0.00 multi_runsingle > 0.28 331.38 0.98 10006806 0.00 0.00 > client_tracing_function > 0.24 332.21 0.83 151951 0.00 0.00 Curl_num_addresses > 0.22 332.97 0.77 188075672 0.00 0.00 addbyter > 0.19 333.64 0.67 841840 0.00 0.00 Curl_readwrite > 0.19 334.29 0.65 id_function > 0.18 334.90 0.61 39843 0.00 0.01 curl_multi_socket_all > > > > 4. yes, it is correct for 0.56. If use same URL list by replacing https to > http, I can get CAPS 1500 and throughput 870Mbps with CPU% < 60% > > > 2012/3/6 Robert Iakobashvili <cor...@gm...> > >> Several questions: >> >> 1. Could you please run top and see if there are other processes burning >> CPU? >> 2. What is the number of loading clients that you use? >> 3. Could you try curl-loader-0.52 if it compiles for you? >> If not, please try to place curl-loader-7.19.7.tar.gz and >> c-ares-16.0.tar.gz, >> update their versions in Makefile and rebuild by make cleanall; make >> >> 4. Is it correct to say that for HTTP URLs (not HTTPS) you are getting >> a good throughput >> and CAPS with curl-loader-0.56? >> >> Thanks, >> Robert >> >> On Tue, Mar 6, 2012 at 3:52 AM, Fred Huang <di...@gm...> wrote: >> > TIMEOUT -1 -> 60 works a little better but no performance improvement on >> > CAPS or throughput. >> > >> > test 5: >> > enable c-ares; >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); >> > >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> *.com to >> > one IP address >> > number of domain names in URLs: 780,000 >> > number of client: 1000 >> > CPU usage: 74% >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> > 74.0 0.0 0.7 4.2 0.0 3530.6Mb 250.8Mb >> > SSL TPS: 360 >> > SSL throughput: 220Mbps >> > >> > # gprof /usr/bin/curl-loader gmon.out -p | head -50 >> > Flat profile: >> > Each sample counts as 0.01 seconds. >> > % cumulative self self total >> > time seconds seconds calls s/call s/call name >> > 26.63 23.13 23.13 119226 0.00 0.00 >> > Curl_hash_clean_with_criterium >> > 20.34 40.80 17.67 317792146 0.00 0.00 >> > hostcache_timestamp_remove >> > 11.27 50.59 9.79 2579903 0.00 0.00 Curl_hash_pick >> > 7.84 57.40 6.81 120606 0.00 0.00 create_conn >> > 5.65 62.31 4.91 119117401 0.00 0.00 >> Curl_str_key_compare >> > 4.98 66.64 4.33 329829 0.00 0.00 Curl_hash_add >> > 2.80 69.08 2.44 4476606 0.00 0.00 dprintf_formatf >> > 2.57 71.31 2.23 124019 0.00 0.00 ConnectionStore >> > 1.50 72.61 1.30 740997609 0.00 0.00 >> ares__is_list_empty >> > 1.16 73.61 1.01 locking_function >> > 0.76 74.27 0.66 238744 0.00 0.00 ares_cancel >> > 0.70 74.88 0.61 457564 0.00 0.00 >> > curl_multi_socket_action >> > 0.67 75.46 0.58 118616 0.00 0.00 >> > curl_multi_remove_handle >> > 0.58 75.97 0.51 8211502 0.00 0.00 >> > client_tracing_function >> > 0.53 76.43 0.46 733561 0.00 0.00 Curl_readwrite >> > 0.51 76.87 0.44 2183179 0.00 0.00 multi_runsingle >> > 0.51 77.31 0.44 166585567 0.00 0.00 addbyter >> > 0.44 77.69 0.38 2175805 0.00 0.00 Curl_raw_equal >> > 0.43 78.06 0.38 id_function >> > 0.40 78.41 0.35 29789346 0.00 0.00 Curl_socket_check >> > 0.40 78.76 0.35 479671 0.00 0.00 >> ossl_connect_common >> > 0.40 79.11 0.35 490639 0.00 0.00 >> ares__init_list_node >> > 0.33 79.40 0.29 94197670 0.00 0.00 Curl_raw_toupper >> > 0.33 79.69 0.29 3268031 0.00 0.00 Curl_pgrsUpdate >> > 0.28 79.93 0.24 8797 0.00 0.00 curl_multi_perform >> > 0.27 80.17 0.24 2233641 0.00 0.00 Curl_infof >> > 0.26 80.40 0.23 1839837 0.00 0.00 Curl_splay >> > 0.24 80.61 0.21 121713028 0.00 0.00 curl_strequal >> > 0.21 80.79 0.18 243333 0.00 0.00 >> Curl_ssl_getsessionid >> > 0.18 80.95 0.16 122180 0.00 0.00 Curl_num_addresses >> > 0.17 81.09 0.15 8271666 0.00 0.00 Curl_debug >> > 0.16 81.23 0.14 18902452 0.00 0.00 alloc_addbyter >> > 0.16 81.37 0.14 8851 0.00 0.00 >> curl_multi_socket_all >> > 0.14 81.50 0.13 2236769 0.00 0.00 Curl_read >> > 0.14 81.62 0.12 2271700 0.00 0.00 fd_key_compare >> > 0.14 81.74 0.12 323167 0.00 0.00 event_del >> > 0.14 81.86 0.12 205436 0.00 0.00 read_udp_packets >> > 0.14 81.98 0.12 200979 0.00 0.00 processfds >> > 0.13 82.09 0.11 123886 0.00 0.00 >> > Curl_if_is_interface_name >> > 0.13 82.20 0.11 120350 0.00 0.00 Curl_http >> > 0.13 82.31 0.11 ssl_tls_trace >> > 0.12 82.41 0.10 119229 0.00 0.00 >> Curl_splayremovebyaddr >> > 0.12 82.51 0.10 7798717 0.00 0.00 Curl_raw_nequal >> > 0.12 82.61 0.10 784485 0.00 0.00 event_add >> > 0.10 82.70 0.09 643482 0.00 0.00 Curl_expire >> > >> > >> > >> > >> > >> > test 6: >> > enable c-ares; >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0 >> > >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> *.com to >> > one IP address >> > number of domain names in URLs: 780,000 >> > number of client: 1000 >> > CPU usage: 86% >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> > 85.9 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb >> > SSL TPS: 360 >> > SSL throughput: 200Mbps >> > >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> > Flat profile: >> > Each sample counts as 0.01 seconds. >> > % cumulative self self total >> > time seconds seconds calls s/call s/call name >> > 27.19 19.96 19.96 103472 0.00 0.00 >> > Curl_hash_clean_with_criterium >> > 21.73 35.91 15.95 274168981 0.00 0.00 >> > hostcache_timestamp_remove >> > 10.33 43.49 7.58 2256441 0.00 0.00 Curl_hash_pick >> > 7.21 48.78 5.29 104540 0.00 0.00 create_conn >> > 6.20 53.33 4.55 103413405 0.00 0.00 >> Curl_str_key_compare >> > 4.88 56.91 3.58 287109 0.00 0.00 Curl_hash_add >> > 3.02 59.13 2.22 3797855 0.00 0.00 dprintf_formatf >> > 2.10 60.67 1.54 107711 0.00 0.00 ConnectionStore >> > 1.65 61.88 1.21 642725327 0.00 0.00 >> ares__is_list_empty >> > 1.19 62.76 0.88 locking_function >> > 0.87 63.40 0.64 207817 0.00 0.00 ares_cancel >> > 0.56 63.81 0.41 404108 0.00 0.00 >> > curl_multi_socket_action >> > 0.54 64.21 0.40 425603 0.00 0.00 >> ares__init_list_node >> > 0.53 64.60 0.39 7068112 0.00 0.00 >> > client_tracing_function >> > 0.49 64.96 0.36 1908587 0.00 0.00 multi_runsingle >> > >> > >> > >> > test 7: >> > enable c-ares; >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0 >> > >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> *.com to >> > one IP address >> > number of domain names in URLs: 780,000 >> > number of client: 1000 >> > CPU usage: 86% >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> > 86.6 0.0 0.6 3.3 0.0 3624.8Mb 263.0Mb >> > SSL TPS: 300 >> > SSL throughput: 200Mbps >> > >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> > Flat profile: >> > Each sample counts as 0.01 seconds. >> > % cumulative self self total >> > time seconds seconds calls s/call s/call name >> > 39.83 369.71 369.71 207173 0.00 0.00 >> > Curl_hash_clean_with_criterium >> > 24.27 594.98 225.27 3198337733 0.00 0.00 >> > hostcache_timestamp_remove >> > 13.75 722.59 127.61 4312864 0.00 0.00 Curl_hash_pick >> > 9.19 807.87 85.28 546507 0.00 0.00 Curl_hash_add >> > 6.14 864.88 57.01 1022419684 0.00 0.00 >> Curl_str_key_compare >> > 2.06 884.00 19.12 203749 0.00 0.00 create_conn >> > 0.54 888.99 4.99 213306 0.00 0.00 ConnectionStore >> > 0.49 893.50 4.51 7331251 0.00 0.00 dprintf_formatf >> > 0.29 896.16 2.66 1287182329 0.00 0.00 >> ares__is_list_empty >> > 0.23 898.28 2.12 locking_function >> > 0.20 900.10 1.82 212210 0.00 0.00 Curl_num_addresses >> > 0.17 901.66 1.56 198385 0.00 0.00 >> > curl_multi_remove_handle >> > 0.12 902.81 1.15 411114 0.00 0.00 ares_cancel >> > 0.12 903.95 1.14 3683448 0.00 0.00 multi_runsingle >> > 0.11 904.97 1.03 13664120 0.00 0.00 >> > client_tracing_function >> > >> > >> > test 8: >> > enable c-ares; >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0 >> > >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> *.com to >> > one IP address >> > number of domain names in URLs: 780,000 >> > number of client: 1000 >> > CPU usage: 75% >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> > 75.2 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb >> > SSL TPS: 380 >> > SSL throughput: 220Mbps >> > >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> > Flat profile: >> > Each sample counts as 0.01 seconds. >> > % cumulative self self total >> > time seconds seconds calls s/call s/call name >> > 27.58 12.31 12.31 71140 0.00 0.00 >> > Curl_hash_clean_with_criterium >> > 19.09 20.83 8.52 176729692 0.00 0.00 >> > hostcache_timestamp_remove >> > 10.44 25.49 4.66 1589309 0.00 0.00 Curl_hash_pick >> > 6.85 28.55 3.06 73204 0.00 0.00 create_conn >> > 5.56 31.03 2.48 67620530 0.00 0.00 >> Curl_str_key_compare >> > 4.41 33.00 1.97 200684 0.00 0.00 Curl_hash_add >> > 3.56 34.59 1.59 2717956 0.00 0.00 dprintf_formatf >> > 1.72 35.36 0.77 75247 0.00 0.00 ConnectionStore >> > 1.70 36.12 0.76 444342256 0.00 0.00 >> ares__is_list_empty >> > 1.23 36.67 0.55 locking_function >> > 1.16 37.19 0.52 143342 0.00 0.00 ares_cancel >> > 0.99 37.63 0.44 72211 0.00 0.00 >> > curl_multi_remove_handle >> > 0.74 37.96 0.33 1333371 0.00 0.00 multi_runsingle >> > 0.68 38.27 0.31 101147798 0.00 0.00 addbyter >> > 0.58 38.53 0.26 288737 0.00 0.00 >> > curl_multi_socket_action >> > >> > >> > >> > test 9: >> > enable c-ares; >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0 >> > >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> *.com to >> > one IP address >> > number of domain names in URLs: 780,000 >> > number of client: 1000 >> > CPU usage: 75% >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> > 75.1 0.0 0.6 4.1 0.0 3545.2Mb 260.1Mb >> > SSL TPS: 380 >> > SSL throughput: 200Mbps >> > >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> > Flat profile: >> > Each sample counts as 0.01 seconds. >> > % cumulative self self total >> > time seconds seconds calls s/call s/call name >> > 34.00 31.13 31.13 72495 0.00 0.00 >> > Curl_hash_clean_with_criterium >> > 23.64 52.78 21.65 386070412 0.00 0.00 >> > hostcache_timestamp_remove >> > 13.28 64.94 12.16 1619440 0.00 0.00 Curl_hash_pick >> > 5.92 70.36 5.42 137653129 0.00 0.00 >> Curl_str_key_compare >> > 5.50 75.40 5.04 203018 0.00 0.00 Curl_hash_add >> > 4.55 79.57 4.17 74618 0.00 0.00 create_conn >> > 1.70 81.13 1.56 2757148 0.00 0.00 dprintf_formatf >> > 0.98 82.03 0.90 76945 0.00 0.00 ConnectionStore >> > 0.95 82.90 0.87 455775100 0.00 0.00 >> ares__is_list_empty >> > 0.57 83.42 0.53 locking_function >> > 0.39 83.78 0.36 146742 0.00 0.00 ares_cancel >> > 0.35 84.10 0.32 102968806 0.00 0.00 addbyter >> > 0.35 84.42 0.32 456587 0.00 0.00 Curl_readwrite >> > 0.35 84.74 0.32 73133 0.00 0.00 >> > curl_multi_remove_handle >> > 0.34 85.05 0.31 1360545 0.00 0.00 multi_runsingle >> > >> > >> > 2012/3/6 Robert Iakobashvili <cor...@gm...> >> >> >> >> If it will not be helpful this means that recent >> >> changes in libcurl library have broken DNS interface. >> >> >> >> You can try a previos version and the latest libcurl, like: >> >> >> >> http://curl.haxx.se/download/curl-7.23.0.tar.gz >> >> and >> >> http://curl.haxx.se/downloadcurl-7.24.0.tar.gz >> >> >> >> by: >> >> >> >> 1. downloading a libcurl version and placing it to >> >> curl-loader-0.56/packages ; >> >> 2. correcting the Makefile in curl-loader-0.56 >> >> CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24 or CURL_VER:=7.23 >> >> 3. make cleanall; make >> >> >> >> Please, send us your observations. >> >> Thanks! >> >> Robert >> >> >> >> On Mon, Mar 5, 2012 at 6:57 PM, Robert Iakobashvili < >> cor...@gm...> >> >> wrote: >> >> > If so, please try to change >> >> > >> >> > loader.c, line 578 which looks like: >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); >> >> > >> >> > 1. instead of -1 try to set some value like 60 >> >> > 2. make clean; make >> >> > >> >> > Thanks, >> >> > Robert >> >> > >> >> > On Mon, Mar 5, 2012 at 6:19 PM, Fred Huang <di...@gm...> wrote: >> >> >> >> >> >> Looks like the bottleneck comes from libcurl other than c-ares... >> >> >> test 3 (no ares patch applied): >> >> >> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> *.com >> >> >> to one IP address >> >> >> number of domain names in URLs: 780,000 >> >> >> number of client: 1000 >> >> >> CPU usage: 85% >> >> >> cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> >> 85.6 0.0 0.8 5.9 0.0 3334.3Mb 112.0Mb >> >> >> SSL TPS: 320 >> >> >> SSL throughput: 200Mbps >> >> >> >> >> >> # gprof /usr/bin/curl-loader gmon.out -p | head -50 >> >> >> Flat profile: >> >> >> Each sample counts as 0.01 seconds. >> >> >> % cumulative self self total >> >> >> time seconds seconds calls s/call s/call name >> >> >> 40.80 120.16 120.16 116079 0.00 0.00 >> >> >> Curl_hash_clean_with_criterium >> >> >> 24.46 192.18 72.02 983166323 0.00 0.00 >> >> >> hostcache_timestamp_remove >> >> >> 13.30 231.35 39.17 2059341 0.00 0.00 Curl_hash_pick >> >> >> 6.45 250.35 19.00 240087 0.00 0.00 Curl_hash_add >> >> >> 4.81 264.51 14.16 334257102 0.00 0.00 >> >> >> Curl_str_key_compare >> >> >> 3.40 274.52 10.01 119514 0.00 0.00 create_conn >> >> >> 0.93 277.25 2.73 4478910 0.00 0.00 >> dprintf_formatf >> >> >> 0.90 279.91 2.66 122209 0.00 0.00 >> ConnectionStore >> >> >> 0.40 281.09 1.18 >> locking_function >> >> >> 0.27 281.88 0.79 114759 0.00 0.00 >> >> >> curl_multi_remove_handle >> >> >> 0.21 282.51 0.63 7993664 0.00 0.00 >> >> >> client_tracing_function >> >> >> 0.21 283.12 0.61 122685 0.00 0.00 >> >> >> Curl_num_addresses >> >> >> 0.19 283.68 0.56 397028 0.00 0.00 >> >> >> curl_multi_socket_action >> >> >> 0.19 284.23 0.55 2036116 0.00 0.00 >> multi_runsingle >> >> >> 0.17 284.72 0.49 703687 0.00 0.00 Curl_readwrite >> >> >> 0.16 285.19 0.47 id_function >> >> >> 0.15 285.62 0.43 29692029 0.00 0.00 >> Curl_socket_check >> >> >> 0.14 286.02 0.40 91973856 0.00 0.00 >> Curl_raw_toupper >> >> >> 0.13 286.40 0.38 164664144 0.00 0.00 addbyter >> >> >> 0.13 286.77 0.37 2180797 0.00 0.00 Curl_infof >> >> >> 0.11 287.10 0.33 122675 0.00 0.00 >> >> >> Curl_if_is_interface_name >> >> >> 0.10 287.39 0.29 471318 0.00 0.00 >> >> >> ossl_connect_common >> >> >> 0.10 287.67 0.28 2113528 0.00 0.00 Curl_raw_equal >> >> >> 0.09 287.93 0.26 8676 0.00 0.02 >> >> >> curl_multi_socket_all >> >> >> 0.08 288.17 0.24 1663262 0.00 0.00 Curl_splay >> >> >> 0.08 288.41 0.24 8505 0.00 0.01 >> >> >> curl_multi_perform >> >> >> 0.07 288.63 0.22 3114576 0.00 0.00 >> Curl_pgrsUpdate >> >> >> 0.06 288.80 0.17 119637 0.00 0.00 >> singleipconnect >> >> >> 0.06 288.97 0.17 7994818 0.00 0.00 Curl_debug >> >> >> 0.05 289.13 0.16 235820 0.00 0.00 event_del >> >> >> 0.05 289.29 0.16 119393071 0.00 0.00 curl_strequal >> >> >> 0.05 289.45 0.16 18535473 0.00 0.00 alloc_addbyter >> >> >> 0.05 289.61 0.16 356074 0.00 0.00 Curl_hash_str >> >> >> 0.05 289.76 0.15 1797233 0.00 0.00 fd_key_compare >> >> >> 0.05 289.91 0.15 234966 0.00 0.00 >> >> >> Curl_ssl_getsessionid >> >> >> 0.04 290.04 0.13 585853 0.00 0.00 >> >> >> Curl_llist_destroy >> >> >> 0.04 290.16 0.12 240826 0.00 0.00 Curl_resolv >> >> >> 0.03 290.26 0.10 7514328 0.00 0.00 >> Curl_raw_nequal >> >> >> 0.03 290.36 0.10 2214436 0.00 0.00 Curl_read >> >> >> 0.03 290.46 0.10 116872 0.00 0.00 >> >> >> Curl_http_readwrite_headers >> >> >> 0.03 290.56 0.10 ssl_tls_trace >> >> >> 0.03 290.66 0.10 8013097 0.00 0.00 scan_response >> >> >> 0.03 290.75 0.09 29488555 0.00 0.00 curlx_tvdiff >> >> >> 0.03 290.84 0.09 2142468 0.00 0.00 Curl_setopt >> >> >> 0.03 290.93 0.09 476299 0.00 0.00 Curl_expire >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> 2012/3/5 Robert Iakobashvili <cor...@gm...> >> >> >>> >> >> >>> Dear Fred Huang, >> >> >>> >> >> >>> Thanks for your posting. >> >> >>> >> >> >>> Could you please check if this is due to some badness at cares? >> >> >>> Thanks. >> >> >>> >> >> >>> Let's exclude usage of cares based DNS resolving by applying the >> patch >> >> >>> attached >> >> >>> and use instead global DNS cache. >> >> >>> >> >> >>> cd ./curl-loader-0.56 >> >> >>> patch -p1 < path-to/disable-cares-use-global-dns.patch >> >> >>> make cleanall; make >> >> >>> >> >> >>> See if it helps. >> >> >>> >> >> >>> Global cache, unfortunately, is not thread safe. >> >> >>> >> >> >>> If it helps, we'll try to restore the previous version of cares. >> >> >>> >> >> >>> Thanks, >> >> >>> Robert >> >> >>> >> >> >>> On Mon, Mar 5, 2012 at 4:28 PM, Fred Huang <di...@gm...> >> wrote: >> >> >>>> >> >> >>>> test 1: >> >> >>>> >> >> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> >>>> *.com to one IP address >> >> >>>> number of domain names in all https URLs: 780,000 >> >> >>>> number of client: 1000 >> >> >>>> CPU usage: 78% >> >> >>>> cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> >>>> 78.8 0.0 0.6 3.8 0.0 3640.9Mb 282.1Mb >> >> >>>> SSL TPS: 300 >> >> >>>> SSL throughput: 200Mbps >> >> >>>> >> >> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50 >> >> >>>> Flat profile: >> >> >>>> Each sample counts as 0.01 seconds. >> >> >>>> % cumulative self self total >> >> >>>> time seconds seconds calls s/call s/call name >> >> >>>> 38.55 127.10 127.10 130975 0.00 0.00 >> >> >>>> Curl_hash_clean_with_criterium >> >> >>>> 24.63 208.32 81.22 1247528407 0.00 0.00 >> >> >>>> hostcache_timestamp_remove >> >> >>>> 13.08 251.44 43.12 2791988 0.00 0.00 >> Curl_hash_pick >> >> >>>> 7.30 275.51 24.07 352490 0.00 0.00 >> Curl_hash_add >> >> >>>> 6.01 295.31 19.80 410760770 0.00 0.00 >> >> >>>> Curl_str_key_compare >> >> >>>> 2.95 305.04 9.73 131015 0.00 0.00 create_conn >> >> >>>> 0.88 307.94 2.90 4841390 0.00 0.00 >> dprintf_formatf >> >> >>>> 0.74 310.38 2.44 136036 0.00 0.00 >> ConnectionStore >> >> >>>> 0.48 311.96 1.58 814604911 0.00 0.00 >> >> >>>> ares__is_list_empty >> >> >>>> 0.36 313.16 1.20 >> >> >>>> locking_function >> >> >>>> 0.27 314.06 0.90 261608 0.00 0.00 ares_cancel >> >> >>>> 0.22 314.79 0.73 127417 0.00 0.00 >> >> >>>> curl_multi_remove_handle >> >> >>>> 0.19 315.40 0.62 8906301 0.00 0.00 >> >> >>>> client_tracing_function >> >> >>>> 0.17 315.97 0.57 135305 0.00 0.00 >> >> >>>> Curl_num_addresses >> >> >>>> 0.17 316.54 0.57 181937907 0.00 0.00 addbyter >> >> >>>> 0.17 317.09 0.55 2374363 0.00 0.00 >> multi_runsingle >> >> >>>> 0.16 317.62 0.53 541954 0.00 0.00 >> >> >>>> ares__init_list_node >> >> >>>> 0.16 318.14 0.52 494412 0.00 0.00 >> >> >>>> curl_multi_socket_action >> >> >>>> 0.14 318.60 0.46 33578897 0.00 0.00 >> >> >>>> Curl_socket_check >> >> >>>> 0.13 319.04 0.44 102951052 0.00 0.00 >> >> >>>> Curl_raw_toupper >> >> >>>> 0.13 319.48 0.44 787558 0.00 0.00 >> Curl_readwrite >> >> >>>> 0.13 319.91 0.44 id_function >> >> >>>> 0.12 320.29 0.38 391390 0.00 0.00 >> Curl_hash_str >> >> >>>> 0.09 320.58 0.29 9216 0.00 0.00 >> >> >>>> curl_multi_perform >> >> >>>> 0.08 320.86 0.28 2350908 0.00 0.00 >> Curl_raw_equal >> >> >>>> 0.08 321.14 0.28 2019415 0.00 0.00 Curl_splay >> >> >>>> 0.08 321.40 0.26 520625 0.00 0.00 >> >> >>>> ossl_connect_common >> >> >>>> 0.08 321.65 0.25 3532992 0.00 0.00 >> Curl_pgrsUpdate >> >> >>>> 0.07 321.87 0.22 131778537 0.00 0.00 >> curl_strequal >> >> >>>> 0.05 322.05 0.18 8954118 0.00 0.00 >> scan_response >> >> >>>> 0.05 322.23 0.18 533644 0.00 0.00 >> >> >>>> ares_expand_name >> >> >>>> 0.05 322.40 0.17 2477345 0.00 0.00 >> fd_key_compare >> >> >>>> 0.05 322.57 0.17 2429135 0.00 0.00 Curl_infof >> >> >>>> 0.05 322.74 0.17 130966 0.00 0.00 >> singleipconnect >> >> >>>> 0.05 322.90 0.16 9292 0.00 0.00 >> >> >>>> curl_multi_socket_all >> >> >>>> 0.05 323.06 0.16 270137 0.00 0.00 >> >> >>>> ares__get_hostent >> >> >>>> 0.05 323.21 0.15 8943657 0.00 0.00 Curl_debug >> >> >>>> 0.04 323.35 0.14 262493 0.00 0.00 >> >> >>>> Curl_ssl_getsessionid >> >> >>>> 0.04 323.48 0.13 566006 0.00 0.00 >> socket_callback >> >> >>>> 0.04 323.61 0.13 33983864 0.00 0.00 curlx_tvdiff >> >> >>>> 0.04 323.74 0.13 132853 0.00 0.00 >> >> >>>> Curl_http_readwrite_headers >> >> >>>> 0.04 323.86 0.12 344032 0.00 0.00 epoll_del >> >> >>>> 0.04 323.98 0.12 223477 0.00 0.00 Curl_poll >> >> >>>> 0.04 324.10 0.12 8526280 0.00 0.00 >> Curl_raw_nequal >> >> >>>> 0.03 324.21 0.11 348839 0.00 0.00 event_del >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> test 2: >> >> >>>> >> >> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> >>>> *.com to one IP address >> >> >>>> number of domain names in all https URLs: 1 >> >> >>>> number of client: 1000 >> >> >>>> CPU usage: 75% >> >> >>>> SSL TPS: 1300 >> >> >>>> SSL... [truncated message content] |
|
From: Robert I. <cor...@gm...> - 2012-03-06 16:40:51
|
Dear Fred, If I understand correctly your observations, the issue is not in: 1. DNS resolving; 2. Version of cares or libcurl; but rather in Less effective handling of HTTPS versus HTTP in libcurl. Please, confirm that my understanding is correct. HTTPS URLs are using an extra locking in openssl with -t option that could be making the issue. If you are using -t option, try to load without it and let's say with 300 clients. If you are not using -t, thus the only way to proceed is to share your observations with libcurl people at libcurl development <cur...@co...> Thank you. Best wishes. Robert On Tue, Mar 6, 2012 at 6:17 PM, Fred Huang <di...@gm...> wrote: > 1. the major CPU consumer is curl-loader. here is the top: > top - 07:36:36 up 23:31, 5 users, load average: 4.32, 2.26, 0.91 > Tasks: 237 total, 2 running, 235 sleeping, 0 stopped, 0 zombie > Cpu0 : 76.6%us, 3.6%sy, 0.0%ni, 19.4%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0% > Cpu1 : 72.4%us, 4.0%sy, 0.0%ni, 22.9%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0% > Cpu2 : 65.8%us, 3.3%sy, 0.0%ni, 30.6%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0% > Cpu3 : 75.2%us, 4.3%sy, 0.0%ni, 19.5%id, 0.0%wa, 0.0%hi, 1.0%si, 0.0% > Cpu4 : 79.1%us, 4.0%sy, 0.0%ni, 16.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0% > Cpu5 : 75.8%us, 4.3%sy, 0.0%ni, 19.2%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0% > Cpu6 : 74.1%us, 5.0%sy, 0.0%ni, 20.3%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0% > Cpu7 : 78.7%us, 4.0%sy, 0.0%ni, 16.6%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0% > Mem: 16209932k total, 3829500k used, 12380432k free, 99112k buffers > Swap: 18448376k total, 0k used, 18448376k free, 320284k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 30795 root 20 0 3200m 2.8g 2296 S 79.2 18.4 21:26.84 curl-loader > 1832 nobody 20 0 13944 1736 580 S 0.1 0.0 0:32.65 dnsmasq > 9 root 20 0 0 0 0 S 0.0 0.0 0:00.53 ksoftirqd/1 > 30825 root 20 0 15220 1360 960 R 0.0 0.0 0:00.42 top > 1 root 20 0 19400 1568 1256 S 0.0 0.0 0:00.75 init > 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd > 3 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0 > 4 root 20 0 0 0 0 S 0.0 0.0 0:00.06 ksoftirqd/0 > 5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0 > > > 2. 1000 loading clients > > 3. yes, tried 0.52 but no lucky -- CAPS 380, throughput 200Mbps, CPU 90% > top - 08:17:10 up 1 day, 12 min, 8 users, load average: 4.66, 3.39, 1.88 > Tasks: 245 total, 2 running, 243 sleeping, 0 stopped, 0 zombie > Cpu0 : 82.2%us, 4.0%sy, 0.0%ni, 13.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0% > Cpu1 : 86.1%us, 4.3%sy, 0.0%ni, 7.9%id, 0.0%wa, 0.0%hi, 1.7%si, 0.0% > Cpu2 : 85.3%us, 4.0%sy, 0.0%ni, 10.0%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0% > Cpu3 : 81.8%us, 3.0%sy, 0.0%ni, 14.2%id, 0.0%wa, 0.0%hi, 1.0%si, 0.0% > Cpu4 : 83.7%us, 4.0%sy, 0.0%ni, 12.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0% > Cpu5 : 92.7%us, 3.3%sy, 0.0%ni, 3.3%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0% > Cpu6 : 89.7%us, 2.0%sy, 0.0%ni, 8.0%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0% > Cpu7 : 88.1%us, 3.6%sy, 0.0%ni, 6.6%id, 0.0%wa, 0.0%hi, 1.7%si, 0.0% > Mem: 16209932k total, 1795748k used, 14414184k free, 102824k buffers > Swap: 18448376k total, 0k used, 18448376k free, 382988k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 12236 root 20 0 1150m 894m 2572 S 90.7 5.7 51:11.91 curl-loader > 1832 nobody 20 0 13944 1736 580 S 0.1 0.0 0:47.04 dnsmasq > > ]# gprof /usr/bin/curl-loader gmon.out -p | head -20 > Flat profile: > Each sample counts as 0.01 seconds. > % cumulative self self total > time seconds seconds calls s/call s/call name > 43.92 151.95 151.95 157295 0.00 0.00 > Curl_hash_clean_with_criterium > 22.90 231.17 79.22 1712190971 0.00 0.00 > hostcache_timestamp_remove > 13.15 276.68 45.51 3260321 0.00 0.00 Curl_hash_pick > 7.51 302.65 25.97 394171 0.00 0.00 Curl_hash_add > 5.98 323.34 20.68 567437518 0.00 0.00 Curl_str_key_compare > 1.00 326.80 3.46 5380220 0.00 0.00 dprintf_formatf > 0.41 328.21 1.41 locking_function > 0.32 329.32 1.11 469222138 0.00 0.00 ares__is_list_empty > 0.31 330.40 1.08 3284508 0.00 0.00 multi_runsingle > 0.28 331.38 0.98 10006806 0.00 0.00 > client_tracing_function > 0.24 332.21 0.83 151951 0.00 0.00 Curl_num_addresses > 0.22 332.97 0.77 188075672 0.00 0.00 addbyter > 0.19 333.64 0.67 841840 0.00 0.00 Curl_readwrite > 0.19 334.29 0.65 id_function > 0.18 334.90 0.61 39843 0.00 0.01 curl_multi_socket_all > > > > 4. yes, it is correct for 0.56. If use same URL list by replacing https to > http, I can get CAPS 1500 and throughput 870Mbps with CPU% < 60% > > > 2012/3/6 Robert Iakobashvili <cor...@gm...> >> >> Several questions: >> >> 1. Could you please run top and see if there are other processes burning >> CPU? >> 2. What is the number of loading clients that you use? >> 3. Could you try curl-loader-0.52 if it compiles for you? >> If not, please try to place curl-loader-7.19.7.tar.gz and >> c-ares-16.0.tar.gz, >> update their versions in Makefile and rebuild by make cleanall; make >> >> 4. Is it correct to say that for HTTP URLs (not HTTPS) you are getting >> a good throughput >> and CAPS with curl-loader-0.56? >> >> Thanks, >> Robert >> >> On Tue, Mar 6, 2012 at 3:52 AM, Fred Huang <di...@gm...> wrote: >> > TIMEOUT -1 -> 60 works a little better but no performance improvement on >> > CAPS or throughput. >> > >> > test 5: >> > enable c-ares; >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); >> > >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com >> > to >> > one IP address >> > number of domain names in URLs: 780,000 >> > number of client: 1000 >> > CPU usage: 74% >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> > 74.0 0.0 0.7 4.2 0.0 3530.6Mb 250.8Mb >> > SSL TPS: 360 >> > SSL throughput: 220Mbps >> > >> > # gprof /usr/bin/curl-loader gmon.out -p | head -50 >> > Flat profile: >> > Each sample counts as 0.01 seconds. >> > % cumulative self self total >> > time seconds seconds calls s/call s/call name >> > 26.63 23.13 23.13 119226 0.00 0.00 >> > Curl_hash_clean_with_criterium >> > 20.34 40.80 17.67 317792146 0.00 0.00 >> > hostcache_timestamp_remove >> > 11.27 50.59 9.79 2579903 0.00 0.00 Curl_hash_pick >> > 7.84 57.40 6.81 120606 0.00 0.00 create_conn >> > 5.65 62.31 4.91 119117401 0.00 0.00 >> > Curl_str_key_compare >> > 4.98 66.64 4.33 329829 0.00 0.00 Curl_hash_add >> > 2.80 69.08 2.44 4476606 0.00 0.00 dprintf_formatf >> > 2.57 71.31 2.23 124019 0.00 0.00 ConnectionStore >> > 1.50 72.61 1.30 740997609 0.00 0.00 >> > ares__is_list_empty >> > 1.16 73.61 1.01 locking_function >> > 0.76 74.27 0.66 238744 0.00 0.00 ares_cancel >> > 0.70 74.88 0.61 457564 0.00 0.00 >> > curl_multi_socket_action >> > 0.67 75.46 0.58 118616 0.00 0.00 >> > curl_multi_remove_handle >> > 0.58 75.97 0.51 8211502 0.00 0.00 >> > client_tracing_function >> > 0.53 76.43 0.46 733561 0.00 0.00 Curl_readwrite >> > 0.51 76.87 0.44 2183179 0.00 0.00 multi_runsingle >> > 0.51 77.31 0.44 166585567 0.00 0.00 addbyter >> > 0.44 77.69 0.38 2175805 0.00 0.00 Curl_raw_equal >> > 0.43 78.06 0.38 id_function >> > 0.40 78.41 0.35 29789346 0.00 0.00 Curl_socket_check >> > 0.40 78.76 0.35 479671 0.00 0.00 >> > ossl_connect_common >> > 0.40 79.11 0.35 490639 0.00 0.00 >> > ares__init_list_node >> > 0.33 79.40 0.29 94197670 0.00 0.00 Curl_raw_toupper >> > 0.33 79.69 0.29 3268031 0.00 0.00 Curl_pgrsUpdate >> > 0.28 79.93 0.24 8797 0.00 0.00 curl_multi_perform >> > 0.27 80.17 0.24 2233641 0.00 0.00 Curl_infof >> > 0.26 80.40 0.23 1839837 0.00 0.00 Curl_splay >> > 0.24 80.61 0.21 121713028 0.00 0.00 curl_strequal >> > 0.21 80.79 0.18 243333 0.00 0.00 >> > Curl_ssl_getsessionid >> > 0.18 80.95 0.16 122180 0.00 0.00 Curl_num_addresses >> > 0.17 81.09 0.15 8271666 0.00 0.00 Curl_debug >> > 0.16 81.23 0.14 18902452 0.00 0.00 alloc_addbyter >> > 0.16 81.37 0.14 8851 0.00 0.00 >> > curl_multi_socket_all >> > 0.14 81.50 0.13 2236769 0.00 0.00 Curl_read >> > 0.14 81.62 0.12 2271700 0.00 0.00 fd_key_compare >> > 0.14 81.74 0.12 323167 0.00 0.00 event_del >> > 0.14 81.86 0.12 205436 0.00 0.00 read_udp_packets >> > 0.14 81.98 0.12 200979 0.00 0.00 processfds >> > 0.13 82.09 0.11 123886 0.00 0.00 >> > Curl_if_is_interface_name >> > 0.13 82.20 0.11 120350 0.00 0.00 Curl_http >> > 0.13 82.31 0.11 ssl_tls_trace >> > 0.12 82.41 0.10 119229 0.00 0.00 >> > Curl_splayremovebyaddr >> > 0.12 82.51 0.10 7798717 0.00 0.00 Curl_raw_nequal >> > 0.12 82.61 0.10 784485 0.00 0.00 event_add >> > 0.10 82.70 0.09 643482 0.00 0.00 Curl_expire >> > >> > >> > >> > >> > >> > test 6: >> > enable c-ares; >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0 >> > >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com >> > to >> > one IP address >> > number of domain names in URLs: 780,000 >> > number of client: 1000 >> > CPU usage: 86% >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> > 85.9 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb >> > SSL TPS: 360 >> > SSL throughput: 200Mbps >> > >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> > Flat profile: >> > Each sample counts as 0.01 seconds. >> > % cumulative self self total >> > time seconds seconds calls s/call s/call name >> > 27.19 19.96 19.96 103472 0.00 0.00 >> > Curl_hash_clean_with_criterium >> > 21.73 35.91 15.95 274168981 0.00 0.00 >> > hostcache_timestamp_remove >> > 10.33 43.49 7.58 2256441 0.00 0.00 Curl_hash_pick >> > 7.21 48.78 5.29 104540 0.00 0.00 create_conn >> > 6.20 53.33 4.55 103413405 0.00 0.00 >> > Curl_str_key_compare >> > 4.88 56.91 3.58 287109 0.00 0.00 Curl_hash_add >> > 3.02 59.13 2.22 3797855 0.00 0.00 dprintf_formatf >> > 2.10 60.67 1.54 107711 0.00 0.00 ConnectionStore >> > 1.65 61.88 1.21 642725327 0.00 0.00 >> > ares__is_list_empty >> > 1.19 62.76 0.88 locking_function >> > 0.87 63.40 0.64 207817 0.00 0.00 ares_cancel >> > 0.56 63.81 0.41 404108 0.00 0.00 >> > curl_multi_socket_action >> > 0.54 64.21 0.40 425603 0.00 0.00 >> > ares__init_list_node >> > 0.53 64.60 0.39 7068112 0.00 0.00 >> > client_tracing_function >> > 0.49 64.96 0.36 1908587 0.00 0.00 multi_runsingle >> > >> > >> > >> > test 7: >> > enable c-ares; >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0 >> > >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com >> > to >> > one IP address >> > number of domain names in URLs: 780,000 >> > number of client: 1000 >> > CPU usage: 86% >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> > 86.6 0.0 0.6 3.3 0.0 3624.8Mb 263.0Mb >> > SSL TPS: 300 >> > SSL throughput: 200Mbps >> > >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> > Flat profile: >> > Each sample counts as 0.01 seconds. >> > % cumulative self self total >> > time seconds seconds calls s/call s/call name >> > 39.83 369.71 369.71 207173 0.00 0.00 >> > Curl_hash_clean_with_criterium >> > 24.27 594.98 225.27 3198337733 0.00 0.00 >> > hostcache_timestamp_remove >> > 13.75 722.59 127.61 4312864 0.00 0.00 Curl_hash_pick >> > 9.19 807.87 85.28 546507 0.00 0.00 Curl_hash_add >> > 6.14 864.88 57.01 1022419684 0.00 0.00 >> > Curl_str_key_compare >> > 2.06 884.00 19.12 203749 0.00 0.00 create_conn >> > 0.54 888.99 4.99 213306 0.00 0.00 ConnectionStore >> > 0.49 893.50 4.51 7331251 0.00 0.00 dprintf_formatf >> > 0.29 896.16 2.66 1287182329 0.00 0.00 >> > ares__is_list_empty >> > 0.23 898.28 2.12 locking_function >> > 0.20 900.10 1.82 212210 0.00 0.00 Curl_num_addresses >> > 0.17 901.66 1.56 198385 0.00 0.00 >> > curl_multi_remove_handle >> > 0.12 902.81 1.15 411114 0.00 0.00 ares_cancel >> > 0.12 903.95 1.14 3683448 0.00 0.00 multi_runsingle >> > 0.11 904.97 1.03 13664120 0.00 0.00 >> > client_tracing_function >> > >> > >> > test 8: >> > enable c-ares; >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0 >> > >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com >> > to >> > one IP address >> > number of domain names in URLs: 780,000 >> > number of client: 1000 >> > CPU usage: 75% >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> > 75.2 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb >> > SSL TPS: 380 >> > SSL throughput: 220Mbps >> > >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> > Flat profile: >> > Each sample counts as 0.01 seconds. >> > % cumulative self self total >> > time seconds seconds calls s/call s/call name >> > 27.58 12.31 12.31 71140 0.00 0.00 >> > Curl_hash_clean_with_criterium >> > 19.09 20.83 8.52 176729692 0.00 0.00 >> > hostcache_timestamp_remove >> > 10.44 25.49 4.66 1589309 0.00 0.00 Curl_hash_pick >> > 6.85 28.55 3.06 73204 0.00 0.00 create_conn >> > 5.56 31.03 2.48 67620530 0.00 0.00 >> > Curl_str_key_compare >> > 4.41 33.00 1.97 200684 0.00 0.00 Curl_hash_add >> > 3.56 34.59 1.59 2717956 0.00 0.00 dprintf_formatf >> > 1.72 35.36 0.77 75247 0.00 0.00 ConnectionStore >> > 1.70 36.12 0.76 444342256 0.00 0.00 >> > ares__is_list_empty >> > 1.23 36.67 0.55 locking_function >> > 1.16 37.19 0.52 143342 0.00 0.00 ares_cancel >> > 0.99 37.63 0.44 72211 0.00 0.00 >> > curl_multi_remove_handle >> > 0.74 37.96 0.33 1333371 0.00 0.00 multi_runsingle >> > 0.68 38.27 0.31 101147798 0.00 0.00 addbyter >> > 0.58 38.53 0.26 288737 0.00 0.00 >> > curl_multi_socket_action >> > >> > >> > >> > test 9: >> > enable c-ares; >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0 >> > >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com >> > to >> > one IP address >> > number of domain names in URLs: 780,000 >> > number of client: 1000 >> > CPU usage: 75% >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> > 75.1 0.0 0.6 4.1 0.0 3545.2Mb 260.1Mb >> > SSL TPS: 380 >> > SSL throughput: 200Mbps >> > >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> > Flat profile: >> > Each sample counts as 0.01 seconds. >> > % cumulative self self total >> > time seconds seconds calls s/call s/call name >> > 34.00 31.13 31.13 72495 0.00 0.00 >> > Curl_hash_clean_with_criterium >> > 23.64 52.78 21.65 386070412 0.00 0.00 >> > hostcache_timestamp_remove >> > 13.28 64.94 12.16 1619440 0.00 0.00 Curl_hash_pick >> > 5.92 70.36 5.42 137653129 0.00 0.00 >> > Curl_str_key_compare >> > 5.50 75.40 5.04 203018 0.00 0.00 Curl_hash_add >> > 4.55 79.57 4.17 74618 0.00 0.00 create_conn >> > 1.70 81.13 1.56 2757148 0.00 0.00 dprintf_formatf >> > 0.98 82.03 0.90 76945 0.00 0.00 ConnectionStore >> > 0.95 82.90 0.87 455775100 0.00 0.00 >> > ares__is_list_empty >> > 0.57 83.42 0.53 locking_function >> > 0.39 83.78 0.36 146742 0.00 0.00 ares_cancel >> > 0.35 84.10 0.32 102968806 0.00 0.00 addbyter >> > 0.35 84.42 0.32 456587 0.00 0.00 Curl_readwrite >> > 0.35 84.74 0.32 73133 0.00 0.00 >> > curl_multi_remove_handle >> > 0.34 85.05 0.31 1360545 0.00 0.00 multi_runsingle >> > >> > >> > 2012/3/6 Robert Iakobashvili <cor...@gm...> >> >> >> >> If it will not be helpful this means that recent >> >> changes in libcurl library have broken DNS interface. >> >> >> >> You can try a previos version and the latest libcurl, like: >> >> >> >> http://curl.haxx.se/download/curl-7.23.0.tar.gz >> >> and >> >> http://curl.haxx.se/downloadcurl-7.24.0.tar.gz >> >> >> >> by: >> >> >> >> 1. downloading a libcurl version and placing it to >> >> curl-loader-0.56/packages ; >> >> 2. correcting the Makefile in curl-loader-0.56 >> >> CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24 or CURL_VER:=7.23 >> >> 3. make cleanall; make >> >> >> >> Please, send us your observations. >> >> Thanks! >> >> Robert >> >> >> >> On Mon, Mar 5, 2012 at 6:57 PM, Robert Iakobashvili >> >> <cor...@gm...> >> >> wrote: >> >> > If so, please try to change >> >> > >> >> > loader.c, line 578 which looks like: >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); >> >> > >> >> > 1. instead of -1 try to set some value like 60 >> >> > 2. make clean; make >> >> > >> >> > Thanks, >> >> > Robert >> >> > >> >> > On Mon, Mar 5, 2012 at 6:19 PM, Fred Huang <di...@gm...> wrote: >> >> >> >> >> >> Looks like the bottleneck comes from libcurl other than c-ares... >> >> >> test 3 (no ares patch applied): >> >> >> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> >> *.com >> >> >> to one IP address >> >> >> number of domain names in URLs: 780,000 >> >> >> number of client: 1000 >> >> >> CPU usage: 85% >> >> >> cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> >> 85.6 0.0 0.8 5.9 0.0 3334.3Mb 112.0Mb >> >> >> SSL TPS: 320 >> >> >> SSL throughput: 200Mbps >> >> >> >> >> >> # gprof /usr/bin/curl-loader gmon.out -p | head -50 >> >> >> Flat profile: >> >> >> Each sample counts as 0.01 seconds. >> >> >> % cumulative self self total >> >> >> time seconds seconds calls s/call s/call name >> >> >> 40.80 120.16 120.16 116079 0.00 0.00 >> >> >> Curl_hash_clean_with_criterium >> >> >> 24.46 192.18 72.02 983166323 0.00 0.00 >> >> >> hostcache_timestamp_remove >> >> >> 13.30 231.35 39.17 2059341 0.00 0.00 Curl_hash_pick >> >> >> 6.45 250.35 19.00 240087 0.00 0.00 Curl_hash_add >> >> >> 4.81 264.51 14.16 334257102 0.00 0.00 >> >> >> Curl_str_key_compare >> >> >> 3.40 274.52 10.01 119514 0.00 0.00 create_conn >> >> >> 0.93 277.25 2.73 4478910 0.00 0.00 >> >> >> dprintf_formatf >> >> >> 0.90 279.91 2.66 122209 0.00 0.00 >> >> >> ConnectionStore >> >> >> 0.40 281.09 1.18 >> >> >> locking_function >> >> >> 0.27 281.88 0.79 114759 0.00 0.00 >> >> >> curl_multi_remove_handle >> >> >> 0.21 282.51 0.63 7993664 0.00 0.00 >> >> >> client_tracing_function >> >> >> 0.21 283.12 0.61 122685 0.00 0.00 >> >> >> Curl_num_addresses >> >> >> 0.19 283.68 0.56 397028 0.00 0.00 >> >> >> curl_multi_socket_action >> >> >> 0.19 284.23 0.55 2036116 0.00 0.00 >> >> >> multi_runsingle >> >> >> 0.17 284.72 0.49 703687 0.00 0.00 Curl_readwrite >> >> >> 0.16 285.19 0.47 id_function >> >> >> 0.15 285.62 0.43 29692029 0.00 0.00 >> >> >> Curl_socket_check >> >> >> 0.14 286.02 0.40 91973856 0.00 0.00 >> >> >> Curl_raw_toupper >> >> >> 0.13 286.40 0.38 164664144 0.00 0.00 addbyter >> >> >> 0.13 286.77 0.37 2180797 0.00 0.00 Curl_infof >> >> >> 0.11 287.10 0.33 122675 0.00 0.00 >> >> >> Curl_if_is_interface_name >> >> >> 0.10 287.39 0.29 471318 0.00 0.00 >> >> >> ossl_connect_common >> >> >> 0.10 287.67 0.28 2113528 0.00 0.00 Curl_raw_equal >> >> >> 0.09 287.93 0.26 8676 0.00 0.02 >> >> >> curl_multi_socket_all >> >> >> 0.08 288.17 0.24 1663262 0.00 0.00 Curl_splay >> >> >> 0.08 288.41 0.24 8505 0.00 0.01 >> >> >> curl_multi_perform >> >> >> 0.07 288.63 0.22 3114576 0.00 0.00 >> >> >> Curl_pgrsUpdate >> >> >> 0.06 288.80 0.17 119637 0.00 0.00 >> >> >> singleipconnect >> >> >> 0.06 288.97 0.17 7994818 0.00 0.00 Curl_debug >> >> >> 0.05 289.13 0.16 235820 0.00 0.00 event_del >> >> >> 0.05 289.29 0.16 119393071 0.00 0.00 curl_strequal >> >> >> 0.05 289.45 0.16 18535473 0.00 0.00 alloc_addbyter >> >> >> 0.05 289.61 0.16 356074 0.00 0.00 Curl_hash_str >> >> >> 0.05 289.76 0.15 1797233 0.00 0.00 fd_key_compare >> >> >> 0.05 289.91 0.15 234966 0.00 0.00 >> >> >> Curl_ssl_getsessionid >> >> >> 0.04 290.04 0.13 585853 0.00 0.00 >> >> >> Curl_llist_destroy >> >> >> 0.04 290.16 0.12 240826 0.00 0.00 Curl_resolv >> >> >> 0.03 290.26 0.10 7514328 0.00 0.00 >> >> >> Curl_raw_nequal >> >> >> 0.03 290.36 0.10 2214436 0.00 0.00 Curl_read >> >> >> 0.03 290.46 0.10 116872 0.00 0.00 >> >> >> Curl_http_readwrite_headers >> >> >> 0.03 290.56 0.10 ssl_tls_trace >> >> >> 0.03 290.66 0.10 8013097 0.00 0.00 scan_response >> >> >> 0.03 290.75 0.09 29488555 0.00 0.00 curlx_tvdiff >> >> >> 0.03 290.84 0.09 2142468 0.00 0.00 Curl_setopt >> >> >> 0.03 290.93 0.09 476299 0.00 0.00 Curl_expire >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> 2012/3/5 Robert Iakobashvili <cor...@gm...> >> >> >>> >> >> >>> Dear Fred Huang, >> >> >>> >> >> >>> Thanks for your posting. >> >> >>> >> >> >>> Could you please check if this is due to some badness at cares? >> >> >>> Thanks. >> >> >>> >> >> >>> Let's exclude usage of cares based DNS resolving by applying the >> >> >>> patch >> >> >>> attached >> >> >>> and use instead global DNS cache. >> >> >>> >> >> >>> cd ./curl-loader-0.56 >> >> >>> patch -p1 < path-to/disable-cares-use-global-dns.patch >> >> >>> make cleanall; make >> >> >>> >> >> >>> See if it helps. >> >> >>> >> >> >>> Global cache, unfortunately, is not thread safe. >> >> >>> >> >> >>> If it helps, we'll try to restore the previous version of cares. >> >> >>> >> >> >>> Thanks, >> >> >>> Robert >> >> >>> >> >> >>> On Mon, Mar 5, 2012 at 4:28 PM, Fred Huang <di...@gm...> >> >> >>> wrote: >> >> >>>> >> >> >>>> test 1: >> >> >>>> >> >> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> >>>> *.com to one IP address >> >> >>>> number of domain names in all https URLs: 780,000 >> >> >>>> number of client: 1000 >> >> >>>> CPU usage: 78% >> >> >>>> cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> >>>> 78.8 0.0 0.6 3.8 0.0 3640.9Mb 282.1Mb >> >> >>>> SSL TPS: 300 >> >> >>>> SSL throughput: 200Mbps >> >> >>>> >> >> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50 >> >> >>>> Flat profile: >> >> >>>> Each sample counts as 0.01 seconds. >> >> >>>> % cumulative self self total >> >> >>>> time seconds seconds calls s/call s/call name >> >> >>>> 38.55 127.10 127.10 130975 0.00 0.00 >> >> >>>> Curl_hash_clean_with_criterium >> >> >>>> 24.63 208.32 81.22 1247528407 0.00 0.00 >> >> >>>> hostcache_timestamp_remove >> >> >>>> 13.08 251.44 43.12 2791988 0.00 0.00 >> >> >>>> Curl_hash_pick >> >> >>>> 7.30 275.51 24.07 352490 0.00 0.00 >> >> >>>> Curl_hash_add >> >> >>>> 6.01 295.31 19.80 410760770 0.00 0.00 >> >> >>>> Curl_str_key_compare >> >> >>>> 2.95 305.04 9.73 131015 0.00 0.00 create_conn >> >> >>>> 0.88 307.94 2.90 4841390 0.00 0.00 >> >> >>>> dprintf_formatf >> >> >>>> 0.74 310.38 2.44 136036 0.00 0.00 >> >> >>>> ConnectionStore >> >> >>>> 0.48 311.96 1.58 814604911 0.00 0.00 >> >> >>>> ares__is_list_empty >> >> >>>> 0.36 313.16 1.20 >> >> >>>> locking_function >> >> >>>> 0.27 314.06 0.90 261608 0.00 0.00 ares_cancel >> >> >>>> 0.22 314.79 0.73 127417 0.00 0.00 >> >> >>>> curl_multi_remove_handle >> >> >>>> 0.19 315.40 0.62 8906301 0.00 0.00 >> >> >>>> client_tracing_function >> >> >>>> 0.17 315.97 0.57 135305 0.00 0.00 >> >> >>>> Curl_num_addresses >> >> >>>> 0.17 316.54 0.57 181937907 0.00 0.00 addbyter >> >> >>>> 0.17 317.09 0.55 2374363 0.00 0.00 >> >> >>>> multi_runsingle >> >> >>>> 0.16 317.62 0.53 541954 0.00 0.00 >> >> >>>> ares__init_list_node >> >> >>>> 0.16 318.14 0.52 494412 0.00 0.00 >> >> >>>> curl_multi_socket_action >> >> >>>> 0.14 318.60 0.46 33578897 0.00 0.00 >> >> >>>> Curl_socket_check >> >> >>>> 0.13 319.04 0.44 102951052 0.00 0.00 >> >> >>>> Curl_raw_toupper >> >> >>>> 0.13 319.48 0.44 787558 0.00 0.00 >> >> >>>> Curl_readwrite >> >> >>>> 0.13 319.91 0.44 id_function >> >> >>>> 0.12 320.29 0.38 391390 0.00 0.00 >> >> >>>> Curl_hash_str >> >> >>>> 0.09 320.58 0.29 9216 0.00 0.00 >> >> >>>> curl_multi_perform >> >> >>>> 0.08 320.86 0.28 2350908 0.00 0.00 >> >> >>>> Curl_raw_equal >> >> >>>> 0.08 321.14 0.28 2019415 0.00 0.00 Curl_splay >> >> >>>> 0.08 321.40 0.26 520625 0.00 0.00 >> >> >>>> ossl_connect_common >> >> >>>> 0.08 321.65 0.25 3532992 0.00 0.00 >> >> >>>> Curl_pgrsUpdate >> >> >>>> 0.07 321.87 0.22 131778537 0.00 0.00 >> >> >>>> curl_strequal >> >> >>>> 0.05 322.05 0.18 8954118 0.00 0.00 >> >> >>>> scan_response >> >> >>>> 0.05 322.23 0.18 533644 0.00 0.00 >> >> >>>> ares_expand_name >> >> >>>> 0.05 322.40 0.17 2477345 0.00 0.00 >> >> >>>> fd_key_compare >> >> >>>> 0.05 322.57 0.17 2429135 0.00 0.00 Curl_infof >> >> >>>> 0.05 322.74 0.17 130966 0.00 0.00 >> >> >>>> singleipconnect >> >> >>>> 0.05 322.90 0.16 9292 0.00 0.00 >> >> >>>> curl_multi_socket_all >> >> >>>> 0.05 323.06 0.16 270137 0.00 0.00 >> >> >>>> ares__get_hostent >> >> >>>> 0.05 323.21 0.15 8943657 0.00 0.00 Curl_debug >> >> >>>> 0.04 323.35 0.14 262493 0.00 0.00 >> >> >>>> Curl_ssl_getsessionid >> >> >>>> 0.04 323.48 0.13 566006 0.00 0.00 >> >> >>>> socket_callback >> >> >>>> 0.04 323.61 0.13 33983864 0.00 0.00 curlx_tvdiff >> >> >>>> 0.04 323.74 0.13 132853 0.00 0.00 >> >> >>>> Curl_http_readwrite_headers >> >> >>>> 0.04 323.86 0.12 344032 0.00 0.00 epoll_del >> >> >>>> 0.04 323.98 0.12 223477 0.00 0.00 Curl_poll >> >> >>>> 0.04 324.10 0.12 8526280 0.00 0.00 >> >> >>>> Curl_raw_nequal >> >> >>>> 0.03 324.21 0.11 348839 0.00 0.00 event_del >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> test 2: >> >> >>>> >> >> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> >>>> *.com to one IP address >> >> >>>> number of domain names in all https URLs: 1 >> >> >>>> number of client: 1000 >> >> >>>> CPU usage: 75% >> >> >>>> SSL TPS: 1300 >> >> >>>> SSL throughput: 700Mbps >> >> >>>> >> >> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50 >> >> >>>> Flat profile: >> >> >>>> Each sample counts as 0.01 seconds. >> >> >>>> % cumulative self self total >> >> >>>> time seconds seconds calls s/call s/call name >> >> >>>> 8.98 2.45 2.45 5127312 0.00 0.00 >> >> >>>> dprintf_formatf >> >> >>>> 8.90 4.88 2.43 1292796503 0.00 0.00 >> >> >>>> ares__is_list... [truncated message content] |
|
From: Fred H. <di...@gm...> - 2012-03-07 00:44:00
|
yes, I use -t 8 option because the test box has 8 logic CPUs (Dell 990, Intel Core i7 2600, 16GB memory). if I do not use -t option, the CPU used by curl-loader will be 100% on one of 8 logic CPU (12% of total CPU), and the CAPS drops to 100: # gprof /usr/bin/curl-loader gmon.out -p | head -20 Flat profile: Each sample counts as 0.01 seconds. % cumulative self self total time seconds seconds calls s/call s/call name 34.12 2.74 2.74 17385 0.00 0.00 create_conn 11.21 3.64 0.90 30998583 0.00 0.00 hostcache_timestamp_remove 9.84 4.43 0.79 13433 0.00 0.00 Curl_hash_clean_with_criterium 7.97 5.07 0.64 381818 0.00 0.00 Curl_hash_pick 5.85 5.54 0.47 17385 0.00 0.00 ConnectionStore 5.11 5.95 0.41 16385 0.00 0.00 curl_multi_remove_handle 4.11 6.28 0.33 16509889 0.00 0.00 Curl_str_key_compare 3.36 6.55 0.27 23959914 0.00 0.00 Curl_socket_check 3.24 6.81 0.26 56703 0.00 0.00 Curl_hash_add 2.62 7.02 0.21 635750 0.00 0.00 dprintf_formatf 1.25 7.12 0.10 92421190 0.00 0.00 ares__is_list_empty 0.75 7.18 0.06 97957 0.00 0.00 Curl_readwrite 0.62 7.23 0.05 23422260 0.00 0.00 addbyter 0.62 7.28 0.05 1136775 0.00 0.00 client_tracing_function 0.50 7.32 0.04 3091375 0.00 0.00 ares__init_list_head I also tried to use *curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 20);* but get no improvement on CAPS. Regards, Fred 2012/3/7 Robert Iakobashvili <cor...@gm...> > Dear Fred, > > If I understand correctly your observations, the issue is not in: > > 1. DNS resolving; > 2. Version of cares or libcurl; > > but rather in > > Less effective handling of HTTPS versus HTTP in libcurl. > > Please, confirm that my understanding is correct. > > > HTTPS URLs are using an extra locking in openssl with -t option > that could be making the issue. > If you are using -t option, try to load without it and let's say with > 300 clients. > > If you are not using -t, thus the only way to proceed is to share > your observations with libcurl people at > > libcurl development <cur...@co...> > > Thank you. > > Best wishes. > Robert > > On Tue, Mar 6, 2012 at 6:17 PM, Fred Huang <di...@gm...> wrote: > > 1. the major CPU consumer is curl-loader. here is the top: > > top - 07:36:36 up 23:31, 5 users, load average: 4.32, 2.26, 0.91 > > Tasks: 237 total, 2 running, 235 sleeping, 0 stopped, 0 zombie > > Cpu0 : 76.6%us, 3.6%sy, 0.0%ni, 19.4%id, 0.0%wa, 0.0%hi, 0.3%si, > 0.0% > > Cpu1 : 72.4%us, 4.0%sy, 0.0%ni, 22.9%id, 0.0%wa, 0.0%hi, 0.7%si, > 0.0% > > Cpu2 : 65.8%us, 3.3%sy, 0.0%ni, 30.6%id, 0.0%wa, 0.0%hi, 0.3%si, > 0.0% > > Cpu3 : 75.2%us, 4.3%sy, 0.0%ni, 19.5%id, 0.0%wa, 0.0%hi, 1.0%si, > 0.0% > > Cpu4 : 79.1%us, 4.0%sy, 0.0%ni, 16.8%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0% > > Cpu5 : 75.8%us, 4.3%sy, 0.0%ni, 19.2%id, 0.0%wa, 0.0%hi, 0.7%si, > 0.0% > > Cpu6 : 74.1%us, 5.0%sy, 0.0%ni, 20.3%id, 0.0%wa, 0.0%hi, 0.7%si, > 0.0% > > Cpu7 : 78.7%us, 4.0%sy, 0.0%ni, 16.6%id, 0.0%wa, 0.0%hi, 0.7%si, > 0.0% > > Mem: 16209932k total, 3829500k used, 12380432k free, 99112k buffers > > Swap: 18448376k total, 0k used, 18448376k free, 320284k cached > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > 30795 root 20 0 3200m 2.8g 2296 S 79.2 18.4 21:26.84 curl-loader > > 1832 nobody 20 0 13944 1736 580 S 0.1 0.0 0:32.65 dnsmasq > > 9 root 20 0 0 0 0 S 0.0 0.0 0:00.53 ksoftirqd/1 > > 30825 root 20 0 15220 1360 960 R 0.0 0.0 0:00.42 top > > 1 root 20 0 19400 1568 1256 S 0.0 0.0 0:00.75 init > > 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd > > 3 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0 > > 4 root 20 0 0 0 0 S 0.0 0.0 0:00.06 ksoftirqd/0 > > 5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0 > > > > > > 2. 1000 loading clients > > > > 3. yes, tried 0.52 but no lucky -- CAPS 380, throughput 200Mbps, CPU 90% > > top - 08:17:10 up 1 day, 12 min, 8 users, load average: 4.66, 3.39, > 1.88 > > Tasks: 245 total, 2 running, 243 sleeping, 0 stopped, 0 zombie > > Cpu0 : 82.2%us, 4.0%sy, 0.0%ni, 13.9%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0% > > Cpu1 : 86.1%us, 4.3%sy, 0.0%ni, 7.9%id, 0.0%wa, 0.0%hi, 1.7%si, > 0.0% > > Cpu2 : 85.3%us, 4.0%sy, 0.0%ni, 10.0%id, 0.0%wa, 0.0%hi, 0.7%si, > 0.0% > > Cpu3 : 81.8%us, 3.0%sy, 0.0%ni, 14.2%id, 0.0%wa, 0.0%hi, 1.0%si, > 0.0% > > Cpu4 : 83.7%us, 4.0%sy, 0.0%ni, 12.3%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0% > > Cpu5 : 92.7%us, 3.3%sy, 0.0%ni, 3.3%id, 0.0%wa, 0.0%hi, 0.7%si, > 0.0% > > Cpu6 : 89.7%us, 2.0%sy, 0.0%ni, 8.0%id, 0.0%wa, 0.0%hi, 0.3%si, > 0.0% > > Cpu7 : 88.1%us, 3.6%sy, 0.0%ni, 6.6%id, 0.0%wa, 0.0%hi, 1.7%si, > 0.0% > > Mem: 16209932k total, 1795748k used, 14414184k free, 102824k buffers > > Swap: 18448376k total, 0k used, 18448376k free, 382988k cached > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > 12236 root 20 0 1150m 894m 2572 S 90.7 5.7 51:11.91 curl-loader > > 1832 nobody 20 0 13944 1736 580 S 0.1 0.0 0:47.04 dnsmasq > > > > ]# gprof /usr/bin/curl-loader gmon.out -p | head -20 > > Flat profile: > > Each sample counts as 0.01 seconds. > > % cumulative self self total > > time seconds seconds calls s/call s/call name > > 43.92 151.95 151.95 157295 0.00 0.00 > > Curl_hash_clean_with_criterium > > 22.90 231.17 79.22 1712190971 0.00 0.00 > > hostcache_timestamp_remove > > 13.15 276.68 45.51 3260321 0.00 0.00 Curl_hash_pick > > 7.51 302.65 25.97 394171 0.00 0.00 Curl_hash_add > > 5.98 323.34 20.68 567437518 0.00 0.00 > Curl_str_key_compare > > 1.00 326.80 3.46 5380220 0.00 0.00 dprintf_formatf > > 0.41 328.21 1.41 locking_function > > 0.32 329.32 1.11 469222138 0.00 0.00 > ares__is_list_empty > > 0.31 330.40 1.08 3284508 0.00 0.00 multi_runsingle > > 0.28 331.38 0.98 10006806 0.00 0.00 > > client_tracing_function > > 0.24 332.21 0.83 151951 0.00 0.00 Curl_num_addresses > > 0.22 332.97 0.77 188075672 0.00 0.00 addbyter > > 0.19 333.64 0.67 841840 0.00 0.00 Curl_readwrite > > 0.19 334.29 0.65 id_function > > 0.18 334.90 0.61 39843 0.00 0.01 > curl_multi_socket_all > > > > > > > > 4. yes, it is correct for 0.56. If use same URL list by replacing https > to > > http, I can get CAPS 1500 and throughput 870Mbps with CPU% < 60% > > > > > > 2012/3/6 Robert Iakobashvili <cor...@gm...> > >> > >> Several questions: > >> > >> 1. Could you please run top and see if there are other processes burning > >> CPU? > >> 2. What is the number of loading clients that you use? > >> 3. Could you try curl-loader-0.52 if it compiles for you? > >> If not, please try to place curl-loader-7.19.7.tar.gz and > >> c-ares-16.0.tar.gz, > >> update their versions in Makefile and rebuild by make cleanall; make > >> > >> 4. Is it correct to say that for HTTP URLs (not HTTPS) you are getting > >> a good throughput > >> and CAPS with curl-loader-0.56? > >> > >> Thanks, > >> Robert > >> > >> On Tue, Mar 6, 2012 at 3:52 AM, Fred Huang <di...@gm...> wrote: > >> > TIMEOUT -1 -> 60 works a little better but no performance improvement > on > >> > CAPS or throughput. > >> > > >> > test 5: > >> > enable c-ares; > >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); > >> > > >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve > *.com > >> > to > >> > one IP address > >> > number of domain names in URLs: 780,000 > >> > number of client: 1000 > >> > CPU usage: 74% > >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached > >> > 74.0 0.0 0.7 4.2 0.0 3530.6Mb 250.8Mb > >> > SSL TPS: 360 > >> > SSL throughput: 220Mbps > >> > > >> > # gprof /usr/bin/curl-loader gmon.out -p | head -50 > >> > Flat profile: > >> > Each sample counts as 0.01 seconds. > >> > % cumulative self self total > >> > time seconds seconds calls s/call s/call name > >> > 26.63 23.13 23.13 119226 0.00 0.00 > >> > Curl_hash_clean_with_criterium > >> > 20.34 40.80 17.67 317792146 0.00 0.00 > >> > hostcache_timestamp_remove > >> > 11.27 50.59 9.79 2579903 0.00 0.00 Curl_hash_pick > >> > 7.84 57.40 6.81 120606 0.00 0.00 create_conn > >> > 5.65 62.31 4.91 119117401 0.00 0.00 > >> > Curl_str_key_compare > >> > 4.98 66.64 4.33 329829 0.00 0.00 Curl_hash_add > >> > 2.80 69.08 2.44 4476606 0.00 0.00 dprintf_formatf > >> > 2.57 71.31 2.23 124019 0.00 0.00 ConnectionStore > >> > 1.50 72.61 1.30 740997609 0.00 0.00 > >> > ares__is_list_empty > >> > 1.16 73.61 1.01 locking_function > >> > 0.76 74.27 0.66 238744 0.00 0.00 ares_cancel > >> > 0.70 74.88 0.61 457564 0.00 0.00 > >> > curl_multi_socket_action > >> > 0.67 75.46 0.58 118616 0.00 0.00 > >> > curl_multi_remove_handle > >> > 0.58 75.97 0.51 8211502 0.00 0.00 > >> > client_tracing_function > >> > 0.53 76.43 0.46 733561 0.00 0.00 Curl_readwrite > >> > 0.51 76.87 0.44 2183179 0.00 0.00 multi_runsingle > >> > 0.51 77.31 0.44 166585567 0.00 0.00 addbyter > >> > 0.44 77.69 0.38 2175805 0.00 0.00 Curl_raw_equal > >> > 0.43 78.06 0.38 id_function > >> > 0.40 78.41 0.35 29789346 0.00 0.00 > Curl_socket_check > >> > 0.40 78.76 0.35 479671 0.00 0.00 > >> > ossl_connect_common > >> > 0.40 79.11 0.35 490639 0.00 0.00 > >> > ares__init_list_node > >> > 0.33 79.40 0.29 94197670 0.00 0.00 Curl_raw_toupper > >> > 0.33 79.69 0.29 3268031 0.00 0.00 Curl_pgrsUpdate > >> > 0.28 79.93 0.24 8797 0.00 0.00 > curl_multi_perform > >> > 0.27 80.17 0.24 2233641 0.00 0.00 Curl_infof > >> > 0.26 80.40 0.23 1839837 0.00 0.00 Curl_splay > >> > 0.24 80.61 0.21 121713028 0.00 0.00 curl_strequal > >> > 0.21 80.79 0.18 243333 0.00 0.00 > >> > Curl_ssl_getsessionid > >> > 0.18 80.95 0.16 122180 0.00 0.00 > Curl_num_addresses > >> > 0.17 81.09 0.15 8271666 0.00 0.00 Curl_debug > >> > 0.16 81.23 0.14 18902452 0.00 0.00 alloc_addbyter > >> > 0.16 81.37 0.14 8851 0.00 0.00 > >> > curl_multi_socket_all > >> > 0.14 81.50 0.13 2236769 0.00 0.00 Curl_read > >> > 0.14 81.62 0.12 2271700 0.00 0.00 fd_key_compare > >> > 0.14 81.74 0.12 323167 0.00 0.00 event_del > >> > 0.14 81.86 0.12 205436 0.00 0.00 read_udp_packets > >> > 0.14 81.98 0.12 200979 0.00 0.00 processfds > >> > 0.13 82.09 0.11 123886 0.00 0.00 > >> > Curl_if_is_interface_name > >> > 0.13 82.20 0.11 120350 0.00 0.00 Curl_http > >> > 0.13 82.31 0.11 ssl_tls_trace > >> > 0.12 82.41 0.10 119229 0.00 0.00 > >> > Curl_splayremovebyaddr > >> > 0.12 82.51 0.10 7798717 0.00 0.00 Curl_raw_nequal > >> > 0.12 82.61 0.10 784485 0.00 0.00 event_add > >> > 0.10 82.70 0.09 643482 0.00 0.00 Curl_expire > >> > > >> > > >> > > >> > > >> > > >> > test 6: > >> > enable c-ares; > >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); > >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0 > >> > > >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve > *.com > >> > to > >> > one IP address > >> > number of domain names in URLs: 780,000 > >> > number of client: 1000 > >> > CPU usage: 86% > >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached > >> > 85.9 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb > >> > SSL TPS: 360 > >> > SSL throughput: 200Mbps > >> > > >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 > >> > Flat profile: > >> > Each sample counts as 0.01 seconds. > >> > % cumulative self self total > >> > time seconds seconds calls s/call s/call name > >> > 27.19 19.96 19.96 103472 0.00 0.00 > >> > Curl_hash_clean_with_criterium > >> > 21.73 35.91 15.95 274168981 0.00 0.00 > >> > hostcache_timestamp_remove > >> > 10.33 43.49 7.58 2256441 0.00 0.00 Curl_hash_pick > >> > 7.21 48.78 5.29 104540 0.00 0.00 create_conn > >> > 6.20 53.33 4.55 103413405 0.00 0.00 > >> > Curl_str_key_compare > >> > 4.88 56.91 3.58 287109 0.00 0.00 Curl_hash_add > >> > 3.02 59.13 2.22 3797855 0.00 0.00 dprintf_formatf > >> > 2.10 60.67 1.54 107711 0.00 0.00 ConnectionStore > >> > 1.65 61.88 1.21 642725327 0.00 0.00 > >> > ares__is_list_empty > >> > 1.19 62.76 0.88 locking_function > >> > 0.87 63.40 0.64 207817 0.00 0.00 ares_cancel > >> > 0.56 63.81 0.41 404108 0.00 0.00 > >> > curl_multi_socket_action > >> > 0.54 64.21 0.40 425603 0.00 0.00 > >> > ares__init_list_node > >> > 0.53 64.60 0.39 7068112 0.00 0.00 > >> > client_tracing_function > >> > 0.49 64.96 0.36 1908587 0.00 0.00 multi_runsingle > >> > > >> > > >> > > >> > test 7: > >> > enable c-ares; > >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); > >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0 > >> > > >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve > *.com > >> > to > >> > one IP address > >> > number of domain names in URLs: 780,000 > >> > number of client: 1000 > >> > CPU usage: 86% > >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached > >> > 86.6 0.0 0.6 3.3 0.0 3624.8Mb 263.0Mb > >> > SSL TPS: 300 > >> > SSL throughput: 200Mbps > >> > > >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 > >> > Flat profile: > >> > Each sample counts as 0.01 seconds. > >> > % cumulative self self total > >> > time seconds seconds calls s/call s/call name > >> > 39.83 369.71 369.71 207173 0.00 0.00 > >> > Curl_hash_clean_with_criterium > >> > 24.27 594.98 225.27 3198337733 0.00 0.00 > >> > hostcache_timestamp_remove > >> > 13.75 722.59 127.61 4312864 0.00 0.00 Curl_hash_pick > >> > 9.19 807.87 85.28 546507 0.00 0.00 Curl_hash_add > >> > 6.14 864.88 57.01 1022419684 0.00 0.00 > >> > Curl_str_key_compare > >> > 2.06 884.00 19.12 203749 0.00 0.00 create_conn > >> > 0.54 888.99 4.99 213306 0.00 0.00 ConnectionStore > >> > 0.49 893.50 4.51 7331251 0.00 0.00 dprintf_formatf > >> > 0.29 896.16 2.66 1287182329 0.00 0.00 > >> > ares__is_list_empty > >> > 0.23 898.28 2.12 locking_function > >> > 0.20 900.10 1.82 212210 0.00 0.00 > Curl_num_addresses > >> > 0.17 901.66 1.56 198385 0.00 0.00 > >> > curl_multi_remove_handle > >> > 0.12 902.81 1.15 411114 0.00 0.00 ares_cancel > >> > 0.12 903.95 1.14 3683448 0.00 0.00 multi_runsingle > >> > 0.11 904.97 1.03 13664120 0.00 0.00 > >> > client_tracing_function > >> > > >> > > >> > test 8: > >> > enable c-ares; > >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); > >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0 > >> > > >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve > *.com > >> > to > >> > one IP address > >> > number of domain names in URLs: 780,000 > >> > number of client: 1000 > >> > CPU usage: 75% > >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached > >> > 75.2 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb > >> > SSL TPS: 380 > >> > SSL throughput: 220Mbps > >> > > >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 > >> > Flat profile: > >> > Each sample counts as 0.01 seconds. > >> > % cumulative self self total > >> > time seconds seconds calls s/call s/call name > >> > 27.58 12.31 12.31 71140 0.00 0.00 > >> > Curl_hash_clean_with_criterium > >> > 19.09 20.83 8.52 176729692 0.00 0.00 > >> > hostcache_timestamp_remove > >> > 10.44 25.49 4.66 1589309 0.00 0.00 Curl_hash_pick > >> > 6.85 28.55 3.06 73204 0.00 0.00 create_conn > >> > 5.56 31.03 2.48 67620530 0.00 0.00 > >> > Curl_str_key_compare > >> > 4.41 33.00 1.97 200684 0.00 0.00 Curl_hash_add > >> > 3.56 34.59 1.59 2717956 0.00 0.00 dprintf_formatf > >> > 1.72 35.36 0.77 75247 0.00 0.00 ConnectionStore > >> > 1.70 36.12 0.76 444342256 0.00 0.00 > >> > ares__is_list_empty > >> > 1.23 36.67 0.55 locking_function > >> > 1.16 37.19 0.52 143342 0.00 0.00 ares_cancel > >> > 0.99 37.63 0.44 72211 0.00 0.00 > >> > curl_multi_remove_handle > >> > 0.74 37.96 0.33 1333371 0.00 0.00 multi_runsingle > >> > 0.68 38.27 0.31 101147798 0.00 0.00 addbyter > >> > 0.58 38.53 0.26 288737 0.00 0.00 > >> > curl_multi_socket_action > >> > > >> > > >> > > >> > test 9: > >> > enable c-ares; > >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); > >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0 > >> > > >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve > *.com > >> > to > >> > one IP address > >> > number of domain names in URLs: 780,000 > >> > number of client: 1000 > >> > CPU usage: 75% > >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached > >> > 75.1 0.0 0.6 4.1 0.0 3545.2Mb 260.1Mb > >> > SSL TPS: 380 > >> > SSL throughput: 200Mbps > >> > > >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 > >> > Flat profile: > >> > Each sample counts as 0.01 seconds. > >> > % cumulative self self total > >> > time seconds seconds calls s/call s/call name > >> > 34.00 31.13 31.13 72495 0.00 0.00 > >> > Curl_hash_clean_with_criterium > >> > 23.64 52.78 21.65 386070412 0.00 0.00 > >> > hostcache_timestamp_remove > >> > 13.28 64.94 12.16 1619440 0.00 0.00 Curl_hash_pick > >> > 5.92 70.36 5.42 137653129 0.00 0.00 > >> > Curl_str_key_compare > >> > 5.50 75.40 5.04 203018 0.00 0.00 Curl_hash_add > >> > 4.55 79.57 4.17 74618 0.00 0.00 create_conn > >> > 1.70 81.13 1.56 2757148 0.00 0.00 dprintf_formatf > >> > 0.98 82.03 0.90 76945 0.00 0.00 ConnectionStore > >> > 0.95 82.90 0.87 455775100 0.00 0.00 > >> > ares__is_list_empty > >> > 0.57 83.42 0.53 locking_function > >> > 0.39 83.78 0.36 146742 0.00 0.00 ares_cancel > >> > 0.35 84.10 0.32 102968806 0.00 0.00 addbyter > >> > 0.35 84.42 0.32 456587 0.00 0.00 Curl_readwrite > >> > 0.35 84.74 0.32 73133 0.00 0.00 > >> > curl_multi_remove_handle > >> > 0.34 85.05 0.31 1360545 0.00 0.00 multi_runsingle > >> > > >> > > >> > 2012/3/6 Robert Iakobashvili <cor...@gm...> > >> >> > >> >> If it will not be helpful this means that recent > >> >> changes in libcurl library have broken DNS interface. > >> >> > >> >> You can try a previos version and the latest libcurl, like: > >> >> > >> >> http://curl.haxx.se/download/curl-7.23.0.tar.gz > >> >> and > >> >> http://curl.haxx.se/downloadcurl-7.24.0.tar.gz > >> >> > >> >> by: > >> >> > >> >> 1. downloading a libcurl version and placing it to > >> >> curl-loader-0.56/packages ; > >> >> 2. correcting the Makefile in curl-loader-0.56 > >> >> CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24 or CURL_VER:=7.23 > >> >> 3. make cleanall; make > >> >> > >> >> Please, send us your observations. > >> >> Thanks! > >> >> Robert > >> >> > >> >> On Mon, Mar 5, 2012 at 6:57 PM, Robert Iakobashvili > >> >> <cor...@gm...> > >> >> wrote: > >> >> > If so, please try to change > >> >> > > >> >> > loader.c, line 578 which looks like: > >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); > >> >> > > >> >> > 1. instead of -1 try to set some value like 60 > >> >> > 2. make clean; make > >> >> > > >> >> > Thanks, > >> >> > Robert > >> >> > > >> >> > On Mon, Mar 5, 2012 at 6:19 PM, Fred Huang <di...@gm...> > wrote: > >> >> >> > >> >> >> Looks like the bottleneck comes from libcurl other than c-ares... > >> >> >> test 3 (no ares patch applied): > >> >> >> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve > >> >> >> *.com > >> >> >> to one IP address > >> >> >> number of domain names in URLs: 780,000 > >> >> >> number of client: 1000 > >> >> >> CPU usage: 85% > >> >> >> cpu% irq% sirq% sys% iowt% mem_used buf&cached > >> >> >> 85.6 0.0 0.8 5.9 0.0 3334.3Mb 112.0Mb > >> >> >> SSL TPS: 320 > >> >> >> SSL throughput: 200Mbps > >> >> >> > >> >> >> # gprof /usr/bin/curl-loader gmon.out -p | head -50 > >> >> >> Flat profile: > >> >> >> Each sample counts as 0.01 seconds. > >> >> >> % cumulative self self total > >> >> >> time seconds seconds calls s/call s/call name > >> >> >> 40.80 120.16 120.16 116079 0.00 0.00 > >> >> >> Curl_hash_clean_with_criterium > >> >> >> 24.46 192.18 72.02 983166323 0.00 0.00 > >> >> >> hostcache_timestamp_remove > >> >> >> 13.30 231.35 39.17 2059341 0.00 0.00 > Curl_hash_pick > >> >> >> 6.45 250.35 19.00 240087 0.00 0.00 > Curl_hash_add > >> >> >> 4.81 264.51 14.16 334257102 0.00 0.00 > >> >> >> Curl_str_key_compare > >> >> >> 3.40 274.52 10.01 119514 0.00 0.00 create_conn > >> >> >> 0.93 277.25 2.73 4478910 0.00 0.00 > >> >> >> dprintf_formatf > >> >> >> 0.90 279.91 2.66 122209 0.00 0.00 > >> >> >> ConnectionStore > >> >> >> 0.40 281.09 1.18 > >> >> >> locking_function > >> >> >> 0.27 281.88 0.79 114759 0.00 0.00 > >> >> >> curl_multi_remove_handle > >> >> >> 0.21 282.51 0.63 7993664 0.00 0.00 > >> >> >> client_tracing_function > >> >> >> 0.21 283.12 0.61 122685 0.00 0.00 > >> >> >> Curl_num_addresses > >> >> >> 0.19 283.68 0.56 397028 0.00 0.00 > >> >> >> curl_multi_socket_action > >> >> >> 0.19 284.23 0.55 2036116 0.00 0.00 > >> >> >> multi_runsingle > >> >> >> 0.17 284.72 0.49 703687 0.00 0.00 > Curl_readwrite > >> >> >> 0.16 285.19 0.47 id_function > >> >> >> 0.15 285.62 0.43 29692029 0.00 0.00 > >> >> >> Curl_socket_check > >> >> >> 0.14 286.02 0.40 91973856 0.00 0.00 > >> >> >> Curl_raw_toupper > >> >> >> 0.13 286.40 0.38 164664144 0.00 0.00 addbyter > >> >> >> 0.13 286.77 0.37 2180797 0.00 0.00 Curl_infof > >> >> >> 0.11 287.10 0.33 122675 0.00 0.00 > >> >> >> Curl_if_is_interface_name > >> >> >> 0.10 287.39 0.29 471318 0.00 0.00 > >> >> >> ossl_connect_common > >> >> >> 0.10 287.67 0.28 2113528 0.00 0.00 > Curl_raw_equal > >> >> >> 0.09 287.93 0.26 8676 0.00 0.02 > >> >> >> curl_multi_socket_all > >> >> >> 0.08 288.17 0.24 1663262 0.00 0.00 Curl_splay > >> >> >> 0.08 288.41 0.24 8505 0.00 0.01 > >> >> >> curl_multi_perform > >> >> >> 0.07 288.63 0.22 3114576 0.00 0.00 > >> >> >> Curl_pgrsUpdate > >> >> >> 0.06 288.80 0.17 119637 0.00 0.00 > >> >> >> singleipconnect > >> >> >> 0.06 288.97 0.17 7994818 0.00 0.00 Curl_debug > >> >> >> 0.05 289.13 0.16 235820 0.00 0.00 event_del > >> >> >> 0.05 289.29 0.16 119393071 0.00 0.00 > curl_strequal > >> >> >> 0.05 289.45 0.16 18535473 0.00 0.00 > alloc_addbyter > >> >> >> 0.05 289.61 0.16 356074 0.00 0.00 > Curl_hash_str > >> >> >> 0.05 289.76 0.15 1797233 0.00 0.00 > fd_key_compare > >> >> >> 0.05 289.91 0.15 234966 0.00 0.00 > >> >> >> Curl_ssl_getsessionid > >> >> >> 0.04 290.04 0.13 585853 0.00 0.00 > >> >> >> Curl_llist_destroy > >> >> >> 0.04 290.16 0.12 240826 0.00 0.00 Curl_resolv > >> >> >> 0.03 290.26 0.10 7514328 0.00 0.00 > >> >> >> Curl_raw_nequal > >> >> >> 0.03 290.36 0.10 2214436 0.00 0.00 Curl_read > >> >> >> 0.03 290.46 0.10 116872 0.00 0.00 > >> >> >> Curl_http_readwrite_headers > >> >> >> 0.03 290.56 0.10 > ssl_tls_trace > >> >> >> 0.03 290.66 0.10 8013097 0.00 0.00 > scan_response > >> >> >> 0.03 290.75 0.09 29488555 0.00 0.00 curlx_tvdiff > >> >> >> 0.03 290.84 0.09 2142468 0.00 0.00 Curl_setopt > >> >> >> 0.03 290.93 0.09 476299 0.00 0.00 Curl_expire > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> 2012/3/5 Robert Iakobashvili <cor...@gm...> > >> >> >>> > >> >> >>> Dear Fred Huang, > >> >> >>> > >> >> >>> Thanks for your posting. > >> >> >>> > >> >> >>> Could you please check if this is due to some badness at cares? > >> >> >>> Thanks. > >> >> >>> > >> >> >>> Let's exclude usage of cares based DNS resolving by applying the > >> >> >>> patch > >> >> >>> attached > >> >> >>> and use instead global DNS cache. > >> >> >>> > >> >> >>> cd ./curl-loader-0.56 > >> >> >>> patch -p1 < path-to/disable-cares-use-global-dns.patch > >> >> >>> make cleanall; make > >> >> >>> > >> >> >>> See if it helps. > >> >> >>> > >> >> >>> Global cache, unfortunately, is not thread safe. > >> >> >>> > >> >> >>> If it helps, we'll try to restore the previous version of cares. > >> >> >>> > >> >> >>> Thanks, > >> >> >>> Robert > >> >> >>> > >> >> >>> On Mon, Mar 5, 2012 at 4:28 PM, Fred Huang <di...@gm...> > >> >> >>> wrote: > >> >> >>>> > >> >> >>>> test 1: > >> >> >>>> > >> >> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, > resolve > >> >> >>>> *.com to one IP address > >> >> >>>> number of domain names in all https URLs: 780,000 > >> >> >>>> number of client: 1000 > >> >> >>>> CPU usage: 78% > >> >> >>>> cpu% irq% sirq% sys% iowt% mem_used buf&cached > >> >> >>>> 78.8 0.0 0.6 3.8 0.0 3640.9Mb 282.1Mb > >> >> >>>> SSL TPS: 300 > >> >> >>>> SSL throughput: 200Mbps > >> >> >>>> > >> >> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50 > >> >> >>>> Flat profile: > >> >> >>>> Each sample counts as 0.01 seconds. > >> >> >>>> % cumulative self self total > >> >> >>>> time seconds seconds calls s/call s/call name > >> >> >>>> 38.55 127.10 127.10 130975 0.00 0.00 > >> >> >>>> Curl_hash_clean_with_criterium > >> >> >>>> 24.63 208.32 81.22 1247528407 0.00 0.00 > >> >> >>>> hostcache_timestamp_remove > >> >> >>>> 13.08 251.44 43.12 2791988 0.00 0.00 > >> >> >>>> Curl_hash_pick > >> >> >>>> 7.30 275.51 24.07 352490 0.00 0.00 > >> >> >>>> Curl_hash_add > >> >> >>>> 6.01 295.31 19.80 410760770 0.00 0.00 > >> >> >>>> Curl_str_key_compare > >> >> >>>> 2.95 305.04 9.73 131015 0.00 0.00 > create_conn > >> >> >>>> 0.88 307.94 2.90 4841390 0.00 0.00 > >> >> >>>> dprintf_formatf > >> >> >>>> 0.74 310.38 2.44 136036 0.00 0.00 > >> >> >>>> ConnectionStore > >> >> >>>> 0.48 311.96 1.58 814604911 0.00 0.00 > >> >> >>>> ares__is_list_empty > >> >> >>>> 0.36 313.16 1.20 > >> >> >>>> locking_function > >> >> >>>> 0.27 314.06 0.90 261608 0.00 0.00 > ares_cancel > >> >> >>>> 0.22 314.79 0.73 127417 0.00 0.00 > >> >> >>>> curl_multi_remove_handle > >> >> >>>> 0.19 315.40 0.62 8906301 0.00 0.00 > >> >> >>>> client_tracing_function > >> >> >>>> 0.17 315.97 0.57 135305 0.00 0.00 > >> >> >>>> Curl_num_addresses > >> >> >>>> 0.17 316.54 0.57 181937907 0.00 0.00 addbyter > >> >> >>>> 0.17 317.09 0.55 2374363 0.00 0.00 > >> >> >>>> multi_runsingle > >> >> >>>> 0.16 317.62 0.53 541954 0.00 0.00 > >> >> >>>> ares__init_list_node > >> >> >>>> 0.16 318.14 0.52 494412 0.00 0.00 > >> >> >>>> curl_multi_socket_action > >> >> >>>> 0.14 318.60 0.46 33578897 0.00 0.00 > >> >> >>>> Curl_socket_check > >> >> >>>> 0.13 319.04 0.44 102951052 0.00 0.00 > >> >> >>>> Curl_raw_toupper > >> >> >>>> 0.13 3... [truncated message content] |
|
From: Robert I. <cor...@gm...> - 2012-03-07 06:34:28
|
Sorry, Fred. I'm poised to think that thread-locking with openssl is not the issue here but rather libcurl interaction with openssl is to blame. The fix could be done only in libcurl or even in openssl. If you could manage to compile openssl with -pg -O2 and to pass these openssl binaries to linking, perhaps, you could see more in your profiling. Sincerely, Robert On Wed, Mar 7, 2012 at 2:43 AM, Fred Huang <di...@gm...> wrote: > yes, I use -t 8 option because the test box has 8 logic CPUs (Dell 990, > Intel Core i7 2600, 16GB memory). > if I do not use -t option, the CPU used by curl-loader will be 100% on one > of 8 logic CPU (12% of total CPU), and the CAPS drops to 100: > # gprof /usr/bin/curl-loader gmon.out -p | head -20 > Flat profile: > Each sample counts as 0.01 seconds. > % cumulative self self total > time seconds seconds calls s/call s/call name > 34.12 2.74 2.74 17385 0.00 0.00 create_conn > 11.21 3.64 0.90 30998583 0.00 0.00 > hostcache_timestamp_remove > 9.84 4.43 0.79 13433 0.00 0.00 > Curl_hash_clean_with_criterium > 7.97 5.07 0.64 381818 0.00 0.00 Curl_hash_pick > 5.85 5.54 0.47 17385 0.00 0.00 ConnectionStore > 5.11 5.95 0.41 16385 0.00 0.00 > curl_multi_remove_handle > 4.11 6.28 0.33 16509889 0.00 0.00 Curl_str_key_compare > 3.36 6.55 0.27 23959914 0.00 0.00 Curl_socket_check > 3.24 6.81 0.26 56703 0.00 0.00 Curl_hash_add > 2.62 7.02 0.21 635750 0.00 0.00 dprintf_formatf > 1.25 7.12 0.10 92421190 0.00 0.00 ares__is_list_empty > 0.75 7.18 0.06 97957 0.00 0.00 Curl_readwrite > 0.62 7.23 0.05 23422260 0.00 0.00 addbyter > 0.62 7.28 0.05 1136775 0.00 0.00 > client_tracing_function > 0.50 7.32 0.04 3091375 0.00 0.00 ares__init_list_head > > I also tried to use > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 20); > but get no improvement on CAPS. > > > Regards, > Fred > > 2012/3/7 Robert Iakobashvili <cor...@gm...> >> >> Dear Fred, >> >> If I understand correctly your observations, the issue is not in: >> >> 1. DNS resolving; >> 2. Version of cares or libcurl; >> >> but rather in >> >> Less effective handling of HTTPS versus HTTP in libcurl. >> >> Please, confirm that my understanding is correct. >> >> >> HTTPS URLs are using an extra locking in openssl with -t option >> that could be making the issue. >> If you are using -t option, try to load without it and let's say with >> 300 clients. >> >> If you are not using -t, thus the only way to proceed is to share >> your observations with libcurl people at >> >> libcurl development <cur...@co...> >> >> Thank you. >> >> Best wishes. >> Robert >> >> On Tue, Mar 6, 2012 at 6:17 PM, Fred Huang <di...@gm...> wrote: >> > 1. the major CPU consumer is curl-loader. here is the top: >> > top - 07:36:36 up 23:31, 5 users, load average: 4.32, 2.26, 0.91 >> > Tasks: 237 total, 2 running, 235 sleeping, 0 stopped, 0 zombie >> > Cpu0 : 76.6%us, 3.6%sy, 0.0%ni, 19.4%id, 0.0%wa, 0.0%hi, 0.3%si, >> > 0.0% >> > Cpu1 : 72.4%us, 4.0%sy, 0.0%ni, 22.9%id, 0.0%wa, 0.0%hi, 0.7%si, >> > 0.0% >> > Cpu2 : 65.8%us, 3.3%sy, 0.0%ni, 30.6%id, 0.0%wa, 0.0%hi, 0.3%si, >> > 0.0% >> > Cpu3 : 75.2%us, 4.3%sy, 0.0%ni, 19.5%id, 0.0%wa, 0.0%hi, 1.0%si, >> > 0.0% >> > Cpu4 : 79.1%us, 4.0%sy, 0.0%ni, 16.8%id, 0.0%wa, 0.0%hi, 0.0%si, >> > 0.0% >> > Cpu5 : 75.8%us, 4.3%sy, 0.0%ni, 19.2%id, 0.0%wa, 0.0%hi, 0.7%si, >> > 0.0% >> > Cpu6 : 74.1%us, 5.0%sy, 0.0%ni, 20.3%id, 0.0%wa, 0.0%hi, 0.7%si, >> > 0.0% >> > Cpu7 : 78.7%us, 4.0%sy, 0.0%ni, 16.6%id, 0.0%wa, 0.0%hi, 0.7%si, >> > 0.0% >> > Mem: 16209932k total, 3829500k used, 12380432k free, 99112k buffers >> > Swap: 18448376k total, 0k used, 18448376k free, 320284k cached >> > >> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >> > 30795 root 20 0 3200m 2.8g 2296 S 79.2 18.4 21:26.84 curl-loader >> > 1832 nobody 20 0 13944 1736 580 S 0.1 0.0 0:32.65 dnsmasq >> > 9 root 20 0 0 0 0 S 0.0 0.0 0:00.53 ksoftirqd/1 >> > 30825 root 20 0 15220 1360 960 R 0.0 0.0 0:00.42 top >> > 1 root 20 0 19400 1568 1256 S 0.0 0.0 0:00.75 init >> > 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd >> > 3 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0 >> > 4 root 20 0 0 0 0 S 0.0 0.0 0:00.06 ksoftirqd/0 >> > 5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0 >> > >> > >> > 2. 1000 loading clients >> > >> > 3. yes, tried 0.52 but no lucky -- CAPS 380, throughput 200Mbps, CPU 90% >> > top - 08:17:10 up 1 day, 12 min, 8 users, load average: 4.66, 3.39, >> > 1.88 >> > Tasks: 245 total, 2 running, 243 sleeping, 0 stopped, 0 zombie >> > Cpu0 : 82.2%us, 4.0%sy, 0.0%ni, 13.9%id, 0.0%wa, 0.0%hi, 0.0%si, >> > 0.0% >> > Cpu1 : 86.1%us, 4.3%sy, 0.0%ni, 7.9%id, 0.0%wa, 0.0%hi, 1.7%si, >> > 0.0% >> > Cpu2 : 85.3%us, 4.0%sy, 0.0%ni, 10.0%id, 0.0%wa, 0.0%hi, 0.7%si, >> > 0.0% >> > Cpu3 : 81.8%us, 3.0%sy, 0.0%ni, 14.2%id, 0.0%wa, 0.0%hi, 1.0%si, >> > 0.0% >> > Cpu4 : 83.7%us, 4.0%sy, 0.0%ni, 12.3%id, 0.0%wa, 0.0%hi, 0.0%si, >> > 0.0% >> > Cpu5 : 92.7%us, 3.3%sy, 0.0%ni, 3.3%id, 0.0%wa, 0.0%hi, 0.7%si, >> > 0.0% >> > Cpu6 : 89.7%us, 2.0%sy, 0.0%ni, 8.0%id, 0.0%wa, 0.0%hi, 0.3%si, >> > 0.0% >> > Cpu7 : 88.1%us, 3.6%sy, 0.0%ni, 6.6%id, 0.0%wa, 0.0%hi, 1.7%si, >> > 0.0% >> > Mem: 16209932k total, 1795748k used, 14414184k free, 102824k buffers >> > Swap: 18448376k total, 0k used, 18448376k free, 382988k cached >> > >> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >> > 12236 root 20 0 1150m 894m 2572 S 90.7 5.7 51:11.91 curl-loader >> > 1832 nobody 20 0 13944 1736 580 S 0.1 0.0 0:47.04 dnsmasq >> > >> > ]# gprof /usr/bin/curl-loader gmon.out -p | head -20 >> > Flat profile: >> > Each sample counts as 0.01 seconds. >> > % cumulative self self total >> > time seconds seconds calls s/call s/call name >> > 43.92 151.95 151.95 157295 0.00 0.00 >> > Curl_hash_clean_with_criterium >> > 22.90 231.17 79.22 1712190971 0.00 0.00 >> > hostcache_timestamp_remove >> > 13.15 276.68 45.51 3260321 0.00 0.00 Curl_hash_pick >> > 7.51 302.65 25.97 394171 0.00 0.00 Curl_hash_add >> > 5.98 323.34 20.68 567437518 0.00 0.00 >> > Curl_str_key_compare >> > 1.00 326.80 3.46 5380220 0.00 0.00 dprintf_formatf >> > 0.41 328.21 1.41 locking_function >> > 0.32 329.32 1.11 469222138 0.00 0.00 >> > ares__is_list_empty >> > 0.31 330.40 1.08 3284508 0.00 0.00 multi_runsingle >> > 0.28 331.38 0.98 10006806 0.00 0.00 >> > client_tracing_function >> > 0.24 332.21 0.83 151951 0.00 0.00 Curl_num_addresses >> > 0.22 332.97 0.77 188075672 0.00 0.00 addbyter >> > 0.19 333.64 0.67 841840 0.00 0.00 Curl_readwrite >> > 0.19 334.29 0.65 id_function >> > 0.18 334.90 0.61 39843 0.00 0.01 >> > curl_multi_socket_all >> > >> > >> > >> > 4. yes, it is correct for 0.56. If use same URL list by replacing https >> > to >> > http, I can get CAPS 1500 and throughput 870Mbps with CPU% < 60% >> > >> > >> > 2012/3/6 Robert Iakobashvili <cor...@gm...> >> >> >> >> Several questions: >> >> >> >> 1. Could you please run top and see if there are other processes >> >> burning >> >> CPU? >> >> 2. What is the number of loading clients that you use? >> >> 3. Could you try curl-loader-0.52 if it compiles for you? >> >> If not, please try to place curl-loader-7.19.7.tar.gz and >> >> c-ares-16.0.tar.gz, >> >> update their versions in Makefile and rebuild by make cleanall; make >> >> >> >> 4. Is it correct to say that for HTTP URLs (not HTTPS) you are getting >> >> a good throughput >> >> and CAPS with curl-loader-0.56? >> >> >> >> Thanks, >> >> Robert >> >> >> >> On Tue, Mar 6, 2012 at 3:52 AM, Fred Huang <di...@gm...> wrote: >> >> > TIMEOUT -1 -> 60 works a little better but no performance improvement >> >> > on >> >> > CAPS or throughput. >> >> > >> >> > test 5: >> >> > enable c-ares; >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); >> >> > >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> > *.com >> >> > to >> >> > one IP address >> >> > number of domain names in URLs: 780,000 >> >> > number of client: 1000 >> >> > CPU usage: 74% >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> > 74.0 0.0 0.7 4.2 0.0 3530.6Mb 250.8Mb >> >> > SSL TPS: 360 >> >> > SSL throughput: 220Mbps >> >> > >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -50 >> >> > Flat profile: >> >> > Each sample counts as 0.01 seconds. >> >> > % cumulative self self total >> >> > time seconds seconds calls s/call s/call name >> >> > 26.63 23.13 23.13 119226 0.00 0.00 >> >> > Curl_hash_clean_with_criterium >> >> > 20.34 40.80 17.67 317792146 0.00 0.00 >> >> > hostcache_timestamp_remove >> >> > 11.27 50.59 9.79 2579903 0.00 0.00 Curl_hash_pick >> >> > 7.84 57.40 6.81 120606 0.00 0.00 create_conn >> >> > 5.65 62.31 4.91 119117401 0.00 0.00 >> >> > Curl_str_key_compare >> >> > 4.98 66.64 4.33 329829 0.00 0.00 Curl_hash_add >> >> > 2.80 69.08 2.44 4476606 0.00 0.00 dprintf_formatf >> >> > 2.57 71.31 2.23 124019 0.00 0.00 ConnectionStore >> >> > 1.50 72.61 1.30 740997609 0.00 0.00 >> >> > ares__is_list_empty >> >> > 1.16 73.61 1.01 >> >> > locking_function >> >> > 0.76 74.27 0.66 238744 0.00 0.00 ares_cancel >> >> > 0.70 74.88 0.61 457564 0.00 0.00 >> >> > curl_multi_socket_action >> >> > 0.67 75.46 0.58 118616 0.00 0.00 >> >> > curl_multi_remove_handle >> >> > 0.58 75.97 0.51 8211502 0.00 0.00 >> >> > client_tracing_function >> >> > 0.53 76.43 0.46 733561 0.00 0.00 Curl_readwrite >> >> > 0.51 76.87 0.44 2183179 0.00 0.00 multi_runsingle >> >> > 0.51 77.31 0.44 166585567 0.00 0.00 addbyter >> >> > 0.44 77.69 0.38 2175805 0.00 0.00 Curl_raw_equal >> >> > 0.43 78.06 0.38 id_function >> >> > 0.40 78.41 0.35 29789346 0.00 0.00 >> >> > Curl_socket_check >> >> > 0.40 78.76 0.35 479671 0.00 0.00 >> >> > ossl_connect_common >> >> > 0.40 79.11 0.35 490639 0.00 0.00 >> >> > ares__init_list_node >> >> > 0.33 79.40 0.29 94197670 0.00 0.00 >> >> > Curl_raw_toupper >> >> > 0.33 79.69 0.29 3268031 0.00 0.00 Curl_pgrsUpdate >> >> > 0.28 79.93 0.24 8797 0.00 0.00 >> >> > curl_multi_perform >> >> > 0.27 80.17 0.24 2233641 0.00 0.00 Curl_infof >> >> > 0.26 80.40 0.23 1839837 0.00 0.00 Curl_splay >> >> > 0.24 80.61 0.21 121713028 0.00 0.00 curl_strequal >> >> > 0.21 80.79 0.18 243333 0.00 0.00 >> >> > Curl_ssl_getsessionid >> >> > 0.18 80.95 0.16 122180 0.00 0.00 >> >> > Curl_num_addresses >> >> > 0.17 81.09 0.15 8271666 0.00 0.00 Curl_debug >> >> > 0.16 81.23 0.14 18902452 0.00 0.00 alloc_addbyter >> >> > 0.16 81.37 0.14 8851 0.00 0.00 >> >> > curl_multi_socket_all >> >> > 0.14 81.50 0.13 2236769 0.00 0.00 Curl_read >> >> > 0.14 81.62 0.12 2271700 0.00 0.00 fd_key_compare >> >> > 0.14 81.74 0.12 323167 0.00 0.00 event_del >> >> > 0.14 81.86 0.12 205436 0.00 0.00 >> >> > read_udp_packets >> >> > 0.14 81.98 0.12 200979 0.00 0.00 processfds >> >> > 0.13 82.09 0.11 123886 0.00 0.00 >> >> > Curl_if_is_interface_name >> >> > 0.13 82.20 0.11 120350 0.00 0.00 Curl_http >> >> > 0.13 82.31 0.11 ssl_tls_trace >> >> > 0.12 82.41 0.10 119229 0.00 0.00 >> >> > Curl_splayremovebyaddr >> >> > 0.12 82.51 0.10 7798717 0.00 0.00 Curl_raw_nequal >> >> > 0.12 82.61 0.10 784485 0.00 0.00 event_add >> >> > 0.10 82.70 0.09 643482 0.00 0.00 Curl_expire >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > test 6: >> >> > enable c-ares; >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); >> >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0 >> >> > >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> > *.com >> >> > to >> >> > one IP address >> >> > number of domain names in URLs: 780,000 >> >> > number of client: 1000 >> >> > CPU usage: 86% >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> > 85.9 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb >> >> > SSL TPS: 360 >> >> > SSL throughput: 200Mbps >> >> > >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> >> > Flat profile: >> >> > Each sample counts as 0.01 seconds. >> >> > % cumulative self self total >> >> > time seconds seconds calls s/call s/call name >> >> > 27.19 19.96 19.96 103472 0.00 0.00 >> >> > Curl_hash_clean_with_criterium >> >> > 21.73 35.91 15.95 274168981 0.00 0.00 >> >> > hostcache_timestamp_remove >> >> > 10.33 43.49 7.58 2256441 0.00 0.00 Curl_hash_pick >> >> > 7.21 48.78 5.29 104540 0.00 0.00 create_conn >> >> > 6.20 53.33 4.55 103413405 0.00 0.00 >> >> > Curl_str_key_compare >> >> > 4.88 56.91 3.58 287109 0.00 0.00 Curl_hash_add >> >> > 3.02 59.13 2.22 3797855 0.00 0.00 dprintf_formatf >> >> > 2.10 60.67 1.54 107711 0.00 0.00 ConnectionStore >> >> > 1.65 61.88 1.21 642725327 0.00 0.00 >> >> > ares__is_list_empty >> >> > 1.19 62.76 0.88 >> >> > locking_function >> >> > 0.87 63.40 0.64 207817 0.00 0.00 ares_cancel >> >> > 0.56 63.81 0.41 404108 0.00 0.00 >> >> > curl_multi_socket_action >> >> > 0.54 64.21 0.40 425603 0.00 0.00 >> >> > ares__init_list_node >> >> > 0.53 64.60 0.39 7068112 0.00 0.00 >> >> > client_tracing_function >> >> > 0.49 64.96 0.36 1908587 0.00 0.00 multi_runsingle >> >> > >> >> > >> >> > >> >> > test 7: >> >> > enable c-ares; >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); >> >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0 >> >> > >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> > *.com >> >> > to >> >> > one IP address >> >> > number of domain names in URLs: 780,000 >> >> > number of client: 1000 >> >> > CPU usage: 86% >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> > 86.6 0.0 0.6 3.3 0.0 3624.8Mb 263.0Mb >> >> > SSL TPS: 300 >> >> > SSL throughput: 200Mbps >> >> > >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> >> > Flat profile: >> >> > Each sample counts as 0.01 seconds. >> >> > % cumulative self self total >> >> > time seconds seconds calls s/call s/call name >> >> > 39.83 369.71 369.71 207173 0.00 0.00 >> >> > Curl_hash_clean_with_criterium >> >> > 24.27 594.98 225.27 3198337733 0.00 0.00 >> >> > hostcache_timestamp_remove >> >> > 13.75 722.59 127.61 4312864 0.00 0.00 Curl_hash_pick >> >> > 9.19 807.87 85.28 546507 0.00 0.00 Curl_hash_add >> >> > 6.14 864.88 57.01 1022419684 0.00 0.00 >> >> > Curl_str_key_compare >> >> > 2.06 884.00 19.12 203749 0.00 0.00 create_conn >> >> > 0.54 888.99 4.99 213306 0.00 0.00 ConnectionStore >> >> > 0.49 893.50 4.51 7331251 0.00 0.00 dprintf_formatf >> >> > 0.29 896.16 2.66 1287182329 0.00 0.00 >> >> > ares__is_list_empty >> >> > 0.23 898.28 2.12 >> >> > locking_function >> >> > 0.20 900.10 1.82 212210 0.00 0.00 >> >> > Curl_num_addresses >> >> > 0.17 901.66 1.56 198385 0.00 0.00 >> >> > curl_multi_remove_handle >> >> > 0.12 902.81 1.15 411114 0.00 0.00 ares_cancel >> >> > 0.12 903.95 1.14 3683448 0.00 0.00 multi_runsingle >> >> > 0.11 904.97 1.03 13664120 0.00 0.00 >> >> > client_tracing_function >> >> > >> >> > >> >> > test 8: >> >> > enable c-ares; >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); >> >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0 >> >> > >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> > *.com >> >> > to >> >> > one IP address >> >> > number of domain names in URLs: 780,000 >> >> > number of client: 1000 >> >> > CPU usage: 75% >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> > 75.2 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb >> >> > SSL TPS: 380 >> >> > SSL throughput: 220Mbps >> >> > >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> >> > Flat profile: >> >> > Each sample counts as 0.01 seconds. >> >> > % cumulative self self total >> >> > time seconds seconds calls s/call s/call name >> >> > 27.58 12.31 12.31 71140 0.00 0.00 >> >> > Curl_hash_clean_with_criterium >> >> > 19.09 20.83 8.52 176729692 0.00 0.00 >> >> > hostcache_timestamp_remove >> >> > 10.44 25.49 4.66 1589309 0.00 0.00 Curl_hash_pick >> >> > 6.85 28.55 3.06 73204 0.00 0.00 create_conn >> >> > 5.56 31.03 2.48 67620530 0.00 0.00 >> >> > Curl_str_key_compare >> >> > 4.41 33.00 1.97 200684 0.00 0.00 Curl_hash_add >> >> > 3.56 34.59 1.59 2717956 0.00 0.00 dprintf_formatf >> >> > 1.72 35.36 0.77 75247 0.00 0.00 ConnectionStore >> >> > 1.70 36.12 0.76 444342256 0.00 0.00 >> >> > ares__is_list_empty >> >> > 1.23 36.67 0.55 >> >> > locking_function >> >> > 1.16 37.19 0.52 143342 0.00 0.00 ares_cancel >> >> > 0.99 37.63 0.44 72211 0.00 0.00 >> >> > curl_multi_remove_handle >> >> > 0.74 37.96 0.33 1333371 0.00 0.00 multi_runsingle >> >> > 0.68 38.27 0.31 101147798 0.00 0.00 addbyter >> >> > 0.58 38.53 0.26 288737 0.00 0.00 >> >> > curl_multi_socket_action >> >> > >> >> > >> >> > >> >> > test 9: >> >> > enable c-ares; >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); >> >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0 >> >> > >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> > *.com >> >> > to >> >> > one IP address >> >> > number of domain names in URLs: 780,000 >> >> > number of client: 1000 >> >> > CPU usage: 75% >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> > 75.1 0.0 0.6 4.1 0.0 3545.2Mb 260.1Mb >> >> > SSL TPS: 380 >> >> > SSL throughput: 200Mbps >> >> > >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> >> > Flat profile: >> >> > Each sample counts as 0.01 seconds. >> >> > % cumulative self self total >> >> > time seconds seconds calls s/call s/call name >> >> > 34.00 31.13 31.13 72495 0.00 0.00 >> >> > Curl_hash_clean_with_criterium >> >> > 23.64 52.78 21.65 386070412 0.00 0.00 >> >> > hostcache_timestamp_remove >> >> > 13.28 64.94 12.16 1619440 0.00 0.00 Curl_hash_pick >> >> > 5.92 70.36 5.42 137653129 0.00 0.00 >> >> > Curl_str_key_compare >> >> > 5.50 75.40 5.04 203018 0.00 0.00 Curl_hash_add >> >> > 4.55 79.57 4.17 74618 0.00 0.00 create_conn >> >> > 1.70 81.13 1.56 2757148 0.00 0.00 dprintf_formatf >> >> > 0.98 82.03 0.90 76945 0.00 0.00 ConnectionStore >> >> > 0.95 82.90 0.87 455775100 0.00 0.00 >> >> > ares__is_list_empty >> >> > 0.57 83.42 0.53 >> >> > locking_function >> >> > 0.39 83.78 0.36 146742 0.00 0.00 ares_cancel >> >> > 0.35 84.10 0.32 102968806 0.00 0.00 addbyter >> >> > 0.35 84.42 0.32 456587 0.00 0.00 Curl_readwrite >> >> > 0.35 84.74 0.32 73133 0.00 0.00 >> >> > curl_multi_remove_handle >> >> > 0.34 85.05 0.31 1360545 0.00 0.00 multi_runsingle >> >> > >> >> > >> >> > 2012/3/6 Robert Iakobashvili <cor...@gm...> >> >> >> >> >> >> If it will not be helpful this means that recent >> >> >> changes in libcurl library have broken DNS interface. >> >> >> >> >> >> You can try a previos version and the latest libcurl, like: >> >> >> >> >> >> http://curl.haxx.se/download/curl-7.23.0.tar.gz >> >> >> and >> >> >> http://curl.haxx.se/downloadcurl-7.24.0.tar.gz >> >> >> >> >> >> by: >> >> >> >> >> >> 1. downloading a libcurl version and placing it to >> >> >> curl-loader-0.56/packages ; >> >> >> 2. correcting the Makefile in curl-loader-0.56 >> >> >> CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24 or CURL_VER:=7.23 >> >> >> 3. make cleanall; make >> >> >> >> >> >> Please, send us your observations. >> >> >> Thanks! >> >> >> Robert >> >> >> >> >> >> On Mon, Mar 5, 2012 at 6:57 PM, Robert Iakobashvili >> >> >> <cor...@gm...> >> >> >> wrote: >> >> >> > If so, please try to change >> >> >> > >> >> >> > loader.c, line 578 which looks like: >> >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); >> >> >> > >> >> >> > 1. instead of -1 try to set some value like 60 >> >> >> > 2. make clean; make >> >> >> > >> >> >> > Thanks, >> >> >> > Robert >> >> >> > >> >> >> > On Mon, Mar 5, 2012 at 6:19 PM, Fred Huang <di...@gm...> >> >> >> > wrote: >> >> >> >> >> >> >> >> Looks like the bottleneck comes from libcurl other than c-ares... >> >> >> >> test 3 (no ares patch applied): >> >> >> >> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> >> >> *.com >> >> >> >> to one IP address >> >> >> >> number of domain names in URLs: 780,000 >> >> >> >> number of client: 1000 >> >> >> >> CPU usage: 85% >> >> >> >> cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> >> >> 85.6 0.0 0.8 5.9 0.0 3334.3Mb 112.0Mb >> >> >> >> SSL TPS: 320 >> >> >> >> SSL throughput: 200Mbps >> >> >> >> >> >> >> >> # gprof /usr/bin/curl-loader gmon.out -p | head -50 >> >> >> >> Flat profile: >> >> >> >> Each sample counts as 0.01 seconds. >> >> >> >> % cumulative self self total >> >> >> >> time seconds seconds calls s/call s/call name >> >> >> >> 40.80 120.16 120.16 116079 0.00 0.00 >> >> >> >> Curl_hash_clean_with_criterium >> >> >> >> 24.46 192.18 72.02 983166323 0.00 0.00 >> >> >> >> hostcache_timestamp_remove >> >> >> >> 13.30 231.35 39.17 2059341 0.00 0.00 >> >> >> >> Curl_hash_pick >> >> >> >> 6.45 250.35 19.00 240087 0.00 0.00 >> >> >> >> Curl_hash_add >> >> >> >> 4.81 264.51 14.16 334257102 0.00 0.00 >> >> >> >> Curl_str_key_compare >> >> >> >> 3.40 274.52 10.01 119514 0.00 0.00 create_conn >> >> >> >> 0.93 277.25 2.73 4478910 0.00 0.00 >> >> >> >> dprintf_formatf >> >> >> >> 0.90 279.91 2.66 122209 0.00 0.00 >> >> >> >> ConnectionStore >> >> >> >> 0.40 281.09 1.18 >> >> >> >> locking_function >> >> >> >> 0.27 281.88 0.79 114759 0.00 0.00 >> >> >> >> curl_multi_remove_handle >> >> >> >> 0.21 282.51 0.63 7993664 0.00 0.00 >> >> >> >> client_tracing_function >> >> >> >> 0.21 283.12 0.61 122685 0.00 0.00 >> >> >> >> Curl_num_addresses >> >> >> >> 0.19 283.68 0.56 397028 0.00 0.00 >> >> >> >> curl_multi_socket_action >> >> >> >> 0.19 284.23 0.55 2036116 0.00 0.00 >> >> >> >> multi_runsingle >> >> >> >> 0.17 284.72 0.49 703687 0.00 0.00 >> >> >> >> Curl_readwrite >> >> >> >> 0.16 285.19 0.47 id_function >> >> >> >> 0.15 285.62 0.43 29692029 0.00 0.00 >> >> >> >> Curl_socket_check >> >> >> >> 0.14 286.02 0.40 91973856 0.00 0.00 >> >> >> >> Curl_raw_toupper >> >> >> >> 0.13 286.40 0.38 164664144 0.00 0.00 addbyter >> >> >> >> 0.13 286.77 0.37 2180797 0.00 0.00 Curl_infof >> >> >> >> 0.11 287.10 0.33 122675 0.00 0.00 >> >> >> >> Curl_if_is_interface_name >> >> >> >> 0.10 287.39 0.29 471318 0.00 0.00 >> >> >> >> ossl_connect_common >> >> >> >> 0.10 287.67 0.28 2113528 0.00 0.00 >> >> >> >> Curl_raw_equal >> >> >> >> 0.09 287.93 0.26 8676 0.00 0.02 >> >> >> >> curl_multi_socket_all >> >> >> >> 0.08 288.17 0.24 1663262 0.00 0.00 Curl_splay >> >> >> >> 0.08 288.41 0.24 8505 0.00 0.01 >> >> >> >> curl_multi_perform >> >> >> >> 0.07 288.63 0.22 3114576 0.00 0.00 >> >> >> >> Curl_pgrsUpdate >> >> >> >> 0.06 288.80 0.17 119637 0.00 0.00 >> >> >> >> singleipconnect >> >> >> >> 0.06 288.97 0.17 7994818 0.00 0.00 Curl_debug >> >> >> >> 0.05 289.13 0.16 235820 0.00 0.00 event_del >> >> >> >> 0.05 289.29 0.16 119393071 0.00 0.00 >> >> >> >> curl_strequal >> >> >> >> 0.05 289.45 0.16 18535473 0.00 0.00 >> >> >> >> alloc_addbyter >> >> >> >> 0.05 289.61 0.16 356074 0.00 0.00 >> >> >> >> Curl_hash_str >> >> >> >> 0.05 289.76 0.15 1797233 0.00 0.00 >> >> >> >> fd_key_compare >> >> >> >> 0.05 289.91 0.15 234966 0.00 0.00 >> >> >> >> Curl_ssl_getsessionid >> >> >> >> 0.04 290.04 0.13 585853 0.00 0.00 >> >> >> >> Curl_llist_destroy >> >> >> >> 0.04 290.16 0.12 240826 0.00 0.00 Curl_resolv >> >> >> >> 0.03 290.26 0.10 7514328 0.00 0.00 >> >> >> >> Curl_raw_nequal >> >> >> >> 0.03 290.36 0.10 2214436 0.00 0.00 Curl_read >> >> >> >> 0.03 290.46 0.10 116872 0.00 0.00 >> >> >> >> Curl_http_readwrite_headers >> >> >> >> 0.03 290.56 0.10 >> >> >> >> ssl_tls_trace >> >> >> >> 0.03 290.66 0.10 8013097 0.00 0.00 >> >> >> >> scan_response >> >> >> >> 0.03 290.75 0.09 29488555 0.00 0.00 >> >> >> >> curlx_tvdiff >> >> >> >> 0.03 290.84 0.09 2142468 0.00 0.00 Curl_setopt >> >> >> >> 0.03 290.93 0.09 476299 0.00 0.00 Curl_expire >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> 2012/3/5 Robert Iakobashvili <cor...@gm...> >> >> >> >>> >> >> >> >>> Dear Fred Huang, >> >> >> >>> >> >> >> >>> Thanks for your posting. >> >> >> >>> >> >> >> >>> Could you please check if this is due to some badness at cares? >> >> >> >>> Thanks. >> >> >> >>> >> >> >> >>> Let's exclude usage of cares based DNS resolving by applying the >> >> >> >>> patch >> >> >> >>> attached >> >> >> >>> and use instead global DNS cache. >> >> >> >>> >> >> >> >>> cd ./curl-loader-0.56 >> >> >> >>> patch -p1 < path-to/disable-cares-use-global-dns.patch >> >> >> >>> make cleanall; make >> >> >> >>> >> >> >> >>> See if it helps. >> >> >> >>> >> >> >> >>> Global cache, unfortunately, is not thread safe. >> >> >> >>> >> >> >> >>> If it helps, we'll try to restore the previous version of cares. >> >> >> >>> >> >> >> >>> Thanks, >> >> >> >>> Robert >> >> >> >>> >> >> >> >>> On Mon, Mar 5, 2012 at 4:28 PM, Fred Huang <di...@gm...> >> >> >> >>> wrote: >> >> >> >>>> >> >> >> >>>> test 1: >> >> >> >>>> >> >> >> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, >> >> >> >>>> resolve >> >> >> >>>> *.com to one IP address >> >> >> >>>> number of domain names in all https URLs: 780,000 >> >> >> >>>> number of client: 1000 >> >> >> >>>> CPU usage: 78% >> >> >> >>>> cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> >> >>>> 78.8 0.0 0.6 3.8 0.0 3640.9Mb 282.1Mb >> >> >> >>>> SSL TPS: 300 >> >> >> >>>> SSL throughput: 200Mbps >> >> >> >>>> >> >> >> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50 >> >> >> >>>> Flat profile: >> >> >> >>>> Each sample counts as 0.01 seconds. >> >> >> >>>> % cumulative self self total >> >> >> >>>> time seconds seconds calls s/call s/call name >> >> >> >>>> 38.55 127.10 127.10 130975 0.00 0.00 >> >> >> >>>> Curl_hash_clean_with_criterium >> >> >> >>>> 24.63 208.32 81.22 1247528407 0.00 0.00 >> >> >> >>>> hostcache_timestamp_remove >> >> >> >>>> 13.08 251.44 43.12 2791988 0.00 0.00 >> >> >> >>>> Curl_hash_pick >> >> >> >>>> 7.30 275.51 24.07 352490 0.00 0.00 >> >> >> >>>> Curl_hash_add >> >> >> >>>> 6.01 295.31 19.80 410760770 0.00 0.00 >> >> >> >>>> Curl_str_key_compare >> >> >> >>>> 2.95 305.04 9.73 131015 0.00 0.00 >> >> >> >>>> create_conn >> >> >> >>>> 0.88 307.94 2.90 4841390 0.00 0.00... [truncated message content] |
|
From: Robert I. <cor...@gm...> - 2012-03-07 06:38:08
|
Fred, On more suggestion, which is a general one. If N is the number of your logical CPUs, usually, I'd recommend to run 2*N + B threads. Thus, you can try: -t 16 -t 32 and it MAY increased a bit CAPS. Thanks, Robert On Wed, Mar 7, 2012 at 2:43 AM, Fred Huang <di...@gm...> wrote: > yes, I use -t 8 option because the test box has 8 logic CPUs (Dell 990, > Intel Core i7 2600, 16GB memory). > if I do not use -t option, the CPU used by curl-loader will be 100% on one > of 8 logic CPU (12% of total CPU), and the CAPS drops to 100: > # gprof /usr/bin/curl-loader gmon.out -p | head -20 > Flat profile: > Each sample counts as 0.01 seconds. > % cumulative self self total > time seconds seconds calls s/call s/call name > 34.12 2.74 2.74 17385 0.00 0.00 create_conn > 11.21 3.64 0.90 30998583 0.00 0.00 > hostcache_timestamp_remove > 9.84 4.43 0.79 13433 0.00 0.00 > Curl_hash_clean_with_criterium > 7.97 5.07 0.64 381818 0.00 0.00 Curl_hash_pick > 5.85 5.54 0.47 17385 0.00 0.00 ConnectionStore > 5.11 5.95 0.41 16385 0.00 0.00 > curl_multi_remove_handle > 4.11 6.28 0.33 16509889 0.00 0.00 Curl_str_key_compare > 3.36 6.55 0.27 23959914 0.00 0.00 Curl_socket_check > 3.24 6.81 0.26 56703 0.00 0.00 Curl_hash_add > 2.62 7.02 0.21 635750 0.00 0.00 dprintf_formatf > 1.25 7.12 0.10 92421190 0.00 0.00 ares__is_list_empty > 0.75 7.18 0.06 97957 0.00 0.00 Curl_readwrite > 0.62 7.23 0.05 23422260 0.00 0.00 addbyter > 0.62 7.28 0.05 1136775 0.00 0.00 > client_tracing_function > 0.50 7.32 0.04 3091375 0.00 0.00 ares__init_list_head > > I also tried to use > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 20); > but get no improvement on CAPS. > > > Regards, > Fred > > 2012/3/7 Robert Iakobashvili <cor...@gm...> >> >> Dear Fred, >> >> If I understand correctly your observations, the issue is not in: >> >> 1. DNS resolving; >> 2. Version of cares or libcurl; >> >> but rather in >> >> Less effective handling of HTTPS versus HTTP in libcurl. >> >> Please, confirm that my understanding is correct. >> >> >> HTTPS URLs are using an extra locking in openssl with -t option >> that could be making the issue. >> If you are using -t option, try to load without it and let's say with >> 300 clients. >> >> If you are not using -t, thus the only way to proceed is to share >> your observations with libcurl people at >> >> libcurl development <cur...@co...> >> >> Thank you. >> >> Best wishes. >> Robert >> >> On Tue, Mar 6, 2012 at 6:17 PM, Fred Huang <di...@gm...> wrote: >> > 1. the major CPU consumer is curl-loader. here is the top: >> > top - 07:36:36 up 23:31, 5 users, load average: 4.32, 2.26, 0.91 >> > Tasks: 237 total, 2 running, 235 sleeping, 0 stopped, 0 zombie >> > Cpu0 : 76.6%us, 3.6%sy, 0.0%ni, 19.4%id, 0.0%wa, 0.0%hi, 0.3%si, >> > 0.0% >> > Cpu1 : 72.4%us, 4.0%sy, 0.0%ni, 22.9%id, 0.0%wa, 0.0%hi, 0.7%si, >> > 0.0% >> > Cpu2 : 65.8%us, 3.3%sy, 0.0%ni, 30.6%id, 0.0%wa, 0.0%hi, 0.3%si, >> > 0.0% >> > Cpu3 : 75.2%us, 4.3%sy, 0.0%ni, 19.5%id, 0.0%wa, 0.0%hi, 1.0%si, >> > 0.0% >> > Cpu4 : 79.1%us, 4.0%sy, 0.0%ni, 16.8%id, 0.0%wa, 0.0%hi, 0.0%si, >> > 0.0% >> > Cpu5 : 75.8%us, 4.3%sy, 0.0%ni, 19.2%id, 0.0%wa, 0.0%hi, 0.7%si, >> > 0.0% >> > Cpu6 : 74.1%us, 5.0%sy, 0.0%ni, 20.3%id, 0.0%wa, 0.0%hi, 0.7%si, >> > 0.0% >> > Cpu7 : 78.7%us, 4.0%sy, 0.0%ni, 16.6%id, 0.0%wa, 0.0%hi, 0.7%si, >> > 0.0% >> > Mem: 16209932k total, 3829500k used, 12380432k free, 99112k buffers >> > Swap: 18448376k total, 0k used, 18448376k free, 320284k cached >> > >> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >> > 30795 root 20 0 3200m 2.8g 2296 S 79.2 18.4 21:26.84 curl-loader >> > 1832 nobody 20 0 13944 1736 580 S 0.1 0.0 0:32.65 dnsmasq >> > 9 root 20 0 0 0 0 S 0.0 0.0 0:00.53 ksoftirqd/1 >> > 30825 root 20 0 15220 1360 960 R 0.0 0.0 0:00.42 top >> > 1 root 20 0 19400 1568 1256 S 0.0 0.0 0:00.75 init >> > 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd >> > 3 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0 >> > 4 root 20 0 0 0 0 S 0.0 0.0 0:00.06 ksoftirqd/0 >> > 5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0 >> > >> > >> > 2. 1000 loading clients >> > >> > 3. yes, tried 0.52 but no lucky -- CAPS 380, throughput 200Mbps, CPU 90% >> > top - 08:17:10 up 1 day, 12 min, 8 users, load average: 4.66, 3.39, >> > 1.88 >> > Tasks: 245 total, 2 running, 243 sleeping, 0 stopped, 0 zombie >> > Cpu0 : 82.2%us, 4.0%sy, 0.0%ni, 13.9%id, 0.0%wa, 0.0%hi, 0.0%si, >> > 0.0% >> > Cpu1 : 86.1%us, 4.3%sy, 0.0%ni, 7.9%id, 0.0%wa, 0.0%hi, 1.7%si, >> > 0.0% >> > Cpu2 : 85.3%us, 4.0%sy, 0.0%ni, 10.0%id, 0.0%wa, 0.0%hi, 0.7%si, >> > 0.0% >> > Cpu3 : 81.8%us, 3.0%sy, 0.0%ni, 14.2%id, 0.0%wa, 0.0%hi, 1.0%si, >> > 0.0% >> > Cpu4 : 83.7%us, 4.0%sy, 0.0%ni, 12.3%id, 0.0%wa, 0.0%hi, 0.0%si, >> > 0.0% >> > Cpu5 : 92.7%us, 3.3%sy, 0.0%ni, 3.3%id, 0.0%wa, 0.0%hi, 0.7%si, >> > 0.0% >> > Cpu6 : 89.7%us, 2.0%sy, 0.0%ni, 8.0%id, 0.0%wa, 0.0%hi, 0.3%si, >> > 0.0% >> > Cpu7 : 88.1%us, 3.6%sy, 0.0%ni, 6.6%id, 0.0%wa, 0.0%hi, 1.7%si, >> > 0.0% >> > Mem: 16209932k total, 1795748k used, 14414184k free, 102824k buffers >> > Swap: 18448376k total, 0k used, 18448376k free, 382988k cached >> > >> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >> > 12236 root 20 0 1150m 894m 2572 S 90.7 5.7 51:11.91 curl-loader >> > 1832 nobody 20 0 13944 1736 580 S 0.1 0.0 0:47.04 dnsmasq >> > >> > ]# gprof /usr/bin/curl-loader gmon.out -p | head -20 >> > Flat profile: >> > Each sample counts as 0.01 seconds. >> > % cumulative self self total >> > time seconds seconds calls s/call s/call name >> > 43.92 151.95 151.95 157295 0.00 0.00 >> > Curl_hash_clean_with_criterium >> > 22.90 231.17 79.22 1712190971 0.00 0.00 >> > hostcache_timestamp_remove >> > 13.15 276.68 45.51 3260321 0.00 0.00 Curl_hash_pick >> > 7.51 302.65 25.97 394171 0.00 0.00 Curl_hash_add >> > 5.98 323.34 20.68 567437518 0.00 0.00 >> > Curl_str_key_compare >> > 1.00 326.80 3.46 5380220 0.00 0.00 dprintf_formatf >> > 0.41 328.21 1.41 locking_function >> > 0.32 329.32 1.11 469222138 0.00 0.00 >> > ares__is_list_empty >> > 0.31 330.40 1.08 3284508 0.00 0.00 multi_runsingle >> > 0.28 331.38 0.98 10006806 0.00 0.00 >> > client_tracing_function >> > 0.24 332.21 0.83 151951 0.00 0.00 Curl_num_addresses >> > 0.22 332.97 0.77 188075672 0.00 0.00 addbyter >> > 0.19 333.64 0.67 841840 0.00 0.00 Curl_readwrite >> > 0.19 334.29 0.65 id_function >> > 0.18 334.90 0.61 39843 0.00 0.01 >> > curl_multi_socket_all >> > >> > >> > >> > 4. yes, it is correct for 0.56. If use same URL list by replacing https >> > to >> > http, I can get CAPS 1500 and throughput 870Mbps with CPU% < 60% >> > >> > >> > 2012/3/6 Robert Iakobashvili <cor...@gm...> >> >> >> >> Several questions: >> >> >> >> 1. Could you please run top and see if there are other processes >> >> burning >> >> CPU? >> >> 2. What is the number of loading clients that you use? >> >> 3. Could you try curl-loader-0.52 if it compiles for you? >> >> If not, please try to place curl-loader-7.19.7.tar.gz and >> >> c-ares-16.0.tar.gz, >> >> update their versions in Makefile and rebuild by make cleanall; make >> >> >> >> 4. Is it correct to say that for HTTP URLs (not HTTPS) you are getting >> >> a good throughput >> >> and CAPS with curl-loader-0.56? >> >> >> >> Thanks, >> >> Robert >> >> >> >> On Tue, Mar 6, 2012 at 3:52 AM, Fred Huang <di...@gm...> wrote: >> >> > TIMEOUT -1 -> 60 works a little better but no performance improvement >> >> > on >> >> > CAPS or throughput. >> >> > >> >> > test 5: >> >> > enable c-ares; >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); >> >> > >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> > *.com >> >> > to >> >> > one IP address >> >> > number of domain names in URLs: 780,000 >> >> > number of client: 1000 >> >> > CPU usage: 74% >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> > 74.0 0.0 0.7 4.2 0.0 3530.6Mb 250.8Mb >> >> > SSL TPS: 360 >> >> > SSL throughput: 220Mbps >> >> > >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -50 >> >> > Flat profile: >> >> > Each sample counts as 0.01 seconds. >> >> > % cumulative self self total >> >> > time seconds seconds calls s/call s/call name >> >> > 26.63 23.13 23.13 119226 0.00 0.00 >> >> > Curl_hash_clean_with_criterium >> >> > 20.34 40.80 17.67 317792146 0.00 0.00 >> >> > hostcache_timestamp_remove >> >> > 11.27 50.59 9.79 2579903 0.00 0.00 Curl_hash_pick >> >> > 7.84 57.40 6.81 120606 0.00 0.00 create_conn >> >> > 5.65 62.31 4.91 119117401 0.00 0.00 >> >> > Curl_str_key_compare >> >> > 4.98 66.64 4.33 329829 0.00 0.00 Curl_hash_add >> >> > 2.80 69.08 2.44 4476606 0.00 0.00 dprintf_formatf >> >> > 2.57 71.31 2.23 124019 0.00 0.00 ConnectionStore >> >> > 1.50 72.61 1.30 740997609 0.00 0.00 >> >> > ares__is_list_empty >> >> > 1.16 73.61 1.01 >> >> > locking_function >> >> > 0.76 74.27 0.66 238744 0.00 0.00 ares_cancel >> >> > 0.70 74.88 0.61 457564 0.00 0.00 >> >> > curl_multi_socket_action >> >> > 0.67 75.46 0.58 118616 0.00 0.00 >> >> > curl_multi_remove_handle >> >> > 0.58 75.97 0.51 8211502 0.00 0.00 >> >> > client_tracing_function >> >> > 0.53 76.43 0.46 733561 0.00 0.00 Curl_readwrite >> >> > 0.51 76.87 0.44 2183179 0.00 0.00 multi_runsingle >> >> > 0.51 77.31 0.44 166585567 0.00 0.00 addbyter >> >> > 0.44 77.69 0.38 2175805 0.00 0.00 Curl_raw_equal >> >> > 0.43 78.06 0.38 id_function >> >> > 0.40 78.41 0.35 29789346 0.00 0.00 >> >> > Curl_socket_check >> >> > 0.40 78.76 0.35 479671 0.00 0.00 >> >> > ossl_connect_common >> >> > 0.40 79.11 0.35 490639 0.00 0.00 >> >> > ares__init_list_node >> >> > 0.33 79.40 0.29 94197670 0.00 0.00 >> >> > Curl_raw_toupper >> >> > 0.33 79.69 0.29 3268031 0.00 0.00 Curl_pgrsUpdate >> >> > 0.28 79.93 0.24 8797 0.00 0.00 >> >> > curl_multi_perform >> >> > 0.27 80.17 0.24 2233641 0.00 0.00 Curl_infof >> >> > 0.26 80.40 0.23 1839837 0.00 0.00 Curl_splay >> >> > 0.24 80.61 0.21 121713028 0.00 0.00 curl_strequal >> >> > 0.21 80.79 0.18 243333 0.00 0.00 >> >> > Curl_ssl_getsessionid >> >> > 0.18 80.95 0.16 122180 0.00 0.00 >> >> > Curl_num_addresses >> >> > 0.17 81.09 0.15 8271666 0.00 0.00 Curl_debug >> >> > 0.16 81.23 0.14 18902452 0.00 0.00 alloc_addbyter >> >> > 0.16 81.37 0.14 8851 0.00 0.00 >> >> > curl_multi_socket_all >> >> > 0.14 81.50 0.13 2236769 0.00 0.00 Curl_read >> >> > 0.14 81.62 0.12 2271700 0.00 0.00 fd_key_compare >> >> > 0.14 81.74 0.12 323167 0.00 0.00 event_del >> >> > 0.14 81.86 0.12 205436 0.00 0.00 >> >> > read_udp_packets >> >> > 0.14 81.98 0.12 200979 0.00 0.00 processfds >> >> > 0.13 82.09 0.11 123886 0.00 0.00 >> >> > Curl_if_is_interface_name >> >> > 0.13 82.20 0.11 120350 0.00 0.00 Curl_http >> >> > 0.13 82.31 0.11 ssl_tls_trace >> >> > 0.12 82.41 0.10 119229 0.00 0.00 >> >> > Curl_splayremovebyaddr >> >> > 0.12 82.51 0.10 7798717 0.00 0.00 Curl_raw_nequal >> >> > 0.12 82.61 0.10 784485 0.00 0.00 event_add >> >> > 0.10 82.70 0.09 643482 0.00 0.00 Curl_expire >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > test 6: >> >> > enable c-ares; >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); >> >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0 >> >> > >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> > *.com >> >> > to >> >> > one IP address >> >> > number of domain names in URLs: 780,000 >> >> > number of client: 1000 >> >> > CPU usage: 86% >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> > 85.9 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb >> >> > SSL TPS: 360 >> >> > SSL throughput: 200Mbps >> >> > >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> >> > Flat profile: >> >> > Each sample counts as 0.01 seconds. >> >> > % cumulative self self total >> >> > time seconds seconds calls s/call s/call name >> >> > 27.19 19.96 19.96 103472 0.00 0.00 >> >> > Curl_hash_clean_with_criterium >> >> > 21.73 35.91 15.95 274168981 0.00 0.00 >> >> > hostcache_timestamp_remove >> >> > 10.33 43.49 7.58 2256441 0.00 0.00 Curl_hash_pick >> >> > 7.21 48.78 5.29 104540 0.00 0.00 create_conn >> >> > 6.20 53.33 4.55 103413405 0.00 0.00 >> >> > Curl_str_key_compare >> >> > 4.88 56.91 3.58 287109 0.00 0.00 Curl_hash_add >> >> > 3.02 59.13 2.22 3797855 0.00 0.00 dprintf_formatf >> >> > 2.10 60.67 1.54 107711 0.00 0.00 ConnectionStore >> >> > 1.65 61.88 1.21 642725327 0.00 0.00 >> >> > ares__is_list_empty >> >> > 1.19 62.76 0.88 >> >> > locking_function >> >> > 0.87 63.40 0.64 207817 0.00 0.00 ares_cancel >> >> > 0.56 63.81 0.41 404108 0.00 0.00 >> >> > curl_multi_socket_action >> >> > 0.54 64.21 0.40 425603 0.00 0.00 >> >> > ares__init_list_node >> >> > 0.53 64.60 0.39 7068112 0.00 0.00 >> >> > client_tracing_function >> >> > 0.49 64.96 0.36 1908587 0.00 0.00 multi_runsingle >> >> > >> >> > >> >> > >> >> > test 7: >> >> > enable c-ares; >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); >> >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0 >> >> > >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> > *.com >> >> > to >> >> > one IP address >> >> > number of domain names in URLs: 780,000 >> >> > number of client: 1000 >> >> > CPU usage: 86% >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> > 86.6 0.0 0.6 3.3 0.0 3624.8Mb 263.0Mb >> >> > SSL TPS: 300 >> >> > SSL throughput: 200Mbps >> >> > >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> >> > Flat profile: >> >> > Each sample counts as 0.01 seconds. >> >> > % cumulative self self total >> >> > time seconds seconds calls s/call s/call name >> >> > 39.83 369.71 369.71 207173 0.00 0.00 >> >> > Curl_hash_clean_with_criterium >> >> > 24.27 594.98 225.27 3198337733 0.00 0.00 >> >> > hostcache_timestamp_remove >> >> > 13.75 722.59 127.61 4312864 0.00 0.00 Curl_hash_pick >> >> > 9.19 807.87 85.28 546507 0.00 0.00 Curl_hash_add >> >> > 6.14 864.88 57.01 1022419684 0.00 0.00 >> >> > Curl_str_key_compare >> >> > 2.06 884.00 19.12 203749 0.00 0.00 create_conn >> >> > 0.54 888.99 4.99 213306 0.00 0.00 ConnectionStore >> >> > 0.49 893.50 4.51 7331251 0.00 0.00 dprintf_formatf >> >> > 0.29 896.16 2.66 1287182329 0.00 0.00 >> >> > ares__is_list_empty >> >> > 0.23 898.28 2.12 >> >> > locking_function >> >> > 0.20 900.10 1.82 212210 0.00 0.00 >> >> > Curl_num_addresses >> >> > 0.17 901.66 1.56 198385 0.00 0.00 >> >> > curl_multi_remove_handle >> >> > 0.12 902.81 1.15 411114 0.00 0.00 ares_cancel >> >> > 0.12 903.95 1.14 3683448 0.00 0.00 multi_runsingle >> >> > 0.11 904.97 1.03 13664120 0.00 0.00 >> >> > client_tracing_function >> >> > >> >> > >> >> > test 8: >> >> > enable c-ares; >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); >> >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0 >> >> > >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> > *.com >> >> > to >> >> > one IP address >> >> > number of domain names in URLs: 780,000 >> >> > number of client: 1000 >> >> > CPU usage: 75% >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> > 75.2 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb >> >> > SSL TPS: 380 >> >> > SSL throughput: 220Mbps >> >> > >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> >> > Flat profile: >> >> > Each sample counts as 0.01 seconds. >> >> > % cumulative self self total >> >> > time seconds seconds calls s/call s/call name >> >> > 27.58 12.31 12.31 71140 0.00 0.00 >> >> > Curl_hash_clean_with_criterium >> >> > 19.09 20.83 8.52 176729692 0.00 0.00 >> >> > hostcache_timestamp_remove >> >> > 10.44 25.49 4.66 1589309 0.00 0.00 Curl_hash_pick >> >> > 6.85 28.55 3.06 73204 0.00 0.00 create_conn >> >> > 5.56 31.03 2.48 67620530 0.00 0.00 >> >> > Curl_str_key_compare >> >> > 4.41 33.00 1.97 200684 0.00 0.00 Curl_hash_add >> >> > 3.56 34.59 1.59 2717956 0.00 0.00 dprintf_formatf >> >> > 1.72 35.36 0.77 75247 0.00 0.00 ConnectionStore >> >> > 1.70 36.12 0.76 444342256 0.00 0.00 >> >> > ares__is_list_empty >> >> > 1.23 36.67 0.55 >> >> > locking_function >> >> > 1.16 37.19 0.52 143342 0.00 0.00 ares_cancel >> >> > 0.99 37.63 0.44 72211 0.00 0.00 >> >> > curl_multi_remove_handle >> >> > 0.74 37.96 0.33 1333371 0.00 0.00 multi_runsingle >> >> > 0.68 38.27 0.31 101147798 0.00 0.00 addbyter >> >> > 0.58 38.53 0.26 288737 0.00 0.00 >> >> > curl_multi_socket_action >> >> > >> >> > >> >> > >> >> > test 9: >> >> > enable c-ares; >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); >> >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0 >> >> > >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> > *.com >> >> > to >> >> > one IP address >> >> > number of domain names in URLs: 780,000 >> >> > number of client: 1000 >> >> > CPU usage: 75% >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> > 75.1 0.0 0.6 4.1 0.0 3545.2Mb 260.1Mb >> >> > SSL TPS: 380 >> >> > SSL throughput: 200Mbps >> >> > >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> >> > Flat profile: >> >> > Each sample counts as 0.01 seconds. >> >> > % cumulative self self total >> >> > time seconds seconds calls s/call s/call name >> >> > 34.00 31.13 31.13 72495 0.00 0.00 >> >> > Curl_hash_clean_with_criterium >> >> > 23.64 52.78 21.65 386070412 0.00 0.00 >> >> > hostcache_timestamp_remove >> >> > 13.28 64.94 12.16 1619440 0.00 0.00 Curl_hash_pick >> >> > 5.92 70.36 5.42 137653129 0.00 0.00 >> >> > Curl_str_key_compare >> >> > 5.50 75.40 5.04 203018 0.00 0.00 Curl_hash_add >> >> > 4.55 79.57 4.17 74618 0.00 0.00 create_conn >> >> > 1.70 81.13 1.56 2757148 0.00 0.00 dprintf_formatf >> >> > 0.98 82.03 0.90 76945 0.00 0.00 ConnectionStore >> >> > 0.95 82.90 0.87 455775100 0.00 0.00 >> >> > ares__is_list_empty >> >> > 0.57 83.42 0.53 >> >> > locking_function >> >> > 0.39 83.78 0.36 146742 0.00 0.00 ares_cancel >> >> > 0.35 84.10 0.32 102968806 0.00 0.00 addbyter >> >> > 0.35 84.42 0.32 456587 0.00 0.00 Curl_readwrite >> >> > 0.35 84.74 0.32 73133 0.00 0.00 >> >> > curl_multi_remove_handle >> >> > 0.34 85.05 0.31 1360545 0.00 0.00 multi_runsingle >> >> > >> >> > >> >> > 2012/3/6 Robert Iakobashvili <cor...@gm...> >> >> >> >> >> >> If it will not be helpful this means that recent >> >> >> changes in libcurl library have broken DNS interface. >> >> >> >> >> >> You can try a previos version and the latest libcurl, like: >> >> >> >> >> >> http://curl.haxx.se/download/curl-7.23.0.tar.gz >> >> >> and >> >> >> http://curl.haxx.se/downloadcurl-7.24.0.tar.gz >> >> >> >> >> >> by: >> >> >> >> >> >> 1. downloading a libcurl version and placing it to >> >> >> curl-loader-0.56/packages ; >> >> >> 2. correcting the Makefile in curl-loader-0.56 >> >> >> CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24 or CURL_VER:=7.23 >> >> >> 3. make cleanall; make >> >> >> >> >> >> Please, send us your observations. >> >> >> Thanks! >> >> >> Robert >> >> >> >> >> >> On Mon, Mar 5, 2012 at 6:57 PM, Robert Iakobashvili >> >> >> <cor...@gm...> >> >> >> wrote: >> >> >> > If so, please try to change >> >> >> > >> >> >> > loader.c, line 578 which looks like: >> >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); >> >> >> > >> >> >> > 1. instead of -1 try to set some value like 60 >> >> >> > 2. make clean; make >> >> >> > >> >> >> > Thanks, >> >> >> > Robert >> >> >> > >> >> >> > On Mon, Mar 5, 2012 at 6:19 PM, Fred Huang <di...@gm...> >> >> >> > wrote: >> >> >> >> >> >> >> >> Looks like the bottleneck comes from libcurl other than c-ares... >> >> >> >> test 3 (no ares patch applied): >> >> >> >> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> >> >> *.com >> >> >> >> to one IP address >> >> >> >> number of domain names in URLs: 780,000 >> >> >> >> number of client: 1000 >> >> >> >> CPU usage: 85% >> >> >> >> cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> >> >> 85.6 0.0 0.8 5.9 0.0 3334.3Mb 112.0Mb >> >> >> >> SSL TPS: 320 >> >> >> >> SSL throughput: 200Mbps >> >> >> >> >> >> >> >> # gprof /usr/bin/curl-loader gmon.out -p | head -50 >> >> >> >> Flat profile: >> >> >> >> Each sample counts as 0.01 seconds. >> >> >> >> % cumulative self self total >> >> >> >> time seconds seconds calls s/call s/call name >> >> >> >> 40.80 120.16 120.16 116079 0.00 0.00 >> >> >> >> Curl_hash_clean_with_criterium >> >> >> >> 24.46 192.18 72.02 983166323 0.00 0.00 >> >> >> >> hostcache_timestamp_remove >> >> >> >> 13.30 231.35 39.17 2059341 0.00 0.00 >> >> >> >> Curl_hash_pick >> >> >> >> 6.45 250.35 19.00 240087 0.00 0.00 >> >> >> >> Curl_hash_add >> >> >> >> 4.81 264.51 14.16 334257102 0.00 0.00 >> >> >> >> Curl_str_key_compare >> >> >> >> 3.40 274.52 10.01 119514 0.00 0.00 create_conn >> >> >> >> 0.93 277.25 2.73 4478910 0.00 0.00 >> >> >> >> dprintf_formatf >> >> >> >> 0.90 279.91 2.66 122209 0.00 0.00 >> >> >> >> ConnectionStore >> >> >> >> 0.40 281.09 1.18 >> >> >> >> locking_function >> >> >> >> 0.27 281.88 0.79 114759 0.00 0.00 >> >> >> >> curl_multi_remove_handle >> >> >> >> 0.21 282.51 0.63 7993664 0.00 0.00 >> >> >> >> client_tracing_function >> >> >> >> 0.21 283.12 0.61 122685 0.00 0.00 >> >> >> >> Curl_num_addresses >> >> >> >> 0.19 283.68 0.56 397028 0.00 0.00 >> >> >> >> curl_multi_socket_action >> >> >> >> 0.19 284.23 0.55 2036116 0.00 0.00 >> >> >> >> multi_runsingle >> >> >> >> 0.17 284.72 0.49 703687 0.00 0.00 >> >> >> >> Curl_readwrite >> >> >> >> 0.16 285.19 0.47 id_function >> >> >> >> 0.15 285.62 0.43 29692029 0.00 0.00 >> >> >> >> Curl_socket_check >> >> >> >> 0.14 286.02 0.40 91973856 0.00 0.00 >> >> >> >> Curl_raw_toupper >> >> >> >> 0.13 286.40 0.38 164664144 0.00 0.00 addbyter >> >> >> >> 0.13 286.77 0.37 2180797 0.00 0.00 Curl_infof >> >> >> >> 0.11 287.10 0.33 122675 0.00 0.00 >> >> >> >> Curl_if_is_interface_name >> >> >> >> 0.10 287.39 0.29 471318 0.00 0.00 >> >> >> >> ossl_connect_common >> >> >> >> 0.10 287.67 0.28 2113528 0.00 0.00 >> >> >> >> Curl_raw_equal >> >> >> >> 0.09 287.93 0.26 8676 0.00 0.02 >> >> >> >> curl_multi_socket_all >> >> >> >> 0.08 288.17 0.24 1663262 0.00 0.00 Curl_splay >> >> >> >> 0.08 288.41 0.24 8505 0.00 0.01 >> >> >> >> curl_multi_perform >> >> >> >> 0.07 288.63 0.22 3114576 0.00 0.00 >> >> >> >> Curl_pgrsUpdate >> >> >> >> 0.06 288.80 0.17 119637 0.00 0.00 >> >> >> >> singleipconnect >> >> >> >> 0.06 288.97 0.17 7994818 0.00 0.00 Curl_debug >> >> >> >> 0.05 289.13 0.16 235820 0.00 0.00 event_del >> >> >> >> 0.05 289.29 0.16 119393071 0.00 0.00 >> >> >> >> curl_strequal >> >> >> >> 0.05 289.45 0.16 18535473 0.00 0.00 >> >> >> >> alloc_addbyter >> >> >> >> 0.05 289.61 0.16 356074 0.00 0.00 >> >> >> >> Curl_hash_str >> >> >> >> 0.05 289.76 0.15 1797233 0.00 0.00 >> >> >> >> fd_key_compare >> >> >> >> 0.05 289.91 0.15 234966 0.00 0.00 >> >> >> >> Curl_ssl_getsessionid >> >> >> >> 0.04 290.04 0.13 585853 0.00 0.00 >> >> >> >> Curl_llist_destroy >> >> >> >> 0.04 290.16 0.12 240826 0.00 0.00 Curl_resolv >> >> >> >> 0.03 290.26 0.10 7514328 0.00 0.00 >> >> >> >> Curl_raw_nequal >> >> >> >> 0.03 290.36 0.10 2214436 0.00 0.00 Curl_read >> >> >> >> 0.03 290.46 0.10 116872 0.00 0.00 >> >> >> >> Curl_http_readwrite_headers >> >> >> >> 0.03 290.56 0.10 >> >> >> >> ssl_tls_trace >> >> >> >> 0.03 290.66 0.10 8013097 0.00 0.00 >> >> >> >> scan_response >> >> >> >> 0.03 290.75 0.09 29488555 0.00 0.00 >> >> >> >> curlx_tvdiff >> >> >> >> 0.03 290.84 0.09 2142468 0.00 0.00 Curl_setopt >> >> >> >> 0.03 290.93 0.09 476299 0.00 0.00 Curl_expire >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> 2012/3/5 Robert Iakobashvili <cor...@gm...> >> >> >> >>> >> >> >> >>> Dear Fred Huang, >> >> >> >>> >> >> >> >>> Thanks for your posting. >> >> >> >>> >> >> >> >>> Could you please check if this is due to some badness at cares? >> >> >> >>> Thanks. >> >> >> >>> >> >> >> >>> Let's exclude usage of cares based DNS resolving by applying the >> >> >> >>> patch >> >> >> >>> attached >> >> >> >>> and use instead global DNS cache. >> >> >> >>> >> >> >> >>> cd ./curl-loader-0.56 >> >> >> >>> patch -p1 < path-to/disable-cares-use-global-dns.patch >> >> >> >>> make cleanall; make >> >> >> >>> >> >> >> >>> See if it helps. >> >> >> >>> >> >> >> >>> Global cache, unfortunately, is not thread safe. >> >> >> >>> >> >> >> >>> If it helps, we'll try to restore the previous version of cares. >> >> >> >>> >> >> >> >>> Thanks, >> >> >> >>> Robert >> >> >> >>> >> >> >> >>> On Mon, Mar 5, 2012 at 4:28 PM, Fred Huang <di...@gm...> >> >> >> >>> wrote: >> >> >> >>>> >> >> >> >>>> test 1: >> >> >> >>>> >> >> >> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, >> >> >> >>>> resolve >> >> >> >>>> *.com to one IP address >> >> >> >>>> number of domain names in all https URLs: 780,000 >> >> >> >>>> number of client: 1000 >> >> >> >>>> CPU usage: 78% >> >> >> >>>> cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> >> >>>> 78.8 0.0 0.6 3.8 0.0 3640.9Mb 282.1Mb >> >> >> >>>> SSL TPS: 300 >> >> >> >>>> SSL throughput: 200Mbps >> >> >> >>>> >> >> >> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50 >> >> >> >>>> Flat profile: >> >> >> >>>> Each sample counts as 0.01 seconds. >> >> >> >>>> % cumulative self self total >> >> >> >>>> time seconds seconds calls s/call s/call name >> >> >> >>>> 38.55 127.10 127.10 130975 0.00 0.00 >> >> >> >>>> Curl_hash_clean_with_criterium >> >> >> >>>> 24.63 208.32 81.22 1247528407 0.00 0.00 >> >> >> >>>> hostcache_timestamp_remove >> >> >> >>>> 13.08 251.44 43.12 2791988 0.00 0.00 >> >> >> >>>> Curl_hash_pick >> >> >> >>>> 7.30 275.51 24.07 352490 0.00 0.00 >> >> >> >>>> Curl_hash_add >> >> >> >>>> 6.01 295.31 19.80 410760770 0.00 0.00 >> >> >> >>>> Curl_str_key_compare >> >> >> >>>> 2.95 305.04 9.73 131015 0.00 0.00 >> >> >> >>>> create_conn >> >> >> >>>> 0.88 307.94 2.90 4841390 0.00 0.00 >> >> >> >>>> dprintf_formatf >> >> >> >>>> 0.74 310.38 2.44 136036 0.00 0.00 >> >> >> >>>> ConnectionStore >> >> >> >>>> 0... [truncated message content] |
|
From: Fred H. <di...@gm...> - 2012-03-07 13:49:19
|
I tried -t 16 or -t 32, the performance drops to approx. 50 CAPS... 2012/3/7 Robert Iakobashvili <cor...@gm...> > Fred, > > On more suggestion, which is a general one. > > If N is the number of your logical CPUs, > usually, I'd recommend to run 2*N + B threads. > > Thus, you can try: > > -t 16 > > -t 32 > > and it MAY increased a bit CAPS. > > Thanks, > Robert > > On Wed, Mar 7, 2012 at 2:43 AM, Fred Huang <di...@gm...> wrote: > > yes, I use -t 8 option because the test box has 8 logic CPUs (Dell 990, > > Intel Core i7 2600, 16GB memory). > > if I do not use -t option, the CPU used by curl-loader will be 100% on > one > > of 8 logic CPU (12% of total CPU), and the CAPS drops to 100: > > # gprof /usr/bin/curl-loader gmon.out -p | head -20 > > Flat profile: > > Each sample counts as 0.01 seconds. > > % cumulative self self total > > time seconds seconds calls s/call s/call name > > 34.12 2.74 2.74 17385 0.00 0.00 create_conn > > 11.21 3.64 0.90 30998583 0.00 0.00 > > hostcache_timestamp_remove > > 9.84 4.43 0.79 13433 0.00 0.00 > > Curl_hash_clean_with_criterium > > 7.97 5.07 0.64 381818 0.00 0.00 Curl_hash_pick > > 5.85 5.54 0.47 17385 0.00 0.00 ConnectionStore > > 5.11 5.95 0.41 16385 0.00 0.00 > > curl_multi_remove_handle > > 4.11 6.28 0.33 16509889 0.00 0.00 > Curl_str_key_compare > > 3.36 6.55 0.27 23959914 0.00 0.00 Curl_socket_check > > 3.24 6.81 0.26 56703 0.00 0.00 Curl_hash_add > > 2.62 7.02 0.21 635750 0.00 0.00 dprintf_formatf > > 1.25 7.12 0.10 92421190 0.00 0.00 ares__is_list_empty > > 0.75 7.18 0.06 97957 0.00 0.00 Curl_readwrite > > 0.62 7.23 0.05 23422260 0.00 0.00 addbyter > > 0.62 7.28 0.05 1136775 0.00 0.00 > > client_tracing_function > > 0.50 7.32 0.04 3091375 0.00 0.00 > ares__init_list_head > > > > I also tried to use > > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 20); > > but get no improvement on CAPS. > > > > > > Regards, > > Fred > > > > 2012/3/7 Robert Iakobashvili <cor...@gm...> > >> > >> Dear Fred, > >> > >> If I understand correctly your observations, the issue is not in: > >> > >> 1. DNS resolving; > >> 2. Version of cares or libcurl; > >> > >> but rather in > >> > >> Less effective handling of HTTPS versus HTTP in libcurl. > >> > >> Please, confirm that my understanding is correct. > >> > >> > >> HTTPS URLs are using an extra locking in openssl with -t option > >> that could be making the issue. > >> If you are using -t option, try to load without it and let's say with > >> 300 clients. > >> > >> If you are not using -t, thus the only way to proceed is to share > >> your observations with libcurl people at > >> > >> libcurl development <cur...@co...> > >> > >> Thank you. > >> > >> Best wishes. > >> Robert > >> > >> On Tue, Mar 6, 2012 at 6:17 PM, Fred Huang <di...@gm...> wrote: > >> > 1. the major CPU consumer is curl-loader. here is the top: > >> > top - 07:36:36 up 23:31, 5 users, load average: 4.32, 2.26, 0.91 > >> > Tasks: 237 total, 2 running, 235 sleeping, 0 stopped, 0 zombie > >> > Cpu0 : 76.6%us, 3.6%sy, 0.0%ni, 19.4%id, 0.0%wa, 0.0%hi, 0.3%si, > >> > 0.0% > >> > Cpu1 : 72.4%us, 4.0%sy, 0.0%ni, 22.9%id, 0.0%wa, 0.0%hi, 0.7%si, > >> > 0.0% > >> > Cpu2 : 65.8%us, 3.3%sy, 0.0%ni, 30.6%id, 0.0%wa, 0.0%hi, 0.3%si, > >> > 0.0% > >> > Cpu3 : 75.2%us, 4.3%sy, 0.0%ni, 19.5%id, 0.0%wa, 0.0%hi, 1.0%si, > >> > 0.0% > >> > Cpu4 : 79.1%us, 4.0%sy, 0.0%ni, 16.8%id, 0.0%wa, 0.0%hi, 0.0%si, > >> > 0.0% > >> > Cpu5 : 75.8%us, 4.3%sy, 0.0%ni, 19.2%id, 0.0%wa, 0.0%hi, 0.7%si, > >> > 0.0% > >> > Cpu6 : 74.1%us, 5.0%sy, 0.0%ni, 20.3%id, 0.0%wa, 0.0%hi, 0.7%si, > >> > 0.0% > >> > Cpu7 : 78.7%us, 4.0%sy, 0.0%ni, 16.6%id, 0.0%wa, 0.0%hi, 0.7%si, > >> > 0.0% > >> > Mem: 16209932k total, 3829500k used, 12380432k free, 99112k > buffers > >> > Swap: 18448376k total, 0k used, 18448376k free, 320284k > cached > >> > > >> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > >> > 30795 root 20 0 3200m 2.8g 2296 S 79.2 18.4 21:26.84 > curl-loader > >> > 1832 nobody 20 0 13944 1736 580 S 0.1 0.0 0:32.65 dnsmasq > >> > 9 root 20 0 0 0 0 S 0.0 0.0 0:00.53 > ksoftirqd/1 > >> > 30825 root 20 0 15220 1360 960 R 0.0 0.0 0:00.42 top > >> > 1 root 20 0 19400 1568 1256 S 0.0 0.0 0:00.75 init > >> > 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd > >> > 3 root RT 0 0 0 0 S 0.0 0.0 0:00.00 > migration/0 > >> > 4 root 20 0 0 0 0 S 0.0 0.0 0:00.06 > ksoftirqd/0 > >> > 5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 > migration/0 > >> > > >> > > >> > 2. 1000 loading clients > >> > > >> > 3. yes, tried 0.52 but no lucky -- CAPS 380, throughput 200Mbps, CPU > 90% > >> > top - 08:17:10 up 1 day, 12 min, 8 users, load average: 4.66, 3.39, > >> > 1.88 > >> > Tasks: 245 total, 2 running, 243 sleeping, 0 stopped, 0 zombie > >> > Cpu0 : 82.2%us, 4.0%sy, 0.0%ni, 13.9%id, 0.0%wa, 0.0%hi, 0.0%si, > >> > 0.0% > >> > Cpu1 : 86.1%us, 4.3%sy, 0.0%ni, 7.9%id, 0.0%wa, 0.0%hi, 1.7%si, > >> > 0.0% > >> > Cpu2 : 85.3%us, 4.0%sy, 0.0%ni, 10.0%id, 0.0%wa, 0.0%hi, 0.7%si, > >> > 0.0% > >> > Cpu3 : 81.8%us, 3.0%sy, 0.0%ni, 14.2%id, 0.0%wa, 0.0%hi, 1.0%si, > >> > 0.0% > >> > Cpu4 : 83.7%us, 4.0%sy, 0.0%ni, 12.3%id, 0.0%wa, 0.0%hi, 0.0%si, > >> > 0.0% > >> > Cpu5 : 92.7%us, 3.3%sy, 0.0%ni, 3.3%id, 0.0%wa, 0.0%hi, 0.7%si, > >> > 0.0% > >> > Cpu6 : 89.7%us, 2.0%sy, 0.0%ni, 8.0%id, 0.0%wa, 0.0%hi, 0.3%si, > >> > 0.0% > >> > Cpu7 : 88.1%us, 3.6%sy, 0.0%ni, 6.6%id, 0.0%wa, 0.0%hi, 1.7%si, > >> > 0.0% > >> > Mem: 16209932k total, 1795748k used, 14414184k free, 102824k > buffers > >> > Swap: 18448376k total, 0k used, 18448376k free, 382988k > cached > >> > > >> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > >> > 12236 root 20 0 1150m 894m 2572 S 90.7 5.7 51:11.91 > curl-loader > >> > 1832 nobody 20 0 13944 1736 580 S 0.1 0.0 0:47.04 dnsmasq > >> > > >> > ]# gprof /usr/bin/curl-loader gmon.out -p | head -20 > >> > Flat profile: > >> > Each sample counts as 0.01 seconds. > >> > % cumulative self self total > >> > time seconds seconds calls s/call s/call name > >> > 43.92 151.95 151.95 157295 0.00 0.00 > >> > Curl_hash_clean_with_criterium > >> > 22.90 231.17 79.22 1712190971 0.00 0.00 > >> > hostcache_timestamp_remove > >> > 13.15 276.68 45.51 3260321 0.00 0.00 Curl_hash_pick > >> > 7.51 302.65 25.97 394171 0.00 0.00 Curl_hash_add > >> > 5.98 323.34 20.68 567437518 0.00 0.00 > >> > Curl_str_key_compare > >> > 1.00 326.80 3.46 5380220 0.00 0.00 dprintf_formatf > >> > 0.41 328.21 1.41 locking_function > >> > 0.32 329.32 1.11 469222138 0.00 0.00 > >> > ares__is_list_empty > >> > 0.31 330.40 1.08 3284508 0.00 0.00 multi_runsingle > >> > 0.28 331.38 0.98 10006806 0.00 0.00 > >> > client_tracing_function > >> > 0.24 332.21 0.83 151951 0.00 0.00 > Curl_num_addresses > >> > 0.22 332.97 0.77 188075672 0.00 0.00 addbyter > >> > 0.19 333.64 0.67 841840 0.00 0.00 Curl_readwrite > >> > 0.19 334.29 0.65 id_function > >> > 0.18 334.90 0.61 39843 0.00 0.01 > >> > curl_multi_socket_all > >> > > >> > > >> > > >> > 4. yes, it is correct for 0.56. If use same URL list by replacing > https > >> > to > >> > http, I can get CAPS 1500 and throughput 870Mbps with CPU% < 60% > >> > > >> > > >> > 2012/3/6 Robert Iakobashvili <cor...@gm...> > >> >> > >> >> Several questions: > >> >> > >> >> 1. Could you please run top and see if there are other processes > >> >> burning > >> >> CPU? > >> >> 2. What is the number of loading clients that you use? > >> >> 3. Could you try curl-loader-0.52 if it compiles for you? > >> >> If not, please try to place curl-loader-7.19.7.tar.gz and > >> >> c-ares-16.0.tar.gz, > >> >> update their versions in Makefile and rebuild by make cleanall; make > >> >> > >> >> 4. Is it correct to say that for HTTP URLs (not HTTPS) you are > getting > >> >> a good throughput > >> >> and CAPS with curl-loader-0.56? > >> >> > >> >> Thanks, > >> >> Robert > >> >> > >> >> On Tue, Mar 6, 2012 at 3:52 AM, Fred Huang <di...@gm...> wrote: > >> >> > TIMEOUT -1 -> 60 works a little better but no performance > improvement > >> >> > on > >> >> > CAPS or throughput. > >> >> > > >> >> > test 5: > >> >> > enable c-ares; > >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); > >> >> > > >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve > >> >> > *.com > >> >> > to > >> >> > one IP address > >> >> > number of domain names in URLs: 780,000 > >> >> > number of client: 1000 > >> >> > CPU usage: 74% > >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached > >> >> > 74.0 0.0 0.7 4.2 0.0 3530.6Mb 250.8Mb > >> >> > SSL TPS: 360 > >> >> > SSL throughput: 220Mbps > >> >> > > >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -50 > >> >> > Flat profile: > >> >> > Each sample counts as 0.01 seconds. > >> >> > % cumulative self self total > >> >> > time seconds seconds calls s/call s/call name > >> >> > 26.63 23.13 23.13 119226 0.00 0.00 > >> >> > Curl_hash_clean_with_criterium > >> >> > 20.34 40.80 17.67 317792146 0.00 0.00 > >> >> > hostcache_timestamp_remove > >> >> > 11.27 50.59 9.79 2579903 0.00 0.00 > Curl_hash_pick > >> >> > 7.84 57.40 6.81 120606 0.00 0.00 create_conn > >> >> > 5.65 62.31 4.91 119117401 0.00 0.00 > >> >> > Curl_str_key_compare > >> >> > 4.98 66.64 4.33 329829 0.00 0.00 Curl_hash_add > >> >> > 2.80 69.08 2.44 4476606 0.00 0.00 > dprintf_formatf > >> >> > 2.57 71.31 2.23 124019 0.00 0.00 > ConnectionStore > >> >> > 1.50 72.61 1.30 740997609 0.00 0.00 > >> >> > ares__is_list_empty > >> >> > 1.16 73.61 1.01 > >> >> > locking_function > >> >> > 0.76 74.27 0.66 238744 0.00 0.00 ares_cancel > >> >> > 0.70 74.88 0.61 457564 0.00 0.00 > >> >> > curl_multi_socket_action > >> >> > 0.67 75.46 0.58 118616 0.00 0.00 > >> >> > curl_multi_remove_handle > >> >> > 0.58 75.97 0.51 8211502 0.00 0.00 > >> >> > client_tracing_function > >> >> > 0.53 76.43 0.46 733561 0.00 0.00 > Curl_readwrite > >> >> > 0.51 76.87 0.44 2183179 0.00 0.00 > multi_runsingle > >> >> > 0.51 77.31 0.44 166585567 0.00 0.00 addbyter > >> >> > 0.44 77.69 0.38 2175805 0.00 0.00 > Curl_raw_equal > >> >> > 0.43 78.06 0.38 id_function > >> >> > 0.40 78.41 0.35 29789346 0.00 0.00 > >> >> > Curl_socket_check > >> >> > 0.40 78.76 0.35 479671 0.00 0.00 > >> >> > ossl_connect_common > >> >> > 0.40 79.11 0.35 490639 0.00 0.00 > >> >> > ares__init_list_node > >> >> > 0.33 79.40 0.29 94197670 0.00 0.00 > >> >> > Curl_raw_toupper > >> >> > 0.33 79.69 0.29 3268031 0.00 0.00 > Curl_pgrsUpdate > >> >> > 0.28 79.93 0.24 8797 0.00 0.00 > >> >> > curl_multi_perform > >> >> > 0.27 80.17 0.24 2233641 0.00 0.00 Curl_infof > >> >> > 0.26 80.40 0.23 1839837 0.00 0.00 Curl_splay > >> >> > 0.24 80.61 0.21 121713028 0.00 0.00 > curl_strequal > >> >> > 0.21 80.79 0.18 243333 0.00 0.00 > >> >> > Curl_ssl_getsessionid > >> >> > 0.18 80.95 0.16 122180 0.00 0.00 > >> >> > Curl_num_addresses > >> >> > 0.17 81.09 0.15 8271666 0.00 0.00 Curl_debug > >> >> > 0.16 81.23 0.14 18902452 0.00 0.00 > alloc_addbyter > >> >> > 0.16 81.37 0.14 8851 0.00 0.00 > >> >> > curl_multi_socket_all > >> >> > 0.14 81.50 0.13 2236769 0.00 0.00 Curl_read > >> >> > 0.14 81.62 0.12 2271700 0.00 0.00 > fd_key_compare > >> >> > 0.14 81.74 0.12 323167 0.00 0.00 event_del > >> >> > 0.14 81.86 0.12 205436 0.00 0.00 > >> >> > read_udp_packets > >> >> > 0.14 81.98 0.12 200979 0.00 0.00 processfds > >> >> > 0.13 82.09 0.11 123886 0.00 0.00 > >> >> > Curl_if_is_interface_name > >> >> > 0.13 82.20 0.11 120350 0.00 0.00 Curl_http > >> >> > 0.13 82.31 0.11 ssl_tls_trace > >> >> > 0.12 82.41 0.10 119229 0.00 0.00 > >> >> > Curl_splayremovebyaddr > >> >> > 0.12 82.51 0.10 7798717 0.00 0.00 > Curl_raw_nequal > >> >> > 0.12 82.61 0.10 784485 0.00 0.00 event_add > >> >> > 0.10 82.70 0.09 643482 0.00 0.00 Curl_expire > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > test 6: > >> >> > enable c-ares; > >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); > >> >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0 > >> >> > > >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve > >> >> > *.com > >> >> > to > >> >> > one IP address > >> >> > number of domain names in URLs: 780,000 > >> >> > number of client: 1000 > >> >> > CPU usage: 86% > >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached > >> >> > 85.9 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb > >> >> > SSL TPS: 360 > >> >> > SSL throughput: 200Mbps > >> >> > > >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 > >> >> > Flat profile: > >> >> > Each sample counts as 0.01 seconds. > >> >> > % cumulative self self total > >> >> > time seconds seconds calls s/call s/call name > >> >> > 27.19 19.96 19.96 103472 0.00 0.00 > >> >> > Curl_hash_clean_with_criterium > >> >> > 21.73 35.91 15.95 274168981 0.00 0.00 > >> >> > hostcache_timestamp_remove > >> >> > 10.33 43.49 7.58 2256441 0.00 0.00 > Curl_hash_pick > >> >> > 7.21 48.78 5.29 104540 0.00 0.00 create_conn > >> >> > 6.20 53.33 4.55 103413405 0.00 0.00 > >> >> > Curl_str_key_compare > >> >> > 4.88 56.91 3.58 287109 0.00 0.00 Curl_hash_add > >> >> > 3.02 59.13 2.22 3797855 0.00 0.00 > dprintf_formatf > >> >> > 2.10 60.67 1.54 107711 0.00 0.00 > ConnectionStore > >> >> > 1.65 61.88 1.21 642725327 0.00 0.00 > >> >> > ares__is_list_empty > >> >> > 1.19 62.76 0.88 > >> >> > locking_function > >> >> > 0.87 63.40 0.64 207817 0.00 0.00 ares_cancel > >> >> > 0.56 63.81 0.41 404108 0.00 0.00 > >> >> > curl_multi_socket_action > >> >> > 0.54 64.21 0.40 425603 0.00 0.00 > >> >> > ares__init_list_node > >> >> > 0.53 64.60 0.39 7068112 0.00 0.00 > >> >> > client_tracing_function > >> >> > 0.49 64.96 0.36 1908587 0.00 0.00 > multi_runsingle > >> >> > > >> >> > > >> >> > > >> >> > test 7: > >> >> > enable c-ares; > >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); > >> >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0 > >> >> > > >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve > >> >> > *.com > >> >> > to > >> >> > one IP address > >> >> > number of domain names in URLs: 780,000 > >> >> > number of client: 1000 > >> >> > CPU usage: 86% > >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached > >> >> > 86.6 0.0 0.6 3.3 0.0 3624.8Mb 263.0Mb > >> >> > SSL TPS: 300 > >> >> > SSL throughput: 200Mbps > >> >> > > >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 > >> >> > Flat profile: > >> >> > Each sample counts as 0.01 seconds. > >> >> > % cumulative self self total > >> >> > time seconds seconds calls s/call s/call name > >> >> > 39.83 369.71 369.71 207173 0.00 0.00 > >> >> > Curl_hash_clean_with_criterium > >> >> > 24.27 594.98 225.27 3198337733 0.00 0.00 > >> >> > hostcache_timestamp_remove > >> >> > 13.75 722.59 127.61 4312864 0.00 0.00 > Curl_hash_pick > >> >> > 9.19 807.87 85.28 546507 0.00 0.00 Curl_hash_add > >> >> > 6.14 864.88 57.01 1022419684 0.00 0.00 > >> >> > Curl_str_key_compare > >> >> > 2.06 884.00 19.12 203749 0.00 0.00 create_conn > >> >> > 0.54 888.99 4.99 213306 0.00 0.00 > ConnectionStore > >> >> > 0.49 893.50 4.51 7331251 0.00 0.00 > dprintf_formatf > >> >> > 0.29 896.16 2.66 1287182329 0.00 0.00 > >> >> > ares__is_list_empty > >> >> > 0.23 898.28 2.12 > >> >> > locking_function > >> >> > 0.20 900.10 1.82 212210 0.00 0.00 > >> >> > Curl_num_addresses > >> >> > 0.17 901.66 1.56 198385 0.00 0.00 > >> >> > curl_multi_remove_handle > >> >> > 0.12 902.81 1.15 411114 0.00 0.00 ares_cancel > >> >> > 0.12 903.95 1.14 3683448 0.00 0.00 > multi_runsingle > >> >> > 0.11 904.97 1.03 13664120 0.00 0.00 > >> >> > client_tracing_function > >> >> > > >> >> > > >> >> > test 8: > >> >> > enable c-ares; > >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); > >> >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0 > >> >> > > >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve > >> >> > *.com > >> >> > to > >> >> > one IP address > >> >> > number of domain names in URLs: 780,000 > >> >> > number of client: 1000 > >> >> > CPU usage: 75% > >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached > >> >> > 75.2 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb > >> >> > SSL TPS: 380 > >> >> > SSL throughput: 220Mbps > >> >> > > >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 > >> >> > Flat profile: > >> >> > Each sample counts as 0.01 seconds. > >> >> > % cumulative self self total > >> >> > time seconds seconds calls s/call s/call name > >> >> > 27.58 12.31 12.31 71140 0.00 0.00 > >> >> > Curl_hash_clean_with_criterium > >> >> > 19.09 20.83 8.52 176729692 0.00 0.00 > >> >> > hostcache_timestamp_remove > >> >> > 10.44 25.49 4.66 1589309 0.00 0.00 > Curl_hash_pick > >> >> > 6.85 28.55 3.06 73204 0.00 0.00 create_conn > >> >> > 5.56 31.03 2.48 67620530 0.00 0.00 > >> >> > Curl_str_key_compare > >> >> > 4.41 33.00 1.97 200684 0.00 0.00 Curl_hash_add > >> >> > 3.56 34.59 1.59 2717956 0.00 0.00 > dprintf_formatf > >> >> > 1.72 35.36 0.77 75247 0.00 0.00 > ConnectionStore > >> >> > 1.70 36.12 0.76 444342256 0.00 0.00 > >> >> > ares__is_list_empty > >> >> > 1.23 36.67 0.55 > >> >> > locking_function > >> >> > 1.16 37.19 0.52 143342 0.00 0.00 ares_cancel > >> >> > 0.99 37.63 0.44 72211 0.00 0.00 > >> >> > curl_multi_remove_handle > >> >> > 0.74 37.96 0.33 1333371 0.00 0.00 > multi_runsingle > >> >> > 0.68 38.27 0.31 101147798 0.00 0.00 addbyter > >> >> > 0.58 38.53 0.26 288737 0.00 0.00 > >> >> > curl_multi_socket_action > >> >> > > >> >> > > >> >> > > >> >> > test 9: > >> >> > enable c-ares; > >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); > >> >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0 > >> >> > > >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve > >> >> > *.com > >> >> > to > >> >> > one IP address > >> >> > number of domain names in URLs: 780,000 > >> >> > number of client: 1000 > >> >> > CPU usage: 75% > >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached > >> >> > 75.1 0.0 0.6 4.1 0.0 3545.2Mb 260.1Mb > >> >> > SSL TPS: 380 > >> >> > SSL throughput: 200Mbps > >> >> > > >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 > >> >> > Flat profile: > >> >> > Each sample counts as 0.01 seconds. > >> >> > % cumulative self self total > >> >> > time seconds seconds calls s/call s/call name > >> >> > 34.00 31.13 31.13 72495 0.00 0.00 > >> >> > Curl_hash_clean_with_criterium > >> >> > 23.64 52.78 21.65 386070412 0.00 0.00 > >> >> > hostcache_timestamp_remove > >> >> > 13.28 64.94 12.16 1619440 0.00 0.00 > Curl_hash_pick > >> >> > 5.92 70.36 5.42 137653129 0.00 0.00 > >> >> > Curl_str_key_compare > >> >> > 5.50 75.40 5.04 203018 0.00 0.00 Curl_hash_add > >> >> > 4.55 79.57 4.17 74618 0.00 0.00 create_conn > >> >> > 1.70 81.13 1.56 2757148 0.00 0.00 > dprintf_formatf > >> >> > 0.98 82.03 0.90 76945 0.00 0.00 > ConnectionStore > >> >> > 0.95 82.90 0.87 455775100 0.00 0.00 > >> >> > ares__is_list_empty > >> >> > 0.57 83.42 0.53 > >> >> > locking_function > >> >> > 0.39 83.78 0.36 146742 0.00 0.00 ares_cancel > >> >> > 0.35 84.10 0.32 102968806 0.00 0.00 addbyter > >> >> > 0.35 84.42 0.32 456587 0.00 0.00 > Curl_readwrite > >> >> > 0.35 84.74 0.32 73133 0.00 0.00 > >> >> > curl_multi_remove_handle > >> >> > 0.34 85.05 0.31 1360545 0.00 0.00 > multi_runsingle > >> >> > > >> >> > > >> >> > 2012/3/6 Robert Iakobashvili <cor...@gm...> > >> >> >> > >> >> >> If it will not be helpful this means that recent > >> >> >> changes in libcurl library have broken DNS interface. > >> >> >> > >> >> >> You can try a previos version and the latest libcurl, like: > >> >> >> > >> >> >> http://curl.haxx.se/download/curl-7.23.0.tar.gz > >> >> >> and > >> >> >> http://curl.haxx.se/downloadcurl-7.24.0.tar.gz > >> >> >> > >> >> >> by: > >> >> >> > >> >> >> 1. downloading a libcurl version and placing it to > >> >> >> curl-loader-0.56/packages ; > >> >> >> 2. correcting the Makefile in curl-loader-0.56 > >> >> >> CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24 or CURL_VER:=7.23 > >> >> >> 3. make cleanall; make > >> >> >> > >> >> >> Please, send us your observations. > >> >> >> Thanks! > >> >> >> Robert > >> >> >> > >> >> >> On Mon, Mar 5, 2012 at 6:57 PM, Robert Iakobashvili > >> >> >> <cor...@gm...> > >> >> >> wrote: > >> >> >> > If so, please try to change > >> >> >> > > >> >> >> > loader.c, line 578 which looks like: > >> >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); > >> >> >> > > >> >> >> > 1. instead of -1 try to set some value like 60 > >> >> >> > 2. make clean; make > >> >> >> > > >> >> >> > Thanks, > >> >> >> > Robert > >> >> >> > > >> >> >> > On Mon, Mar 5, 2012 at 6:19 PM, Fred Huang <di...@gm...> > >> >> >> > wrote: > >> >> >> >> > >> >> >> >> Looks like the bottleneck comes from libcurl other than > c-ares... > >> >> >> >> test 3 (no ares patch applied): > >> >> >> >> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, > resolve > >> >> >> >> *.com > >> >> >> >> to one IP address > >> >> >> >> number of domain names in URLs: 780,000 > >> >> >> >> number of client: 1000 > >> >> >> >> CPU usage: 85% > >> >> >> >> cpu% irq% sirq% sys% iowt% mem_used buf&cached > >> >> >> >> 85.6 0.0 0.8 5.9 0.0 3334.3Mb 112.0Mb > >> >> >> >> SSL TPS: 320 > >> >> >> >> SSL throughput: 200Mbps > >> >> >> >> > >> >> >> >> # gprof /usr/bin/curl-loader gmon.out -p | head -50 > >> >> >> >> Flat profile: > >> >> >> >> Each sample counts as 0.01 seconds. > >> >> >> >> % cumulative self self total > >> >> >> >> time seconds seconds calls s/call s/call name > >> >> >> >> 40.80 120.16 120.16 116079 0.00 0.00 > >> >> >> >> Curl_hash_clean_with_criterium > >> >> >> >> 24.46 192.18 72.02 983166323 0.00 0.00 > >> >> >> >> hostcache_timestamp_remove > >> >> >> >> 13.30 231.35 39.17 2059341 0.00 0.00 > >> >> >> >> Curl_hash_pick > >> >> >> >> 6.45 250.35 19.00 240087 0.00 0.00 > >> >> >> >> Curl_hash_add > >> >> >> >> 4.81 264.51 14.16 334257102 0.00 0.00 > >> >> >> >> Curl_str_key_compare > >> >> >> >> 3.40 274.52 10.01 119514 0.00 0.00 > create_conn > >> >> >> >> 0.93 277.25 2.73 4478910 0.00 0.00 > >> >> >> >> dprintf_formatf > >> >> >> >> 0.90 279.91 2.66 122209 0.00 0.00 > >> >> >> >> ConnectionStore > >> >> >> >> 0.40 281.09 1.18 > >> >> >> >> locking_function > >> >> >> >> 0.27 281.88 0.79 114759 0.00 0.00 > >> >> >> >> curl_multi_remove_handle > >> >> >> >> 0.21 282.51 0.63 7993664 0.00 0.00 > >> >> >> >> client_tracing_function > >> >> >> >> 0.21 283.12 0.61 122685 0.00 0.00 > >> >> >> >> Curl_num_addresses > >> >> >> >> 0.19 283.68 0.56 397028 0.00 0.00 > >> >> >> >> curl_multi_socket_action > >> >> >> >> 0.19 284.23 0.55 2036116 0.00 0.00 > >> >> >> >> multi_runsingle > >> >> >> >> 0.17 284.72 0.49 703687 0.00 0.00 > >> >> >> >> Curl_readwrite > >> >> >> >> 0.16 285.19 0.47 > id_function > >> >> >> >> 0.15 285.62 0.43 29692029 0.00 0.00 > >> >> >> >> Curl_socket_check > >> >> >> >> 0.14 286.02 0.40 91973856 0.00 0.00 > >> >> >> >> Curl_raw_toupper > >> >> >> >> 0.13 286.40 0.38 164664144 0.00 0.00 addbyter > >> >> >> >> 0.13 286.77 0.37 2180797 0.00 0.00 > Curl_infof > >> >> >> >> 0.11 287.10 0.33 122675 0.00 0.00 > >> >> >> >> Curl_if_is_interface_name > >> >> >> >> 0.10 287.39 0.29 471318 0.00 0.00 > >> >> >> >> ossl_connect_common > >> >> >> >> 0.10 287.67 0.28 2113528 0.00 0.00 > >> >> >> >> Curl_raw_equal > >> >> >> >> 0.09 287.93 0.26 8676 0.00 0.02 > >> >> >> >> curl_multi_socket_all > >> >> >> >> 0.08 288.17 0.24 1663262 0.00 0.00 > Curl_splay > >> >> >> >> 0.08 288.41 0.24 8505 0.00 0.01 > >> >> >> >> curl_multi_perform > >> >> >> >> 0.07 288.63 0.22 3114576 0.00 0.00 > >> >> >> >> Curl_pgrsUpdate > >> >> >> >> 0.06 288.80 0.17 119637 0.00 0.00 > >> >> >> >> singleipconnect > >> >> >> >> 0.06 288.97 0.17 7994818 0.00 0.00 > Curl_debug > >> >> >> >> 0.05 289.13 0.16 235820 0.00 0.00 event_del > >> >> >> >> 0.05 289.29 0.16 119393071 0.00 0.00 > >> >> >> >> curl_strequal > >> >> >> >> 0.05 289.45 0.16 18535473 0.00 0.00 > >> >> >> >> alloc_addbyter > >> >> >> >> 0.05 289.61 0.16 356074 0.00 0.00 > >> >> >> >> Curl_hash_str > >> >> >> >> 0.05 289.76 0.15 1797233 0.00 0.00 > >> >> >> >> fd_key_compare > >> >> >> >> 0.05 289.91 0.15 234966 0.00 0.00 > >> >> >> >> Curl_ssl_getsessionid > >> >> >> >> 0.04 290.04 0.13 585853 0.00 0.00 > >> >> >> >> Curl_llist_destroy > >> >> >> >> 0.04 290.16 0.12 240826 0.00 0.00 > Curl_resolv > >> >> >> >> 0.03 290.26 0.10 7514328 0.00 0.00 > >> >> >> >> Curl_raw_nequal > >> >> >> >> 0.03 290.36 0.10 2214436 0.00 0.00 Curl_read > >> >> >> >> 0.03 290.46 0.10 116872 0.00 0.00 > >> >> >> >> Curl_http_readwrite_headers > >> >> >> >> 0.03 290.56 0.10 > >> >> >> >> ssl_tls_trace > >> >> >> >> 0.03 290.66 0.10 8013097 0.00 0.00 > >> >> >> >> scan_response > >> >> >> >> 0.03 290.75 0.09 29488555 0.00 0.00 > >> >> >> >> curlx_tvdiff > >> >> >> >> 0.03 290.84 0.09 2142468 0.00 0.00 > Curl_setopt > >> >> >> >> 0.03 290.93 0.09 476299 0.00 0.00 > Curl_expire > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> 2012/3/5 Robert Iakobashvili <cor...@gm...> > >> >> >> >>> > >> >> >> >>> Dear Fred Huang, > >> >> >> >>> > >> >> >> >>> Thanks for your posting. > >> >> >> >>> > >> >> >> >>> Could you please check if this is due to some badness at > cares? > >> >> >> >>> Thanks. > >> >> >> >>> > >> >> >> >>> Let's exclude usage of cares based DNS resolving by applying > the > >> >> >> >>> patch > >> >> >> >>> attached > >> >> >> >>> and use instead global DNS cache. > >> >> >> >>> > >> >> >> >>> cd ./curl-loader-0.56 > >> >> >> >>> patch -p1 < path-to/disable-cares-use-global-dns.patch > >> >> >> >>> make cleanall; make > >> >> >> >>> > >> >> >> >>> See if it helps. > >> >> >> >>> > >> >> >> >>> Global cache, unfortunately, is not thread safe. > >> >> >> >>> > >> >> >> >>> If it helps, we'll try to restore the previous version of > cares. > >> >> >> >>> > >> >> >> >>> Thanks, > >> >> >> >>> Robert > >> >> >> >>> > >> >> >> >>> On Mon, Mar 5, 2012 at 4:28 PM, Fred Huang <di...@gm...> > >> >> >> >>> wrote: > >> >> >> >>>> > >> >> >> >>>> test 1: > >> >> >> >>>> > >> >> >> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dn... [truncated message content] |
|
From: Robert I. <cor...@gm...> - 2012-03-07 15:09:04
|
I see. On Wed, Mar 7, 2012 at 3:49 PM, Fred Huang <di...@gm...> wrote: > I tried -t 16 or -t 32, the performance drops to approx. 50 CAPS... > > > 2012/3/7 Robert Iakobashvili <cor...@gm...> >> >> Fred, >> >> On more suggestion, which is a general one. >> >> If N is the number of your logical CPUs, >> usually, I'd recommend to run 2*N + B threads. >> >> Thus, you can try: >> >> -t 16 >> >> -t 32 >> >> and it MAY increased a bit CAPS. >> >> Thanks, >> Robert >> >> On Wed, Mar 7, 2012 at 2:43 AM, Fred Huang <di...@gm...> wrote: >> > yes, I use -t 8 option because the test box has 8 logic CPUs (Dell 990, >> > Intel Core i7 2600, 16GB memory). >> > if I do not use -t option, the CPU used by curl-loader will be 100% on >> > one >> > of 8 logic CPU (12% of total CPU), and the CAPS drops to 100: >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> > Flat profile: >> > Each sample counts as 0.01 seconds. >> > % cumulative self self total >> > time seconds seconds calls s/call s/call name >> > 34.12 2.74 2.74 17385 0.00 0.00 create_conn >> > 11.21 3.64 0.90 30998583 0.00 0.00 >> > hostcache_timestamp_remove >> > 9.84 4.43 0.79 13433 0.00 0.00 >> > Curl_hash_clean_with_criterium >> > 7.97 5.07 0.64 381818 0.00 0.00 Curl_hash_pick >> > 5.85 5.54 0.47 17385 0.00 0.00 ConnectionStore >> > 5.11 5.95 0.41 16385 0.00 0.00 >> > curl_multi_remove_handle >> > 4.11 6.28 0.33 16509889 0.00 0.00 >> > Curl_str_key_compare >> > 3.36 6.55 0.27 23959914 0.00 0.00 Curl_socket_check >> > 3.24 6.81 0.26 56703 0.00 0.00 Curl_hash_add >> > 2.62 7.02 0.21 635750 0.00 0.00 dprintf_formatf >> > 1.25 7.12 0.10 92421190 0.00 0.00 >> > ares__is_list_empty >> > 0.75 7.18 0.06 97957 0.00 0.00 Curl_readwrite >> > 0.62 7.23 0.05 23422260 0.00 0.00 addbyter >> > 0.62 7.28 0.05 1136775 0.00 0.00 >> > client_tracing_function >> > 0.50 7.32 0.04 3091375 0.00 0.00 >> > ares__init_list_head >> > >> > I also tried to use >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 20); >> > but get no improvement on CAPS. >> > >> > >> > Regards, >> > Fred >> > >> > 2012/3/7 Robert Iakobashvili <cor...@gm...> >> >> >> >> Dear Fred, >> >> >> >> If I understand correctly your observations, the issue is not in: >> >> >> >> 1. DNS resolving; >> >> 2. Version of cares or libcurl; >> >> >> >> but rather in >> >> >> >> Less effective handling of HTTPS versus HTTP in libcurl. >> >> >> >> Please, confirm that my understanding is correct. >> >> >> >> >> >> HTTPS URLs are using an extra locking in openssl with -t option >> >> that could be making the issue. >> >> If you are using -t option, try to load without it and let's say with >> >> 300 clients. >> >> >> >> If you are not using -t, thus the only way to proceed is to share >> >> your observations with libcurl people at >> >> >> >> libcurl development <cur...@co...> >> >> >> >> Thank you. >> >> >> >> Best wishes. >> >> Robert >> >> >> >> On Tue, Mar 6, 2012 at 6:17 PM, Fred Huang <di...@gm...> wrote: >> >> > 1. the major CPU consumer is curl-loader. here is the top: >> >> > top - 07:36:36 up 23:31, 5 users, load average: 4.32, 2.26, 0.91 >> >> > Tasks: 237 total, 2 running, 235 sleeping, 0 stopped, 0 zombie >> >> > Cpu0 : 76.6%us, 3.6%sy, 0.0%ni, 19.4%id, 0.0%wa, 0.0%hi, >> >> > 0.3%si, >> >> > 0.0% >> >> > Cpu1 : 72.4%us, 4.0%sy, 0.0%ni, 22.9%id, 0.0%wa, 0.0%hi, >> >> > 0.7%si, >> >> > 0.0% >> >> > Cpu2 : 65.8%us, 3.3%sy, 0.0%ni, 30.6%id, 0.0%wa, 0.0%hi, >> >> > 0.3%si, >> >> > 0.0% >> >> > Cpu3 : 75.2%us, 4.3%sy, 0.0%ni, 19.5%id, 0.0%wa, 0.0%hi, >> >> > 1.0%si, >> >> > 0.0% >> >> > Cpu4 : 79.1%us, 4.0%sy, 0.0%ni, 16.8%id, 0.0%wa, 0.0%hi, >> >> > 0.0%si, >> >> > 0.0% >> >> > Cpu5 : 75.8%us, 4.3%sy, 0.0%ni, 19.2%id, 0.0%wa, 0.0%hi, >> >> > 0.7%si, >> >> > 0.0% >> >> > Cpu6 : 74.1%us, 5.0%sy, 0.0%ni, 20.3%id, 0.0%wa, 0.0%hi, >> >> > 0.7%si, >> >> > 0.0% >> >> > Cpu7 : 78.7%us, 4.0%sy, 0.0%ni, 16.6%id, 0.0%wa, 0.0%hi, >> >> > 0.7%si, >> >> > 0.0% >> >> > Mem: 16209932k total, 3829500k used, 12380432k free, 99112k >> >> > buffers >> >> > Swap: 18448376k total, 0k used, 18448376k free, 320284k >> >> > cached >> >> > >> >> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >> >> > 30795 root 20 0 3200m 2.8g 2296 S 79.2 18.4 21:26.84 >> >> > curl-loader >> >> > 1832 nobody 20 0 13944 1736 580 S 0.1 0.0 0:32.65 dnsmasq >> >> > 9 root 20 0 0 0 0 S 0.0 0.0 0:00.53 >> >> > ksoftirqd/1 >> >> > 30825 root 20 0 15220 1360 960 R 0.0 0.0 0:00.42 top >> >> > 1 root 20 0 19400 1568 1256 S 0.0 0.0 0:00.75 init >> >> > 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd >> >> > 3 root RT 0 0 0 0 S 0.0 0.0 0:00.00 >> >> > migration/0 >> >> > 4 root 20 0 0 0 0 S 0.0 0.0 0:00.06 >> >> > ksoftirqd/0 >> >> > 5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 >> >> > migration/0 >> >> > >> >> > >> >> > 2. 1000 loading clients >> >> > >> >> > 3. yes, tried 0.52 but no lucky -- CAPS 380, throughput 200Mbps, CPU >> >> > 90% >> >> > top - 08:17:10 up 1 day, 12 min, 8 users, load average: 4.66, 3.39, >> >> > 1.88 >> >> > Tasks: 245 total, 2 running, 243 sleeping, 0 stopped, 0 zombie >> >> > Cpu0 : 82.2%us, 4.0%sy, 0.0%ni, 13.9%id, 0.0%wa, 0.0%hi, >> >> > 0.0%si, >> >> > 0.0% >> >> > Cpu1 : 86.1%us, 4.3%sy, 0.0%ni, 7.9%id, 0.0%wa, 0.0%hi, >> >> > 1.7%si, >> >> > 0.0% >> >> > Cpu2 : 85.3%us, 4.0%sy, 0.0%ni, 10.0%id, 0.0%wa, 0.0%hi, >> >> > 0.7%si, >> >> > 0.0% >> >> > Cpu3 : 81.8%us, 3.0%sy, 0.0%ni, 14.2%id, 0.0%wa, 0.0%hi, >> >> > 1.0%si, >> >> > 0.0% >> >> > Cpu4 : 83.7%us, 4.0%sy, 0.0%ni, 12.3%id, 0.0%wa, 0.0%hi, >> >> > 0.0%si, >> >> > 0.0% >> >> > Cpu5 : 92.7%us, 3.3%sy, 0.0%ni, 3.3%id, 0.0%wa, 0.0%hi, >> >> > 0.7%si, >> >> > 0.0% >> >> > Cpu6 : 89.7%us, 2.0%sy, 0.0%ni, 8.0%id, 0.0%wa, 0.0%hi, >> >> > 0.3%si, >> >> > 0.0% >> >> > Cpu7 : 88.1%us, 3.6%sy, 0.0%ni, 6.6%id, 0.0%wa, 0.0%hi, >> >> > 1.7%si, >> >> > 0.0% >> >> > Mem: 16209932k total, 1795748k used, 14414184k free, 102824k >> >> > buffers >> >> > Swap: 18448376k total, 0k used, 18448376k free, 382988k >> >> > cached >> >> > >> >> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >> >> > 12236 root 20 0 1150m 894m 2572 S 90.7 5.7 51:11.91 >> >> > curl-loader >> >> > 1832 nobody 20 0 13944 1736 580 S 0.1 0.0 0:47.04 dnsmasq >> >> > >> >> > ]# gprof /usr/bin/curl-loader gmon.out -p | head -20 >> >> > Flat profile: >> >> > Each sample counts as 0.01 seconds. >> >> > % cumulative self self total >> >> > time seconds seconds calls s/call s/call name >> >> > 43.92 151.95 151.95 157295 0.00 0.00 >> >> > Curl_hash_clean_with_criterium >> >> > 22.90 231.17 79.22 1712190971 0.00 0.00 >> >> > hostcache_timestamp_remove >> >> > 13.15 276.68 45.51 3260321 0.00 0.00 Curl_hash_pick >> >> > 7.51 302.65 25.97 394171 0.00 0.00 Curl_hash_add >> >> > 5.98 323.34 20.68 567437518 0.00 0.00 >> >> > Curl_str_key_compare >> >> > 1.00 326.80 3.46 5380220 0.00 0.00 dprintf_formatf >> >> > 0.41 328.21 1.41 >> >> > locking_function >> >> > 0.32 329.32 1.11 469222138 0.00 0.00 >> >> > ares__is_list_empty >> >> > 0.31 330.40 1.08 3284508 0.00 0.00 multi_runsingle >> >> > 0.28 331.38 0.98 10006806 0.00 0.00 >> >> > client_tracing_function >> >> > 0.24 332.21 0.83 151951 0.00 0.00 >> >> > Curl_num_addresses >> >> > 0.22 332.97 0.77 188075672 0.00 0.00 addbyter >> >> > 0.19 333.64 0.67 841840 0.00 0.00 Curl_readwrite >> >> > 0.19 334.29 0.65 id_function >> >> > 0.18 334.90 0.61 39843 0.00 0.01 >> >> > curl_multi_socket_all >> >> > >> >> > >> >> > >> >> > 4. yes, it is correct for 0.56. If use same URL list by replacing >> >> > https >> >> > to >> >> > http, I can get CAPS 1500 and throughput 870Mbps with CPU% < 60% >> >> > >> >> > >> >> > 2012/3/6 Robert Iakobashvili <cor...@gm...> >> >> >> >> >> >> Several questions: >> >> >> >> >> >> 1. Could you please run top and see if there are other processes >> >> >> burning >> >> >> CPU? >> >> >> 2. What is the number of loading clients that you use? >> >> >> 3. Could you try curl-loader-0.52 if it compiles for you? >> >> >> If not, please try to place curl-loader-7.19.7.tar.gz and >> >> >> c-ares-16.0.tar.gz, >> >> >> update their versions in Makefile and rebuild by make cleanall; make >> >> >> >> >> >> 4. Is it correct to say that for HTTP URLs (not HTTPS) you are >> >> >> getting >> >> >> a good throughput >> >> >> and CAPS with curl-loader-0.56? >> >> >> >> >> >> Thanks, >> >> >> Robert >> >> >> >> >> >> On Tue, Mar 6, 2012 at 3:52 AM, Fred Huang <di...@gm...> wrote: >> >> >> > TIMEOUT -1 -> 60 works a little better but no performance >> >> >> > improvement >> >> >> > on >> >> >> > CAPS or throughput. >> >> >> > >> >> >> > test 5: >> >> >> > enable c-ares; >> >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); >> >> >> > >> >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> >> > *.com >> >> >> > to >> >> >> > one IP address >> >> >> > number of domain names in URLs: 780,000 >> >> >> > number of client: 1000 >> >> >> > CPU usage: 74% >> >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> >> > 74.0 0.0 0.7 4.2 0.0 3530.6Mb 250.8Mb >> >> >> > SSL TPS: 360 >> >> >> > SSL throughput: 220Mbps >> >> >> > >> >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -50 >> >> >> > Flat profile: >> >> >> > Each sample counts as 0.01 seconds. >> >> >> > % cumulative self self total >> >> >> > time seconds seconds calls s/call s/call name >> >> >> > 26.63 23.13 23.13 119226 0.00 0.00 >> >> >> > Curl_hash_clean_with_criterium >> >> >> > 20.34 40.80 17.67 317792146 0.00 0.00 >> >> >> > hostcache_timestamp_remove >> >> >> > 11.27 50.59 9.79 2579903 0.00 0.00 >> >> >> > Curl_hash_pick >> >> >> > 7.84 57.40 6.81 120606 0.00 0.00 create_conn >> >> >> > 5.65 62.31 4.91 119117401 0.00 0.00 >> >> >> > Curl_str_key_compare >> >> >> > 4.98 66.64 4.33 329829 0.00 0.00 >> >> >> > Curl_hash_add >> >> >> > 2.80 69.08 2.44 4476606 0.00 0.00 >> >> >> > dprintf_formatf >> >> >> > 2.57 71.31 2.23 124019 0.00 0.00 >> >> >> > ConnectionStore >> >> >> > 1.50 72.61 1.30 740997609 0.00 0.00 >> >> >> > ares__is_list_empty >> >> >> > 1.16 73.61 1.01 >> >> >> > locking_function >> >> >> > 0.76 74.27 0.66 238744 0.00 0.00 ares_cancel >> >> >> > 0.70 74.88 0.61 457564 0.00 0.00 >> >> >> > curl_multi_socket_action >> >> >> > 0.67 75.46 0.58 118616 0.00 0.00 >> >> >> > curl_multi_remove_handle >> >> >> > 0.58 75.97 0.51 8211502 0.00 0.00 >> >> >> > client_tracing_function >> >> >> > 0.53 76.43 0.46 733561 0.00 0.00 >> >> >> > Curl_readwrite >> >> >> > 0.51 76.87 0.44 2183179 0.00 0.00 >> >> >> > multi_runsingle >> >> >> > 0.51 77.31 0.44 166585567 0.00 0.00 addbyter >> >> >> > 0.44 77.69 0.38 2175805 0.00 0.00 >> >> >> > Curl_raw_equal >> >> >> > 0.43 78.06 0.38 id_function >> >> >> > 0.40 78.41 0.35 29789346 0.00 0.00 >> >> >> > Curl_socket_check >> >> >> > 0.40 78.76 0.35 479671 0.00 0.00 >> >> >> > ossl_connect_common >> >> >> > 0.40 79.11 0.35 490639 0.00 0.00 >> >> >> > ares__init_list_node >> >> >> > 0.33 79.40 0.29 94197670 0.00 0.00 >> >> >> > Curl_raw_toupper >> >> >> > 0.33 79.69 0.29 3268031 0.00 0.00 >> >> >> > Curl_pgrsUpdate >> >> >> > 0.28 79.93 0.24 8797 0.00 0.00 >> >> >> > curl_multi_perform >> >> >> > 0.27 80.17 0.24 2233641 0.00 0.00 Curl_infof >> >> >> > 0.26 80.40 0.23 1839837 0.00 0.00 Curl_splay >> >> >> > 0.24 80.61 0.21 121713028 0.00 0.00 >> >> >> > curl_strequal >> >> >> > 0.21 80.79 0.18 243333 0.00 0.00 >> >> >> > Curl_ssl_getsessionid >> >> >> > 0.18 80.95 0.16 122180 0.00 0.00 >> >> >> > Curl_num_addresses >> >> >> > 0.17 81.09 0.15 8271666 0.00 0.00 Curl_debug >> >> >> > 0.16 81.23 0.14 18902452 0.00 0.00 >> >> >> > alloc_addbyter >> >> >> > 0.16 81.37 0.14 8851 0.00 0.00 >> >> >> > curl_multi_socket_all >> >> >> > 0.14 81.50 0.13 2236769 0.00 0.00 Curl_read >> >> >> > 0.14 81.62 0.12 2271700 0.00 0.00 >> >> >> > fd_key_compare >> >> >> > 0.14 81.74 0.12 323167 0.00 0.00 event_del >> >> >> > 0.14 81.86 0.12 205436 0.00 0.00 >> >> >> > read_udp_packets >> >> >> > 0.14 81.98 0.12 200979 0.00 0.00 processfds >> >> >> > 0.13 82.09 0.11 123886 0.00 0.00 >> >> >> > Curl_if_is_interface_name >> >> >> > 0.13 82.20 0.11 120350 0.00 0.00 Curl_http >> >> >> > 0.13 82.31 0.11 >> >> >> > ssl_tls_trace >> >> >> > 0.12 82.41 0.10 119229 0.00 0.00 >> >> >> > Curl_splayremovebyaddr >> >> >> > 0.12 82.51 0.10 7798717 0.00 0.00 >> >> >> > Curl_raw_nequal >> >> >> > 0.12 82.61 0.10 784485 0.00 0.00 event_add >> >> >> > 0.10 82.70 0.09 643482 0.00 0.00 Curl_expire >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > test 6: >> >> >> > enable c-ares; >> >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); >> >> >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0 >> >> >> > >> >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> >> > *.com >> >> >> > to >> >> >> > one IP address >> >> >> > number of domain names in URLs: 780,000 >> >> >> > number of client: 1000 >> >> >> > CPU usage: 86% >> >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> >> > 85.9 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb >> >> >> > SSL TPS: 360 >> >> >> > SSL throughput: 200Mbps >> >> >> > >> >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> >> >> > Flat profile: >> >> >> > Each sample counts as 0.01 seconds. >> >> >> > % cumulative self self total >> >> >> > time seconds seconds calls s/call s/call name >> >> >> > 27.19 19.96 19.96 103472 0.00 0.00 >> >> >> > Curl_hash_clean_with_criterium >> >> >> > 21.73 35.91 15.95 274168981 0.00 0.00 >> >> >> > hostcache_timestamp_remove >> >> >> > 10.33 43.49 7.58 2256441 0.00 0.00 >> >> >> > Curl_hash_pick >> >> >> > 7.21 48.78 5.29 104540 0.00 0.00 create_conn >> >> >> > 6.20 53.33 4.55 103413405 0.00 0.00 >> >> >> > Curl_str_key_compare >> >> >> > 4.88 56.91 3.58 287109 0.00 0.00 >> >> >> > Curl_hash_add >> >> >> > 3.02 59.13 2.22 3797855 0.00 0.00 >> >> >> > dprintf_formatf >> >> >> > 2.10 60.67 1.54 107711 0.00 0.00 >> >> >> > ConnectionStore >> >> >> > 1.65 61.88 1.21 642725327 0.00 0.00 >> >> >> > ares__is_list_empty >> >> >> > 1.19 62.76 0.88 >> >> >> > locking_function >> >> >> > 0.87 63.40 0.64 207817 0.00 0.00 ares_cancel >> >> >> > 0.56 63.81 0.41 404108 0.00 0.00 >> >> >> > curl_multi_socket_action >> >> >> > 0.54 64.21 0.40 425603 0.00 0.00 >> >> >> > ares__init_list_node >> >> >> > 0.53 64.60 0.39 7068112 0.00 0.00 >> >> >> > client_tracing_function >> >> >> > 0.49 64.96 0.36 1908587 0.00 0.00 >> >> >> > multi_runsingle >> >> >> > >> >> >> > >> >> >> > >> >> >> > test 7: >> >> >> > enable c-ares; >> >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); >> >> >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0 >> >> >> > >> >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> >> > *.com >> >> >> > to >> >> >> > one IP address >> >> >> > number of domain names in URLs: 780,000 >> >> >> > number of client: 1000 >> >> >> > CPU usage: 86% >> >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> >> > 86.6 0.0 0.6 3.3 0.0 3624.8Mb 263.0Mb >> >> >> > SSL TPS: 300 >> >> >> > SSL throughput: 200Mbps >> >> >> > >> >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> >> >> > Flat profile: >> >> >> > Each sample counts as 0.01 seconds. >> >> >> > % cumulative self self total >> >> >> > time seconds seconds calls s/call s/call name >> >> >> > 39.83 369.71 369.71 207173 0.00 0.00 >> >> >> > Curl_hash_clean_with_criterium >> >> >> > 24.27 594.98 225.27 3198337733 0.00 0.00 >> >> >> > hostcache_timestamp_remove >> >> >> > 13.75 722.59 127.61 4312864 0.00 0.00 >> >> >> > Curl_hash_pick >> >> >> > 9.19 807.87 85.28 546507 0.00 0.00 >> >> >> > Curl_hash_add >> >> >> > 6.14 864.88 57.01 1022419684 0.00 0.00 >> >> >> > Curl_str_key_compare >> >> >> > 2.06 884.00 19.12 203749 0.00 0.00 create_conn >> >> >> > 0.54 888.99 4.99 213306 0.00 0.00 >> >> >> > ConnectionStore >> >> >> > 0.49 893.50 4.51 7331251 0.00 0.00 >> >> >> > dprintf_formatf >> >> >> > 0.29 896.16 2.66 1287182329 0.00 0.00 >> >> >> > ares__is_list_empty >> >> >> > 0.23 898.28 2.12 >> >> >> > locking_function >> >> >> > 0.20 900.10 1.82 212210 0.00 0.00 >> >> >> > Curl_num_addresses >> >> >> > 0.17 901.66 1.56 198385 0.00 0.00 >> >> >> > curl_multi_remove_handle >> >> >> > 0.12 902.81 1.15 411114 0.00 0.00 ares_cancel >> >> >> > 0.12 903.95 1.14 3683448 0.00 0.00 >> >> >> > multi_runsingle >> >> >> > 0.11 904.97 1.03 13664120 0.00 0.00 >> >> >> > client_tracing_function >> >> >> > >> >> >> > >> >> >> > test 8: >> >> >> > enable c-ares; >> >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); >> >> >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0 >> >> >> > >> >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> >> > *.com >> >> >> > to >> >> >> > one IP address >> >> >> > number of domain names in URLs: 780,000 >> >> >> > number of client: 1000 >> >> >> > CPU usage: 75% >> >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> >> > 75.2 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb >> >> >> > SSL TPS: 380 >> >> >> > SSL throughput: 220Mbps >> >> >> > >> >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> >> >> > Flat profile: >> >> >> > Each sample counts as 0.01 seconds. >> >> >> > % cumulative self self total >> >> >> > time seconds seconds calls s/call s/call name >> >> >> > 27.58 12.31 12.31 71140 0.00 0.00 >> >> >> > Curl_hash_clean_with_criterium >> >> >> > 19.09 20.83 8.52 176729692 0.00 0.00 >> >> >> > hostcache_timestamp_remove >> >> >> > 10.44 25.49 4.66 1589309 0.00 0.00 >> >> >> > Curl_hash_pick >> >> >> > 6.85 28.55 3.06 73204 0.00 0.00 create_conn >> >> >> > 5.56 31.03 2.48 67620530 0.00 0.00 >> >> >> > Curl_str_key_compare >> >> >> > 4.41 33.00 1.97 200684 0.00 0.00 >> >> >> > Curl_hash_add >> >> >> > 3.56 34.59 1.59 2717956 0.00 0.00 >> >> >> > dprintf_formatf >> >> >> > 1.72 35.36 0.77 75247 0.00 0.00 >> >> >> > ConnectionStore >> >> >> > 1.70 36.12 0.76 444342256 0.00 0.00 >> >> >> > ares__is_list_empty >> >> >> > 1.23 36.67 0.55 >> >> >> > locking_function >> >> >> > 1.16 37.19 0.52 143342 0.00 0.00 ares_cancel >> >> >> > 0.99 37.63 0.44 72211 0.00 0.00 >> >> >> > curl_multi_remove_handle >> >> >> > 0.74 37.96 0.33 1333371 0.00 0.00 >> >> >> > multi_runsingle >> >> >> > 0.68 38.27 0.31 101147798 0.00 0.00 addbyter >> >> >> > 0.58 38.53 0.26 288737 0.00 0.00 >> >> >> > curl_multi_socket_action >> >> >> > >> >> >> > >> >> >> > >> >> >> > test 9: >> >> >> > enable c-ares; >> >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); >> >> >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0 >> >> >> > >> >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> >> > *.com >> >> >> > to >> >> >> > one IP address >> >> >> > number of domain names in URLs: 780,000 >> >> >> > number of client: 1000 >> >> >> > CPU usage: 75% >> >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> >> > 75.1 0.0 0.6 4.1 0.0 3545.2Mb 260.1Mb >> >> >> > SSL TPS: 380 >> >> >> > SSL throughput: 200Mbps >> >> >> > >> >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> >> >> > Flat profile: >> >> >> > Each sample counts as 0.01 seconds. >> >> >> > % cumulative self self total >> >> >> > time seconds seconds calls s/call s/call name >> >> >> > 34.00 31.13 31.13 72495 0.00 0.00 >> >> >> > Curl_hash_clean_with_criterium >> >> >> > 23.64 52.78 21.65 386070412 0.00 0.00 >> >> >> > hostcache_timestamp_remove >> >> >> > 13.28 64.94 12.16 1619440 0.00 0.00 >> >> >> > Curl_hash_pick >> >> >> > 5.92 70.36 5.42 137653129 0.00 0.00 >> >> >> > Curl_str_key_compare >> >> >> > 5.50 75.40 5.04 203018 0.00 0.00 >> >> >> > Curl_hash_add >> >> >> > 4.55 79.57 4.17 74618 0.00 0.00 create_conn >> >> >> > 1.70 81.13 1.56 2757148 0.00 0.00 >> >> >> > dprintf_formatf >> >> >> > 0.98 82.03 0.90 76945 0.00 0.00 >> >> >> > ConnectionStore >> >> >> > 0.95 82.90 0.87 455775100 0.00 0.00 >> >> >> > ares__is_list_empty >> >> >> > 0.57 83.42 0.53 >> >> >> > locking_function >> >> >> > 0.39 83.78 0.36 146742 0.00 0.00 ares_cancel >> >> >> > 0.35 84.10 0.32 102968806 0.00 0.00 addbyter >> >> >> > 0.35 84.42 0.32 456587 0.00 0.00 >> >> >> > Curl_readwrite >> >> >> > 0.35 84.74 0.32 73133 0.00 0.00 >> >> >> > curl_multi_remove_handle >> >> >> > 0.34 85.05 0.31 1360545 0.00 0.00 >> >> >> > multi_runsingle >> >> >> > >> >> >> > >> >> >> > 2012/3/6 Robert Iakobashvili <cor...@gm...> >> >> >> >> >> >> >> >> If it will not be helpful this means that recent >> >> >> >> changes in libcurl library have broken DNS interface. >> >> >> >> >> >> >> >> You can try a previos version and the latest libcurl, like: >> >> >> >> >> >> >> >> http://curl.haxx.se/download/curl-7.23.0.tar.gz >> >> >> >> and >> >> >> >> http://curl.haxx.se/downloadcurl-7.24.0.tar.gz >> >> >> >> >> >> >> >> by: >> >> >> >> >> >> >> >> 1. downloading a libcurl version and placing it to >> >> >> >> curl-loader-0.56/packages ; >> >> >> >> 2. correcting the Makefile in curl-loader-0.56 >> >> >> >> CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24 or CURL_VER:=7.23 >> >> >> >> 3. make cleanall; make >> >> >> >> >> >> >> >> Please, send us your observations. >> >> >> >> Thanks! >> >> >> >> Robert >> >> >> >> >> >> >> >> On Mon, Mar 5, 2012 at 6:57 PM, Robert Iakobashvili >> >> >> >> <cor...@gm...> >> >> >> >> wrote: >> >> >> >> > If so, please try to change >> >> >> >> > >> >> >> >> > loader.c, line 578 which looks like: >> >> >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); >> >> >> >> > >> >> >> >> > 1. instead of -1 try to set some value like 60 >> >> >> >> > 2. make clean; make >> >> >> >> > >> >> >> >> > Thanks, >> >> >> >> > Robert >> >> >> >> > >> >> >> >> > On Mon, Mar 5, 2012 at 6:19 PM, Fred Huang <di...@gm...> >> >> >> >> > wrote: >> >> >> >> >> >> >> >> >> >> Looks like the bottleneck comes from libcurl other than >> >> >> >> >> c-ares... >> >> >> >> >> test 3 (no ares patch applied): >> >> >> >> >> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, >> >> >> >> >> resolve >> >> >> >> >> *.com >> >> >> >> >> to one IP address >> >> >> >> >> number of domain names in URLs: 780,000 >> >> >> >> >> number of client: 1000 >> >> >> >> >> CPU usage: 85% >> >> >> >> >> cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> >> >> >> 85.6 0.0 0.8 5.9 0.0 3334.3Mb 112.0Mb >> >> >> >> >> SSL TPS: 320 >> >> >> >> >> SSL throughput: 200Mbps >> >> >> >> >> >> >> >> >> >> # gprof /usr/bin/curl-loader gmon.out -p | head -50 >> >> >> >> >> Flat profile: >> >> >> >> >> Each sample counts as 0.01 seconds. >> >> >> >> >> % cumulative self self total >> >> >> >> >> time seconds seconds calls s/call s/call name >> >> >> >> >> 40.80 120.16 120.16 116079 0.00 0.00 >> >> >> >> >> Curl_hash_clean_with_criterium >> >> >> >> >> 24.46 192.18 72.02 983166323 0.00 0.00 >> >> >> >> >> hostcache_timestamp_remove >> >> >> >> >> 13.30 231.35 39.17 2059341 0.00 0.00 >> >> >> >> >> Curl_hash_pick >> >> >> >> >> 6.45 250.35 19.00 240087 0.00 0.00 >> >> >> >> >> Curl_hash_add >> >> >> >> >> 4.81 264.51 14.16 334257102 0.00 0.00 >> >> >> >> >> Curl_str_key_compare >> >> >> >> >> 3.40 274.52 10.01 119514 0.00 0.00 >> >> >> >> >> create_conn >> >> >> >> >> 0.93 277.25 2.73 4478910 0.00 0.00 >> >> >> >> >> dprintf_formatf >> >> >> >> >> 0.90 279.91 2.66 122209 0.00 0.00 >> >> >> >> >> ConnectionStore >> >> >> >> >> 0.40 281.09 1.18 >> >> >> >> >> locking_function >> >> >> >> >> 0.27 281.88 0.79 114759 0.00 0.00 >> >> >> >> >> curl_multi_remove_handle >> >> >> >> >> 0.21 282.51 0.63 7993664 0.00 0.00 >> >> >> >> >> client_tracing_function >> >> >> >> >> 0.21 283.12 0.61 122685 0.00 0.00 >> >> >> >> >> Curl_num_addresses >> >> >> >> >> 0.19 283.68 0.56 397028 0.00 0.00 >> >> >> >> >> curl_multi_socket_action >> >> >> >> >> 0.19 284.23 0.55 2036116 0.00 0.00 >> >> >> >> >> multi_runsingle >> >> >> >> >> 0.17 284.72 0.49 703687 0.00 0.00 >> >> >> >> >> Curl_readwrite >> >> >> >> >> 0.16 285.19 0.47 >> >> >> >> >> id_function >> >> >> >> >> 0.15 285.62 0.43 29692029 0.00 0.00 >> >> >> >> >> Curl_socket_check >> >> >> >> >> 0.14 286.02 0.40 91973856 0.00 0.00 >> >> >> >> >> Curl_raw_toupper >> >> >> >> >> 0.13 286.40 0.38 164664144 0.00 0.00 >> >> >> >> >> addbyter >> >> >> >> >> 0.13 286.77 0.37 2180797 0.00 0.00 >> >> >> >> >> Curl_infof >> >> >> >> >> 0.11 287.10 0.33 122675 0.00 0.00 >> >> >> >> >> Curl_if_is_interface_name >> >> >> >> >> 0.10 287.39 0.29 471318 0.00 0.00 >> >> >> >> >> ossl_connect_common >> >> >> >> >> 0.10 287.67 0.28 2113528 0.00 0.00 >> >> >> >> >> Curl_raw_equal >> >> >> >> >> 0.09 287.93 0.26 8676 0.00 0.02 >> >> >> >> >> curl_multi_socket_all >> >> >> >> >> 0.08 288.17 0.24 1663262 0.00 0.00 >> >> >> >> >> Curl_splay >> >> >> >> >> 0.08 288.41 0.24 8505 0.00 0.01 >> >> >> >> >> curl_multi_perform >> >> >> >> >> 0.07 288.63 0.22 3114576 0.00 0.00 >> >> >> >> >> Curl_pgrsUpdate >> >> >> >> >> 0.06 288.80 0.17 119637 0.00 0.00 >> >> >> >> >> singleipconnect >> >> >> >> >> 0.06 288.97 0.17 7994818 0.00 0.00 >> >> >> >> >> Curl_debug >> >> >> >> >> 0.05 289.13 0.16 235820 0.00 0.00 >> >> >> >> >> event_del >> >> >> >> >> 0.05 289.29 0.16 119393071 0.00 0.00 >> >> >> >> >> curl_strequal >> >> >> >> >> 0.05 289.45 0.16 18535473 0.00 0.00 >> >> >> >> >> alloc_addbyter >> >> >> >> >> 0.05 289.61 0.16 356074 0.00 0.00 >> >> >> >> >> Curl_hash_str >> >> >> >> >> 0.05 289.76 0.15 1797233 0.00 0.00 >> >> >> >> >> fd_key_compare >> >> >> >> >> 0.05 289.91 0.15 234966 0.00 0.00 >> >> >> >> >> Curl_ssl_getsessionid >> >> >> >> >> 0.04 290.04 0.13 585853 0.00 0.00 >> >> >> >> >> Curl_llist_destroy >> >> >> >> >> 0.04 290.16 0.12 240826 0.00 0.00 >> >> >> >> >> Curl_resolv >> >> >> >> >> 0.03 290.26 0.10 7514328 0.00 0.00 >> >> >> >> >> Curl_raw_nequal >> >> >> >> >> 0.03 290.36 0.10 2214436 0.00 0.00 >> >> >> >> >> Curl_read >> >> >> >> >> 0.03 290.46 0.10 116872 0.00 0.00 >> >> >> >> >> Curl_http_readwrite_headers >> >> >> >> >> 0.03 290.56 0.10 >> >> >> >> >> ssl_tls_trace >> >> >> >> >> 0.03 290.66 0.10 8013097 0.00 0.00 >> >> >> >> >> scan_response >> >> >> >> >> 0.03 290.75 0.09 29488555 0.00 0.00 >> >> >> >> >> curlx_tvdiff >> >> >> >> >> 0.03 290.84 0.09 2142468 0.00 ... [truncated message content] |
|
From: Fred H. <di...@gm...> - 2012-11-24 15:00:18
|
hi all, is it possible to use adns (http://www.chiark.greenend.org.uk/~ian/adns/) to do dns resolving instead of libc-ares? all my test results shows that lib-cares can do no more than 300 queries per second. however, using 'dig +short -f hostnames.list &>/dev/null' against same dns server I can get more 7500 queries per second. I think libc-ares is also limited by the system API of gethostbyname. About ADNS lib: Many clients for DNS resolution are coded poorly.Most UNIX systems provide an implementation of gethostbyname (the DNS client API—application program interface), which cannot concurrently handle multiple outstanding requests. Therefore, the crawler cannot issue many resolution requests together and poll at a later time for completion of individual requests, which is critical for acceptable performance. Furthermore, if the system-provided client is used, there is no way to distribute load among a number of DNS servers. For all these reasons, many crawlers choose to include their own custom client for DNS name resolution. The Mercator crawler from Compaq System Research Center reduced the time spent in DNS from as high as 87% to a modest 25% by implementing a custom client. The ADNS asynchronous DNS client library is ideal for use in crawlers. In spite of these optimizations, a large-scale crawler will spend a substantial fraction of its network time not waiting for Http data transfer, but for address resolution. For every hostname that has not been resolved before (which happens frequently with crawlers), the local DNS may have to go across many network hops to fill its cache for the first time. To overlap this unavoidable delay with useful work, prefetching can be used. When a page that has just been fetched is parsed, a stream of HREFs is extracted. Right at this time, that is, even before any of the corresponding URLs are fetched, hostnames are extracted from the HREF targets, and DNS resolution requests are made to the caching server. The prefetching client is usually implemented using UDP instead of TCP, and it does not wait for resolution to be completed. The request serves only to fill the DNS cache so that resolution will be fast when the page is actually needed later on. ===end 2012/3/5 Fred Huang <di...@gm...> > test 1: > > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com > to one IP address > number of domain names in all https URLs: 780,000 > number of client: 1000 > CPU usage: 78% > cpu% irq% sirq% sys% iowt% mem_used buf&cached > 78.8 0.0 0.6 3.8 0.0 3640.9Mb 282.1Mb > SSL TPS: 300 > SSL throughput: 200Mbps > > # gprof /usr/bin/curl-loader gmon.out -p | head -50 > Flat profile: > Each sample counts as 0.01 seconds. > % cumulative self self total > time seconds seconds calls s/call s/call name > * 38.55 127.10 127.10 130975 0.00 0.00 > Curl_hash_clean_with_criterium* > * 24.63 208.32 81.22 1247528407 0.00 0.00 > hostcache_timestamp_remove* > * 13.08 251.44 43.12 2791988 0.00 0.00 Curl_hash_pick* > 7.30 275.51 24.07 352490 0.00 0.00 Curl_hash_add > 6.01 295.31 19.80 410760770 0.00 0.00 Curl_str_key_compare > 2.95 305.04 9.73 131015 0.00 0.00 create_conn > 0.88 307.94 2.90 4841390 0.00 0.00 dprintf_formatf > 0.74 310.38 2.44 136036 0.00 0.00 ConnectionStore > 0.48 311.96 1.58 814604911 0.00 0.00 ares__is_list_empty > 0.36 313.16 1.20 locking_function > 0.27 314.06 0.90 261608 0.00 0.00 ares_cancel > 0.22 314.79 0.73 127417 0.00 0.00 > curl_multi_remove_handle > 0.19 315.40 0.62 8906301 0.00 0.00 > client_tracing_function > 0.17 315.97 0.57 135305 0.00 0.00 Curl_num_addresses > 0.17 316.54 0.57 181937907 0.00 0.00 addbyter > 0.17 317.09 0.55 2374363 0.00 0.00 multi_runsingle > 0.16 317.62 0.53 541954 0.00 0.00 ares__init_list_node > 0.16 318.14 0.52 494412 0.00 0.00 > curl_multi_socket_action > 0.14 318.60 0.46 33578897 0.00 0.00 Curl_socket_check > 0.13 319.04 0.44 102951052 0.00 0.00 Curl_raw_toupper > 0.13 319.48 0.44 787558 0.00 0.00 Curl_readwrite > 0.13 319.91 0.44 id_function > 0.12 320.29 0.38 391390 0.00 0.00 Curl_hash_str > 0.09 320.58 0.29 9216 0.00 0.00 curl_multi_perform > 0.08 320.86 0.28 2350908 0.00 0.00 Curl_raw_equal > 0.08 321.14 0.28 2019415 0.00 0.00 Curl_splay > 0.08 321.40 0.26 520625 0.00 0.00 ossl_connect_common > 0.08 321.65 0.25 3532992 0.00 0.00 Curl_pgrsUpdate > 0.07 321.87 0.22 131778537 0.00 0.00 curl_strequal > 0.05 322.05 0.18 8954118 0.00 0.00 scan_response > 0.05 322.23 0.18 533644 0.00 0.00 ares_expand_name > 0.05 322.40 0.17 2477345 0.00 0.00 fd_key_compare > 0.05 322.57 0.17 2429135 0.00 0.00 Curl_infof > 0.05 322.74 0.17 130966 0.00 0.00 singleipconnect > 0.05 322.90 0.16 9292 0.00 0.00 curl_multi_socket_all > 0.05 323.06 0.16 270137 0.00 0.00 ares__get_hostent > 0.05 323.21 0.15 8943657 0.00 0.00 Curl_debug > 0.04 323.35 0.14 262493 0.00 0.00 Curl_ssl_getsessionid > 0.04 323.48 0.13 566006 0.00 0.00 socket_callback > 0.04 323.61 0.13 33983864 0.00 0.00 curlx_tvdiff > 0.04 323.74 0.13 132853 0.00 0.00 > Curl_http_readwrite_headers > 0.04 323.86 0.12 344032 0.00 0.00 epoll_del > 0.04 323.98 0.12 223477 0.00 0.00 Curl_poll > 0.04 324.10 0.12 8526280 0.00 0.00 Curl_raw_nequal > 0.03 324.21 0.11 348839 0.00 0.00 event_del > > > > > test 2: > > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com > to one IP address > number of domain names in all https URLs: 1 > number of client: 1000 > CPU usage: 75% > SSL TPS: 1300 > SSL throughput: 700Mbps > > # gprof /usr/bin/curl-loader gmon.out -p | head -50 > Flat profile: > Each sample counts as 0.01 seconds. > % cumulative self self total > time seconds seconds calls s/call s/call name > 8.98 2.45 2.45 5127312 0.00 0.00 dprintf_formatf > 8.90 4.88 2.43 1292796503 0.00 0.00 ares__is_list_empty > 6.12 6.55 1.67 322461 0.00 0.00 create_conn > 4.58 7.80 1.25 419872 0.00 0.00 ares_cancel > 4.10 8.92 1.12 2100162 0.00 0.00 Curl_readwrite > 3.92 9.99 1.07 4527388 0.00 0.00 Curl_hash_pick > 3.43 10.93 0.94 locking_function > 3.04 11.76 0.83 909759 0.00 0.00 > curl_multi_socket_action > 3.00 12.58 0.82 97252 0.00 0.00 > Curl_hash_clean_with_criterium > 2.71 13.32 0.74 179704016 0.00 0.00 Curl_raw_toupper > 2.68 14.05 0.73 4646524 0.00 0.00 multi_runsingle > 2.44 14.71 0.67 13862992 0.00 0.00 > client_tracing_function > 2.42 15.37 0.66 319735 0.00 0.00 > curl_multi_remove_handle > 2.42 16.03 0.66 22 0.03 0.03 ares__init_list_node > 1.94 16.56 0.53 9930244 0.00 0.00 > hostcache_timestamp_remove > 1.94 17.09 0.53 169320088 0.00 0.00 addbyter > 1.80 17.58 0.49 id_function > 1.36 17.95 0.37 7057146 0.00 0.00 Curl_pgrsUpdate > 1.36 18.32 0.37 16835 0.00 0.00 curl_multi_socket_all > 1.25 18.66 0.34 16719 0.00 0.00 curl_multi_perform > 1.21 18.99 0.33 329403 0.00 0.00 > Curl_http_readwrite_headers > 1.17 19.31 0.32 4036624 0.00 0.00 Curl_splay > 1.14 19.62 0.31 20838902 0.00 0.00 Curl_raw_nequal > 0.82 19.85 0.23 2764123 0.00 0.00 Curl_infof > 0.81 20.07 0.22 5933857 0.00 0.00 ossl_recv > 0.77 20.28 0.21 323112 0.00 0.00 > Curl_splayremovebyaddr > 0.75 20.48 0.21 5943991 0.00 0.00 Curl_read > 0.70 20.67 0.19 425174 0.00 0.00 event_del > 0.70 20.86 0.19 98652 0.00 0.00 > Curl_if_is_interface_name > 0.70 21.05 0.19 8677733 0.00 0.00 Curl_socket_check > 0.59 21.21 0.16 4129554 0.00 0.00 fd_key_compare > 0.57 21.37 0.16 14017019 0.00 0.00 Curl_debug > 0.55 21.52 0.15 9376861 0.00 0.00 stat_data_in_add > 0.55 21.67 0.15 6840468 0.00 0.00 Curl_timeleft > 0.55 21.82 0.15 2054602 0.00 0.00 Curl_raw_equal > 0.55 21.97 0.15 1291611 0.00 0.00 Curl_expire > 0.51 22.11 0.14 16570063 0.00 0.00 curlx_tvnow > 0.51 22.25 0.14 13863284 0.00 0.00 scan_response > 0.51 22.39 0.14 5498864 0.00 0.00 Curl_setopt > 0.48 22.52 0.13 97490291 0.00 0.00 curl_strequal > 0.44 22.64 0.12 324253 0.00 0.00 Curl_http > 0.44 22.76 0.12 394386 0.00 0.00 ossl_connect_common > 0.40 22.87 0.11 3581335 0.00 0.00 Curl_getinfo > 0.38 22.97 0.11 26983092 0.00 0.00 alloc_addbyter > 0.37 23.07 0.10 8729044 0.00 0.00 Curl_client_write > > |
|
From: Robert I. <cor...@gm...> - 2012-11-24 15:06:43
|
Hi Fred, I think you can address this suggestion and your observations to the lists of cares and curl-development. Take care, Thanks Robert On Sat, Nov 24, 2012 at 5:00 PM, Fred Huang <di...@gm...> wrote: > hi all, > > is it possible to use adns (http://www.chiark.greenend.org.uk/~ian/adns/) to > do dns resolving instead of libc-ares? all my test results shows that > lib-cares can do no more than 300 queries per second. however, using 'dig > +short -f hostnames.list &>/dev/null' against same dns server I can get more > 7500 queries per second. I think libc-ares is also limited by the system API > of gethostbyname. > > About ADNS lib: > Many clients for DNS resolution are coded poorly.Most UNIX systems > provide an implementation of gethostbyname (the DNS client API—application > program interface), which cannot concurrently handle multiple outstanding > requests. Therefore, the crawler cannot issue many resolution requests > together and poll at a later time for completion of individual requests, > which is critical for acceptable performance. Furthermore, if the > system-provided client is used, there is no way to distribute load among a > number of DNS servers. For all these reasons, many crawlers choose to > include their own custom client for DNS name resolution. The Mercator > crawler from Compaq System Research Center reduced the time spent in DNS > from as high as 87% to a modest 25% by implementing a custom client. The > ADNS asynchronous DNS client library is ideal for use in crawlers. > In spite of these optimizations, a large-scale crawler will spend a > substantial fraction of its network time not waiting for Http data transfer, > but for address resolution. For every hostname that has not been resolved > before (which happens frequently with crawlers), the local DNS may have to > go across many network hops to fill its cache for the first time. To overlap > this unavoidable delay with useful work, prefetching can be used. When a > page that has just been fetched is parsed, a stream of HREFs is extracted. > Right at this time, that is, even before any of the corresponding URLs are > fetched, hostnames are extracted from the HREF targets, and DNS resolution > requests are made to the caching server. The prefetching client is usually > implemented using UDP instead of TCP, and it does not wait for resolution > to be completed. The request serves only to fill the DNS cache so that > resolution will be fast when the page is actually needed later on. > > ===end > > > 2012/3/5 Fred Huang <di...@gm...> >> >> test 1: >> >> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com to >> one IP address >> number of domain names in all https URLs: 780,000 >> number of client: 1000 >> CPU usage: 78% >> cpu% irq% sirq% sys% iowt% mem_used buf&cached >> 78.8 0.0 0.6 3.8 0.0 3640.9Mb 282.1Mb >> SSL TPS: 300 >> SSL throughput: 200Mbps >> >> # gprof /usr/bin/curl-loader gmon.out -p | head -50 >> Flat profile: >> Each sample counts as 0.01 seconds. >> % cumulative self self total >> time seconds seconds calls s/call s/call name >> 38.55 127.10 127.10 130975 0.00 0.00 >> Curl_hash_clean_with_criterium >> 24.63 208.32 81.22 1247528407 0.00 0.00 >> hostcache_timestamp_remove >> 13.08 251.44 43.12 2791988 0.00 0.00 Curl_hash_pick >> 7.30 275.51 24.07 352490 0.00 0.00 Curl_hash_add >> 6.01 295.31 19.80 410760770 0.00 0.00 >> Curl_str_key_compare >> 2.95 305.04 9.73 131015 0.00 0.00 create_conn >> 0.88 307.94 2.90 4841390 0.00 0.00 dprintf_formatf >> 0.74 310.38 2.44 136036 0.00 0.00 ConnectionStore >> 0.48 311.96 1.58 814604911 0.00 0.00 ares__is_list_empty >> 0.36 313.16 1.20 locking_function >> 0.27 314.06 0.90 261608 0.00 0.00 ares_cancel >> 0.22 314.79 0.73 127417 0.00 0.00 >> curl_multi_remove_handle >> 0.19 315.40 0.62 8906301 0.00 0.00 >> client_tracing_function >> 0.17 315.97 0.57 135305 0.00 0.00 Curl_num_addresses >> 0.17 316.54 0.57 181937907 0.00 0.00 addbyter >> 0.17 317.09 0.55 2374363 0.00 0.00 multi_runsingle >> 0.16 317.62 0.53 541954 0.00 0.00 ares__init_list_node >> 0.16 318.14 0.52 494412 0.00 0.00 >> curl_multi_socket_action >> 0.14 318.60 0.46 33578897 0.00 0.00 Curl_socket_check >> 0.13 319.04 0.44 102951052 0.00 0.00 Curl_raw_toupper >> 0.13 319.48 0.44 787558 0.00 0.00 Curl_readwrite >> 0.13 319.91 0.44 id_function >> 0.12 320.29 0.38 391390 0.00 0.00 Curl_hash_str >> 0.09 320.58 0.29 9216 0.00 0.00 curl_multi_perform >> 0.08 320.86 0.28 2350908 0.00 0.00 Curl_raw_equal >> 0.08 321.14 0.28 2019415 0.00 0.00 Curl_splay >> 0.08 321.40 0.26 520625 0.00 0.00 ossl_connect_common >> 0.08 321.65 0.25 3532992 0.00 0.00 Curl_pgrsUpdate >> 0.07 321.87 0.22 131778537 0.00 0.00 curl_strequal >> 0.05 322.05 0.18 8954118 0.00 0.00 scan_response >> 0.05 322.23 0.18 533644 0.00 0.00 ares_expand_name >> 0.05 322.40 0.17 2477345 0.00 0.00 fd_key_compare >> 0.05 322.57 0.17 2429135 0.00 0.00 Curl_infof >> 0.05 322.74 0.17 130966 0.00 0.00 singleipconnect >> 0.05 322.90 0.16 9292 0.00 0.00 >> curl_multi_socket_all >> 0.05 323.06 0.16 270137 0.00 0.00 ares__get_hostent >> 0.05 323.21 0.15 8943657 0.00 0.00 Curl_debug >> 0.04 323.35 0.14 262493 0.00 0.00 >> Curl_ssl_getsessionid >> 0.04 323.48 0.13 566006 0.00 0.00 socket_callback >> 0.04 323.61 0.13 33983864 0.00 0.00 curlx_tvdiff >> 0.04 323.74 0.13 132853 0.00 0.00 >> Curl_http_readwrite_headers >> 0.04 323.86 0.12 344032 0.00 0.00 epoll_del >> 0.04 323.98 0.12 223477 0.00 0.00 Curl_poll >> 0.04 324.10 0.12 8526280 0.00 0.00 Curl_raw_nequal >> 0.03 324.21 0.11 348839 0.00 0.00 event_del >> >> >> >> >> test 2: >> >> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com to >> one IP address >> number of domain names in all https URLs: 1 >> number of client: 1000 >> CPU usage: 75% >> SSL TPS: 1300 >> SSL throughput: 700Mbps >> >> # gprof /usr/bin/curl-loader gmon.out -p | head -50 >> Flat profile: >> Each sample counts as 0.01 seconds. >> % cumulative self self total >> time seconds seconds calls s/call s/call name >> 8.98 2.45 2.45 5127312 0.00 0.00 dprintf_formatf >> 8.90 4.88 2.43 1292796503 0.00 0.00 >> ares__is_list_empty >> 6.12 6.55 1.67 322461 0.00 0.00 create_conn >> 4.58 7.80 1.25 419872 0.00 0.00 ares_cancel >> 4.10 8.92 1.12 2100162 0.00 0.00 Curl_readwrite >> 3.92 9.99 1.07 4527388 0.00 0.00 Curl_hash_pick >> 3.43 10.93 0.94 locking_function >> 3.04 11.76 0.83 909759 0.00 0.00 >> curl_multi_socket_action >> 3.00 12.58 0.82 97252 0.00 0.00 >> Curl_hash_clean_with_criterium >> 2.71 13.32 0.74 179704016 0.00 0.00 Curl_raw_toupper >> 2.68 14.05 0.73 4646524 0.00 0.00 multi_runsingle >> 2.44 14.71 0.67 13862992 0.00 0.00 >> client_tracing_function >> 2.42 15.37 0.66 319735 0.00 0.00 >> curl_multi_remove_handle >> 2.42 16.03 0.66 22 0.03 0.03 ares__init_list_node >> 1.94 16.56 0.53 9930244 0.00 0.00 >> hostcache_timestamp_remove >> 1.94 17.09 0.53 169320088 0.00 0.00 addbyter >> 1.80 17.58 0.49 id_function >> 1.36 17.95 0.37 7057146 0.00 0.00 Curl_pgrsUpdate >> 1.36 18.32 0.37 16835 0.00 0.00 >> curl_multi_socket_all >> 1.25 18.66 0.34 16719 0.00 0.00 curl_multi_perform >> 1.21 18.99 0.33 329403 0.00 0.00 >> Curl_http_readwrite_headers >> 1.17 19.31 0.32 4036624 0.00 0.00 Curl_splay >> 1.14 19.62 0.31 20838902 0.00 0.00 Curl_raw_nequal >> 0.82 19.85 0.23 2764123 0.00 0.00 Curl_infof >> 0.81 20.07 0.22 5933857 0.00 0.00 ossl_recv >> 0.77 20.28 0.21 323112 0.00 0.00 >> Curl_splayremovebyaddr >> 0.75 20.48 0.21 5943991 0.00 0.00 Curl_read >> 0.70 20.67 0.19 425174 0.00 0.00 event_del >> 0.70 20.86 0.19 98652 0.00 0.00 >> Curl_if_is_interface_name >> 0.70 21.05 0.19 8677733 0.00 0.00 Curl_socket_check >> 0.59 21.21 0.16 4129554 0.00 0.00 fd_key_compare >> 0.57 21.37 0.16 14017019 0.00 0.00 Curl_debug >> 0.55 21.52 0.15 9376861 0.00 0.00 stat_data_in_add >> 0.55 21.67 0.15 6840468 0.00 0.00 Curl_timeleft >> 0.55 21.82 0.15 2054602 0.00 0.00 Curl_raw_equal >> 0.55 21.97 0.15 1291611 0.00 0.00 Curl_expire >> 0.51 22.11 0.14 16570063 0.00 0.00 curlx_tvnow >> 0.51 22.25 0.14 13863284 0.00 0.00 scan_response >> 0.51 22.39 0.14 5498864 0.00 0.00 Curl_setopt >> 0.48 22.52 0.13 97490291 0.00 0.00 curl_strequal >> 0.44 22.64 0.12 324253 0.00 0.00 Curl_http >> 0.44 22.76 0.12 394386 0.00 0.00 ossl_connect_common >> 0.40 22.87 0.11 3581335 0.00 0.00 Curl_getinfo >> 0.38 22.97 0.11 26983092 0.00 0.00 alloc_addbyter >> 0.37 23.07 0.10 8729044 0.00 0.00 Curl_client_write >> > > > ------------------------------------------------------------------------------ > Monitor your physical, virtual and cloud infrastructure from a single > web console. Get in-depth insight into apps, servers, databases, vmware, > SAP, cloud infrastructure, etc. Download 30-day Free Trial. > Pricing starts from $795 for 25 servers or applications! > http://p.sf.net/sfu/zoho_dev2dev_nov > _______________________________________________ > curl-loader-devel mailing list > cur...@li... > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > -- Regards, Robert Iakobashvili, Ph.D. Home: http://www.ghotit.com ......................................................... Ghotit Dyslexia -> Das Ist Real Writer ......................................................... |
|
From: Fred H. <di...@gm...> - 2012-11-26 07:08:28
|
yes. I will follow up. one more thing I see is that cares driven by libcurl sends queries in req/resp/req/resp/req/resp mode while adnshost sends queries in req,req,req/resp, resp,resp/... mode. that is to say, libcares doesnot use multi-querying in the wire line. req/resp/req/resp mode has a limit between 250~600 queries/s, while req,req,req/resp, resp,resp/... mode can reach 10,000 queries/s 2012/11/24 Robert Iakobashvili <cor...@gm...> > Hi Fred, > I think you can address this suggestion and your observations to the lists > of cares and curl-development. > > Take care, > Thanks > Robert > > On Sat, Nov 24, 2012 at 5:00 PM, Fred Huang <di...@gm...> wrote: > > hi all, > > > > is it possible to use adns (http://www.chiark.greenend.org.uk/~ian/adns/) > to > > do dns resolving instead of libc-ares? all my test results shows that > > lib-cares can do no more than 300 queries per second. however, using 'dig > > +short -f hostnames.list &>/dev/null' against same dns server I can get > more > > 7500 queries per second. I think libc-ares is also limited by the system > API > > of gethostbyname. > > > > About ADNS lib: > > Many clients for DNS resolution are coded poorly.Most UNIX systems > > provide an implementation of gethostbyname (the DNS client > API—application > > program interface), which cannot concurrently handle multiple outstanding > > requests. Therefore, the crawler cannot issue many resolution requests > > together and poll at a later time for completion of individual requests, > > which is critical for acceptable performance. Furthermore, if the > > system-provided client is used, there is no way to distribute load among > a > > number of DNS servers. For all these reasons, many crawlers choose to > > include their own custom client for DNS name resolution. The Mercator > > crawler from Compaq System Research Center reduced the time spent in DNS > > from as high as 87% to a modest 25% by implementing a custom client. The > > ADNS asynchronous DNS client library is ideal for use in crawlers. > > In spite of these optimizations, a large-scale crawler will spend a > > substantial fraction of its network time not waiting for Http data > transfer, > > but for address resolution. For every hostname that has not been resolved > > before (which happens frequently with crawlers), the local DNS may have > to > > go across many network hops to fill its cache for the first time. To > overlap > > this unavoidable delay with useful work, prefetching can be used. When a > > page that has just been fetched is parsed, a stream of HREFs is > extracted. > > Right at this time, that is, even before any of the corresponding URLs > are > > fetched, hostnames are extracted from the HREF targets, and DNS > resolution > > requests are made to the caching server. The prefetching client is > usually > > implemented using UDP instead of TCP, and it does not wait for > resolution > > to be completed. The request serves only to fill the DNS cache so that > > resolution will be fast when the page is actually needed later on. > > > > ===end > > > > > > 2012/3/5 Fred Huang <di...@gm...> > >> > >> test 1: > >> > >> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve > *.com to > >> one IP address > >> number of domain names in all https URLs: 780,000 > >> number of client: 1000 > >> CPU usage: 78% > >> cpu% irq% sirq% sys% iowt% mem_used buf&cached > >> 78.8 0.0 0.6 3.8 0.0 3640.9Mb 282.1Mb > >> SSL TPS: 300 > >> SSL throughput: 200Mbps > >> > >> # gprof /usr/bin/curl-loader gmon.out -p | head -50 > >> Flat profile: > >> Each sample counts as 0.01 seconds. > >> % cumulative self self total > >> time seconds seconds calls s/call s/call name > >> 38.55 127.10 127.10 130975 0.00 0.00 > >> Curl_hash_clean_with_criterium > >> 24.63 208.32 81.22 1247528407 0.00 0.00 > >> hostcache_timestamp_remove > >> 13.08 251.44 43.12 2791988 0.00 0.00 Curl_hash_pick > >> 7.30 275.51 24.07 352490 0.00 0.00 Curl_hash_add > >> 6.01 295.31 19.80 410760770 0.00 0.00 > >> Curl_str_key_compare > >> 2.95 305.04 9.73 131015 0.00 0.00 create_conn > >> 0.88 307.94 2.90 4841390 0.00 0.00 dprintf_formatf > >> 0.74 310.38 2.44 136036 0.00 0.00 ConnectionStore > >> 0.48 311.96 1.58 814604911 0.00 0.00 > ares__is_list_empty > >> 0.36 313.16 1.20 locking_function > >> 0.27 314.06 0.90 261608 0.00 0.00 ares_cancel > >> 0.22 314.79 0.73 127417 0.00 0.00 > >> curl_multi_remove_handle > >> 0.19 315.40 0.62 8906301 0.00 0.00 > >> client_tracing_function > >> 0.17 315.97 0.57 135305 0.00 0.00 Curl_num_addresses > >> 0.17 316.54 0.57 181937907 0.00 0.00 addbyter > >> 0.17 317.09 0.55 2374363 0.00 0.00 multi_runsingle > >> 0.16 317.62 0.53 541954 0.00 0.00 > ares__init_list_node > >> 0.16 318.14 0.52 494412 0.00 0.00 > >> curl_multi_socket_action > >> 0.14 318.60 0.46 33578897 0.00 0.00 Curl_socket_check > >> 0.13 319.04 0.44 102951052 0.00 0.00 Curl_raw_toupper > >> 0.13 319.48 0.44 787558 0.00 0.00 Curl_readwrite > >> 0.13 319.91 0.44 id_function > >> 0.12 320.29 0.38 391390 0.00 0.00 Curl_hash_str > >> 0.09 320.58 0.29 9216 0.00 0.00 curl_multi_perform > >> 0.08 320.86 0.28 2350908 0.00 0.00 Curl_raw_equal > >> 0.08 321.14 0.28 2019415 0.00 0.00 Curl_splay > >> 0.08 321.40 0.26 520625 0.00 0.00 > ossl_connect_common > >> 0.08 321.65 0.25 3532992 0.00 0.00 Curl_pgrsUpdate > >> 0.07 321.87 0.22 131778537 0.00 0.00 curl_strequal > >> 0.05 322.05 0.18 8954118 0.00 0.00 scan_response > >> 0.05 322.23 0.18 533644 0.00 0.00 ares_expand_name > >> 0.05 322.40 0.17 2477345 0.00 0.00 fd_key_compare > >> 0.05 322.57 0.17 2429135 0.00 0.00 Curl_infof > >> 0.05 322.74 0.17 130966 0.00 0.00 singleipconnect > >> 0.05 322.90 0.16 9292 0.00 0.00 > >> curl_multi_socket_all > >> 0.05 323.06 0.16 270137 0.00 0.00 ares__get_hostent > >> 0.05 323.21 0.15 8943657 0.00 0.00 Curl_debug > >> 0.04 323.35 0.14 262493 0.00 0.00 > >> Curl_ssl_getsessionid > >> 0.04 323.48 0.13 566006 0.00 0.00 socket_callback > >> 0.04 323.61 0.13 33983864 0.00 0.00 curlx_tvdiff > >> 0.04 323.74 0.13 132853 0.00 0.00 > >> Curl_http_readwrite_headers > >> 0.04 323.86 0.12 344032 0.00 0.00 epoll_del > >> 0.04 323.98 0.12 223477 0.00 0.00 Curl_poll > >> 0.04 324.10 0.12 8526280 0.00 0.00 Curl_raw_nequal > >> 0.03 324.21 0.11 348839 0.00 0.00 event_del > >> > >> > >> > >> > >> test 2: > >> > >> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve > *.com to > >> one IP address > >> number of domain names in all https URLs: 1 > >> number of client: 1000 > >> CPU usage: 75% > >> SSL TPS: 1300 > >> SSL throughput: 700Mbps > >> > >> # gprof /usr/bin/curl-loader gmon.out -p | head -50 > >> Flat profile: > >> Each sample counts as 0.01 seconds. > >> % cumulative self self total > >> time seconds seconds calls s/call s/call name > >> 8.98 2.45 2.45 5127312 0.00 0.00 dprintf_formatf > >> 8.90 4.88 2.43 1292796503 0.00 0.00 > >> ares__is_list_empty > >> 6.12 6.55 1.67 322461 0.00 0.00 create_conn > >> 4.58 7.80 1.25 419872 0.00 0.00 ares_cancel > >> 4.10 8.92 1.12 2100162 0.00 0.00 Curl_readwrite > >> 3.92 9.99 1.07 4527388 0.00 0.00 Curl_hash_pick > >> 3.43 10.93 0.94 locking_function > >> 3.04 11.76 0.83 909759 0.00 0.00 > >> curl_multi_socket_action > >> 3.00 12.58 0.82 97252 0.00 0.00 > >> Curl_hash_clean_with_criterium > >> 2.71 13.32 0.74 179704016 0.00 0.00 Curl_raw_toupper > >> 2.68 14.05 0.73 4646524 0.00 0.00 multi_runsingle > >> 2.44 14.71 0.67 13862992 0.00 0.00 > >> client_tracing_function > >> 2.42 15.37 0.66 319735 0.00 0.00 > >> curl_multi_remove_handle > >> 2.42 16.03 0.66 22 0.03 0.03 > ares__init_list_node > >> 1.94 16.56 0.53 9930244 0.00 0.00 > >> hostcache_timestamp_remove > >> 1.94 17.09 0.53 169320088 0.00 0.00 addbyter > >> 1.80 17.58 0.49 id_function > >> 1.36 17.95 0.37 7057146 0.00 0.00 Curl_pgrsUpdate > >> 1.36 18.32 0.37 16835 0.00 0.00 > >> curl_multi_socket_all > >> 1.25 18.66 0.34 16719 0.00 0.00 curl_multi_perform > >> 1.21 18.99 0.33 329403 0.00 0.00 > >> Curl_http_readwrite_headers > >> 1.17 19.31 0.32 4036624 0.00 0.00 Curl_splay > >> 1.14 19.62 0.31 20838902 0.00 0.00 Curl_raw_nequal > >> 0.82 19.85 0.23 2764123 0.00 0.00 Curl_infof > >> 0.81 20.07 0.22 5933857 0.00 0.00 ossl_recv > >> 0.77 20.28 0.21 323112 0.00 0.00 > >> Curl_splayremovebyaddr > >> 0.75 20.48 0.21 5943991 0.00 0.00 Curl_read > >> 0.70 20.67 0.19 425174 0.00 0.00 event_del > >> 0.70 20.86 0.19 98652 0.00 0.00 > >> Curl_if_is_interface_name > >> 0.70 21.05 0.19 8677733 0.00 0.00 Curl_socket_check > >> 0.59 21.21 0.16 4129554 0.00 0.00 fd_key_compare > >> 0.57 21.37 0.16 14017019 0.00 0.00 Curl_debug > >> 0.55 21.52 0.15 9376861 0.00 0.00 stat_data_in_add > >> 0.55 21.67 0.15 6840468 0.00 0.00 Curl_timeleft > >> 0.55 21.82 0.15 2054602 0.00 0.00 Curl_raw_equal > >> 0.55 21.97 0.15 1291611 0.00 0.00 Curl_expire > >> 0.51 22.11 0.14 16570063 0.00 0.00 curlx_tvnow > >> 0.51 22.25 0.14 13863284 0.00 0.00 scan_response > >> 0.51 22.39 0.14 5498864 0.00 0.00 Curl_setopt > >> 0.48 22.52 0.13 97490291 0.00 0.00 curl_strequal > >> 0.44 22.64 0.12 324253 0.00 0.00 Curl_http > >> 0.44 22.76 0.12 394386 0.00 0.00 > ossl_connect_common > >> 0.40 22.87 0.11 3581335 0.00 0.00 Curl_getinfo > >> 0.38 22.97 0.11 26983092 0.00 0.00 alloc_addbyter > >> 0.37 23.07 0.10 8729044 0.00 0.00 Curl_client_write > >> > > > > > > > ------------------------------------------------------------------------------ > > Monitor your physical, virtual and cloud infrastructure from a single > > web console. Get in-depth insight into apps, servers, databases, vmware, > > SAP, cloud infrastructure, etc. Download 30-day Free Trial. > > Pricing starts from $795 for 25 servers or applications! > > http://p.sf.net/sfu/zoho_dev2dev_nov > > _______________________________________________ > > curl-loader-devel mailing list > > cur...@li... > > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > > > > > > -- > Regards, > Robert Iakobashvili, Ph.D. > > Home: http://www.ghotit.com > ......................................................... > Ghotit Dyslexia -> Das Ist Real Writer > ......................................................... > > > ------------------------------------------------------------------------------ > Monitor your physical, virtual and cloud infrastructure from a single > web console. Get in-depth insight into apps, servers, databases, vmware, > SAP, cloud infrastructure, etc. Download 30-day Free Trial. > Pricing starts from $795 for 25 servers or applications! > http://p.sf.net/sfu/zoho_dev2dev_nov > _______________________________________________ > curl-loader-devel mailing list > cur...@li... > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > |
Thanks for helping keep SourceForge clean.
X