Re: curl-loader https: low performance due to DNS query?
Status: Alpha
Brought to you by:
coroberti
Menu
▾
▴
Re: curl-loader https: low performance due to DNS query?
|
From: Robert I. <cor...@gm...> - 2012-03-05 17:12:12
|
If it will not be helpful this means that recent changes in libcurl library have broken DNS interface. You can try a previos version and the latest libcurl, like: http://curl.haxx.se/download/curl-7.23.0.tar.gz and http://curl.haxx.se/downloadcurl-7.24.0.tar.gz by: 1. downloading a libcurl version and placing it to curl-loader-0.56/packages ; 2. correcting the Makefile in curl-loader-0.56 CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24 or CURL_VER:=7.23 3. make cleanall; make Please, send us your observations. Thanks! Robert On Mon, Mar 5, 2012 at 6:57 PM, Robert Iakobashvili <cor...@gm...> wrote: > If so, please try to change > > loader.c, line 578 which looks like: > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); > > 1. instead of -1 try to set some value like 60 > 2. make clean; make > > Thanks, > Robert > > On Mon, Mar 5, 2012 at 6:19 PM, Fred Huang <di...@gm...> wrote: >> >> Looks like the bottleneck comes from libcurl other than c-ares... >> test 3 (no ares patch applied): >> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com to one IP address >> number of domain names in URLs: 780,000 >> number of client: 1000 >> CPU usage: 85% >> cpu% irq% sirq% sys% iowt% mem_used buf&cached >> 85.6 0.0 0.8 5.9 0.0 3334.3Mb 112.0Mb >> SSL TPS: 320 >> SSL throughput: 200Mbps >> >> # gprof /usr/bin/curl-loader gmon.out -p | head -50 >> Flat profile: >> Each sample counts as 0.01 seconds. >> % cumulative self self total >> time seconds seconds calls s/call s/call name >> 40.80 120.16 120.16 116079 0.00 0.00 Curl_hash_clean_with_criterium >> 24.46 192.18 72.02 983166323 0.00 0.00 hostcache_timestamp_remove >> 13.30 231.35 39.17 2059341 0.00 0.00 Curl_hash_pick >> 6.45 250.35 19.00 240087 0.00 0.00 Curl_hash_add >> 4.81 264.51 14.16 334257102 0.00 0.00 Curl_str_key_compare >> 3.40 274.52 10.01 119514 0.00 0.00 create_conn >> 0.93 277.25 2.73 4478910 0.00 0.00 dprintf_formatf >> 0.90 279.91 2.66 122209 0.00 0.00 ConnectionStore >> 0.40 281.09 1.18 locking_function >> 0.27 281.88 0.79 114759 0.00 0.00 curl_multi_remove_handle >> 0.21 282.51 0.63 7993664 0.00 0.00 client_tracing_function >> 0.21 283.12 0.61 122685 0.00 0.00 Curl_num_addresses >> 0.19 283.68 0.56 397028 0.00 0.00 curl_multi_socket_action >> 0.19 284.23 0.55 2036116 0.00 0.00 multi_runsingle >> 0.17 284.72 0.49 703687 0.00 0.00 Curl_readwrite >> 0.16 285.19 0.47 id_function >> 0.15 285.62 0.43 29692029 0.00 0.00 Curl_socket_check >> 0.14 286.02 0.40 91973856 0.00 0.00 Curl_raw_toupper >> 0.13 286.40 0.38 164664144 0.00 0.00 addbyter >> 0.13 286.77 0.37 2180797 0.00 0.00 Curl_infof >> 0.11 287.10 0.33 122675 0.00 0.00 Curl_if_is_interface_name >> 0.10 287.39 0.29 471318 0.00 0.00 ossl_connect_common >> 0.10 287.67 0.28 2113528 0.00 0.00 Curl_raw_equal >> 0.09 287.93 0.26 8676 0.00 0.02 curl_multi_socket_all >> 0.08 288.17 0.24 1663262 0.00 0.00 Curl_splay >> 0.08 288.41 0.24 8505 0.00 0.01 curl_multi_perform >> 0.07 288.63 0.22 3114576 0.00 0.00 Curl_pgrsUpdate >> 0.06 288.80 0.17 119637 0.00 0.00 singleipconnect >> 0.06 288.97 0.17 7994818 0.00 0.00 Curl_debug >> 0.05 289.13 0.16 235820 0.00 0.00 event_del >> 0.05 289.29 0.16 119393071 0.00 0.00 curl_strequal >> 0.05 289.45 0.16 18535473 0.00 0.00 alloc_addbyter >> 0.05 289.61 0.16 356074 0.00 0.00 Curl_hash_str >> 0.05 289.76 0.15 1797233 0.00 0.00 fd_key_compare >> 0.05 289.91 0.15 234966 0.00 0.00 Curl_ssl_getsessionid >> 0.04 290.04 0.13 585853 0.00 0.00 Curl_llist_destroy >> 0.04 290.16 0.12 240826 0.00 0.00 Curl_resolv >> 0.03 290.26 0.10 7514328 0.00 0.00 Curl_raw_nequal >> 0.03 290.36 0.10 2214436 0.00 0.00 Curl_read >> 0.03 290.46 0.10 116872 0.00 0.00 Curl_http_readwrite_headers >> 0.03 290.56 0.10 ssl_tls_trace >> 0.03 290.66 0.10 8013097 0.00 0.00 scan_response >> 0.03 290.75 0.09 29488555 0.00 0.00 curlx_tvdiff >> 0.03 290.84 0.09 2142468 0.00 0.00 Curl_setopt >> 0.03 290.93 0.09 476299 0.00 0.00 Curl_expire >> >> >> >> >> >> >> >> >> 2012/3/5 Robert Iakobashvili <cor...@gm...> >>> >>> Dear Fred Huang, >>> >>> Thanks for your posting. >>> >>> Could you please check if this is due to some badness at cares? >>> Thanks. >>> >>> Let's exclude usage of cares based DNS resolving by applying the patch attached >>> and use instead global DNS cache. >>> >>> cd ./curl-loader-0.56 >>> patch -p1 < path-to/disable-cares-use-global-dns.patch >>> make cleanall; make >>> >>> See if it helps. >>> >>> Global cache, unfortunately, is not thread safe. >>> >>> If it helps, we'll try to restore the previous version of cares. >>> >>> Thanks, >>> Robert >>> >>> On Mon, Mar 5, 2012 at 4:28 PM, Fred Huang <di...@gm...> wrote: >>>> >>>> test 1: >>>> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com to one IP address >>>> number of domain names in all https URLs: 780,000 >>>> number of client: 1000 >>>> CPU usage: 78% >>>> cpu% irq% sirq% sys% iowt% mem_used buf&cached >>>> 78.8 0.0 0.6 3.8 0.0 3640.9Mb 282.1Mb >>>> SSL TPS: 300 >>>> SSL throughput: 200Mbps >>>> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50 >>>> Flat profile: >>>> Each sample counts as 0.01 seconds. >>>> % cumulative self self total >>>> time seconds seconds calls s/call s/call name >>>> 38.55 127.10 127.10 130975 0.00 0.00 Curl_hash_clean_with_criterium >>>> 24.63 208.32 81.22 1247528407 0.00 0.00 hostcache_timestamp_remove >>>> 13.08 251.44 43.12 2791988 0.00 0.00 Curl_hash_pick >>>> 7.30 275.51 24.07 352490 0.00 0.00 Curl_hash_add >>>> 6.01 295.31 19.80 410760770 0.00 0.00 Curl_str_key_compare >>>> 2.95 305.04 9.73 131015 0.00 0.00 create_conn >>>> 0.88 307.94 2.90 4841390 0.00 0.00 dprintf_formatf >>>> 0.74 310.38 2.44 136036 0.00 0.00 ConnectionStore >>>> 0.48 311.96 1.58 814604911 0.00 0.00 ares__is_list_empty >>>> 0.36 313.16 1.20 locking_function >>>> 0.27 314.06 0.90 261608 0.00 0.00 ares_cancel >>>> 0.22 314.79 0.73 127417 0.00 0.00 curl_multi_remove_handle >>>> 0.19 315.40 0.62 8906301 0.00 0.00 client_tracing_function >>>> 0.17 315.97 0.57 135305 0.00 0.00 Curl_num_addresses >>>> 0.17 316.54 0.57 181937907 0.00 0.00 addbyter >>>> 0.17 317.09 0.55 2374363 0.00 0.00 multi_runsingle >>>> 0.16 317.62 0.53 541954 0.00 0.00 ares__init_list_node >>>> 0.16 318.14 0.52 494412 0.00 0.00 curl_multi_socket_action >>>> 0.14 318.60 0.46 33578897 0.00 0.00 Curl_socket_check >>>> 0.13 319.04 0.44 102951052 0.00 0.00 Curl_raw_toupper >>>> 0.13 319.48 0.44 787558 0.00 0.00 Curl_readwrite >>>> 0.13 319.91 0.44 id_function >>>> 0.12 320.29 0.38 391390 0.00 0.00 Curl_hash_str >>>> 0.09 320.58 0.29 9216 0.00 0.00 curl_multi_perform >>>> 0.08 320.86 0.28 2350908 0.00 0.00 Curl_raw_equal >>>> 0.08 321.14 0.28 2019415 0.00 0.00 Curl_splay >>>> 0.08 321.40 0.26 520625 0.00 0.00 ossl_connect_common >>>> 0.08 321.65 0.25 3532992 0.00 0.00 Curl_pgrsUpdate >>>> 0.07 321.87 0.22 131778537 0.00 0.00 curl_strequal >>>> 0.05 322.05 0.18 8954118 0.00 0.00 scan_response >>>> 0.05 322.23 0.18 533644 0.00 0.00 ares_expand_name >>>> 0.05 322.40 0.17 2477345 0.00 0.00 fd_key_compare >>>> 0.05 322.57 0.17 2429135 0.00 0.00 Curl_infof >>>> 0.05 322.74 0.17 130966 0.00 0.00 singleipconnect >>>> 0.05 322.90 0.16 9292 0.00 0.00 curl_multi_socket_all >>>> 0.05 323.06 0.16 270137 0.00 0.00 ares__get_hostent >>>> 0.05 323.21 0.15 8943657 0.00 0.00 Curl_debug >>>> 0.04 323.35 0.14 262493 0.00 0.00 Curl_ssl_getsessionid >>>> 0.04 323.48 0.13 566006 0.00 0.00 socket_callback >>>> 0.04 323.61 0.13 33983864 0.00 0.00 curlx_tvdiff >>>> 0.04 323.74 0.13 132853 0.00 0.00 Curl_http_readwrite_headers >>>> 0.04 323.86 0.12 344032 0.00 0.00 epoll_del >>>> 0.04 323.98 0.12 223477 0.00 0.00 Curl_poll >>>> 0.04 324.10 0.12 8526280 0.00 0.00 Curl_raw_nequal >>>> 0.03 324.21 0.11 348839 0.00 0.00 event_del >>>> >>>> >>>> >>>> >>>> test 2: >>>> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com to one IP address >>>> number of domain names in all https URLs: 1 >>>> number of client: 1000 >>>> CPU usage: 75% >>>> SSL TPS: 1300 >>>> SSL throughput: 700Mbps >>>> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50 >>>> Flat profile: >>>> Each sample counts as 0.01 seconds. >>>> % cumulative self self total >>>> time seconds seconds calls s/call s/call name >>>> 8.98 2.45 2.45 5127312 0.00 0.00 dprintf_formatf >>>> 8.90 4.88 2.43 1292796503 0.00 0.00 ares__is_list_empty >>>> 6.12 6.55 1.67 322461 0.00 0.00 create_conn >>>> 4.58 7.80 1.25 419872 0.00 0.00 ares_cancel >>>> 4.10 8.92 1.12 2100162 0.00 0.00 Curl_readwrite >>>> 3.92 9.99 1.07 4527388 0.00 0.00 Curl_hash_pick >>>> 3.43 10.93 0.94 locking_function >>>> 3.04 11.76 0.83 909759 0.00 0.00 curl_multi_socket_action >>>> 3.00 12.58 0.82 97252 0.00 0.00 Curl_hash_clean_with_criterium >>>> 2.71 13.32 0.74 179704016 0.00 0.00 Curl_raw_toupper >>>> 2.68 14.05 0.73 4646524 0.00 0.00 multi_runsingle >>>> 2.44 14.71 0.67 13862992 0.00 0.00 client_tracing_function >>>> 2.42 15.37 0.66 319735 0.00 0.00 curl_multi_remove_handle >>>> 2.42 16.03 0.66 22 0.03 0.03 ares__init_list_node >>>> 1.94 16.56 0.53 9930244 0.00 0.00 hostcache_timestamp_remove >>>> 1.94 17.09 0.53 169320088 0.00 0.00 addbyter >>>> 1.80 17.58 0.49 id_function >>>> 1.36 17.95 0.37 7057146 0.00 0.00 Curl_pgrsUpdate >>>> 1.36 18.32 0.37 16835 0.00 0.00 curl_multi_socket_all >>>> 1.25 18.66 0.34 16719 0.00 0.00 curl_multi_perform >>>> 1.21 18.99 0.33 329403 0.00 0.00 Curl_http_readwrite_headers >>>> 1.17 19.31 0.32 4036624 0.00 0.00 Curl_splay >>>> 1.14 19.62 0.31 20838902 0.00 0.00 Curl_raw_nequal >>>> 0.82 19.85 0.23 2764123 0.00 0.00 Curl_infof >>>> 0.81 20.07 0.22 5933857 0.00 0.00 ossl_recv >>>> 0.77 20.28 0.21 323112 0.00 0.00 Curl_splayremovebyaddr >>>> 0.75 20.48 0.21 5943991 0.00 0.00 Curl_read >>>> 0.70 20.67 0.19 425174 0.00 0.00 event_del >>>> 0.70 20.86 0.19 98652 0.00 0.00 Curl_if_is_interface_name >>>> 0.70 21.05 0.19 8677733 0.00 0.00 Curl_socket_check >>>> 0.59 21.21 0.16 4129554 0.00 0.00 fd_key_compare >>>> 0.57 21.37 0.16 14017019 0.00 0.00 Curl_debug >>>> 0.55 21.52 0.15 9376861 0.00 0.00 stat_data_in_add >>>> 0.55 21.67 0.15 6840468 0.00 0.00 Curl_timeleft >>>> 0.55 21.82 0.15 2054602 0.00 0.00 Curl_raw_equal >>>> 0.55 21.97 0.15 1291611 0.00 0.00 Curl_expire >>>> 0.51 22.11 0.14 16570063 0.00 0.00 curlx_tvnow >>>> 0.51 22.25 0.14 13863284 0.00 0.00 scan_response >>>> 0.51 22.39 0.14 5498864 0.00 0.00 Curl_setopt >>>> 0.48 22.52 0.13 97490291 0.00 0.00 curl_strequal >>>> 0.44 22.64 0.12 324253 0.00 0.00 Curl_http >>>> 0.44 22.76 0.12 394386 0.00 0.00 ossl_connect_common >>>> 0.40 22.87 0.11 3581335 0.00 0.00 Curl_getinfo >>>> 0.38 22.97 0.11 26983092 0.00 0.00 alloc_addbyter >>>> 0.37 23.07 0.10 8729044 0.00 0.00 Curl_client_write >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> Try before you buy = See our experts in action! >>>> The most comprehensive online learning library for Microsoft developers >>>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, >>>> Metro Style Apps, more. Free future releases when you subscribe now! >>>> http://p.sf.net/sfu/learndevnow-dev2 >>>> _______________________________________________ >>>> curl-loader-devel mailing list >>>> cur...@li... >>>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >>>> >>> >>> >>> >>> -- >>> Regards, >>> Robert Iakobashvili, Ph.D. >>> >>> Home: http://www.ghotit.com >>> Blog: http://www.ghotit.com/category/ghotit-blogs/ >>> Twitter: http://twitter.com/ghotit >>> Facebook: http://facebook.com/ghotit >>> ...................................................................... >>> Ghotit Dyslexia >>> Assistive technology that understands you >>> ...................................................................... >>> >>> >>> ------------------------------------------------------------------------------ >>> Try before you buy = See our experts in action! >>> The most comprehensive online learning library for Microsoft developers >>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, >>> Metro Style Apps, more. Free future releases when you subscribe now! >>> http://p.sf.net/sfu/learndevnow-dev2 >>> _______________________________________________ >>> curl-loader-devel mailing list >>> cur...@li... >>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >>> >> >> >> ------------------------------------------------------------------------------ >> Try before you buy = See our experts in action! >> The most comprehensive online learning library for Microsoft developers >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, >> Metro Style Apps, more. Free future releases when you subscribe now! >> http://p.sf.net/sfu/learndevnow-dev2 >> _______________________________________________ >> curl-loader-devel mailing list >> cur...@li... >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >> > > > > -- > Regards, > Robert Iakobashvili, Ph.D. > > Home: http://www.ghotit.com > Blog: http://www.ghotit.com/category/ghotit-blogs/ > Twitter: http://twitter.com/ghotit > Facebook: http://facebook.com/ghotit > ...................................................................... > Ghotit Dyslexia > Assistive technology that understands you > ...................................................................... -- Regards, Robert Iakobashvili, Ph.D. Home: http://www.ghotit.com Blog: http://www.ghotit.com/category/ghotit-blogs/ Twitter: http://twitter.com/ghotit Facebook: http://facebook.com/ghotit ...................................................................... Ghotit Dyslexia Assistive technology that understands you ...................................................................... |
