Re: curl with c-ares

[Robert I. <cor...@gm...>]

Dear Shao,
I like your profiling.

If you are not using option -t (threads)
you can opt to make a global DNS cache as in
http://sourceforge.net/p/curl-loader/code/591/tree/trunk/curl-loader/loader.c,

Uncomment line 596:

  /*
     If DNS resolving is necesary, global DNS cache is enough,
     otherwise compile libcurl with ares (cares) library support.
     Attention: DNS global cache is not thread-safe, therefore use
     cares for asynchronous DNS lookups.

     curl_easy_setopt (handle, CURLOPT_DNS_USE_GLOBAL_CACHE, 1);
  */

and remove usage of cares in the code.
Still this path will not be stable with -t on multi-core HW.

Try your suggestion and, if it works with -t and without,
please send us your patch.

Kind regards,
Robert


On Tue, Sep 3, 2013 at 12:39 PM, Shao, Shuchao <ss...@we...> wrote:
> Hi,
>
>
>
> I am using curl-loader to simulate 5000 users generating HTTP/s traffic to a
> nginx server, find that curl-loader consumes many time on c-ares initiation.
> Please look the attached gprof output.
>
>
>
> Via checking the libcurl source code, the function Curl_resolver_init() do
> the c-ares initiation, and curl_easy_init() calls this function via
> Curl_open().  So we need to do the c-ares initiation every time when we want
> to initialize a curl handler.
>
> Via checking the C-ares souce code, there are two large ‘for’ loops in
> function ares_init_options(),
>
>                 #define ARES_QID_TABLE_SIZE 2048
>
> #define ARES_TIMEOUT_TABLE_SIZE 1024
>
>   for (i = 0; i < ARES_QID_TABLE_SIZE; i++)
>
>     {
>
>       ares__init_list_head(&(channel->queries_by_qid[i]));
>
>     }
>
>   for (i = 0; i < ARES_TIMEOUT_TABLE_SIZE; i++)
>
>     {
>
>       ares__init_list_head(&(channel->queries_by_timeout[i]));
>
>     }
>
> So, even when we only send a single GET request, curl will call
> ares__init_list_head() thousands times. That’s the reason why curl-loader
> consumes so many time on c-ares initiation.
>
>
>
> My question is if we can do the c-ares initiation only once but not in every
> curl_easy_init()? By that probably we can increase the curl-loader
> performance significantly.
>
>
>
>
>
> Some gprof output:
>
> Flat profile:
>
>
>
> Each sample counts as 0.01 seconds.
>
>   %   cumulative   self              self     total
>
> time   seconds   seconds    calls  ms/call  ms/call  name
>
> 18.46      0.12     0.12  9186352     0.00     0.00  ares__init_list_head
>
> 12.31      0.20     0.08        1    80.00    80.00  build_url_set
>
> 10.77      0.27     0.07    81473     0.00     0.00  dprintf_formatf
>
>   9.23      0.33     0.06     3325     0.02     0.07  ares_init_options
>
>   4.62      0.36     0.03     1770     0.02     0.04  create_conn
>
>   3.08      0.38     0.02   107452     0.00     0.00
> client_tracing_function
>
>   3.08      0.40     0.02    41568     0.00     0.00  fd_key_compare
>
>   3.08      0.42     0.02     5102     0.00     0.01  epoll_dispatch
>
>   3.08      0.44     0.02     3150     0.01     0.01
> ares_library_initialized
>
>   3.08      0.46     0.02     2243     0.01     0.01  ares_cancel
>
>   1.54      0.47     0.01  6918787     0.00     0.00  ares__is_list_empty
>
>   1.54      0.48     0.01   450212     0.00     0.00  alloc_addbyter
>
>   1.54      0.49     0.01   227162     0.00     0.00  aresx_uztosi
>
>   1.54      0.50     0.01   107005     0.00     0.00
> hostcache_timestamp_remove
>
>   1.54      0.51     0.01    56835     0.00     0.00  Curl_pgrsUpdate
>
>   1.54      0.52     0.01    33870     0.00     0.00  Curl_splay
>
>   1.54      0.53     0.01    16682     0.00     0.00  event_add
>
>   1.54      0.54     0.01    14075     0.00     0.00  Curl_readwrite
>
>   1.54      0.55     0.01    12624     0.00     0.00  event_active
>
>   1.54      0.56     0.01    10776     0.00     0.00  ares__get_hostent
>
>   1.54      0.57     0.01     8906     0.00     0.00  ares_expand_name
>
>   1.54      0.58     0.01     3484     0.00     0.01  init_by_resolv_conf
>
>   1.54      0.59     0.01     2062     0.00     0.00  Curl_he2ai
>
>   1.54      0.60     0.01     1663     0.01     0.02  Curl_http
>
>   1.54      0.61     0.01      823     0.01     0.01  next_load_cb_hyper
>
>   1.54      0.62     0.01      727     0.01     0.03
> Curl_hash_clean_with_criterium
>
>   1.54      0.63     0.01      616     0.02     0.02  curl_multi_perform
>
>   1.54      0.64     0.01                             Curl_resolver_cleanup
>
>
>
>
>
> Thanks,
>
> Simon
>
>
>
>
>
>
>
> ------------------------------------------------------------------------------
> Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more!
> Discover the easy way to master current and previous Microsoft technologies
> and advance your career. Get an incredible 1,500+ hours of step-by-step
> tutorial videos with LearnDevNow. Subscribe today and save!
> http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk
> _______________________________________________
> curl-loader-devel mailing list
> cur...@li...
> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>



-- 
Regards,
Robert Iakobashvili, Ph.D.

Home:       http://www.ghotit.com
..............................................
           Ghotit Dyslexia
Das Ist Real Writer & Reader
...............................................