Re: curl with c-ares

[Fred H. <di...@gm...>]

will curl-loader use share interfaces via curl_share_init()?

   1. void set_share_handle(CURL* curl_handle)
   2. {
   3.     static CURLSH* share_handle = NULL;
   4.     if (!share_handle)
   5.     {
   6.         share_handle = curl_share_init();
   7.
           curl_share_setopt(share_handle, CURLSHOPT_SHARE,
CURL_LOCK_DATA_DNS);
   8.     }
   9.     curl_easy_setopt(curl_handle, CURLOPT_SHARE, share_handle);
   10.
       curl_easy_setopt(curl_handle, CURLOPT_DNS_CACHE_TIMEOUT, 60 * 5);
   11. }

*
*
*the libcurl document says it will remove support of global dns cache:*
*15.6 remove CURLOPT_DNS_USE_GLOBAL_CACHE*
*Remove support for a global DNS cache. Anything global is silly, and
we **already
offer the share interface for the same functionality but done **"right". *


2013/9/3 Robert Iakobashvili <cor...@gm...>

> Dear Shao,
> I like your profiling.
>
> If you are not using option -t (threads)
> you can opt to make a global DNS cache as in
>
> http://sourceforge.net/p/curl-loader/code/591/tree/trunk/curl-loader/loader.c
> ,
>
> Uncomment line 596:
>
>   /*
>      If DNS resolving is necesary, global DNS cache is enough,
>      otherwise compile libcurl with ares (cares) library support.
>      Attention: DNS global cache is not thread-safe, therefore use
>      cares for asynchronous DNS lookups.
>
>      curl_easy_setopt (handle, CURLOPT_DNS_USE_GLOBAL_CACHE, 1);
>   */
>
> and remove usage of cares in the code.
> Still this path will not be stable with -t on multi-core HW.
>
> Try your suggestion and, if it works with -t and without,
> please send us your patch.
>
> Kind regards,
> Robert
>
>
> On Tue, Sep 3, 2013 at 12:39 PM, Shao, Shuchao <ss...@we...> wrote:
> > Hi,
> >
> >
> >
> > I am using curl-loader to simulate 5000 users generating HTTP/s traffic
> to a
> > nginx server, find that curl-loader consumes many time on c-ares
> initiation.
> > Please look the attached gprof output.
> >
> >
> >
> > Via checking the libcurl source code, the function Curl_resolver_init()
> do
> > the c-ares initiation, and curl_easy_init() calls this function via
> > Curl_open().  So we need to do the c-ares initiation every time when we
> want
> > to initialize a curl handler.
> >
> > Via checking the C-ares souce code, there are two large ‘for’ loops in
> > function ares_init_options(),
> >
> >                 #define ARES_QID_TABLE_SIZE 2048
> >
> > #define ARES_TIMEOUT_TABLE_SIZE 1024
> >
> >   for (i = 0; i < ARES_QID_TABLE_SIZE; i++)
> >
> >     {
> >
> >       ares__init_list_head(&(channel->queries_by_qid[i]));
> >
> >     }
> >
> >   for (i = 0; i < ARES_TIMEOUT_TABLE_SIZE; i++)
> >
> >     {
> >
> >       ares__init_list_head(&(channel->queries_by_timeout[i]));
> >
> >     }
> >
> > So, even when we only send a single GET request, curl will call
> > ares__init_list_head() thousands times. That’s the reason why curl-loader
> > consumes so many time on c-ares initiation.
> >
> >
> >
> > My question is if we can do the c-ares initiation only once but not in
> every
> > curl_easy_init()? By that probably we can increase the curl-loader
> > performance significantly.
> >
> >
> >
> >
> >
> > Some gprof output:
> >
> > Flat profile:
> >
> >
> >
> > Each sample counts as 0.01 seconds.
> >
> >   %   cumulative   self              self     total
> >
> > time   seconds   seconds    calls  ms/call  ms/call  name
> >
> > 18.46      0.12     0.12  9186352     0.00     0.00  ares__init_list_head
> >
> > 12.31      0.20     0.08        1    80.00    80.00  build_url_set
> >
> > 10.77      0.27     0.07    81473     0.00     0.00  dprintf_formatf
> >
> >   9.23      0.33     0.06     3325     0.02     0.07  ares_init_options
> >
> >   4.62      0.36     0.03     1770     0.02     0.04  create_conn
> >
> >   3.08      0.38     0.02   107452     0.00     0.00
> > client_tracing_function
> >
> >   3.08      0.40     0.02    41568     0.00     0.00  fd_key_compare
> >
> >   3.08      0.42     0.02     5102     0.00     0.01  epoll_dispatch
> >
> >   3.08      0.44     0.02     3150     0.01     0.01
> > ares_library_initialized
> >
> >   3.08      0.46     0.02     2243     0.01     0.01  ares_cancel
> >
> >   1.54      0.47     0.01  6918787     0.00     0.00  ares__is_list_empty
> >
> >   1.54      0.48     0.01   450212     0.00     0.00  alloc_addbyter
> >
> >   1.54      0.49     0.01   227162     0.00     0.00  aresx_uztosi
> >
> >   1.54      0.50     0.01   107005     0.00     0.00
> > hostcache_timestamp_remove
> >
> >   1.54      0.51     0.01    56835     0.00     0.00  Curl_pgrsUpdate
> >
> >   1.54      0.52     0.01    33870     0.00     0.00  Curl_splay
> >
> >   1.54      0.53     0.01    16682     0.00     0.00  event_add
> >
> >   1.54      0.54     0.01    14075     0.00     0.00  Curl_readwrite
> >
> >   1.54      0.55     0.01    12624     0.00     0.00  event_active
> >
> >   1.54      0.56     0.01    10776     0.00     0.00  ares__get_hostent
> >
> >   1.54      0.57     0.01     8906     0.00     0.00  ares_expand_name
> >
> >   1.54      0.58     0.01     3484     0.00     0.01  init_by_resolv_conf
> >
> >   1.54      0.59     0.01     2062     0.00     0.00  Curl_he2ai
> >
> >   1.54      0.60     0.01     1663     0.01     0.02  Curl_http
> >
> >   1.54      0.61     0.01      823     0.01     0.01  next_load_cb_hyper
> >
> >   1.54      0.62     0.01      727     0.01     0.03
> > Curl_hash_clean_with_criterium
> >
> >   1.54      0.63     0.01      616     0.02     0.02  curl_multi_perform
> >
> >   1.54      0.64     0.01
> Curl_resolver_cleanup
> >
> >
> >
> >
> >
> > Thanks,
> >
> > Simon
> >
> >
> >
> >
> >
> >
> >
> >
> ------------------------------------------------------------------------------
> > Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more!
> > Discover the easy way to master current and previous Microsoft
> technologies
> > and advance your career. Get an incredible 1,500+ hours of step-by-step
> > tutorial videos with LearnDevNow. Subscribe today and save!
> >
> http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk
> > _______________________________________________
> > curl-loader-devel mailing list
> > cur...@li...
> > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
> >
>
>
>
> --
> Regards,
> Robert Iakobashvili, Ph.D.
>
> Home:       http://www.ghotit.com
> ..............................................
>            Ghotit Dyslexia
> Das Ist Real Writer & Reader
> ...............................................
>
>
> ------------------------------------------------------------------------------
> Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more!
> Discover the easy way to master current and previous Microsoft technologies
> and advance your career. Get an incredible 1,500+ hours of step-by-step
> tutorial videos with LearnDevNow. Subscribe today and save!
> http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk
> _______________________________________________
> curl-loader-devel mailing list
> cur...@li...
> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>